I am not sure if you'd count me as sufficiently straigtht to offer an opinion but no, it would not bother me.
This reminds me on an incident that happenned a few years ago. I had cycled to somewhere to collect a van, got caught in the rain, then drive the van to a pub (bar for Americans) to collect some equipment. I was dressed in sweat pants and, having hot wet, they were clinging to me more than usual and a guy sitting there with his g/f looked at me and said: "Hey! nice package. Do you mind if I have a feel". I was a little surprised but told him he was welcome to, which he did for a few seconds, and then I got on with collecting what I was there for.
I also remember a conversation I had at work. It turns out a man had been hanging around the gym changing rooms longer than was really necessary and two of the guys in my office had spotted him checking them out. One, who had a long-term girlfriend, said he was flattered, but not interested in doing anything with the guy concerned. The other, single at the time, felt very uncomfortable about it and admitted that he was concerned that he may be giving out some sign that he was not aware of.
My suspicion is that for the person being looked at, this has more to do with how comfortable you are with your sexuality than what your sexuality is.
