My answer is that I do not mind--and even find it flattering--when gay guys admire my penis, as long as they are respectful about it. But there is a fine line between admiring and being intrusive/obnoxious/inappropriate. When guys cross that line, I get uncomfortable and angry and feel violated.
For me, the issue usually arises in the locker room at the gym. My gym happens to have a fair number of gay members and many of them check me out in the locker room. There are a handful of guys that I think of as my "fan club" because they check me out every time and seem to look forward to seeing me there. All of this is well and good--and, as I say, flattering--as long as they keep it at a respectful distance, which most of them do. Some of the guys have really crossed the line, though, with comments, offers for sexual favors, blatantly following me around the whole locker room/shower area while staring very openly, starting to stroke themselves while staring at me, even grabbing at my dick. When such things happen, my response is to either remove myself from the situation or politely but firmly state that I find their conduct inappropriate. Those responses are always effective.
The thing is, about conduct that crosses the line, I would find it inappopriate, offensive, and disrespectful even if done by women. The point is not that it's guys doing it to me, the point is that it is simply inappropriate for any stranger to make aggressive sexual advances without at least some encouragement.
As for straight guys, my penis gets noticed by guys who are apparently straight at the gym, but no straight guys have openly commented on my size since guys I knew in college.