In life we have trade-off's.
Professional bodybuilders are stared at and people make comments to them. It goes with the territory.
There is this one juvenile are in many individuals that never seems to mature and that is when the comment relates to a part of ones genitalia. If you have anything substantially different than what they have, or what they are used to seeing you're going to get comments.
There was a kid when I was in HS who was unusually small and in addition he had a really bad case of gynecomastia (bitch tits) at about 16 years old. He was treated horribly. Today things would be done, but we are talking the late 1960's and people were not at all enlightened on this subject.
If it is different from the perceived norm comments go with the territory. Many would in jest make comments about wishing for that kind of "comment" but in truth it becomes not only "old" but a major annoyance especially if the people commenting are people who have seen you nude a number of times.
I have a friend who is freakishly large. This friend at one point asked me in a capacity as a friend what he should do. Here is what we together came up with. In his case it worked!
He was kind and simply sat the people down and bluntly talked about the down side of his endowment. In his case he could not sit on a conventional toilet without wedging his penis under one leg. If he did not do this, his penis would reach into the water in the toilet bowl. He went on to describe the number of people who were interested in him sexually and then when they actually got what they wanted they were unable to do anything with it. He went on to explain to these people that there is such a thing as "too much of a good thing" and did so in detail. He went on about clothing, about underwear, about swim trunks and gave them the entire run down about special things that they would never consider.
What happened was GREAT! The men who had repeatedly made comments were forced to think. In addition, the word went around the gym where he worked out and all the comments stopped cold after a couple of months because the "rumor mill" pointed out that "bigger beyond a certain point" is not always better.
Don't know how big you are, but this worked once and it might work again!
Good Luck!