I always wore shorts because that was all the other guys wore.
But I still had issues at the school pool from the 4th grade on. Mainly caused by my very early puberty.
I did and still do wear speedos when visiting relatives in Germany and Denmark and never had any trouble there. There may have been an occasional glance, but I didn't notice those 'til later, no one said anything. It might also have something to do do with it being a genetic/family trait, on both sides of the family.
In college I did start wearing speedos in the States as well. Most of my college friends were swimmers; so not wearing speedos actually made me the 'odd man out.' Some people did look, but as long as they didn't say anything, I was fine with it.
Now I usually wear speedos. Usually a size bigger than I need, but with snug fitting leg holes. I usually keep my cock down the center, it looks weird to the side and that's what people really seem to notice. But when swimming and splashing in the surf, it doesn't always stay down the center.
I have only twice had a problem, one of those times it wasn't down the center. Both times women called the cops on me; claimed i was being lewd. One woman said the way I was walking around was obscene (that was the time it slipped to the side in the surf). The women couldn't even bring themselves to say what was "obscene." Both women claimed they were trying to protect their innocent children. They just kept saying stuff like, "Just look at it." The male officer asked, "Look at what?" She said, it," and pointed at my crotch, almost touching it because she looked away while pointing. (It's too bad she didn't, the officers would have HAD to arrest her for that.)
The cops never did anything, other than suggest I wear shorts, while walking around. The one female officer that responded gave me her business card and told me to call her if I needed any help and winked at me. I called her a few weeks later and we dated for a while. One of the male officers that responded the other time looked sort of familiar. I ran into him, at a club some time later. He recognized me; we chatted a bit, danced some and ended up at my place.