The thing with speedos varies tremendously by where you live in the world. Americans seems to be REALLY hung up about speedos. I'm not sure why. Perhaps, people are put off by men being overly sexually objectified, so it's a gender role kind of thing, and women don't want their men noticed. Or maybe speedos are associated with being "gay", so it's good old-fashioned homophobia. There is a really irrational hostility towards them though that borders on being prudish.
Go to some places, and many men walk around town in speedos on a hot summer day. And no one cares or perceives it as sexualized.
It is very possible that the market splintered in the past. As I understand it, speedos were more commonplace in the past, then boardies took over. I want to say the straight masses took to the boardies, because that's what women wanted to see men in. Look at NBA hoops.. bye bye short shorts...
Now, who is left to embrace the speedo? Swimmers and gay men. Swimmers wear them for drag (no pun intended), and gay men like seeing bulge. When fashion changed, who was left behind? That is why I think the populace sees them as gay.
In domiciles where the speedo never went away, it's not an issue. But in the US, the speedo disappeared from fashion in the 80's, and ones left embracing / wearing them probably outed themselves...
I am by no means stating that everyone who wears a speedo is either a swimmer or is gay, but what I am saying is that the populace sees them that way, because fashion has dictated it so. Sad but true.