I think there're a number of things going on. Yes, a lot of it's anecdotal, but it seems too frequent to be random. In no particular order:
As has been said, guys compare & admire each others' muscles & other body parts all the time, often often touching, massaging, etc. while saying how impressed they are. I don't see why dick, balls, & ass are exempt from this. I also think the straight world reads way too much into it. Just because it's a dick instead of a bicep doesn't automatically mean it's more sexual.
Most guys seem to just like being naked. If they're in a space where they feel safe, won't be judged, and won't have any negative consequences it takes about two seconds to get them out of their clothes. It also seems like NON-sexual touching goes through the roof!
I've heard a lot of straight guys complain that their wives, girlfriends, etc are almost squeamish (my word) about sex and have fairly rigid ideas of how sex is 'supposed to be'. You have to read between the lines but I get the impression that a lot of straight guys would like to frot, hump legs, lick armpits, get their balls licked for 20 minutes straight, and just generally get sweaty with maximal skin contact.... but a lot of ladies apparently aren't interested. I bet a lot of straight guys wonder if other guys would be more open.
I think a lot of guys want to just release sexual tension without jeopardizing their romantic relationships.
And who you want to have sex with and who you want to date/marry don't need to overlap.
Think about it. When straight guys look at a group of gay guys what do they see? Depending on the setting,
- Friends are openly affectionate, hanging all over each other with lots of skin contact and nobody automatically assumes it's sexual.
- Guys talking explicitly about having done things that sound like fun but could.
- Guys talking about getting blowjobs from guys who want to see how loud they can make the recipient scream.
- Having sex with someone who agrees that good sex is the goal and 'feelings' are set aside.
To get back to point, I'd say most guys are bi-curious and would at least try it once or twice if they didn't have to worry about any negative consequences
One interesting shift I've noticed among younger people is a tendency to define gay,straight, bi, etc. by whom you DATE. So a striaght guy who jumps into bed with another guy when he's between girlfriends and is lonely, bored, or horny is NOT giving up his straight credentials.