My buddies own basketball shorts. They don't take them off down to just underwear when I'm around. I don't know about them, but I DO hang around in just my underwear when I'm alone. If I have company I'll throw on light shorts.
If I came over and he were wearing just underwear it would get a "what the fuck" moment. I would give him the arched eyebrow like "you got something you need to say, bro?" and he would laugh and say "Hold on I'll go put on some shorts".
I worked with him and sometimes I'd spend the night at his place playing video games and/or drinking and in the morning we'd strip down to boxers to get changed for work. In that context, there was no awkwardness. We took no special effort to be unseeable, but we did take special effort not to ogle. If we're talking we'd continue glancing at one another, but otherwise very businesslike with the putting on of pants.
If your giant dong was hanging out of your always-gapped boxers I'd be uncomfortable to the max. If you had functional button fly I would be like a 2 on a ten scale. Waiting for something to change.
I'm mostly straight, he's straight (the friend with whom I'm most frequently in some state of undress; known the guy since we were 4 and 5).