Depends on brand and style. I wore small til I was about 40, then my hip bones began to spread. Yes, they really do that. A friend who was unusually intelligent ( genius IQ) and my Hubb who was a physician, laughed theirs butts off at me when I told them that as they thought I was using it as an excuse for gaining weight. I haven't gained weight.
After my super-smart friend passed away, mu Hub discovered I was telling the truth and sheepishly told me so. So now I wear anything from a medium to an x-large. If there is a lot of stretch I may wear a M or even a S, MAYBE. But I don't like my junk all squished together. Do NOT wear boxers, EVER (I do like some support) but it depends on the pouch size--I'm not exactly huge, I just like a little looseness there--and fabric. I have and LOVE some women's size 8 black briefs--long story--that are a little snug when first putting them on but the nylonish fabric soon loosens just enough to feel a little freedom there but they kind of mold to your anatomy with out you feeling a thing. They're a hit at the beach! I feel like I am indeed not wearing anything but my stuff is held together just enough to provide safety from being scrunched in pants' creases when sitting or sagging too much down into my crotch recesses and then stick to my thighs. They are wonderful but they're not made anymore. There were a certain style of Barely There. I luckily and bravely bought out all the Kohls stores in the area I liked them so much so now have enough for the rest of my life. A cool side benny is that some have become just enough 'worn' to be a little see-through. Those are the ones I wear to the beach--just a hint of skin color showing through, and I like that so much that I'm usually running around with a bit of a semi showing through. LOVE 'em!