As a few other guys have said, it's b/c we associate them with some of our earliest and most primal memories of gyms. You'd only ever see them in gym locker rooms , which are all-male places. PLus of course, they highlight the package in front so well, AND frame the butt in a really hot way. It's like an wildly titillating invitation that says "ok, this is totally forbidden, but what I really want is for you to throw me over this bench and fuck me right now".....(well, that's what I hear anyway).
And oh yeah, there's the possibility of that intoxicating smell.....100% masculine.