I've alternated back and forth on the facial hair thing over the years. I can't grow in a full beard, splotchy hair folicles on the jawline and cheeks, so I've stuck with a very closely trimmed Van Dyke and longer sideburns for the past few years (See attached.) I've also got alot of grey in my hair now adays so keeping things shorts helps reduce the overall impact of it.
In college I wore a beard and mustache through the winters, but it was itchy and frustrating to say the least.
I also think it depends on the guy. Some guys can pull of facial hair and look really polished and neat, while other guys just look like a mess with facial hair. For me it has to work on the specific guy as part of the total package.
Whisker burn from a night of intense kissing with some stubbly man is great reminder of a good evening though!