When I'm getting head it's usually part of a whole lot more: going down on the girl, fucking and, of course kissing. If I'm not attracted to the person, it's not going to happen. That's my rule.
Now, the exception to the rule. If for some reason, I'm just really horny (and drunk) and the oppoortunity presents itself, the person does not have to be "hot". Really ugly? I hope not.
I do remember being at bike rally in another town with a buddy of mine. He wanted to hook up with a girl he met and asked me to "take care" of her girlfriend. Of course, we were all pretty drunk but I had a whole lot of fun... all night.
Long story short, a couple of phone calls, we agree to have dinner a couple of weeks later. ARGHHHHH! The girl I was with was NOT hot. In fact, if you were being very generous and she didn't have any missing teet, you couldn't even go so far as to call her plain looking.
So, the response to you original post's question:
Does it really matter ? If someone just wanted to give you a BJ but wasn't all that hot, would you turn them down ?
No, it doesn't matter as long as you are certain that's all it is.