Strangely enough, I had a simular conversation with a friend of mine the other day who had noticed that when women speak of a man's size, they tend to only use the descriptors small and huge. I think this is for a couple of reasons. I think the main reason that this happens is that people tend not to talk about something unless it's remarkable. It just so happens that average isn't very remarkable. The other reason for which I can see this occuring is the same as what njqt said, which is that small sample sizes, which is what most women tend to have, can lead to skewed perceptions.
You are right. That which is common or average is not easily remembered.
It's the way above or below average that stands out. That doesn't mean average is no good, though.