I tend to lean towards the "yes" answer in this situation.
I do believe that in today's society, size does matter.
I mean look at all the ad's in today's magazines and even on radio, promoting penis enlargement supplements and methods to increase our size.
bigger is better is always advertised. although I personally, believe that size isn't of that much importance, it's simply a bonus in a relationship.
in any case, even if a woman/man did hide their desire for a bigger cock for their partner, they wouldn't OPENLY admit it, although I'm quite sure that they DO care, or the thought has crossed their minds. The fact that they desire a larger cock doesn't necessarily mean they will necessarily act upon those urges and ditch you for the larger man, although a possibility, if they don't feel "fulfilled"
that's just my opinion.