Nothing really matters in an absolute sense. Perhaps the nearest thing to absolute value (in this philosophical sense, not the mathematical sense), at least in the estimation of members of the kingdom Animalia, might be sugar content: the members of a great number of species would, on average, rank the desirability of many fruits and vegetables in the same order as humans would, on average. (Then there is the complication of obligate carnivores, such as cats, but I digress.)
Value adheres to an object by virtue of its estimation in the mind of an observer. In the human realm, the value of anything is the average of its value in the estimation of all potential human observers. The value of a thing changes according to changes in people's opinions, which are influenced by multifarious factors.
Rather than insisting that a human male's "penis size doesn't really matter," I think it would be more accurate, precise, and honest to say that penis size, in present-day America (or any other group of people, e.g. all living humans), matters less than one's credit score and more than one's hairstyle, for example. (Surely someone will disagree with this example, but let's try to keep in mind that the average of all observers' estimations is more important for defining a thing's value than anyone's particular estimation as an individual.)
Then there is also the complication that, whereas a man may take action to change his credit score or his hairstyle and thereby increase his total value as a person, there is, at present, no satisfactory way to change the size of one's penis.