Evolution has actually resulted in increased penis size though, we have the largest penises of any primate.
You're right about porn, but there really is an evolutionary pressure for larger penises.
Studies of women's preferences indicate that women prefer slightly above-average penises over any other size, and women consistently select an ideal penis in the 6"-7" range, whereas the average penis length is 5.5".
Of course this has virtually no beainrg on our day to day lives, and unless you plan on living for 100s of thousands of years it won't matter to your partners.
There is no proof, only conjecture, regarding the hypothesis of increasing penile length. Why? Because there is no physical evidence. Homo Sapiens goes back about 300,000 years, but we have no specimens beyond mummified remains dating back several thousand, and these provide little insight as to erect size. As body height has zero correlation and zero causation to penis size, the fact that humans have grown taller isn't evidence either.