I think it's a natural part of being a man. I never feel more masculine than whan I am rock-hard and ready for sex, it's a real turn on to see me and know what's coming. What's a bigger turn on is to be rock-hard and huge, I can see it and it's so big and ready. We can theorise: for most of human evolution we were non-monogamous, only in the last thousand years or so has this been a part of humanity. Prior to that, for a man to have offspring, he had to compete with other men. A larger penis will deposit semen closer to the uterus, giving men with larger penises a greater chance of propogating into the next generation. Women get turned on and aroused by girth, so during human development in the absence of loving relationships, a thicker penis has a better chance of repeat bouts of sex has a better chance of propogating.
The other side of size, I feel, is the ready availability of porn, mostly with well-endowed actors. In my lifetime, porn has moved from something fringe to being everywhere. There's no way that an average-sized guy can get away from all of these big dicks, and if his natural evolutionary instinct is to want to be bigger himself, then he can't help but feel insecure.
There's been penis size jokes since the Roman empire, so while women can say 'it doesn't matter to me', clearly it matters to men and always will. I feel that a large penis DOES make me feel more masculine, and that is a nice feeling.