I have a bigger penis and a big ego, and I'm sure that penis size correlates to part of my ego. I enjoy sex and have had a wonderful sex life for many decades and it's still wonderful. Part of my wonderful sex life and the lovely women I've been with is due to my sexual confidence, which is partly fuelled by my size. The largest part of my ego is due to my intelligence and the things I have done beyond sex, and the success I have had in many endeavours. And success breeds success, more ego gives you the encouragement to try difficult things and succeed, which is more satisfying, and so on.
So I believe my size is a part of it, as good relationships and good sex are part of being a man.