I wouldn't go that far, but I think some respondents have lost their sense of proportionality. I don't believe that the quality of my sex life, let alone the quality of my entire life, is driven by the size of my penis. I'm fairly sure that people with small (or even micro-) penises can and do have fulfilling sex lives.
On the other hand, a million dollars would be a life-transforming amount of money for most people. That kind of money allows one to change careers or start a dreamed-of business, send kids to good colleges, retire early, see the world, eliminate all debts, etc, etc. I'm not saying that money solves all the problems in life, by any means, but it certainly has a lot more utility than a large penis does.
I'm pretty sure few will agree here, in the temple of large penis worship, but I'm equally sure that in the real world, there'd be a lot of people jumping to make that swap.