What do you see as the downsides?
The downsides of being large or PE? Both have been covered in exhaustive detail on these boards. In brief.. being bigger than what women consider ideal leads to the possibility of causing physical discomfort to your partners, encountering people who will be afraid to have sex with you, difficulty with oral and anal sex, and so on. There are other issues to... clothing, self-worth issues, et cetera.
As for PE, depends on what exactly you are doing. There's a possibility of damaging you penis, of reducing sensitivity, of ending up with something that looks bloated, distended and unnatural if you pump, of poisoning yourself if you use silicon injections, and other side effects if you use drugs.
Search the boards for these topics and you'll find plenty.