One of the characteristics of men who have larger penises is they often had higher levels of testosterone in adolescence, and this is thought to be genetic. Higher levels of testosterone during development often leads to a slimmer and shorter stature, having started puberty at a younger age and also having less body fat. Other markers are a more noticeable beard growth, a deeper voice and so on.
Testosterone can wane as we grow older, but it's likely that these high testosterone men will have higher levels of testosterone than other men their ages.
Not all men with larger penises have higher levels of testosterone during development, just the majority, and not all men with higher levels of testosterone are shorter and slimmer than average, just the majority. Not all men with higher levels of testosterone have larger penises either, indeed it is only a very small minority.