You seem to be ignoring that testosterone, at certain periods in the womb and during puberty, is what causes the penis to grow. Of course, there is probably a genetic limit too. If T is normal during these periods, more might not cause the penis to grow more. But, a dearth of T due to environmental factors can stunt penis growth. T is used by medical professionals to spur puberty in pathological late puberty or no puberty cases.
Objective masculine characteristics are promoted by testosterone: the rest is social construction. Puberty blockers used by transgender women (biological males) dampen objective masculine characteristics such as genital size, hairyness, muscular development, beard roughness, aggressiveness, etc. Keep in mind that there is wide variation in "normal" testosterone levels: roughly 200 to 900 on typical scales. Thus, there is a large variation in objective masculinity among men.
Some studies have shown African men have higher testosterone levels on average than other races. I cannot vouch for these studies.