Secondary sex characteristics (including penis size) dominate from the genetic side of the Mother. Though we are a combination in many ways of both parents this is the information that is accepted throughout medicine.
I am myself, an example of this, in that I have a great deal of the physical appearance of my Father at the same age, there are however major differences. My Father had little to no facial or body hair. I on the other hand have "wookie genes" and am covered with hair including a heavy beard that climbs up my cheeks a heavy covering of hair on by chest, abdomen, back and shoulders from my butt to my neck. My Father did not have this at all. My Maternal Grandfather was a bear rug. My genital appearance and dimensions are not identical to my Father in that I am larger than he was in a flaccid state and the size of my testicles are probably nearly double of what my Father had. I have seen several male cousins when they were in their 20's nude at pool parties who were the from the blood line of my Dad's Sister. None of these males were on the large side of anything, and, in fact, when it came to testicular size I was quite a bit larger than all of them. Only one of these male cousins had facial hair or body hair. The rest were smooth. This would have been the same genetic pathway that has been considered to be the established norm.
There are some outside environmental factors and some other birth factors which could have an influence. Several genetic problems which happen at random can have an effect on genital size and these of course are exceptions to the rule.
When I was in H.S. there was a neighborhood kid who had Down's Syndrome. As we could, because he was in our age group, we took him around with us and tried to do our best to normalize his teenaged life as best we could. He was protected heavily by all of us and it was the neighborhood guys who taught him how to swim and many other things. His coordination was poor in many areas, but we did everything we could to help. He died at age 46 from a heart defect complicated by the obesity that tends to go with Downs. The same genetic problems that created the Downs Syndrome also created what was close to microphallus in this poor guy. He had good mental function in some areas and poor function in others and we all watched out for him as best we could. There are several other more common genetic conditions that influence size, but, those conditions (Kleinfelters Syndrome being one) are responsive to treatment if caught early enough.