Unlike the gentlemen above, I am less sure about genetics and penis size. Over the decades each generation has gotten physically taller due to better diets, and clearly this is not genetic. However, penis size does not seem to have grown measurably over the years. The other thing about genetics is that characteristics come from both sides of anyone's family (obviously), and one sibling may inherit an average sized penis from one parent (say your father's side) and another sibling a larger sized penis from the other parent (mother for example).
Wiki summed it up as a combination of genetics and environment.