I think penis size is hereditary like most genetic traits like hair and eye colour, but again, like all things genetic, it is most of all the luck of the draw as it were as there are no true garuntees you'll end up getting the same as every other male in your family. Your Dad may indeed just have a small or average dick,your grandpa the whopper and then you inherited the big dick gene from grandpappy and not the daddy as it were.
In any case, a dick is a dick and regardless of size they all do the same job and serve the same purposes.