I think it's determinded by genetics, only because every physical part of you "technically" is genetic, but I think it is a random combination of genes that do it. I haven't seen my dear old father naked (Thank God, I'd go blind -_-) But as far as I've seen there isn' a whole lot going on down there. I think if it comes from any side, it comes from the mother's side, becuase I take almost all of mom's characteristics except my hair, but at least baldness is on her side too, and her dad didn't even start to go bald until he was in his 60s (whew!
I'm a short person, 5'3"or 5'4", but I'm above average down there, so I think it's a totally random gene in and of itself.