Technically as a fetus, you have all the parts to become a man or a woman. It's the presence of one gene (located as might be expected on the Y sex chromosome) that determines which of two tubes survive to form the internal sex organs to form, which guides one organ to develop the external anatomy.
Just to clear that up.

the penis and testicles may not extend during fetal development and the external anatomy may not be complete until later periods, but we all start at the same place and that is being neither male nor female. You could think of it as bipotential sex determinations. And technically, its the lack of the SRY gene found in males that makes the fetus into a female.
It's interesting stuff. With the formation of different internal sex organs, you have the formation of different hormones that clearly have very different effects on males and females. Physically. Males do not have mammary glands, but they do have nipples. Same with females until they reach puberty when progesterone stimulates various events, including, well, "formation"of tits.
One thing I did not was that men could lactate. You learn something new every day.
And my post may not be completely accurate, I'm going by what I remembered on Human Anatomy and Physiology. The reproductive system is what I knew best out of that whole class.