I am breaking my long time lurker status because I think it is important to say that I am in the medical field and you are describing the classical picture of hypogonadism. Other signs would be:
1. small testes (there are rating scales for this if you look online)
2. small penis and prostate
3. lack of scrotal rugae (wrinkles) and pigmentation
4. eunuchoidal skeletal proportions (arm span two inches greater than height, ratio of upper body segment (crown to pubis) to lower body segment (pubis to floor) less than 1)
5. high pitched voice
6. female fat distribution
Some or even none of these symptoms may be present if puberty started but was foreshortened (for instance by an infection or benign tumor compressing your pituitary gland). If your puberty was foreshortened at an early age, this would likely be associated with an abnormally short height.
If you have low testosterone, you would also likely have low libido and never experience temporal balding (loss of hair on the "temples" of your head).
I am NOT saying that you have a medical condition but if there is even a small chance that you do, you should go see your primary care physician. He can do a quick and easy blood test to measure your testosterone levels. Low testosterone is easily treatable with testosterone replacement therapy. If this is the case, the sooner that you intervene, the better. That being said, I think the chances (in the absence of other symptoms that I listed) that you have hypogonadism is low, but I always think that with your health, it is better to be safe than sorry, especially when it is just a matter of a blood test.