Your testes are very sensitive to ambient temperature. Too hot, and sperm production is reduced. Too cold, and the exact same thing.
Your body has a way of controlling the temperature of your testes to a certain extent. When your nuts are too cold, your scrotum will shrivel, pulling your testicles closer to your body to warm them up.
On the flip side, when things get too hot, your scrotum will loosen up and let your testicles hang down away from your body to hopefully cool them off and give them a little more breathing room.
Nothing to fear dude. ;-)