Scrotums (or Scrota, both are acceptable plurals for a word rarely used in plural) came in all shapes and sizes. Most pre-puberty scrota are relatively tight. As puberty enlarges the testicles, the scotum also grows. Since you state that your testicles do hang down when they are very warm, I suspect the issue is not with the size of your scrotal sac, but instead is the tautness of the cremasteric muscle. Your spermatic cord controls the tightness or looseness of your testicles. It regulates the body heat of the testicles, by relaxing in warm environs, and tightening in cold. (Hence the famous "shrinkage" issue of jumping into cold water) The same thing occurs during orgasm, as the testicles pull tightly to the body before ejaculation, then relax.
All that said, I recommend the following. Definetely discuss with your doctor at your next physical. He can easily determine whether this is a problem or not. (In most cases it is not. In some cases it can effect sperm production, as the reflex is due to the body needing to keep sperm temperature regulated for maximum potency.) If it doesnt bother you physically, and you are concerned only because you would prefer lower hanging testicles (some people like that, some dont) then I wouldnt be too concerned. (By the way, in my career, Ive seen as many people with "low hangers" that wanted them tighter, as people like you.
And yes, scrotums do enlarge with age. its more a gravity and time issue than a growth issue.