I was wondering if a man's testicles get smaller with the less frequent sex he has (or no sex for a few months, due to ilness) and can then again get bigger over time when more frequent sex is re-introduced? Is this just more testosterone being produced with activity that can increase the size?
Whether ill or not having sex, it is mostly the LH level that dictates testicle size becaused LH tells the testicle to produce more testosterone. BTW, Follicle Stimulating Hormone (FSH) tell the testicle to produce more sperm. Both FSH and LH are regulated by the Hypothalamus. When men take testosterone (in any form from outside the body) the Hypothalamus notifies the Pituatary Gland to slow down LH and possibly FSH production. The HPTA (Hypothalamus, Pituatary, Testicular Axis) is what makes your question so hard to answer.
Some use HGC or herbal supplements to jump start their hormones and get their nuts back to their normal size and their semen production back to normal. There are also drugs like Clomid that women use to increase fertility that have "off label" use to increase a man's LH level.
LH is prescribed for boys with undescended testicles.
Perhaps someone who uses T can give you a more detailed and up to date explanation. I doubt that "less frequent sex" will decrease testicle size as long as LH levels are ok. A long illness however, will no doubt fuck up the hormones making this possible.