Thats not quite true. Your balls are where your sertoli cells are, which are the stem cells that become sperm. There are also leydig cells, which produce testosterone. The spermazoa then travel to the vas defrens, which are those tube-y things in your ballsack. Thats where they are stored. Then, when youre ready to ejaculate, the sperm are mixed with secretions from your prostate and your cowpers gland, etc to make semen. Bigger balls have more sertoli and leydig cells.
Its true that sperm only make up about 5% of the volume of semen, but if someone has seriously huge balls, it could make a difference that might be noticable. More likely still- there could be a correlation on average between the size of your balls and the productivity of your prostate.