Not in my experience. I'm just past 40. However, most people think I am 10 years younger than I actually am. I go to the gym and lift weights 3 times a week and I watch my diet, so I am in very good physical shape, I have fairly low bodyfat levels and good muscle tone (check pics for proof). I weigh more or less the same as I did in my 20s.
Where I think problems start is when guys get fat and/or their testosterone levels drop. The two are interrelated from what I have read, but I can't remember the physiological reason why. Google it? So they end up with a double-whammy - a layer of belly fat which masks a portion of the penis and a penis which doesn't get quite as erect as it used to. What may also be a factor is that the guy's other half might not really "do it" for them any more, due to her putting on weight, losing her looks or just plain and simple boredom and complacency.
As others have said, I think the key is staying in shape, eating well, not drinking too heavily, not smoking, and having an active sex life (and not relying too much on porn) with a woman who rings your bells.