Physically, the worst thing so far has been a touch of Peyronie's disease. It did freak me for a while but I got over it when I realised no-one else seemed to care.
Apart from that, it's all been fine so far (54 now). There was some backache and the odd joint pain in my late forties, but that's no longer a problem since I started exercising at the gym.
Sex is better than in my youth. I know what I like now, lol, and I seem to have more pulling-power now than I did when 24 or 34. That's more to do with self-confidence than how I actually look, I think.
The worst thing overall though, has been losing friends. Throughout the 90s there were so many aids funerals. My closest friends in my twenties and thirties tended to be older than me, but now I'm 10 - 14 years older than they ever got to be. This century, though, it's been more "usual" causes - heart attacks and cancer, etc. Which makes me appreciate what I've got.
As the wrinkly old man replied when asked by a silly young person "What's it like to be so old?" - It's better than the alternative!