I think it might just be something tall, hung men have difficulty in understanding.
Physical stature gives a man an instinctive confidence that need little reinforcement from the world around him. Short, shy and small men have to work easily twice as hard just to find the same level of instinctive confidence.
I am 6'3", 200 pounds and lean, but that didn't help me lose my virginity. It's like anorexia, it doesn't matter what you really look like, it only matters how you perceive yourself. On paper, I could say I have the kind of body that most women like, but I still couldn't believe that they would like me for some reason. So I don't think being tall or hung would solve your problems.
And sometimes I wonder if being taller makes women more defensive or shy, figuring a tall guy will be more confident and aggressive. So being tall and shy sometimes works against you.
Also, after not having a girlfriend by 21 or so, I felt like I needed someone really nice to kind of prove to everyone that it was worth the wait. Because in some ways I was confident and felt like I deserved a nice girlfriend, so I didn't want to finally settle on someone just "average". I was worried what my friends or family would think. I suppose after a while being with no one looks worse than being with somenoe "average" though.
Another factor for me was a lack of sexual male role models as a teenager. My parents weren't divorced, but were not sexual and I didn't really see any couples (relatives, neighbors, etc.) interacting sexually. So I never really learned how men are expected to approach women. Some girls liked me in high school, but I just had no clue what was acceptable and/or proper. Most people take this for granted that it's some sort of instinct, but I think it's a learned behavior. I generally don't regret anything from the past, but I sometimes look back and wish another guy back then would have taken me aside told me what was up and encouraged me.
So I think it takes several special circumstances for someone to become an older virgin.
This was written from a heterosexual view point, but I imagine being gay wouldn't help things. e.g. I doubt you have many gay couple role models around you.