Seeing as how my wife is 10 years older than me and I've always had a thing towards "mature" women, I personally couldn't see myself with someone significantly younger than myself had circumstances been different in my life.
I guess for me it's not only the obvious sexual aspect, but the fact I would have more in common with someone either my own age or older than much younger. With an older woman, while there is some historical and cultural events as well as musical tastes she may have experienced that I haven't, but for the most part there is also a lot of overlap; while with a much younger person there is a lot more I'd have a hard time relating to I think.
I do think there is a perception in society (sometimes unfairly) that men who gravitate towards much younger women are motivated solely for arm candy & breeding purposes, while for the woman it is financial. I will admit that despite knowing the challenges of an age-disparate relationship and the judgements from others that have sometimes come due to it, I have the same prejudiced reaction myself when I see or hear of some middle-aged or older dude dating a woman in her 20's or even 30's.
But overall, what other people want to do and who they have a relationship is none of my business anyway, though I still would have the perception that a relationship of more than maybe 20 years difference or a middle-aged guy gravitating towards those who closer to the high side of 18, is doing it for unseemly reasons.