I personally have been attracted to men older than me as long as I could remember. I notice a different attitude I take when I'm with someone that I know is even a year older versus a year younger. There's probably a connection with this and my childhood somewhere, considering that my father died when I was six and I had very few (if any other) male role models at home. But that has nothing to do with how I came to the realization that I'm gay, so the connection has to be a small one. In matters of love, age is nothing more than a number. I know people who are younger than me that make more sense intellectually than those who are nearly twice my age. It's the combination of intelligence, personality and in some small way appearance that ultimately makes me look at someone as a friend or hopefully something more.
If you're young, into older men and are truly honest about it, then by all means you should go for it.