Oooh that's a difficult subject.
Accidentally ask a straight guy if he's gay and you run the risk of hurting his masculinity or being punched in the face.
The best way to know if your friend is gay is to… well… get to know him better.
Find common interests and the more comfortable he feels about you then he'll either open up to you, or distance himself if he's getting that "gay vibe" and he's not into that.
Leaving around GQ and Details magazines sitting around as opposed to Maxim or Hustler might work.
You might try asking him if he's ever heard of Fire Island, been to (or wants to go to) Key West, etc. A little stereotypical, but might shed some light.
Also, if there's any relevant news stories dealing with sexuality might be worth working into a conversation. And it's nearing election time. Gay Rights, Defense of Marriage Act, Gay Marriage, all worthy subjects for discussion (and generic enough not to give you away).
Then again, my rule of thumb has always been that if someone has the balls to come up and ask me if I'm gay, the least I can do is be honest and truthful in my reply, even if I'm not out to them. But… that's just me.