Well, barring the fact that I'm straight so I probably wouldn't have sex with you, per se... I have had sex with women who aren't single before. It's not really something that weighs on my conscience to be honest. I may be aiding and abetting the "cheat" but I'm not really the one hurting anybody.
Same reason I wouldn't be mad at the dude if my girlfriend cheated on me (unless it was a friend, that would be a level of betrayal by both, these guys I had never even met).