It sounds like you have a large insecurity around competition from your parnters' exes. Either that, or you have totally bought into the heteronormative model of relationships where men and women can't simply be friends, so being friends with your ex never happens. Well, plenty of us get along fine with people even if they used to fuck our boyfriends, and don't give a crap what you think. But, since you seem to be in some distress over this matter, maybe I can help you see things more clearly.
When you are in a small minority, just a few percent of the overall population, the community you inhabit is small and highly connected. Gay men make up around 2-5 percent of the population, so we have small, well-connected communities. Sometimes it seems like everyone knows everyone, everyone's dated everyone. We date gay men, and are friends with gay men - often the same men! If I had to stop being friends with everyone I once had sex with anytime I was dating someone, I wouldn't have a life outside my relationship.
Also, a real friendship is a rare and valuable thing, and can often last much longer than the time spent dating someone. I went on a couple dates with a guy when we were young - we messed around, then decided we should just be friends. In the 20-some years since then we've stayed close friends and he's been with me through many boyfriends. If I'd kicked him out of my life the next time I dated someone, I would have lost the best friend I ever had. You may think that is unnatural and that makes me weird, but I think it's really sad that you would end a friendship for such a stupid reason.