Well, first of all, let's call sex "hiking". Hiking is fun and it makes you feel good. Hiking is a great way to pass a day with no plans. Hiking is great for blowing off some steam when you're stressed out. Right? Good.
Now let's say you have a friend named "Bob". Bob is a great guy. You and Bob get along really well. You could never be in a relationship with Bob because Bob is... I dunno... Severely OCD (or something else that drives you insane and is absolutely intolerable). However, Bob is the perfect guy to go hiking with. He's usually free when you are. He goes at the same pace as you. You guys have a lot of fun when you hike together. You and Bob have become great friends and sometimes you go out for lunch after you hike. Maybe you go to a movie from time to time because Bob is a pretty cool guy to hang out with. Going out with Bob does not make your heart swell at the sight of him... because Bob is kinda nuts. You and Bob may even go out a couple of times a week because you have such a good time. People may think you and Bob are dating but you know that you're not. Why? Because Bob is psycho. Bob is good for hiking, and lunch, and a movie... not for marrying.
See where I'm going with this? You have to be capable of platonic friendships with men to be capable of sexual friendships with men. You also have to be able to separate sex from the things that go along with it when you have a boyfriend. Think of sex as a hobby, or an interest. Like hiking, or bowling, or... antiquing. Whatever it is that you like to do because it's fun and it makes you feel good.
Then you have to consider the different categories that guys fall into. There are the guys that have no chance, period. They're acquaintances at best. There are the guys like "Bob" who are great guys, but not for you to be with. There are guys that you could probably date for awhile and have a great time with. Still, they're not exactly "marriage material". Then there are the guys who you think you could be in it for the long haul with. For me, the "acquaintances at best" are never an option for a fuckbuddy. The guys who I think have potential for a serious, long term relationship are never options either. Too much risk of screwing that up when you get into a friends with benefits situation (because believe me, guys catch feelings just as much as women do... ugh). So, the options are the "great guys but not to date" or "I could have fun with him for awhile". That way I can pretty much rest assured that I'm not going to fall in love with them but there's still enough attraction to get down
So, maybe this guy just isn't the right guy for that kind of relationship (you'd mentioned that you're thinking you like him "more"). Maybe he is, you just need to make sure that you have your thoughts in order before you go there. Then again, maybe you're just not the type who can build a strong enough wall between sex and romantic feelings. Anyway, I'm sure I'm babbling by now. Hope that helped at least a little.