I'm going to reply to everyone at once, since many of you had overlapping advice anyway. Also, I'll clarify a few things things for those who wanted to offer me more specific advice. In the beginning I wanted to leave it a bit vague so that the responses might be less skewed towards what I think is going on.
Mental component: I know it is definitely part mental. The first time I have sex with anyone I don't feel a damn thing (including my first time ever), but the more I have sex with the same person the more sensation I get. A lot of you were correct about another thing - thinking about actively going after an orgasm tends to preclude one. I think the mental bit is getting in the way, in part because I'm embarrassed about how little I feel, and then the lack of sensation takes me out of the moment and that allows me to think about other things, get self-conscious, etc and it just goes in a vicious cycle. I do need to relax. Perhaps I won't have an orgasm with a partner until I'm in a relationship and feel I can relax, instead of my string of FBs. I know it's not completely mental because I have trouble cumming by myself as well, but thanks for the advice on this component, I'm definitely going to implement it.
Physical issues: Even before I got a vibrator, which I rarely use, I had trouble. But reading all your responses had made me realize that I probably have a physical problem with my clit. All my first orgasms were from internal stimulation. Externally, I have almost no sensation until I am already half-way there, and it offers more of an 'assist' than anything. I can't cum from clitoral stimulation alone. One of the things that makes me think I have a physical issue is that there is a very, very limited range of sensations that I can feel at all and what I can feel seems to be the opposite of what other girls like. Most guys I'm with try a rather fast-paced motion on my clit and I feel nothing. At all. To feel anything, it has to be a fairly low frequency at a pretty specific spot. The same I found is true with my vibe - only the lowest frequency works, the others don't do anything for me. And by that I don't mean the sensation is 'too much' or painful, I mean there might as well not be anything touching me at all. Not entirely sure what that means about me physically, and I know every woman is different, but I've always found it a bit odd.
Personal replies:
@ManlyBanisters: I love the cake analogy! Thank you so much for that, I could not have said it better. One FB of mine once asked me, "What are you getting out of this anyway?" and I wish I could have said something like that. It was the first time my lack of sensitivity was brought up in that way and the question, especially at the time, left me dumbstruck and quite hurt. After he said it, I felt like my enjoyment of the entire process was invalidated because I couldn't cum.
@dolfette: Listening to you talk about what you feel and don't feel - it sounds exactly like me. Nerve damage is a possibility I've considered, but I'm not sure. Is there a test they did for nerve damage, or did you just know because of before and after sensations? I can't have a before and after comparison because the only cause of nerve damage I can think of occurred when I was about four years old. Thank you also for reminding me that I'm not a 'rare freak'. All my female friends and all the girls my FBs have been with are/were highly sensitive. It's hard not to think 'what the hell is wrong with me?' when I hear my female friends talk about their experiences. It's also painful for me to hear whatever guy I'm with talk about how fantastically sensitive his past girls were and how hot is was to see them cum after I had just failed to do so (I don't know why they all do this, perhaps for some sort of self-validation reasons after their 'failure'). I'm really glad you said that because it's honestly something I needed to hear.
@seaside: I've considered the possibility of hormone imbalance before. Do you happen to know which hormone might be out of whack off the top of your head or the website on which you read it? I've looked at the symptoms and it seems low sensitivity would be the only one I have. Most sites don't even list low sensitivity as much as 'low libido', which is not true of me at all.
@teasedsilly: Yep, arousal could be a bit of a problem. When I can get off it's only while watching quite a bit of porn. Good call.
Thanks again to everyone for all your help. As you can tell this has been bothering me for awhile so thank you, thank you, thank you again.