It doesn't happen to me often, but when it does, I think it's like some cosmic pay-off for all of those poor women who never had an orgasm. That's never a problem for my female partners, but we all have to take hits for the other dicks out there once in a while.
Anyway, two is not a very large sampling. I'd guess you are THINKING too much, and if you are jerking off as much as BigGuy suggested, you are probably not accustomed to adjusting to being with a partner. If you've got an sexual encounter coming up, forget about jacking before it, and during the sex get into the rhythms, the feelings, the breathing... get primal, no talking, no thinking, just... sex.