This happened to me once or twice. I think a lot of the issue is if they are a hook-up vs if you are both really into each other. With a hook-up, it's just someone getting off without any substantial emotion/feelings behind it, plus nerves of meeting for the first time.
Btw, I don't consider 5 minutes being a long time for a bj. If you get bored after that, it is definitely you, and the other guy is going to pick up on your mood to rush to get the job done.