I can't say I've had this problem myself but I empathize.
I'm assuming you are young, in which case, you might improve duration with age.
Although, some men don't.
There can be many factors involved with premature ejaculation from psychological to biophysical and it would seem it is already creating a source of stress for you if you are concerned with negative consequences. I would suggest re-framing the situation, rather than focusing on what you are worried about, focus instead on you experiencing what you want, which is longer duration. Our thoughts are behind the experience...