If you are fit and healthy and eat well, then your sex drive should not be adversely affected by age 40. I am 51 myself, and have just returned from a weekend away with my wife where we wore each other out (four times in three days). I do look after myself, I eat well and I am 3kg heavier than when I was 18.
I went through a period of low sex drive, and it turned out to be an illness that wreaked a certain degree of havoc. So I suggest a thorough check-up, and if that doesn't uncover anything then watch what you eat and get into the exercise. We only have one chance at life, and we really can't afford to abuse our bodies.