I can remember several particular encounters from way over 20 years ago in vivid detail. I couldn't tell you one thing about the sex I had 6 weeks ago, or 4...but I can recall everything about the sex 5 weeks ago.
That doesn't mean the sex 20 years ago or 5 weeks ago was better than 6 and 4 weeks ago....it wasn't. It just means that something about the person or events stuck with me. We remember things long-term that we recall several times after they occurred. We tend to forget details we no longer need to remember...unless we have some level of eidetic memory, in which case we can recall everything of which we were conscious.
That being said, sometimes, "I don't remember" is a polite was of saying, "it's none of your business how big my past boyfriend's dicks were."