That seems to be a lot but I have at least half that many (two drawers full). I've honesty not counted and recently did a lot of editing. I still find myself looking a new undies so I understand this affliction - lol. I just discovered "As We Move" and have purchased a few boxer briefs and give them high marks. There's a new company on my radar called "Body Aware" that I keep looking at so there's probably one of those in my future. I try to rotate through mine but have some that were purchased for specific reasons (flying, swimming, biking, etc.) so they get less use.
I'd say do whatever makes you happy as long as it doesn't harm anyone else and isn't detrimental to your life.