Sadly no, it's not my ability to reason that is having the problem, and I'm not necessarily saying that yours is.
Think about it, people see a big one and they're like "whoah, gotta measure this!" but they don't for a small one, implying there is something to be embarrassed about.
It's like saying "no, don't talk about the penises of small guys, it will insult their masculinity" when penis size truly has NOTHING to do with how masculine the man is. How we perceive it is a different matter. But once you say "no, I don't want to be insulting" means that you have already bought into the insulting thought process of "small penises are a joke".