It's kind of like "size can matter" -- just like hair color, face shape, body type. For a lot of people, it doesn't matter; for others, it matters more than anything else.
That said, it does seem a little silly when there are things like making fun of someone for having a small penis. Would we make fun of someone for having a small hand? I mean, something like that shouldn't necessarily be a tragedy either, but understandably, for some people, it might be.
As for Madoff... well, some people deserve a lot of the shit they get. But I would say it reflects poorly upon our society that people would make fun of him for something like that, regardless of whatever he did. Some things just aren't appropriate.
