Though I'm straight and not small, I think I know what you mean, willow78. I suspect something hierarchal is wired into the male brain about size, and genitalia is no exception. We get impresssed by large ones, not impressed by small ones. Same for muscles, height, body size and toys (cars, boats, homes, etc).
Obviously a man isn't really more of a man because his dick is a few inches longer. What defines a man anyway these days?