I think you are as much as less of a man as any of the other factors involved.
People keep saying that it's a social disadvantage. Yes, but isn't there a lot of others as well?
If you see a doctor and a normal computer programmer, would you say the the latter is less of a man? The doctor has more money, more financial security, and (debatable, but just a point) being a doctor is harder than being a computer scientist.
Would you go around telling the computer programmer that he is less of a man?
Hells no! But if you would, then you'll be saying that Bill Gates is more of a man than any of us will ever be, unless there are some multi-billionaires in this forum.
If you don't tell poorer men that they aren't as much of a man, then why would you tell someone if he has a smaller penis he is less of a man.