Nah, I don't think that was all he had going for him. As you point out he had plenty of other attributes w/ or w/o being big.
But since at least one of his conquests, Ms Hayworth, was no ordinary starlet, was presumably choosy in her selection of men and actually married him not just messed around with him I thought that maybe, perhaps, kinda, sorta there could be a connection, no?