Most people don't get chosen to be a model anyway, so how could anybody know if that was the reason? Could be that a model candidate didn't have "good body proportions?" If they have 1 slot for a model, and 100s of people apply, I suspect most will be rejected, for some reason or another. Maybe they are aiming for a certain demographic, so too old, too young, didn't like that minor blemish, didn't smile just right for the camera, too many models already of a certain skin color to achieve their proper demographic mix as to who they think their customers might likely be.
Yes, I imagine it's possible to be "too endowed" to model underwear. I suspect that every underwear catalog is not aiming for the "soft porn" look. If a model is "too endowed," then if the bulge is a bit "bigger than normal," isn't that going to make potential customers who look at the catalog, feel a bit "inadequate?," and that's not the way to put them into a buying mood. I would guess, that "average," or "slightly above average," is the look they are looking for, although I have no experience in any such job field.