I think it comes down to the man's physique and overall ablility to grow hair on his body. If you have more of a gym body you can get away with little/no hair on your chest, back, groin, arm pits, legs, etc. Body builders are, for the most part, smooth everywhere.
Also if you have patchy hair growth (ie: not a Wooley Mamoth) it looks better to take it off as apposed to leaving it on. If you are a hairy beast it would not be worth the hassle.
I myself shave most of my body (chest, back, pits, and groin) because I'm a patchy grower, my physic can support it, and my wife likes it. If I could I would do laser treatments to have it perminantely removed but would cost thousands of dollars. Don't mind spending the 10 minutes a day to take it off (just slightly stubbly in the morning).
To each is own on this but this is what I prefer...