When I was built like you I was climbing two to three grades harder than I can now. If I was 20 lbs less, back to where you are, more of my cock would be visible/useable, falls (ski/climb/bike) wouldn't hurt as much, and I wouldn't have to stop myself when that second helping of pudding looks SO DAMN INVITING!
The only reason to work on changing yourself is if you're personally unhappy, not to please others. We all look for some kind of approval or affirmation, and I think you've got that here. Count up the positive responses vs. the negative and then accept the fact that you're attractive to many, and by bulking up you may just lose the interest of those who prefer skinnier guys. Unless, of course, you think that we're all liars trying to be kind. Speaking for myself, I'm not.
Cheers!