If you do so much bodybuilding that women (and men) stare at you because of your big muscles, you've done too much. Unless they're also bodybuilders (who see you as competition) they are probably staring at you and assuming that you are self obsessed and insecure. This makes you far less attractive than a fit guy with average musculature.
It's difficult not to go for bigger=better in bodybuilding because it such a competitive scene, but you must bear in mind that people who aren't into the scene see it as a freak show. Balance is far more important than bulk. I'm sure you've seen guys who have over developed their legs (for instance) and by doing so have managed to make their torso and arms look skinny (or vice-versa). IMHO you're ok as you are.