The worst sex I ever had was with the guy who had the biggest dick. A guy who sucks in bed is NOT helped by a large penis; IME it often makes it worse (it's better to be mercilessly pounded by a smaller dick, esp. when you start to get dryer and sore).
I think the far ends on both sides of the bell curve are at a disadvantage, either really small or really large. You can work around either one, certainly, but, at least for me, the odds of orgasming during intercourse are greatly diminished from either one. When we start talking the middle of the curve, then , yes, I personally prefer larger, but I've certainly enjoyed being with more average and smaller guys. And I know many girls who prefer guys a little smaller. Hell, I love to give blowjobs, and when I see a smaller penis I get a little thrill- much easier to suck!
Anyway, I think technique is more important than size (and girth is more important than length, also). There is an allure to a larger penis, I think they look nicer for one thing, but honestly, it isn't a make or break thing. If you are Modestly Hung, like you say, you're fine.