As can be seen in my (hopefully somewhat artistic) gallery, I'm not "big" by any stretch of the imagination, so I can't offer any insight from my own experiences. But I can offer from my friend and from what I've read.
I read a book a while back called "The Male Body." It is written by a woman, but there is a whole chapter about penis size. The gist of that chapter is that penis size is a double-edged sword for men. On one hand, men with small penises often feel like they are less of a man than men with big penises. On the other hand, men with big penises often become known for that big penis, and as a result they become sexualized -- only. People come to think of these men as sexual objects, nothing more. People therefore tend to think that men with big penises are not intelligent men.
So, yes, it sounds akin to what women with big breasts may go through. I remember hearing that some women with big breasts have had to have reductions because they felt no one would take them seriously with big breasts.
My best friend is very well endowed. I have talked with him about it, often pressing for insight into how wonderful it must be to be so large. I have finally accepted that I have to drop the issue, and most likely never talk about it again. The reason? It occurred to me that it has hurt him to have been seen only as a sexual object by so many women. The stereotype is that men are just sexual animals. The truth, though, is that men are human beings; sexual, but with more than just sexuality. There is more to my best friend than his big penis.
So, maybe it can all be summed up in this: men with big penises sometimes think that others believe that there is nothing to them but the big penis. Maybe it's like people with a lot of money who can never be sure if their friends like them for themselves, or for their money.
On the other hand, I have to admit this as well. Men DO want their penis to be big. Though my best friend wants others to know there is more to him than his big penis, he has also made it clear that he has enjoyed the hell out of that big penis and is proud of it. He has said, however, that he'd rather receive compliments on something that he himself had to work for and earn, rather than something he was given by an accident of nature.