is that just
maybe he's turned on by the thought, but can't quite come out and say that.
True but rare. Very rare.
Actually, a huge chunk of my clients have this fantasy. It falls into the cuckhold catergory. Like, have you ever imagined how hot it would be to watch your sexy lover having sex with other people, or doing kinky things with other peope and loving it? That would fit into that catergory too. Women do it as well. A very close friend of mine has small breasts, and masturbates to the fantasy that her boyfriend is cheating on her with large-breasted women. Anyway, in my experience, tons of men have cuckhold fantasies, and of those that have that kind of fantasy, 99% of them (regardless of their own penis size) fantasize that the other man or men have larger penises than they do themselves.
Not rare. Not rare at all.
To the original poster:
I think you must continue to be honest, and that includes being honest about this frustration. Maybe you should show him this thread (if you think he'd be okay with knowing you brought your concerns to a "public" forum). Maybe you could show him measurection.com which has tons of advice and support for small and average men, and those who love them. (And by "show" I do mean "accidentally" leaving the browser open to an appropriate thread on that site.) In fact, you yourself might want to head over there, where this question comes up frequently, and where it is answered by men with small and average penises.
Good luck, sweetie.