Give me a break! They talk about dick size all the time. My National Language instructor (equivalent to an English/Language Arts instructor) would digress into tangents about the size of the dick on some guy he had seen recently in a porno video. Most of the guys made almost constant innuendo about the size of the other guys' wieners; it was the mainstay of lunchtime conversation. I cannot count the number of times I was asked whether Americans all had dicks like the ones featured in their pornography, where had all the foreskin gone,
etc. If it wasn't about the size of a dick, it was about the color or shape of one, or the color of my nipples...
On the other hand, the topic seemed to be used less for shaming or disparaging others of any particular size, and more about camaraderie and just enjoying silly fun together.
However, as you have mentioned, when it comes to the context of actual sexual intercourse, they tend to be nonchalant (or at least appear to be so) when penis size is noted explicitly. Frankly, I've never had the gall to ask any of my lovers directly, but, judging from their typical way of speaking and behaving, the answer to a question of why penis size becomes such a non-topic when we are in a situation where the penis will actually be used for a purpose in which size may have some effect would be something like, "It doesn't matter. I want to have sex with you, you want to have sex with me, we are already naked and can see each other's dicks directly. What's the point of stating the obvious?"
Anyway, outside of actual sexual intercourse, Japanese guys at least talk about dicks almost constantly. The one thing you should never try talking about with them is "love"; it's like you've mentioned the Devil or something.