I wish I had an answer for you man. Have you ever given him reason to feel like you were attracted to him? Maybe he got a vibe at some point and got wierd? Just wondering. Maybe he has a self-image or confidence issue? Are you hung huge, maybe he isn't by comparison? There could be many a reason, and the only way to find out would be to ask him directly, specially if you know he's not lying about public nudity...
Also, I had a buddy like that kinda. I didn't really care to see him naked though, but he told me he could NEVER be seen without a tshirt on. He would go to great lenghts to avoid being shirtless even for a second. I never got it. He would tell me that when he had sex he had to have it on still, even with a steady partner... some people just have hang-ups. I did ask him why, but he would never tell me. I figured he had some kind of skin anolmalie he was embarrased about and let it go.