I wouldn't take anything Imus says seriously.
Dan, as far as I have heard, is totally heterosexual. He is very gay friendly, and when guests make blatant homophobic remarks, you can see him getting really mad. He explained once that very close friends (perhaps family members, I'm not sure) are gay and it really pisses him off.
He has quite a reputation for dating very beautiful women, who are not, let us say, intellectually his equal. He was once overheard remarking that while he has lots of very smart women friends, he is afraid to date one, because he is sure he would fall immediately in love and thus end up married, and he just is not ready for that.
Overall, I like Dan, get a bit bent out of joint at him at times, but generally like him. And he is adorable. Also, while it is hard to tell in his suits, he is supposed to have a great body, and works out daily.
As far as his size, no one has said one way or the other that I have heard. If anyone has a jeans or VPL pic of him, many of us would be awful grateful.