I might be hard to begin with, but I've never stayed hard while I'm being fucked. It's pretty normal to not be hard while you are having anal sex, there really is no reason to be hard and I'm pretty sure that your brain learns that pretty quickly, and doesn't bother getting you stiff. It's not that I'm not enjoying it, I am loving it, it's sex, what's not to love. But my dick is not really needed for it and doesn't get used while I get fucked.
Mine is actually very small, which I'm grateful for when I get fucked, as it doesn't get in the way!