I'm also a 5.5 inch guy. And while i now know I am statistically average, I too have always felt like a smaller guy. I attribute that to 3 reasons:
1) when I was in high school and college, the average was said to be 6.5". So I was well below that. Later, the average was said to be 6". And more recently it was dropped to 5.5". I've even seen some stats that say it is 5.25". But since I spent so many early years thinking it was higher, it sort of got ingrained that what I got is below average.
2) I'm a pretty built guy--thick chest, broad shoulders, big arms. So looking in the mirror and seeing how my bone sticks out from this built frame, it just seems short, thick, and stubby.
3) even though I am average in length when hard--and maybe even above average in thickness----soft I'm well below average. In the locker room, I don't have much hang. It's such a contrast being a built confident guy and then in the locker room being one of the smallest guys there. So even though I've never had a complaint about my bone from a woman and know that statistically it is average, it still seems smaller.