For me, it's mostly a temperature thing. In the summer, I usually walk around with a large soft size, but in winter, not so much. If you look at my gallery, those are all done in a toasty envirnment. If my apartment was 3 degrees colder that day, I wouldn't have taken the pics.
One year, when I was still getting used to this, I bought a pair of shorts in spring that seemed to fit me fine. During the first heatwave, I couldn't wear them! I didn't notice until I was already out walking, then I noticed people staring. Man, did I start blushing!
Also, my level of hydration can be a factor, believe it or not.