This was actually an issue that caused me, personally, a great deal of grief. I graduated from school five years ago. I live in New York City, where even the most remote neighborhoods from any real job whatsoever have ridiculous rent costs. I had fifty thousand dollars (more or less) in student loan responsibilities, worked 80 hours on light weeks, and yes, still have to live with my mother.
Even as an educated, working professional, I was on the business end of a great deal of social pressure. Women (even those older than me, who lived with their own parents!) would think I was some sort of serial-killer because I live at home (as of this posting, I'm only in my mid-20s.) My friends whose parents were more well-to-do helped subsidize both their educations and their first apartments while they "figured things out," factors which are conveniently forgotten when they chide me for still not having my own place. I ended up suffering from anxiety attacks and severe depression as a result for a pretty long time, due to the excessive pressure that comes with trying to live up to everybody else's socioeconomic barometer.
The truth is, a lot of people are just plain shallow. Still more folks out there are short-sighted and have no understanding of the sort of things you may have to deal with. However you want to break it down, they simply don't care about you--or whatever it is that put you in your situation.
So, really, why bother caring about them or what they have to think? Just try to make your moves on your own timetable, and remember--everybody has hardships from time to time. Those same folks who treat you like a pariah now run just as much of a risk of being forced to move back home as you do.