Calvin Klein and DKNY and Ralph Lauren do make clothes other than the waif-stuff you see on runways. You might, no
will, be surprised by what you can find, particularly at the outlet stores. Pick a few pieces that look good on you. I'm not suggesting going to buy whole wardrobes. With those few pieces, you can dress-up your more casual items.
REI makes good stuff but it is more casual than what Land's End and Eddie Bauer carry. Both of those stores carry larger sizes as well. As you said you're looking to make your wardrobe nicer and more mature, these stores would be good choices.
Trousers: Always buy trousers that actually fit around your waist, not below the waist where you might fool yourself into thinking you're sizes smaller than you actually are. Correct-fitting trousers will actually make you look less heavy than you are. Favor plain fronts over pleated. Go to Brooks Brothers and get a pair of white ducks and a few khakis. Get them tailored with cuffs because cuffs make trousers drape better and will make you look good. Avoid navy trousers. They make you look like you're wearing uniform. As for jeans: 501s. Buy the REAL 501s, not the preshrunk kind! Buy them 1" larger in the waist and 2" longer in the inseam. Then
shrink them properly. Men look fantastic in 501s. That's why they've lasted so long.
Shirts are easier. Favor solids and verticle stripes. Full-cut shirts will be comfortable and, properly tucked-in, make you look thinner. Cariocas can be dressed-up or down and are very comfortable. Polos are comfortable too. I second the bit about wearing undershirts. A good heavy T-shirt helps your shirts drape properly and will actually help keep you cooler in summer. They should be more form-fitting than your outer shirts. T-shirts you might wear as outer shirts should have a second T-shirt underneath.
Shoes and other accessories aren't easy. Alden makes great shoes. They're not cheap but they're not needlessly marked-up because of some designer name. Brooks Brothers also has good quality shoes and belts to go with them. They're conservatively styled. Coach makes nice items as well. Italian leather goods are also excellent.
If you like wearing all those flannel shirts as outerwear, consider swapping them for (as mentioned) a good jean jacket or, better still, a nice leather jacket. I'm very certain you can find good leather jackets in your area. Brown leather would look best on you.
If you truly want to do this and are serious about it, then get a style consultant for a day. Have them take you shopping and show you what looks good on you. Even if you don't buy much, you will learn what best suits you for future reference.
And for the love of all that is holy, don't leave the house wearing sweats unless you are running or going to and from the gym or going to play sports! Old people do that to go mall walking or visit grandkids, thinking they look cute in their pink sweats with white bunnies on the front. All you need is a matching visor.
I'm not so sure about throwing everything away. I have a friend, many years younger than I, and he was going on and on about how cool New Wave was. I showed him some of my early 80s shirts and he flipped. So I gave them to him. If you can store stuff until it's cool then do so. Then either give it away or sell it on eBay.