EWW....I would NEVER wear underwear my dad or step dad touched/wore.
I don't have a brother, but wouldn't have a problem wearing his underwear if i had one, or possibly male cousins if they were around my age, but those would be the only male relatives i would share with.
As I am gay, I also don't think I would have any problems sharing underwear with a boyfriend or partner if we were the same size as long as they were clean when I put them on...