It is true that often, someone who does not fit neatly into a binary label can be rejected by both sides. I think that really only points to the ignorance of the non-accepter. With that said, though, I think a lot of it depends upon how the bi person views the world around him.
Do I accept bi people? Well, yeah. Am I tolerant of people who cheat on their "other halves?" Not so much. If you are bi, and playing the field, that's one thing. If you are in a relationship, and not totally honest, that's another story.
I have a friend who is truly bisexual (I'm guessing her percentages would be about 60/40 straight/gay), but I call her a serial monogamist. She has been in relationships with both men and with women, but is faithful when she is in a relationship. Unfortunately, she makes some really bad choices about whom she chooses!
I think it is part of her personality - she always wants to fix what's broken. After she got her Ph. D. in chemistry, she worked for several years in industrial plastics, then decided to go back to her original love - medicine. She's now a practicing primary care physician. Still trying to fix what's broken.
But back to the original question here - where would anyone get the idea that it's their business what another person's sexual orientation is, and that anything outside the binary is bad?