Coming from a person who has been through infertility treatments, you can have 10 million sperm in ejaculate and be considered infertile. Because millions more sperm are needed to climb over one another and help each other make their way up to the egg, one or two little guys are highly unlikely to make their way there, even 1 million of them don't account for enough to line the pathway through the vagina.
The vagina is very alkaline in order to protect against bacteria, many sperm are instantly killed when ejaculated inside of there, so many more have to survive. When you think of every crevice of the internal organs and the microscopic nature of sperm they get snagged, caught, stuck and killed in their several day journey.
Think about it, while it only takes one sperm to fertilize an egg, he had to depend on over 10 million of his buddies to help him get there.