Human population has dropped to about 2,000 people, all located in one place, and stayed that way for about 100,000 years until numbers began rising again.
We are all related to each other. Our Eve, the woman all humans can trace their ancestry to, lived about 140,000 years ago. Our Adam, oddly enough, was only about 80,000 years ago. So in strict terms, we really are all related to each other even if extremely distantly. What the magic number is for enough genetic diversity, I don't know.