I would gather that it’s not just a certain group of people, nor 1 universal answer. If the person is unfamiliar with that other persons disability, they may now know their limitations (if any) pertaining to sex or anything physical. They may not be wanting or willing to be a care taker later in life if that is necessary. They might be assholes who don’t want to be seen with “someone different”.
Bottom line is that it’s not just the people you’re asking about that might take issue, I’m. It sure why you pointed them out. Most of the answers will likely be for selfish ones, but it would be better if they are at least honest about it.