The worst they can say is "no thank you" to a date/coffee offer. No, the worst they can do, especially in a work environment, is to threaten you with sexual harassment and try to force you out of your job.