Reality vs. Emotion

I say go for it and meet him. If you feel something, if he feels something, great, you know exactly what is going on. Imho, there will be continual uncertaintity unless you sort out how you both feel properly.