His reply is fallacious. I, like very many men who are predominantly gay, have enjoyed sex with women. Having sex with someone of the opposite sex does not prove that you are straight - anymore than having sex with someone of the same sex proves that you are gay.
Cultural acceptance of physical affection between men in public varies. I teach young adults in Thailand and many of the young men hold hasnds, cuddle, caress and nuzzle one another though they are not gay.