You must defend your boundaries. If you don't, who will? You must tell him why you were upset in the first place. Don't let him interrupt you, not even with an apology. Finish your spiel with the statement that though your boundaries were breached, you feel that you've over reacted, and want to apologize. Then apologize. If he's a grown man, he'll apologize too. Then it should be over, never to come up again.