[Additude]

It seems out of the ordinary that any person would say and do something like that with malicious intent unless there was a specific detrimental reason for doing it. However, this person, based on your story, already knew your wife and knew of you so he was an acquaintance at worst, maybe your wife’s friend? I don't know, but definately not a stranger.

I don't think what was said was intentionally meant in a harmful way.

Why would someone say something like that? Who knows, the booze, the passion, the moment. You were best to ignore it if it bothered you.

A response from you of, "I'll return the favor" would have leveled the field.

Let it go.