This is one of the problems of 'swinging' with singles.
They just don't have anything to lose.
And, it's jerks like him who ruin it for others. A lot of wives wouldn't be as understanding as you and might reject the whole lifestyle after this type of drama.
But if you beat the crap out of him, as he certainly deserves, it'll only validate his claims.
I'd go with simply smiling and saying, "In his dreams", when you hear even a hint from anyone. And, then saying that you hate to put down a friend but he did make some very unwanted advances on many occasions, but, if you were going to be with someone other than your husband you'd want a 'meal' not a 'snack'..., and laugh!
