Wow, allot of mistakes I would think, for sure.
The beginning starts as one of those "one time when we were all liqueured up" stories. Not good at all. Me and my buddies have a few of those. Most, are not good endings.
I don't know about your friends happy long term marriage. But if you saw a look of shock on his face I can imagine why. I don't know about you, but my few close friends I have now (in our non swinging world) would tell their wives, one way or another.
Thats why they are few. I can't stand guys that would put me in any position with, "don't tell my wife".....Anything ! That to me says get the f#ck outa my life then. I ain't hiding from no body, no body is hiding from me. Thats why I have few close friends.
I think you both carry that trait. Or you would have considered his wifes feelings before anyones dick came out. I can only imagine you two have dirt on one another. I can't believe this was not a major issue at the lunch with your wife later. I bet it will though, eventually....
Then the kids bumming ya all out ? Thinking his wife doesn't know about all this, back home

.
Is. F%ckin KARMA... at its best..... If not Divine.
I'm just going to stop here, nothing else matters at this point.
If I'm wrong, can you clarify more ?