Whether you were kidding or not,you did give him the go ahead to be with her...he was kind of rude for not clarifying things with you..and she was inconsiderate as well.But, you are all equally to blame.Clearly, this couple needs to be off limits now.You need to get some rules going about what is acceptable and what is not.Have you ever told him you do not want him talking for hours to any woman? Have you made it clear he's not to fuck anyone without you knowing?Even if you're sleeping right next to them?Another things is...don't let anyone stay the night like that unless you are looking for a polyamory type situation, it muddles the line between swinging and actual polyamory, at least to me, anyway.Swinging is about the sex...it's not about hours on the phone and re-enacting your lovemaking with another person, that is so beyond disrespect....I wouldn't blame you if you knocked him over the head with a skillet...
But, then you'd have to kick yourself in the ass for not stopping this before it got to this point.
Rule#1:
No phone calls other than to make plans to meet...if you want to chat,do it online so you both can read what the convos are about.
Rule#2:
No re-enactments of any kind...your lovemaking is for the 2 of you, not the whole world.
Rule#3:
No SleepOvers...
Whether it is at your home or a hotel, boot the bitch/couple out when you're done...{Sorry, guys...I have to be blunt here}!!
Rule#4:
Communicate, be blatantly honest about your wants and needs, your likes and dislikes...because there is a serious lack of communication going on here.I know you knew that already!!
And lastly...Do not allow yourself to ever be humiliated by anyone...if you sense a red flag...or your instincts tell you something is off...ACT ON THOSE FEELINGS RIGHT THEN!!!!! Sometimes, we just can't believe what we're hearing or seeing and so we go into denial when it's right there before us...I think it's a coping mechanism we humans have.But, if it smells like shit...it probably is shit! 
Good luck, sweety!
Disclaimer:
This is clearly jusy SENSUALITY'S opinion and in no way reflects anyone else's views or opinions in this topic, except when it supports what I'm trying to convey! 