Fire_and_Air, I can tell that you're feeling a lot of conflict. Your feelings of betrayal, anger and jealousy are understandable and entirely appropriate. As gently as I can, I'll say, "He cheated on you." You've stayed with him, and I hope that you have worked it out between the two of you.
I think you need to find a way to break it off with this couple. It doesn't sound as if
you are having much fun with them. I question your boyfriend's motivation somewhat, but that is something you will have to determine yourself.
If you do swing again, avoiding a bad encounter like this one will be helped by one thing: don't, if the sexual attraction isn't there for you.
This thread might interest you. If I were in your shoes, I'm not sure if I would trust my partner not to break the rules again. Before the next time, I would suggest an honest, heart-to-heart with him. Spell out what your boundaries and expectations are and tell him that you expect him to honor them.
-B