[knb2004]

I see three red flags that I'd suggest you address before trying again, most of which have already been hit at least a little.

1- Swinging with friends. While it occasionally works out, it also frequently leads to problems.
2- Swinging with vanillas. They don't understand how this works. As newbies, you don't really get it either. Blind leading the ultra blind.
3- I don't think you guys have communicated enough about this. You should both know, and agree to your boundaries. You shouldn't have to wonder if you crossed a line, you should know where the lines are before you start. You shouldn't have to have an issue with feeling left out, as that is something that should be discussed, at least to a point, ahead of time.

You guys need to talk, the most important thing is you must both be on the same page with regard to your boundaries. Beyond that talk some more, then talk some more, then talk some more.

Then go to a club or a meet-n-greet where you can find bona-fide swingers.