"suck it up" tells me about all I need to hear. Not "let's try again and do it differently", not "what do you feel is missing", not "what do YOU want that we're not doing that we could", etc. It's not uncommon for one to get a faster start into swinging than the other, for lots of different reasons, but saying "suck it up" is not how you work through that finding your groove together.
He's using you on this. As others have said, being a single male in swinging is not an easy thing, and I imagine he knows that. You're his ticket into the couples side of swinging, which is much easier. As to the why, putting on my internet psychologist hat, maybe it's because deep down he's insecure and has a strong need to feel what he consider validation that he gets by having sex with other women? Just my $0.02, so maybe totally off the mark, but whatever it is, it is being destructive to you, him, and the relationship. You should be commended for working to keep things together, and I don't mean to say this lightly, but at some point, you have to start just taking care of you first.