First off, welcome to the site!
I will give you a lot of credit for being honest enough to admit there is an element of unfairness in this situation, with you having your cake but not wanting him to have his. Being able to be that honest with both yourself and someone else is a positive, and if there is a resolution to be found here, that honesty is going to be what you leads you to it.
Here's how I see it, you have found your place in swinging and are perfectly happy in it now. So happy, you really don't want to give it up. He is still trying to find his place, and it sounds like part of the reason for that is that you aren't comfortable with where you think his place is and so as soon as he starts to settle in, you are pushing him a little to get him to try to find a new place that you think you might find more acceptable. Swinging is a team sport and both have to be on board with what the other half does, that is absolutely true, so not trying to say you don't get any say in this at all. But, when you look at it from afar, it does seem like the constant feeling a bit off balance in your swinging life together is all coming from you, while meanwhile you enjoy the stability of having found your place and not having to fight to maintain it.
If you don't think you can ever let him find his place within the same framework of what you enjoy, then yes, I think you probably aren't cut out for swinging, and by having gone down the road this far trying to convince yourself that you are, there is going to be some pain involved, meaning time to end it with the FWB.