We agree. We started because my wife wanted to experience other women. Although we haven't played often, after further discussion, we have ended up playing with bi/f couples so that we could both have our fun.
So far, it's worked out very well for us, with neither experiencing any jealousy issues. We figured fair is fair and if we were going to do this, both of us should get to share equally..
It doesn't sound right to be so one sided
... Ask him exactly what his issue is about your playing with other men and go from there. If he's living out his fantasies through the lifestyle, then BOTH OF YOU should get to live out any fantasies you might have, or stop altogether.