Well, our two cents worth is stick to your guns. The way we approach our fun or swinging for lack of a better word, is definitely not a lifestyle, and we both agree if we aren't both on the same page we won't do it. Participating in this swinging is definitely not worth it if there's any trouble brought into your true lifestyle.
Here's hoping your husband gives you the respect you deserve and allows you to make up your mind with what you want to do. If there's one negative from either of us, it's a no go.