Unregistered,
You've just received advice from a couple of the best on the board. About the only thing I can add to WRNakedRU's post is not only does your husband have no clue of what swinging is about, he appears to also have no clue of what marriage is about.
I'll be blunt. If your husband would divorce you over your inability to enjoy swinging (and remember, you've taken it a step further and even given it a try!), I think I'd beat him in line to the nearest good lawyer and grant him his wish.
As the others have said, you may not have laid out all the details, but if you've told me this was not a base that was covered before marriage and now it's possible grounds for divorce, I can't understand why you've stayed with this insensitive lout for as long as you have. I think your only other option is to wait and see how long it takes him to bring the subject up again. Perhaps he'll drop it, but I'm doubtful. And if he does drop it, I can't help but wonder how long it will be before he cheats on you outright. I wish you all the best of luck, I fear you need it.