In our view, you did nothing wrong. Sometimes that fantasy of swinging and the reality are two different things. Unfortunately, that is not something you can always figure out until after. One thing you should ask yourself is whether your husband has shown this kind of behavior before in more vanilla situations.
One mantra you will see on this board again and again is the importances of communication. Talk to your husband during a time when you can really be open or honest. Don't have to say "you asked me to do it.". Just try to figure out why he is being standoffish and begin to figure out if this lifestyle is right for you.