To the OP - I am sorry for your situation. I think that swinging should be the icing on an already delicious cake. It is so unfortunate that just talking openly about sex in a relationship can be a problem. I really think our society drives a lot of sexual hangups into folks. That said, perhaps you and your wife need less focus on swinging and more focus on some honest communication. If you discover that you aren't a good fit then hopefully you will figure that out through some understanding.
I wish you much luck! Hopefully you find what you need together.