Well.......there is a difference between fantasy and reality. He may have believed that this would always be in the fantasy realm. I think that perhaps he is feeling threatened. A normal response.
You may want to check out this New Swingers forum in depth- you will find much useful information.
You will hear this from the group A LOT- 1. Go at a speed that you are both comfortable with.
2. And talk, talk, talk, and then talk some more
Good luck!