You two seem to be going about this in all the right ways, with consideration and love for each other. Congratulations. Sorry about your being disappointed. I'm sure you already realize this, but being disappointed is certainly better than your husband being upset if he had let something happen he wasn't ready for. Compared to a lot of people, you two are doing great! Again, you probably know all this so please pardon me...
If you keep talking, it's quite possible that you'll have the experiences you both want. In any case, you'll have a lot of really good communication that will enrich your relationship.
Some other people have suggested on this Board that people getting comfortable with the idea of swinging go out to a vanilla bar, not to do anything or pick anyone up, but just to sit together and point out people they like. Just comparing notes on strangers you never talk to can make the possibility seem more real, without any pressure or danger. Oh, and it can be fun
.
I hope you continue to post and participate in the discussions around here.