Welcome to the board mnsnen! Glad you've found us.
I'm not a big fan of one-sided or "open" marriages, where one or both partners has said, "Look, whether you like it or not, I'm going to NEED to fuck other people. If you can't handle that - and I'm sorry if it hurts - you aren't going to be very happy living with me." It can be said in the most respectful way possible, but just the same...it is a case of putting one's self - one's sexual desires - ahead of one's spouse's/partner's needs. I just can't see a significant long-term relationship being built on self-centredness.
I was going to say this might border on polyamory, but he seems to keep saying that he just wants the sex, not the love. This is something you should probably explore further with him. Is he holding this back from you, for fear of hurting you? Is he just
telling you that he doesn't want other emotional relationships? If one is truly polyamorous, it means that they are capable of and prefer multiple emotionally based relationships. This is part of their hardwiring. For them it is fair to say that they cannot control whom they fall in love with. But to say that he cannot control his sexual urges is preposterous.
Sex and food have many parallels. They are drives, they are healthy in moderation, they make you feel good... But when someone cannot be in the same room as a chocolate cake without eating the entire thing, we say that they have a problem. When someone is controlled by their cravings, they are considered addicted. There's a difference between hunger and appetite. Hunger is the body telling you you need to eat, because you will die if you don't; appetite is your mind sending the signal to your stomach to start growling because you've developed a habit of eating a chocolate cake whenever you see one. Need vs. desire.
I seriously doubt that your husband NEEDS to put his penis in another woman's vagina. He just really, really wants to, and he's gotten it into his head that, because he wants it so very badly...well...that must mean that he NEEDS it. The "need" that he has convinced himself of is nothing more than his own mind playing tricks on him. What I've found during our swinging is that the biggest obstacle we have to overcome is our own self. We are our own worst enemies. Welcome to swinging.

Sounds like your husband has run smack into his first personal demon. I guess he wasn't far under the surface. This is something that you two should be taking up with a marital counsellor, seriously. Look for one that is experienced with and supprotive of alternative lifestyles.
Something else I'll mention: I see another problem for you on the horizon, and I'm afraid this one will land in your lap, Mrs. mnsnen. You didn't think you get away scot-free, did you now?

lol
If you feel that anything but strict monogamy is wrong (AKA "sinful"), then sooner or later, you will need to wrestle your own demons. Can you stay happily married to a man who is doing something that you feel is a moral abomination? It means you will eventually have 3 choices. 1) Put up with it, and accept that you will live a life that you will regret. 2) Go your separate ways. You won't be together, but at least you will have been true to yourself. Or 3) Adjust your beliefs.
Personally, I'd go with #3. Most people think that's just blasphemous, but the thing is that I decided that I simply would believe any truth that was self-evident. I only believe what makes sense. So, for me, faith in God and Jesus Christ makes complete sense (otherwise it's a pretty empty and hopeless life to be living). And I believe that a marriage does not need to be monogamous to be happy; only respectful and selfless in nature. The biggest block I had was overcoming the fear of thinking for myself. A bit like being the child who said, "But the Emperor isn't
wearing any clothes!" Once I got over that, things started to make a lot more sense to me, and I realized just how out of wack our cookie-cutter marriage actually was.
It would be helpful if we could hear Mr. mnsnen's side of the story, too.
EDIT>> And a big Dito to MoonLight's post!