This sounds like a disaster waiting to happen! Never do something just to make someone else happy!
I know my husband likes being with other women... and I like women and men too, but we do it together, same room to enjoy it together.
Men do like having sex... PERIOD! The excitement of someone else is a turn on to them, but he shouldnt be hurting you with it. It isnt like you were his first and he wanted to sow his oats so to speak. I married a virgin the first time around, I was his first and he was always curious about other women and how they felt.
I think you two seriously need to talk more about this.