I agree... Take it slowly... Only do what you are comfortable with, but you have to push the limits a little each time. Else you'll never progress to that place where you may feel comfortable with him and another woman...
As long as your husband is comfortable with sharing you while he's not getting any, there is nothing wrong with it. Others will not understand your position as well as you do, and can be very negative about it. The only people who knows if it's wrong or right are you two...
Good luck