I think you would be fine as long as your relationship is an open, honest and secure one.
My husband is smaller than all the men I've ever had and he knows this and feels he can't satisfy me (although he does!) and wants me to be with bigger men. He has no fear that I would leave him for another man. I'm doing it just for the sex and hopefully the man will be too. I have everything I want with my husband and no desire to leave him and two children for a man that has a bigger di@k. I think most women would feel this way.