I agree with intuition. You are putting this guy's feelings and needs above those of your hubby.
BIG TIME. Additionally, if your "guy" can't talk with your hubby about any and all that is going on, then I think you're getting on to the cheating side of the "open relationship" that others of my erstwhile companions are referring to.