We have had this happen once, but we aren't very active "players" anyway ... very hard to find compatible couples, you know. :rollseyes
Anyway, from my perspective, and having been in this position once I feel qualified to share my perspective, I don't understand why you made your husband stop what he was doing each time. I mean, yes, we have a rule that we play together, that we are same room always, but things are not always going to go completely synchronistically and I kind of feel it's unreasonable to think that you would expect them to.
I feel badly for you that this happened to you 3 times and I certainly would establish for future encounters that drinking is to be held at a minimum, and try to foster some sort of communication with potentials that lets them know that they really need to be sure of their abilities, but of course, that gives them reason to become self-conscious in itself, so what to do?????
I just know that I could never pull my husband out of an encounter just because mine wasn't going right on schedule and in tune with his.
In our situation, I just sat back and watched, which I really like to do anyway ....
And let me just add that NO man other than my husband "pulls [me] to him" ... NEVER

Were I placed in that situation, I might have gently told the other man something along the lines of "why don't you join us all over here?" and let him know THAT was where I wanted to be at that time. I am not an ego buster by any means, but if it wasn't working, it wasn't working. There was no need for EVERYONE'S play to be ruined because this one guy had difficulties, quite possibly of his own making.