You are dealing with a very common problem, Snozzberryblu! I have no idea what causes it but a man who has no problem staying "up" a long time with his wife has a problem with another woman. Fortunately, it seems to abate with experience.
When we go on a play date, I take along a Viagra for me and one for the other guy as well. Take it about half-an-hour before play starts. If it turns out to be one of the times your husband doesn't need it, well, it's nice to be a "superman" for awhile.
Tell your husband to relax and be assured this problem has been experienced by most swinging men.
By the way, alcohol does not help.
Mr. Alura