How was the husband behaving? I'm asking because the fact that he went so far as to show up at your door with that offer makes me think the wife picked up on how much HE enjoyed it.
I've heard a lot about people getting that kind of attitude because they feel like their spouse enjoyed the experience too much. I understand that because they're most likely thinking hey, you don't express that much enjoyment with me. I can't think of any examples I've heard where they refused to repeat because someone from the OTHER couple enjoyed the experience "too much" except in cases where it's a swap and they feel as though it wasn't a fair exchange i.e. that they didn't enjoy the experience with the swap partner as much as the other swap partner enjoyed their spouse.