A similar thing has happened to me. We'd been with several couples before this particular evening, and we'd met this couple previously for a drink and gotten along famously. Anyway, it's our second meeting and we're getting cosy next to the fire at their place, a glass of wine or two between us, lots of foreplay and some of the hottest action I'd been witness to was happening live right in front of my eyes... but the little guy just didn't want to play!
I was lucky because they were very understanding, after talking about it with my SO, and writing them an email about it and having a bit of a laugh, we went back a week or so later and I was back in action.
I still don't know what happened.
I guess this is a long way of saying that these things can sometimes just happen, too much external stimulus (I thought it was impossible!) perhaps anxiety in some cases, but don't make it a big deal and it wont be.
Stay honest with everyone, especially your wife, have a laugh about it, and next time you'll be back better than ever