Amy and I had our first experience last week and I was thinking the same thing about this.
We participated in a full swap in different rooms. While I was turned on tremendously by the other woman and FELT turned on, I couldn't seem to get as hard as I normally do. At the time I was thinking, "Oh come on buddy, don't fail me now!", but thinking back, because it was the first time I'd been with a woman other than my wife, I was a little nervous about it and chances are good that had an affect on me, even subconsciously. I knew my wife was upstairs with the husband, but maybe deep down I still had a little anxiety about it.
I'm not too worried about it because it was just our first time and worrying about it might even make more anxiety, which might lead to more problems.
To me, it feels like dating for the first time again, and I was ALWAYS nervous about that.
-Greg