Well,
As many have already said. It happens. I still have issues like this and for us, we have realized that it is going to take more than just a single play date for me to be able to maintain enough firmness to be useful.
I have to be comfortable with my partner to be able to stay firm.
I wish I could help you relax, my wife and the other couple we play with currently keep telling me the same thing, and I haven't figured it out yet.
For me, I have taken the humor approach. I know it is a problem, but I chuckle about it and we all joke about it and still have a good evening. I don't seem to have issues with MrsVan, so I know it is not a physical problem, but a mental one. Maybe try going back to your routine of playing with your wife first before swapping and see if that eases the tension some.
Above all else, just continue to have fun and I am sure it work itself out soon enough.
-Van