If your wife is saying you aren't allowed to play with another woman, then why are you meeting with a couple?
You did jump into things pretty quickly, but you need to talk about parameters with your wife. What is ok, and explain to her that you weren't sure if it was actually ok for you to play with this other woman and it contributed to your anxiety. I mean, your're worried that she will be upset if you play...but she's having a grand old time.
Lots of men do have perfomance issues in the lifestyle (I would wager from nerves? When my SO and I are having sex, he has no problem getting and maintaining an erection...and it's getting better with swinging and being used to being in that situation with a new partner).
Personally, if Jeff and I were to meet up with a couple where the female could full swap, but the man isn't allowed to...we would pass you by. I'm not here to watch my sweetie get his, while we're just sitting by watching and you aren't 'allowed' to fuck me. But that's JMO.
I would clarify things with your wife to make sure you are on the same page.
Maria
