There's a lot of possibilities here. I think you're on the right path, wherever it leads. Lots of communicating with your wife, and not pushing it for now are both excellent ideas. There are many swingers in the lifestyle who never full swap. There's no reason to feel that you have to full swap.
The experiences you keep having sound like a symptom of post traumatic stress disorder. That doesn't mean you're broken. Identifying it, and working through it perhaps with some professional counseling may help.
To say this is jealousy is probably not the right way to describe it. Regardless of whether you are 100% in favor or 100% opposed, watching another man have sex with your wife is a moment of serious, overt input to your brain. You weren't ready for this, and it was too much for you at that moment in time.
I would definitely, emphatically resolve this before ever full swapping again. And, it's ok if you don't full swap. Do what you are comfortable with.