Ahhhh, seems I might be able to help out a little here, seeing as I was once in your wife's position, while my husband was in yours ......
It has EVERYTHING to do with insecurity and shame, in my opinion. BUT let me qualify my opinion by saying that I have it because that was MY problem.
When my husband and I first entered the realm of swinging, I will admit that I did it kind of "under duress" ... because it was what he wanted. It was never anything that I had considered prior to discussing it with my husband. We had gone to a few off-premise clubs, and I admit, I was aroused by the atmosphere, but was not really ready to jump into swapping. Since I am not bi, f/f action was not something I was interested in either.
We took it slow for awhile, but I knew my husband was eager. I probably jumped in a little sooner than I would have liked and our communication was not what it should have been at that time.
When we got hot and heavy into parties and swaps, my preference was total anonymity. I didn't want to know those people's names, I didn't want to see them again, I didn't want to carry on an ongoing "relationship" of any kind with them. I wanted wham, bam, thank you ma'am, one night stands, and that's it.
We did meet one couple through an internet ad. The wife of that couple and my husband chatted frequently on the computer. The phone calls were almost exclusively between the two of them. My husband tried to get me involved, so he wasn't hiding anything at all, but I was jealous and intimidated. We became friends with these people, and as such, in my frame of mind at that time, never played with them. I simply could not. The first time we got together, at our house, for a casual hanging out and getting to know each other evening, SHE was very flirty with my husband and I became very angry!
He could tell and he was avoiding all her attempts and finally she caught the hint and stopped. But it was a very awkward situation for all.
After some time, I told my husband I wanted to stop altogether. I didn't like it, I felt like it was consuming our lives and I was at odds with my religious beliefs. We stopped, for a little over 2 years. We are just now back into it, and I am much more comfortable because I am in it because I want to be, not because my husband wants me to be. And because I am more comfortable, more secure and less guilty feeling, I have absolutely no problem making friendships with people. I now prefer to know the people we swing with, even if only on a casual basis. But I like the idea more now of getting to know people we can hang out with as well as play with on occasion.
I would suggest you and your wife both and each examine your reasons for swinging ... and approach it with brutal honesty. If there are any chinks in your marital armor, or any reason that either of you are uncomfortable with anything at all in this lifestyle, and if your communication is not where it needs to be, which is at 100% honesty and openness, then your swinging life will continually be adversely affected, and then you relationship might become affected in turn.
Though I am not proud ... the problems in our marriage led to some really bad incidents in our marriage. Not caused by swinging per se, but my feelings about swinging and the reasons I wanted to stop contributed to my low self esteem and THAT reached into our marriage. We subsequently suffered issues involving domestic violence, infidelity and later, separation.
I am sorry that I have ventured off onto tangents completely irrelevant to your situation, but I felt it was necessary to get across what our experience was when I felt that way.