The fact that he has never had sex with a woman would cause me to put a stop to this with him. Regardless of how comfortable I might have been with having him involved prior to that knowledge. Guys tend to get overly attached to the first woman they have sex with and the longer a guy has gone without losing his virginity, the more attached they get to the girl from what I have seen. I'd be concerned that he would develop far too many feelings for my wife; while I might not be worried that my wife would leave me for him, I would not want her or I to have to deal with the drama that would come from the situation. I also wouldn't want to burden my friendship with him.
Additionally, I don't think a virgin is going to have the sexual maturity to handle a swinging situation. He may have the personal maturity, but there are a lot of emotions involved in a sexual situation and a virgin has never experienced that first hand.
Far too many things here that would concern me to actually go through with this. If your wife is open to the idea then look for someone else and explore it when you find someone that has some experience.
Either way, good luck and I hope you have fun
