J,
You hit the nail on the head, it is an insecurity issue. And as Lycious said, there is a big difference between swinging alone and swinging in a committed relationship.
Sounds like maybe you guys need to take some more time to solidify your relationship together so that he knows 100% that this is about the two of you having fun and not about you looking for guys who are better than him in any way.
I'm assuming that you allow him to pick women for himself, so remind him of this, and of the fact that you trust him enough to allow him to do so.
Once you do get to the point that you are BOTH ready to swing, you should both agree on the partners (keep in mind that just as we women can see things in other women that guys don't pick up they are the same way with other guys). But if looks are the only thing he is looking at, then there is definately an issue with insecurity.
Julie
http://www.swingersboard.com