You need to be honest with them... and then you never know where it might lead.
Some couples do swing seperately and you might get lucky and find that this is one of them. However, yes since it seems that they are both into you guys you will hurt his feelings.
Your best bet is already past. That would have been to let them know before you made a date that you weren't interested and move on. In the future the two of you might want to work out a signal to communicate to each other at club outings so that you know if you are interested or not without having to go off in a corner and discuss it (or you could just go off in a corner and discuss it). Either way, you need to get that message to each other before you end up in a predicament like the one you are in.
Based on your post, I gather that you realize your chance at even getting with this woman (if they do swing seperately) is pretty much past because you waited too late (ie. until the guy was also interested and thinking there was a chance for a 4-some). So that in mind and knowing that they go to the same club you do and you will have to see them again socially. Your best bet is to go ahead and have dinner with them, get to know them better and hopefully make some friends. Just be honest with them and let them know you aren't interested in swinging with them at this point before they continue to think something more will come out of this. If at that point THEY choose to break off contact, it's on them and not on you.
Julie
http://www.swingersboard.com