I have found that when people ask where we went last night a great answer is:
"We spent Saturday night at the swing club like we do every Saturday night with friends, you should have been there, it was great!"
Now. for people not into swinging they laugh and think you are kidding and leave you alone.
The ones that are into swinging now know you are also and they might show up to party some time.
Either way, it works!