Ok, I admit I'm Mr Paraonoid when it comes to our swinging life

, so I would play it safe. My immediate (but probably incorrect) thought, would be "they are suspicious and are trying to sucker us into admitting we swing". I know that's silly, but hey, paranoia does that to you!
I would express a little surprise, then proceed to handle it much in the way that NakedInSeattle suggested. Just "play along" and see where it goes.
Seperately though, we aren't very keen on playing with our vanilla-life friends. But, that's been discussed already in several other threads.
Let us know what you decide and how it turns out!