I can say it does happen, in our situation it was not bad at all, as we were not "outed". It was another couple that was seeking information for their own consumption. I do have to admit the thought was a little scary at first.
It was actually Swing Lifestyle, not a club that was the "undoing" of our anonymity with the aforementioned couple. The club we go to is a little expensive and I think that this high cost might keep away some of the looky-loo's. Swing Lifestyle on the other hand is free and if someone recognizes a very distinct dress, a tattoo, or your home in some of your pictures you might have some questions to answer. However you might not have a good time doing it.
There is always the CHANCE that you might meet someone you know or someone may find out. This "hobby" is all about meeting people and there is a very thin degree of separation in some places.
I think you should only bet what you have to lose. If your career were to end due to swinging, then by all means DON'T DO IT! But, if it is just personal ridicule you are worried about, ask yourself how much can you take and is it worth the price.