This is true on more than one topic. I work in an extremely religious company. While I have never claimed or agreed to any religion I most definitely do not put it out there that I am an agnostic. Being a swinger isn't going to come up at all. But neither does the topic of sex. AP has it much easier. She works with mostly other women. While it hasn't specifically been asked of her they do know we are pretty adventurous when it comes to sex. While her coworkers know we are pretty open and kinky we believe they would act completely different if the actual term "swinger" was used.
It's odd but it would be more accepted to cheat on a spouse than to be labeled swinger for some reason.