Guess you need to decide which ones you can trust to attend. We have invited both at times, only had one regret over the years (one guy who could not keep his mouth shut with comments like - gee I would like to have her etc; they did not get invited back). You sometimes have to manage the hot tub a little closer
with a mixed group. Our problem is we have Nudist, Textile, and Swingers friends...now that get to be a chore...but makes for a great Halloween parties that last into the night...but since our textile friends know we are nudist (not swingers) we always make sure everyone knows that clothing optional time starts at 10 or 11 PM, and they all know no clothes in the hot tub ever...even had textile friends strip for the hot tub. Most swingers know it not a play date at these parties, unless they have been invited to stay the night up front. Don't stress it, just make sure they know the ground rules. I have even done seperate invitations to the diffrent groups, kind of anal I know. I have the diffrent groups arrive at diffrent times... my wife is more of a planner than me... with big groups she finds a way to ID the diffrent parties, using a diffrent color name badge, party hat etc... when needed.