I agree that this would be a fun idea, but you'd have to make sure everyone was fully aware of the 'dress code'... also, because it's a swingers party, what do you do AFTER the clothing is removed?
I personally would do either wristbands or something similar. That way you could catch people at the door as the host and go over the color designation with them to make sure there were no mistakes.
Cheers!