Dito with all the above.
I think you actually need to go to a swinger club and watch other people around you as to understand how much af a common issue this is.
I believe swinging comes along with a lot of personal changes, several of those going against what our culture rule for us all. No one here will claim they're the perfect couple from the thoot paste TV advertisement, in any case, the concept for "excellence" comes from well grounded, every day things that the couple embrances and threasure, that have very few to do with the cultural dogma for a good marriage. As for us, we trend to be unconfident about "perfect" people and couples who put the esthetic aspects of life over everithing else that define ourselves as human beings.
Of course, there are personal tastes you may or may not fit, but you'd be rejected by lyfestilers just because you two weren't electible for the Baywatch main characters. I know I am not, as most of the lifestylers I've seen.
As a side tought, if you think lifestylers have the chance to be with anyone, someone resembling the cover magazine top model, as much as with normal people: talls, shorts, fat, slimmy, white, black, yellow, blonde, dark... etc. I think it reaches a point were you lack interest in these features, and you look for other things in people, those more subtle things that make them sensual, just the same ones you said you both found in eachother.
So, forget about this... BUT BEWARE... don't you ever dare to wear something green! Everyone knows poeple who wear green outfits worth less than the rest of us as to swing with.