I think it is inappropriate to categorize single males just because they are single or to totally give up on singles. Not all are bad. I (the male half) was a fairly in demand as a single, apparently because I was able to give what the couples desired. And just because I was single didn't make me an incompetent, social inept bafoon.
MrsPlay and I actually met on a swingers website while we were both single. We both had gotten out of rough marriages, and at the time weren't looking for anything serious. Well, things went well, and now we hope we will be together the rest of our lives.
And the issues you are experiencing aren't just limited to single guys. We had several experiences with couples which were exactly like what you had with the singles.