Maybe he's just nervous that he'd lose you. Perhaps he feels that the relationship is tenuous, if for no other reason than that you met through swinging and you could meet someone else.
The comment about "average looks" was insensitive, but we all have moments like that we wish didn't happen. I'd hesitate to base any opinion on him on that alone.
Maybe he just needs time to get his shit together. Give it to him. If he won't shut up about it in the mean time, tell him explicitly that he needs to get his shit together before you move on.
Talking the issues over is fine, but he seems to be giving you pronouncements of what it will be like on a daily basis. Your vote still counts, tell him you feel like he doesn't understand that.
Alternatively, he might be wrestling over the idea of his sexuality. Just a thought. Just because a couple is involved doesn't mean the guys have any physical interaction with each other. Most of us don't. Seems a bit off the wall since you both came from the lifestyle to begin with.
Damn, swinging must be a lot easier on singles.