Without hearing him speak in his own defence, your boyfriend seems like a bit of a jerk. Or, at least pretty obsessed with swinging. This isn't totally uncommon, though, judging by all the married guys with "wives who don't understand" who write us every week. It is something that the two of you will have to wrestle down and come to some understanding on or you are going to have a lot of trouble.
And please don't get me wrong. In spite of all my preaching to you, if I was in your position I would feel exactly the same way, but with less patience and grace.