I am glad you have figured it out what you need to do for your happiness. Overall, this is just a boyfriend, and your happiness and weel-being should be your priority.
Swinging is not for everyone and there is absolutely nothing wrong with that.
Good luck to you!