IMHO he's insecure about your relationship. Thus you can flirt with women but not other men (online or wherever). Other women aren't a threat - some men think
Also.... for the sake of your relationship you both need to stop doing whatever you are doing online and/or in swinging and lifestyle to get a grip on your relationship with each other.
I titled this "flash back" because in my 5 or so years in lifestyle I've seen some pretty bad behaviour from couples.....and I once briefly dated a man who used me so he could be seen as a couple and gain access to more swinger women.
He chatted a lot online to women and if I chatted with men, he'd get very upset. I could never see his chats, he claimed msn wouldn't save them (pffftt). His behaviour made me very uncomfortable and each time that I asked him to stop talking online and that we not continue in swinging until we were comfy with each other, he balked at me and told me I was being insecure.
One thing I've learned and believe...any man who accuses you of being insecure, cheating or who says he's not sure if he can trust you...is doing exactly those things to you. The men who freak out accusing you of cheating are taking the 'best defense is an offense stance'.
I would say he ceases and desists with the swinging and talking online and focuses on just you and him for a bit, or..... toss him and cut your losses.
"Here's my 10 cents my 2 cents is free"