Sounds to me like you need to take a break from swinging, change your phone numbers and work on your relationship. Rebuild the trust between the two of you and a few month down the road you can set some VERY SPECIFIC ground rules about what's acceptable in your relationship. If you both fully understand the rules there will be less chance of hurt feelings.
It sounds like you haven't done a very good job of laying those ground rules so you are both "flying blind" about what would be sexy, or hurtful. Rebuild the trust, lay down clear rules then you'll be ready to return to the game.
Good Luck!