Based on what you said, the fact that you had told him already that nothing was going to happen and all, I would say you are somewhat justified in your feelings. I do however, think you should have told him that it was not alright for him to play. Reason being you were already not feeling well, you were not in the mood due to Comfort level and diabetes.
the only way swinging works is if both are in agreement and communicating. Undercutting a spouse just makes things ruff in the marriage.
I think all of us at one point or another may have given a " do what you want " type answer at one point or another, it just causes strife that we have to talk through and sometimes its ruff.