This isn't that uncommon of a rule really as you can see from these posts:
Do you reserve kissing for your partner only? Do you find kissing to be a normal part of swinging play? Can't Handle heavy kissing between wife and another man Opinions about Kissing?
As far as others getting upset when you tell them of this rule. I think that most people who don't have the rule don't really understand why someone else would. That doesn't mean tho that they should just accept it as your rule and leave it at that. It's up to them to determine whether or not it is a deal-breaker as far as playing with you goes. But they really don't have any right to get upset about it.