It may seem kinda shallow but, most people are who are into swinging at some point are looking for some sort of physical play right? That is not to say that there are not swingers who are just friends. But it is all about preferences. And if someone says they are not into larger people, wouldn't you want to know that from the beginning . It is in all fairness, just being honest with them. I do however agree that there are some rude people out there that need to use a little tact in the way they word there likes and dislikes.
To answer the initial question, size would have something to do with my decision on who to swing with. From what I've seen in the adds and board posts, almost everyone has something that would be a no go for swinging no matter what it is, ie. size, color, str8 or bi, smokers etc. How about the others that no one seems to have a problem with ruling out "Must Be Well Endowed or No Single Males I have yet to hear anyone complain about being shot down for one of them.