Mrs. Denver2some here ...
Personally, I do not really care about a man's size. My husband is larger than average, but I have been with men who are smaller and larger ... and the size never determined my level of interest or satisfaction. In a swinging situation, I am happy if the man (1) can achieve and keep an erection, and (2) has some skill and is attentive.
I think I speak for a lot of women when I say that an attentive lover with a smaller size who is an interesting person wins hands down over a guy with a large penis who is basically "coasting" on having a huge one ("look at my wonderful penis") and not making any effort to please.
Sure, it can be novel to have an encounter with a really, really large fella every now and then ... but most women will tell you that really, really large can hurt if you aren't careful!
I hope this and the other replies here help ease your concerns. Have fun!