When I got involved in swinging five years ago, I was a little apprehensive about my size too. I'm 6' tall, and I'm a grower, not a shower.
On most days I can get a nicely average 6" erection, but at it's worst (especially when it's cold) when the little bugger is soft it can look like a button on a fur coat. I had already been a nudist for 15 years, and was very comfortable being nude around both sexes in a neutral setting. But I was really worried about how my particular showing "down there" would be perceived in a sexually charged setting when it was possible that I would end up nekkid in front of a whole lot of other people.
I needn't have worried. As other posters in this thread have mentioned, women tend to look at the whole package. Even though I was nervous -- and I'm fairly shy anyway -- I just used the good ole first-date strategy: being well-groomed, dressing reasonably well (right down to the freshly laundered underwear), being attentive and a little flirtatious, etc. And of course, when it comes time to play, it's very rarely a case of lick-it and stick-it. In the swinging world, as in the vanilla world, women usually want a man to be able to use other body parts effectively as well. So, the females are most often looking for a well-oiled
machine, not just a single piston.
My first MFM really convinced me size wasn't the be-all and end-all. A couple that I met at some parties -- but whom I had never played with -- invited me to their house for some play. She had happened to see me at my softest and hardest at one of those other parties, so she knew what goods she was buyin'.
However, I'd never seen either of them nekkid. Well, at their house that evening eventually all the clothes were off and her husband had a unit that by all rights belonged on one of your larger species of livestock. Once I got over the initial shock of seeing what I was "competing" against, I got to work. And discovered that I wasn't competing. As far as she was concerned I was a variation on a theme. Seeing as how I've been invited back several times over the years, and that she has acted as a reference for me for a single female and another couple, I guess the variation was acceptable.
By the way, please don't construe any of that as boasting on my part. It's just how I came to the realization that "it's not the size of the dog in the fight..."
(Not that I want to fight: Make love, not war.)