As a woman for whom size DOES matter, I was reluctant to post on this thread.
However, seeing as I am also a woman who has difficulty staying out of anything I consider to be interesting discussion, I have given up fighting myself.
As I said, size DOES matter to me. I haven't been with too many "small" men, but when I have, they have been both small AND ineffective. However, I will also state that at 7", your husband is hardly what I would call small. In fact, being that size matters to me, my "minimum" preference is 7", as my husband is 8, and I realize that not every man is "as lucky as my husband (and me!)"

. As the statistical average for penis size is around 5.75", your husband definitely would fit in among the "well-endowed" category. As Spoo said, if every man at this party was larger than your husband, that was definitely a strange phenomenen indeed.
For those here who have had more experience with swinging, I have noticed that it definitely depends on the way that a man uses his package, no matter what the size. There are those with smaller penises, who can really "rock a world", then there are those with larger penises, who cannot last more than 3 strokes, or they jab too hard with it (hitting that ultra sensitive cervix ... OUCH!), or they cannot maintain an erection. All sorts of problems befall men of all endowments. It's just that I seem to find the ones with most of those problems are the owners of smaller than average penises. :rollseyes So perhaps that's where my bias comes in.
I will state that I have had one play partner that I will not play with again, and the primary reason is that he is very small. He told us 7", in reality it's around 3" fully erect, and very thin (maybe around 1" circumference). He has difficulty achieving an erection, then difficulty maintaining when he does achieve. His sex style is so bizarre ... very without rhythm, just kind of wild and a few thrusts here, nothing, a couple more, then nothing again. It's all so very strange. I have given it three opportunities to improve, but it hasn't so I am not going there again. Incidentally, he is part of a married couple, relatively active in the lifestyle. Though they normally do MFMs, and the wife is constantly in search of "well endowed" men ... wonder why???? lol
I would, as some others have suggested, advise you to ask your husband to read this board, and do some searches on penis size, because there are many threads on that particular subject. Let him work on getting past this feeling of not measuring up, because believe me, he does. And more important than that, you have yourself said he is an outstanding lover. That in itself says all it needs to say.
Let me also add that I am in the minority, apparently in the world at large, as well as on this board. For most women, it seems size plays a very small part in sex for them (no pun intended.)