Tell him pretty much what you have told us.
You like them, they are great to hang out with and you are happy that his wife and your husband have such great chemistry. But that 4 way chemistry is just difficult to find. And that you don't have that level of chemistry between the two of you. It is nothing specific, you can't put your finger on it, it is just not there. And that you hope that it does not ruin things between his wife and the your husband as they are lucky enough to have that chemistry.
Now, all that I have said with the assumption that you and your husband BOTH have been honest with the other couple, i.e. that neither of you have lead them to believe you might warm up to the other man, or that you just take things slowly, just give you some time.
Since they know you play separate and it seems they do as well, then they probably understand that there is not always a 4 way chemistry. Hopefully he/they can understand this and deal with it. If he/they are not mature enough to deal with it then you may loose a friend and your husband may loose a playmate. But, it is either that or take one for the team, which I do not recommend. Or try to keep things as they are which I also do not recommend as it will only get frustrating for the other man in the long run. It is better to be honest and get the air cleared as soon as possible.