I would say talk to your wife and let your concern be known to her. Let her know that this has raised a red flag for you and talk about whether or not you guys should considering breaking ties with them. That may be a little bit extreme, but if you're becomming uncomfortable with him, it may be best if your only future playdates are as two couples.