Well I am originally from a small community (Ironton, Ohio)...so I know how everyone seems to know everyone else's business. And that rumors spread like wildfire.
I think you may need to talk with the neighbor and try to set him straight (give him the same story you gave your son would probably be the best route)...but some people let their imagination run wild and nothing you can say will change their way of thinking. However, there are plenty of partners out there that accuse their spouses of the very things that they are doing because of misplaced guilt and projection...people like your neighbor would drive me nuts. lol
Is the son or the neighbor blabbing to others? Just deny, deny, deny....they have no proof but the words of a 19 year old.
Good luck
Maria