My husband and I almost always reply courteously, either in e-mail, or even in person, that we aren't interested or that simply the 'vibes' are not right. Three-quarters of the time they respect that and all is well. I think that is because we have pretty well screened people to get to the go/no-go point in the first place. By that time we pretty much know how to address the issue.
What makes me mad, and it could just be me, are those that follow up with 'Why not?' To me, if we say 'no thanks', then the subject is over. Maybe its none of their business as to why not. In the very least they need to respect that as our answer.
We have been rejected at times as well, and when that happens we don't push it. If we did, we know SOMEONE was going to be strained somehow. There are other people to get to know without getting bent out of shape over each rejection.