I'd have to say that you are right. Most people with kids should have some idea as to when they would be able to get out to meet you. In this instance it could very well be that they were either not serious about swinging, one part of the couple was unsure about attraction, or they were just luke-warm in general. Is it kinda rude to answer so abruptly.....yep but what can you do?
