I'll echo what others have said. The OP is looking to fulfill a sexual fantasy. Polyamory is more about your emotional relationships. Swingers can be polyamorous and still be involved in swinging and people in polyamorous relationships can be involved in outside sexual contact. In my triad, if one of my partners and I have sex with another person or couple, that person or couple isn't automatically part of our relationship, that's a seperate issue.
To JandY, I wouldn't exactly characterize your relationship with the other couple as a polyamorous one, I'd consider it closed swinging. Unless you have an emotional bond that goes beyond that of friendship with sex with the other couple. There is no hard line though, so if it feels polyamorous to you, I'd be the last person to tell you it isn't
