Well, three things come to mind. Alcohol, nervousness and focus.
Me (M), I can drink a whole lot before I become intoxicated. But, when we're meeting a couple, one's my limit. Anything greater than that, unless the period prior to any activity is a long time, affects performance. And, yes, it does that even if you're taking Viagra, etc.
Nervousness effects us for any first time meeting. Yes, even if we've talked extensively with potential partners. This definitely will contribute to performance anxiety.
Focus is always a factor, too. Some people are more into watching their spuse be pleasured and that can be a problem if they're not concentrating on their immediate partner's pleasure or their own. For our first couple, the lady wanted to be assured that I would pay attention to her rather than watching my wife.
Some of us like long foreplay, concentrating on our partner rather than worrrying about whether we get off. We like long foreplay as part of what "trips our trigger". I don't mean 10 minutes, I mean one or two hours.
There's a lot to compatability. That's why it can be very difficult to find a couple with whom you can hit off. You're dealing with the wants and needs of four people.
You shouldn't be hard on yourself. It's really nobody's fault that you're encountering difficulties. Perhaps you are someone who needs to take it a little slower or perhaps this just isn't for you. Perhaps, hubby should arrange for an mfm for you, so you can be the center of attention. We think you deserve that at least

.