I can't give a lot of advice on this, as I am female. But, there was a couple we were with once and it was their first time swinging. He couldn't keep it up.

I was nice, and not too disappointed. I think it's a normal thing really.
Heck, my dh has the opposite problem-when he's nervous like that, he won't cum forever!!! Our first time together, we went at it on and off for 8 hours!!!!!!! He just wouldn't cum! Our first orgy, he banged this one chick for an hour w/o cumming! It may sound wonderful, but that can be just as frustrating as not being able to get it hard.
I agree with the PP, look into Viagra. It may give you the boost you need. Heck, it could make it so you could go all night!