The same to me happened earlier this week. My wife and I were in a MMF threesome with her regular lover. We've done threesomes before so it wasn't first-night jitters. After being insider her for a while and then switching with her lover, I would get soft when I was out and it would take a while for me to get hard again. Watching the two of them go at it did help.
I even refrained from cumming the last few days, hoping to stay erect for the threesome but it didn't work much. However, when I did cum, I did spurt quite a bit.
I think my problem was fatigue - we have been going to bed late the last couple of days and getting up early. I think if I was well rested, I would have stayed hard.
One thing I did notice was that I did get hard when I was mentally stimulated - we did some roleplaying at the beginning and I was erect the entire time. Also, watching my wife have sex with her lover also excited me. But when I got distracted such as getting tangled in the sheets or hearing our dog bark, I got soft again.
So my suggestion is get plenty of rest and don't get distracted.