Welcome to the Swingers Board, Peter
Are you a cuckold?
Nope, I think you just enjoy getting off, on everyone else getting off around you. Especially, your partner

I can get caught up in the same as well.
I would be concerned about your wifes concerns though. They're valid, if some of the men/women belittle you, and your not catching the vibe.
Ask your wife what she perceives others think of you. Be prepared for the worst and hope for the best, but listen closely to what she says.
fun4ds