I can't say that either of us have ever felt guilt. There have been several times where one or the other of us had what might be perceived as a little more fun than the other. A lot of it depends on stamina and some days it is there and some days it isn't.
As Brit_Pair said, "Provided that you both wanted to go to the club, and you both were happy with what took place while you were there, then you've no need to feel any guilt."
Next time may be your husband's turn to have just as much if not more play pleasure than you do. There is no reason to feel guilty about who gets more or less.....swinging is fun, feeling guilt about it isn't.
Lori