When I first started swinging I felt young in the scene, I was 35. The clubs we were attending and the parties we went to had an average age group of 35-55. There are many different groups of swingers here in Toronto now. In some clubs at 44 I feel positively OLD...in others I feel just right.
I would never say you are too young to swing but as a couple I would suggest you seriously look at why you want to??? At your age I certainly found more than enough sexual thrills outside the lifestyle to keep me busy.
You will always run across those who will not swing with you because of your age and there will be some you will feel the same about. Try for a younger age based club if there is one in your neighborhood.
Good luck, play safe and welcome...