I must commend you on the thought and communication it appears you two have put into this. You have explored the potential issues, and have addressed them. You're not too young based on age, and your ability to express your thoughts indicate you are mentally mature enough to handle this as well.
Now, the bad part: I have to agree 100% with GT. Swinging with a co-worker
might work, but why take the chance when there's quality folks already in the lifestyle? As GT mentioned, playing with people already into swinging, and knowledgeable about swinging takes so many variables off the table.
Good luck to both of you, welcome to the board, and stick around here! We like new board members.