you are a "package deal", and thats that... of course the female half of the other couple must find him attractive, and i suppose if she is'nt into a bigger man then thats not her thing, but.. I'm sure there are plenty of women out there that prefer a bigger man. so you will find someone.
I think the feeling of "taking one for the team" is something that springs to mind here,.. and most often can cause a problem for most.