First off, I think this is something where you need to sit down and talk honestly with your wife and with your friend. First seperately then together. This may be a case where your wife gets upset but I really think it is a risk you have to take.
Talk to your wife about what was going on with his calls and what not and how that bothered you, find out how she felt about it as well. If it didn't bother her, explain to her why it bothered you. That this is a swinging relationship and everything should be about the 3 of you not just two of you (unless it's you and and her). Explain these same factors to your friend. Let him know that you enjoy his friendship and that the swinging expansion of your frienship was great but that he crossed a line. If he can't handle this conversation then perhaps you should cut him off all together.
If you can make him see where he stepped over the line then I would suggest you get all 3 of you together to sit down and set up some ground rules for what should and should not be going on in this relationship. Once you have all agreed to them then you can open the relationship back up to sex. If you can't all agree on them, then I would suggest you leave sex out of the picture and keep him as a friend. I think frienship is much more important than a swinging relationship.
Neither your marriage or a good friendship are worth risking just to have someone to swing with.
Julie
http://www.swingersboard.com