Well, let me chime in with something. This may sound a little selfish, but the name of the topic is "Do size matter to the guys", so here I go....
1) I have never had a woman reject me because of my average-sized penis.
2) I have never been able to have an orgasm because of my penis size.
As such, I would think the answer to the question would be a resounding "NO".