dude, I'm 6.5" and I haven't had any trouble yet satisfying the women I've been with.
When I look down at the feller I don't consider it small, I consider it average just like my height, weight, shoe size or whatever.
it's easy to say "don't worry about it" but they've seen bigger and they've seen smaller, just in the same way you have with breasts. Believe me, I can have just as much (if not more) fun with a flat chested woman than I can with one with huge boobs.