I would do it the day before, even the day of, but only with a certain type of razor. Look for ones with very thin metal wires that run perpendicular to the blades. They make it really easy to shave your scrotum without nicking it. As for timing, the first day or so will give you the smoothest feel.
For some reason, I can shave all over my genital area in every direction, and not get any redness or ingrown hairs. My neck is a different story. On one side I have to shave up, the other side requires down.
On a related note, if you feel you have an excessively hairy chest/stomach/backside, and would like a neater appearance, try using hair clippers instead of shaving. I've found that using a #2 or #1 size guard really tones down the hairy look, and it doesn't feel stubbly. It also doesn't make me look like a pre-pubescent boy.
