I guess another question is what was Dumbledore doing that day that meant that he was occupied with other things, other than picking up Harry? Maybe, he didn't want to risk the possibility of running into Sirius (remember, Dumbledore thought Sirius was the Potter's secret keeper) and getting angry at him. We know what McGonogal was doing - she was looking at the map in Privet Drive, and spying on the Dursleys, as a cat.