Did you think he would do finish off harry if he were there?
I don´t think he was under a invisible cloak or something. He was somewhere else, but not by the Potters.
That doesn´t result in a sense.