I sort of wonder if its linked to what Dumbledore says in book 6, you know about the Dursleys afflicting appalling damage on, by implication, Dudley. But it's not like any of them actually realize what they've done as being really bad... so I guess i got to keep thinking on it.