The Dursleys would tend to avoid him more. They aren't pleasant muggles to be with so Harry is spared some grief. However, that adds to the isolation Harry feels when he's away from Hogwarts. He doesn't have anyone to talk to except Hedwig. So while he's living with the Dursleys, Harry has got to feel like he's in detention.