The books don't say anything about his grandparents. But in one of her latest interviews J.K says that Lilly's parents and James' parents have all died of natural causes. She wanted to make sure that Harry's character was as lonely as possible, or he would have of course lived with them instead of with his aunt and uncle.