But I thought prefects had more power.

And Hagrid could have meant that they were head boy and girl meaning they were better that they behaved better(well not James.)
But remember Lilly and James were both intelligent, but that's at least what they told Harry.

And Hagrid might of also said that because he didn't want Harry to hate his parents.