In Order of the Phoenix either Sirius says that Lupin got the prefect badge and not himself or James, his omission suggesting that Pettigrew wasn't in Gryffindor... unless none of them ever considered him a possibility for prefect.