Tom Riddle was a memory from Memory Land. He could not have won because, as a memory, he had no current insight. He gathered facts from Ginny and strung them together in a rememberd way. He did not understand the significance of the Phoenix or the Sorting Hat because when he was at Hogwarts, Dumbledore was not as important as he grew to be in Harry's time. Riddle underestimated Dumbledore's influence.
As for the curse rebounding, it all has to do with the Prophecy. Voldemort cannot kill Harry because Harry is destined to kill Voldemort. The curse bounced off of Harry and struck Voldemort as an indication of things to come. If Voldemort were wise, he would steer clear of Harry instead of tempting fate. But pride goeth before a fall and Voldemort's pride tells him that he is mightier than fate.
Of course, pride altho goeth before a good story line. So Potterheads, give it up for the greatest of the 7 capital sins--Pride! Without it, we would not have Harry Potter books, movies or chat! Hip Hip Hooray!!!