Well, I figure anyone who looks at old Tom is gonna know he's not exactly all that human. So, naturally, many would be susupicious. Those who have the resources (ahem, Dark family lines, such as the Blacks) would be able to investigate if they were so inspired to do so.
Wouldn't it be ironic if the Hor-wazzit ended up in Sirius's old house with all the things Harry just inherited?