Yeah, I agree with you there is no way Harry could ever be a Horcrux.
1) As you said why sould Voldemort want to destroy his own horcrux especially if it would lead to weakening him and killing him.
2) JK said that he wasn't a horcrux somewhere in a interview I read once.