I think the house was invisible until the fidelus charm was broken then the muggles probably noticed something was wrong and began to investigate. It was probably only once Harry was safe that someone, maybe Prof Dumbledore or Prof McGonnagal, went back to Godrics Hollow to place another charm on the house so that the muggles could no longer see it. Did that make sense?