I think James left the cloak with Dumbledore to use as he saw fit for the order of the pheonix. If he needed someone to be invisible to watch Voldemort or some of the Deatheaters. It may not have been for Dumbledore himself to use but for someone Dumbledore may have wanted to use it for those purposes.