Dumbledore did borrow the cloak. When he found out the cloak had been passed down through the Potter family...he would have realised that this was one of the Hallows.

It's said in the 7th book that normal Invisibility Cloaks lose their invisibility after a few years...but James Potter's one didn't lose it's invisibility

Dumbledore asked to borrow the cloak to have a look at it, and we can only guess that James Potter died before getting it back, and Dumbledore held onto it for Harry...as he knew he would one day come to Hogwarts.