Improve list-files endpoint

[aulneau]

I'm working on some improve Gaia related SDKs for micro-stacks, and the list files endpoint currently is very basic -- every time you call it you need to manually re-fetch if there are many pages of data.

I was thinking at a high level that it'd be great to do a few things:

• change the response to {results: string[], offset: number; limit: number; total: number}
• allow passing these query params:
  • prefix: return files that start with this prefix
  • limit: adjust the limit of results
  • offset: currently page is this param

I think with these changes, it's possible for this endpoint to be vastly improved.

[jcnelson]

As much as I would love to see this, IIRC the API is constrained by the API that the various cloud storage backends offer. Many of them only allow pagination. Maybe there's some kind of page index the node driver could maintain, but even then you'd have to keep it coherent with the upstream servers every so often in case you put something upstream without going through Gaia.

[aulneau]

digging in a bit more, it's actually possible to pass a prefix already (not well documented).

the drivers have the ability to take arbitrary page sizes, but it's not being passed from the endpoint to each driver.

I think the only thing missing would be the ability to enumerate the paths that match a given prefix -- that would help a bit with fetching data in a more efficient way without needing to crawl each page