Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ability to track & restart timeouts? #88

Open
cboettig opened this issue Sep 27, 2023 · 0 comments
Open

ability to track & restart timeouts? #88

cboettig opened this issue Sep 27, 2023 · 0 comments

Comments

@cboettig
Copy link

When a cube is constructed with a large vector of asset URLs (e.g. from a stac catalog), it's not uncommon for some of those resources to time out. The most straight forward example can be produced with the microsoft planetary computer STAC API, where the signed URL signatures expire after 45 minutes. This means that a long-running computation on remote assets can fail silently to get many assets. (A similar issue can occur in other contexts where an arbitrary URL times out, though in such cases we can perhaps just set the GDAL configuration regarding timeout and retries to be more stubborn?) In any event, it would be very helpful if the a list of URLs which fail could be logged and retrieved in some way? (Maybe this already exists?) Given the lazy-eval design this probably couldn't be on function return, but maybe failed URLs could be logged to disk somewhere and a helper function could be used to retrieve (and possibly re-query or re-sign and then requery?) such URLs?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant