-
Notifications
You must be signed in to change notification settings - Fork 50
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[ENH] - Ensure completeness when fetching all pages using REST API #859
Comments
@peytondmurray was this addressed by the environment fetch enhancement you made recently? |
No, this has to do with the pagination machinery, not the underlying fetch. Let's keep this open. |
Coming back to this, I think we can most easily fix this by simply sorting paginated results by time. I'm sure it's possible to do with the other proposed methods, but this just seems simplest to me. @trallard and @krassowski do you have preferences about this? This seems like an easy fix, so once I have your input I'll happily fix this. |
Yes, that's one of the proposed methods :) |
Okay, with two votes for sorting by time, I'll do this. Stay tuned! |
Feature description
Currently the pagination API uses page-based pagination without sorting by time of creation/modification. This leads to incomplete results when iterated over the pages in dynamic systems with multiple modifications happening at the same time when results are fetched.
For example, if during the process of iterating over the pages of environments (
api/v1/environment
) an additional environment is created, either:this is because the environments are sorted by namespace and name. For a more concrete example see the details below.
Say we have five environments A, B, D, E, F, and page size equals 2.
User performs two actions:
If the database commit happens after the second page is fetched, this could lead to the form getting replies:
If many users are creating environments, the admin would be randomly missing some environments.
If the results were sorted by date of environment was created (/last modified if environments can be renamed), then this is not a problem, because the replies could be (assuming A, B, D, E, F were created in this order):
Originally posted in nebari-dev/nebari#2599 (comment)
Instead, one of pagination implementations which guarantees data completeness should be used across conda-store REST API:
Value and/or benefit
No randomly missing items (e.g. environments) in paginated replies.
Anything else?
No response
The text was updated successfully, but these errors were encountered: