Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support returning and resuming partial results #17

Merged
merged 9 commits into from
Nov 4, 2024

Commits on Aug 14, 2024

  1. feat: support returning and resuming partial results

    Add a parameter `keep_failed` to `Dials.list_all()`,
    specifying whether to return the paginated partial results
    when an HTTP request fails.
    
    Add a parameter `resume_from` to `Dials.list_all()`
    to accept paginated partial results and resume the fetching.
    
    Add a field `exception` to `PaginatedBaseModel`
    to record the exception that causes the request to fail.
    
    If applied, users would be able to write something like:
    ```python
    data = None
    data = dials.h1d.list_all(<filter>, keep_failed=True, resume_from=data)
    ```
    and resume by <Up><Enter> from the terminal.
    ShamrockLee committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    3abbb79 View commit details
    Browse the repository at this point in the history

Commits on Oct 29, 2024

  1. feat: support returning and resuming partial results

    Add a parameter `keep_failed` to `Dials.list_all()`,
    specifying whether to return the paginated partial results
    when an HTTP request fails.
    
    Add a parameter `resume_from` to `Dials.list_all()`
    to accept paginated partial results and resume the fetching.
    
    Add a field `exception` to `PaginatedBaseModel`
    to record the exception that causes the request to fail.
    
    If applied, users would be able to write something like:
    ```python
    data = None
    data = dials.h1d.list_all(<filter>, keep_failed=True, resume_from=data)
    ```
    and resume by <Up><Enter> from the terminal.
    ShamrockLee committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    622704b View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    925606f View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    a79bf88 View commit details
    Browse the repository at this point in the history

Commits on Nov 4, 2024

  1. refactor: disable arbitrary_types_allowed in PaginatedBaseModel and s…

    …tore exc as string
    
    - In pydantic v1, model.schema() fails when using arbitrary types. More
    info: pydantic/pydantic#4079
    - Since exception as BaseException has no uses for now, we can safely
    transform it to a string and also store the exc_type for future debug
    gabrielmscampos committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    dc06d4a View commit details
    Browse the repository at this point in the history
  2. feat: allow resuming even if no exception ocurred

    - If we list an endpoint enforcing `max_pages`, we can return the last
    `next` string and resume from there
    gabrielmscampos committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    af89333 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    9ee5030 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    9452388 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    b3faaba View commit details
    Browse the repository at this point in the history