Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow for finer-grained control of concurrency in file transfers #750

Open
wants to merge 17 commits into
base: main
Choose a base branch
from
Open

Allow for finer-grained control of concurrency in file transfers #750

wants to merge 17 commits into from

Commits on Jun 4, 2024

  1. Allow for finer-grained control of concurrency in file transfers

    @moradology
    moradology committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    e760b38 View commit details
    Browse the repository at this point in the history

  2. Update formatting of docstrings

    @moradology
    moradology committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    e262057 View commit details
    Browse the repository at this point in the history

  3. Bit by the snake

    @moradology
    moradology committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    86e8f39 View commit details
    Browse the repository at this point in the history

  4. Unkey after grouping to keep code cleaner

    @moradology
    moradology committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    6fa56ea View commit details
    Browse the repository at this point in the history

  5. Enable sync patching fsspec for file transfers

    @moradology
    moradology committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    7167f96 View commit details
    Browse the repository at this point in the history

  6. Add exponential backoff to avoid rate limiting issues

    @moradology
    moradology committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    e7b60e0 View commit details
    Browse the repository at this point in the history

  7. Linter demands unreachable exception. OK, fine

    @moradology
    moradology committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    487f039 View commit details
    Browse the repository at this point in the history

Commits on Jun 5, 2024

  1. Don't check size to avoid rate limiting issues

    @moradology
    moradology committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    1bceb26 View commit details
    Browse the repository at this point in the history

Commits on Jun 10, 2024

  1. Add flag to filter out already-transferred files

    @moradology
    moradology committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    1b691cc View commit details
    Browse the repository at this point in the history

  2. Use hashlib for determinacy of keyed groups

    @moradology
    moradology committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    7bb3509 View commit details
    Browse the repository at this point in the history

  3. Lint

    @moradology
    moradology committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    a851610 View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2024

  1. Attempt to log the file that is failing during xarray open

    @moradology
    moradology committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    c306c95 View commit details
    Browse the repository at this point in the history

  2. Add error handling and more logging for file transfer

    @moradology
    moradology committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    f78ce0d View commit details
    Browse the repository at this point in the history

  3. lint

    @moradology
    moradology committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    1c9321a View commit details
    Browse the repository at this point in the history

  4. Add verification of copy

    @moradology
    moradology committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    5ee286d View commit details
    Browse the repository at this point in the history

Commits on Jun 13, 2024

  1. Add logging to understand keyerror during index

    @moradology
    moradology committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    71f4061 View commit details
    Browse the repository at this point in the history

  2. Ensure that all URLs are returned

    @moradology
    moradology committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    55a6580 View commit details
    Browse the repository at this point in the history