Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MRG: improve restart by optionally writing batched zipfiles #102

Merged
merged 48 commits into from
Oct 4, 2024
Merged

MRG: improve restart by optionally writing batched zipfiles #102

merged 48 commits into from
Oct 4, 2024

Commits on Jul 24, 2024

  1. init changes for tmpdir

    @bluegenes
    bluegenes committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    10078e4 View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2024

  1. Merge branch 'main' into improve-restart

    @bluegenes
    bluegenes committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    786619f View commit details
    Browse the repository at this point in the history

Commits on Sep 13, 2024

  1. functions compiling, at least :)

    @bluegenes
    bluegenes committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    5ffa71f View commit details
    Browse the repository at this point in the history

  2. better error propagation

    @bluegenes
    bluegenes committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    45fef54 View commit details
    Browse the repository at this point in the history

  3. add tmpdir to urlsketch

    @bluegenes
    bluegenes committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    5420976 View commit details
    Browse the repository at this point in the history

Commits on Sep 27, 2024

  1. init TemplateCollection + associated changes

    @bluegenes
    bluegenes committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    09677a0 View commit details
    Browse the repository at this point in the history

  2. clean up a little

    @bluegenes
    bluegenes committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    a5e79de View commit details
    Browse the repository at this point in the history

  3. more cleanup

    @bluegenes
    bluegenes committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    578e9bb View commit details
    Browse the repository at this point in the history

  4. Template --> Build

    @bluegenes
    bluegenes committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    db0f065 View commit details
    Browse the repository at this point in the history

  5. method to update record, sig after building

    @bluegenes
    bluegenes committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    3b7fa22 View commit details
    Browse the repository at this point in the history

  6. add build_sigs impls

    @bluegenes
    bluegenes committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    622c38b View commit details
    Browse the repository at this point in the history

  7. add manifest new

    @bluegenes
    bluegenes committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    54256a0 View commit details
    Browse the repository at this point in the history

  8. add singleton method

    @bluegenes
    bluegenes committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    588fa8e View commit details
    Browse the repository at this point in the history

  9. use coll better

    @bluegenes
    bluegenes committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    50f9daa View commit details
    Browse the repository at this point in the history

Commits on Sep 29, 2024

  1. async write functions in BuildCollection, BuildManifest

    @bluegenes
    bluegenes committed Sep 29, 2024
    Configuration menu
    Copy the full SHA
    bbdb9cb View commit details
    Browse the repository at this point in the history

  2. update internal_location when writing sigs

    @bluegenes
    bluegenes committed Sep 29, 2024
    Configuration menu
    Copy the full SHA
    c8d7dff View commit details
    Browse the repository at this point in the history

  3. for proteins, ksize needs to be k, not k*3, in manifest

    @bluegenes
    bluegenes committed Sep 29, 2024
    Configuration menu
    Copy the full SHA
    e9e2811 View commit details
    Browse the repository at this point in the history

  4. aha! adjusted_ksize again. Use for sig, just not for manifest

    @bluegenes
    bluegenes committed Sep 29, 2024
    Configuration menu
    Copy the full SHA
    72b858b View commit details
    Browse the repository at this point in the history

  5. clean up now unused build_siginfo

    @bluegenes
    bluegenes committed Sep 29, 2024
    Configuration menu
    Copy the full SHA
    7684e30 View commit details
    Browse the repository at this point in the history

  6. read existing + attempt to filter

    @bluegenes
    bluegenes committed Sep 29, 2024
    Configuration menu
    Copy the full SHA
    97582b7 View commit details
    Browse the repository at this point in the history

  7. add batch reading, comparison logic

    @bluegenes
    bluegenes committed Sep 29, 2024
    Configuration menu
    Copy the full SHA
    cdbc4bc View commit details
    Browse the repository at this point in the history

Commits on Sep 30, 2024

  1. rm zip batching logic

    @bluegenes
    bluegenes committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    3df61df View commit details
    Browse the repository at this point in the history

  2. rm unused fn

    @bluegenes
    bluegenes committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    64fbc29 View commit details
    Browse the repository at this point in the history

  3. use MultiBuildCollection instead of allowing extend of BuildCollection

    @bluegenes
    bluegenes committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    e1e9506 View commit details
    Browse the repository at this point in the history

  4. merge in MultiBuildCollection changes

    @bluegenes
    bluegenes committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    b67ed58 View commit details
    Browse the repository at this point in the history

  5. rm unnecessary is_compat in BuildCollection

    @bluegenes
    bluegenes committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    0a4606c View commit details
    Browse the repository at this point in the history

  6. add back batching logic that was lost in merge to bring in MultiBuild… 

    …Collection
    @bluegenes
    bluegenes committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    2eafeb6 View commit details
    Browse the repository at this point in the history

  7. Merge branch 'main' into batched-zip

    @bluegenes
    bluegenes committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    9777797 View commit details
    Browse the repository at this point in the history

  8. fix batching, add gbsketch batch test

    @bluegenes
    bluegenes committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    70ed5f4 View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2024

  1. test param hashing; turns out it was fine - test sigs were abund!

    @bluegenes
    bluegenes committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    fef279d View commit details
    Browse the repository at this point in the history

  2. add batch logic to urlsketch

    @bluegenes
    bluegenes committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    367b38d View commit details
    Browse the repository at this point in the history

  3. create zips when needed to avoid initializing if no sigs to write

    @bluegenes
    bluegenes committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    7f9e418 View commit details
    Browse the repository at this point in the history

  4. clippy fixes

    @bluegenes
    bluegenes committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    dcb6b4b View commit details
    Browse the repository at this point in the history

  5. Merge branch 'main' into batched-zip

    @bluegenes
    bluegenes committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    71168f9 View commit details
    Browse the repository at this point in the history

  6. no guarantee on sig order in zip batches; mod test to ignore order

    @bluegenes
    bluegenes committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    e5db06e View commit details
    Browse the repository at this point in the history

  7. Merge branch 'main' into batched-zip

    @bluegenes
    bluegenes committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    b257cb8 View commit details
    Browse the repository at this point in the history

  8. add docs

    @bluegenes
    bluegenes committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    980b573 View commit details
    Browse the repository at this point in the history

  9. make sure batch_size is not negative

    @bluegenes
    bluegenes committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    ca1405a View commit details
    Browse the repository at this point in the history

  10. more doc

    @bluegenes
    bluegenes committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    820cc8c View commit details
    Browse the repository at this point in the history

  11. handle (+ overwrite) invalid/incomplete batched zipfiles

    @bluegenes
    bluegenes committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    b4e1673 View commit details
    Browse the repository at this point in the history

  12. clean up test

    @bluegenes
    bluegenes committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    baa8afb View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2024

  1. Merge branch 'main' into batched-zip

    @bluegenes
    bluegenes authored Oct 2, 2024
    Configuration menu
    Copy the full SHA
    7dd6638 View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2024

  1. unify usage considerations

    @bluegenes
    bluegenes committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    b583fb9 View commit details
    Browse the repository at this point in the history

  2. rename Params --> BuildParams

    @bluegenes
    bluegenes committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    9a7f965 View commit details
    Browse the repository at this point in the history

  3. one more rename

    @bluegenes
    bluegenes committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    5a3814f View commit details
    Browse the repository at this point in the history

  4. consistent test naming

    @bluegenes
    bluegenes committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    545fe9b View commit details
    Browse the repository at this point in the history

  5. clarify batch size differences between gbsketch, urlsketch

    @bluegenes
    bluegenes committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    54b6b97 View commit details
    Browse the repository at this point in the history

  6. add version for zip batching

    @bluegenes
    bluegenes committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    4298ff4 View commit details
    Browse the repository at this point in the history