Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added zstandard compression tooling, added to bytes codec for JSON serialization #438

Merged
merged 12 commits into from
Dec 17, 2024

Conversation

dotsdl
Copy link
Member

@dotsdl dotsdl commented Dec 5, 2024

Closes #321.

Tips

  • Comment "pre-commit.ci autofix" to have pre-commit.ci atomically format your PR.
    Since this will create a commit, it is best to make this comment when you are finished with your work.

Checklist

  • Added a news entry

Developers certificate of origin

Copy link
Member

@IAlibay IAlibay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do the NPY_DTYPE and NUMPY codecs also need updating here?

@atravitz atravitz changed the title Added zstandard compression tooling, added to bytes codec for JSON serialization [WIP] Added zstandard compression tooling, added to bytes codec for JSON serialization Dec 9, 2024
Copy link

codecov bot commented Dec 10, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.50%. Comparing base (339143f) to head (b866870).
Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #438   +/-   ##
=======================================
  Coverage   98.49%   98.50%           
=======================================
  Files          36       37    +1     
  Lines        2131     2143   +12     
=======================================
+ Hits         2099     2111   +12     
  Misses         32       32           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@atravitz atravitz changed the title [WIP] Added zstandard compression tooling, added to bytes codec for JSON serialization Added zstandard compression tooling, added to bytes codec for JSON serialization Dec 11, 2024
Copy link
Member Author

@dotsdl dotsdl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking great @atravitz! I gave this a full review, and generalized test_legacy_bytes_uncompressed to apply to all CustomJSONCodingTest subclasses.

Can you add a news entry? After that I think this is good to go!

@dotsdl
Copy link
Member Author

dotsdl commented Dec 17, 2024

Do the NPY_DTYPE and NUMPY codecs also need updating here?

@IAlibay they do not! These convert to bytes, and then the BYTES codec is used to convert these into a string representation. So, this PR compresses any bytes objects being serialized, and will automatically apply to NPY_DTYPE and NUMPY objects when we attempt to serialize them.

Copy link
Member

@IAlibay IAlibay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couple of things, otherwise lgtm

gufe/tests/test_custom_json.py Show resolved Hide resolved
gufe/tests/test_custom_json.py Outdated Show resolved Hide resolved
@atravitz
Copy link
Contributor

pre-commit.ci autofix

@atravitz atravitz merged commit 128da4a into main Dec 17, 2024
13 checks passed
@atravitz atravitz deleted the zst-bytes-compression branch December 17, 2024 22:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add compression approach to Numpy arrays when passed through JSON encoder
3 participants