Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: Fix decompressing all zlib data for NIDS #1394

Merged
merged 1 commit into from
Nov 11, 2024

Conversation

dopplershift
Copy link
Member

Description of Changes

Code was only expanding the buffer by 10000 bytes as it read through. This resulted in blowing the array bounds when the last, noncompressed chunk of data was larger than this. This was easily broken by products with a single compressed chunk, followed by all uncompressed data. A representative data file and smoke test is included.

This was actually triggered by a failure to build Siphon docs trying to access this data over CDMRemote.

PR Checklist

  • Link to any issues that the PR addresses
  • Add labels
  • Open as a draft PR
    until ready for review
  • Make sure GitHub tests pass
  • Mark PR as "Ready for Review"

Code was only expanding the buffer by 10000 bytes as it read through.
This resulted in blowing the array bounds when the last, noncompressed
chunk of data was larger than this. This was easily broken by products
with a single compressed chunk, followed by all uncompressed data. A
representative data file and smoke test is included.
@dopplershift dopplershift added bug Something isn't working radial radial data formats labels Nov 8, 2024
Copy link
Collaborator

@lesserwhirls lesserwhirls left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good! We'll want to see how this works out on the AWS radar server focused TDS to keep an eye on memory usage, but I think we'll be ok.

@lesserwhirls lesserwhirls merged commit 7f24638 into Unidata:maint-5.x Nov 11, 2024
8 checks passed
@dopplershift
Copy link
Member Author

FYI, this particular compression should only impact NIDS/Level 3, there should be no impact on the Level 2 decompression.

@dopplershift dopplershift deleted the fix-nids-deflate branch November 11, 2024 20:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working radial radial data formats
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants