Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[v3] Download special char issue #2366

Merged
merged 3 commits into from
Nov 20, 2024
Merged

Conversation

jace-roell
Copy link
Contributor

What It Does
This PR resolves an issue where special characters could become corrupted when downloading large files. The corruption was caused by each chunk buffer being converted to a string when calling the processNewLines() function and then converted back to a buffer. This PR introduces support for passing buffers directly into the processNewLines() function to prevent corruption.

How to Test
The issue arises when a special character appears as the last character of a stream chunk. To reproduce, create a PDS file containing exclusively special characters with a size larger than the default chunk size (>8KB). In such cases, the last character of each chunk may be corrupted and appear as two unknown characters.

To observe this issue, download a PDS file that meets the criteria using the following example command (assuming the use of IBM-1147 encoding):

zowe zos-file download ds <PDSMEMBER> --encoding "IBM-1147"

where <PDSMEMBER> is a file larger than 8KB containing special characters.

Example Line (about 105 are required in a file to exceed 8KB):
àèéìòùÀÈÉÌÒÙ°àèéìòùÀÈÉÌÒÙ°àèéìòùÀÈÉÌÒÙ°àèéìòùÀÈÉÌÒÙ°àèéìòùÀÈÉÌÒÙ°àèéìòùÀÈÉÌÒÙ°à

Review Checklist
I certify that I have:

Signed-off-by: jace-roell <[email protected]>
@jace-roell jace-roell self-assigned this Nov 18, 2024
@jace-roell jace-roell marked this pull request as ready for review November 18, 2024 16:22
Copy link

codecov bot commented Nov 18, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.27%. Comparing base (33b8d94) to head (e5f82b9).
Report is 5 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #2366   +/-   ##
=======================================
  Coverage   91.27%   91.27%           
=======================================
  Files         636      636           
  Lines       18071    18083   +12     
  Branches     3895     3862   -33     
=======================================
+ Hits        16495    16506   +11     
- Misses       1575     1576    +1     
  Partials        1        1           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.


🚨 Try these New Features:

Copy link
Member

@traeok traeok left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested with the example string in the PR description and was able to reproduce the same output without any data loss.

Thanks Jace for the fix! LGTM

Copy link
Member

@gejohnston gejohnston left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once a question on the V2 PR is resolved and applied in this PR if needed, then this PR looks good to me.

packages/imperative/src/io/src/IO.ts Show resolved Hide resolved
Copy link
Member

@zFernand0 zFernand0 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! 😋

@zFernand0 zFernand0 merged commit b2ae2fb into master Nov 20, 2024
20 checks passed
@zFernand0 zFernand0 deleted the v3-download-special-character-issue branch November 20, 2024 13:57
@zFernand0 zFernand0 added the release-patch Indicates a patch to existing code has been applied label Nov 20, 2024
Copy link

Release succeeded for the master branch. 🎉

The following packages have been published:

Powered by Octorelease 🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release-patch Indicates a patch to existing code has been applied released
Projects
Status: Closed
Development

Successfully merging this pull request may close these issues.

4 participants