Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

decompress jsonl in jsonl reader / filesystem #327

Closed
rudolfix opened this issue Jan 19, 2024 · 2 comments
Closed

decompress jsonl in jsonl reader / filesystem #327

rudolfix opened this issue Jan 19, 2024 · 2 comments
Assignees
Labels
good first issue Good for newcomers source new feature adds new feature to existing source

Comments

@rudolfix
Copy link
Contributor

Which source is this for
filesysten

Describe the data you'd like, be specific, link the docs
filesystem contains a helper (transformer) to read jsonl files (readers.py). please add an option to unzip compressed files

  1. if the file extension is gz, apply decompression automatically (you have the file path in FileItem)
  2. also provide an option to override that behavior, (ie. decompress with true/false/auto)
@rudolfix rudolfix added good first issue Good for newcomers source new feature adds new feature to existing source labels Jan 19, 2024
@IlyaFaer IlyaFaer self-assigned this Jan 30, 2024
@IlyaFaer
Copy link
Contributor

The issue requires this core PR to be merged first
Just a minor change and a test case will be required after that

@rudolfix
Copy link
Contributor Author

@IlyaFaer yeah stuff is merged and stuff is unzipped automatically, closing

@github-project-automation github-project-automation bot moved this from Planned to Ready for Deployment in Verified Sources Jan 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers source new feature adds new feature to existing source
Projects
Status: Ready for Deployment
Development

No branches or pull requests

2 participants