Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use airlift non-native implementations for GZIP, LZ0 and LZ4 codecs #2359

Open
asfimport opened this issue Aug 28, 2019 · 1 comment
Open

Comments

@asfimport
Copy link
Collaborator

@rdblue  pointed me to https://github.com/airlift/aircompressor which provides non-native implementations of compression codecs. It claims to be much faster than native wrappers that parquet uses. This Jira is to track the work needed for exploring using these codecs, getting benchmark results and making changes including not needing to pool compressors and decompressors anymore. Note that this doesn't include SNAPPY since Parquet already has its own non-hadoopy implementation for it. 

Reporter: Samarth Jain / @samarthjain
Assignee: Samarth Jain / @samarthjain

PRs and other links:

Note: This issue was originally created as PARQUET-1643. Please see the migration documentation for further details.

@zhiyun-deng
Copy link

Which "non-hadoopy implementation" is this referring to? Is this referring to SnappyDecompressor in org.apache.parquet.hadoop.codec?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants