Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Explore Zstandard compression #11

Open
sebastian-nagel opened this issue Dec 8, 2021 · 1 comment
Open

Explore Zstandard compression #11

sebastian-nagel opened this issue Dec 8, 2021 · 1 comment

Comments

@sebastian-nagel
Copy link
Contributor

(depends on #10)

Zstandard compression is directly integrated into Parquet (PARQUET-1866). Also Athena now supports Zstd as Parquet compression. Time to explore whether switching from gzip to zstd brings improvements in storage size or query processing speed. In addition to Athena and Spark, support for zstd should be verified also Hive, Arrow, Pandas and other common tools to read/process Parquet files.
@sebastian-nagel
Copy link
Contributor Author

By passing --outputCompression=zstd the Parquet files are compressed using Zstandard compression. However, the zstd-compressed files are slightly bigger compared to gzip. There is no effect on the Parquet file size visible by adjusting the zstd compression level (eg. --conf spark.io.compression.zstd.level=22) - this option might only affect other output formats.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@sebastian-nagel