Skip to content

[FEA] cudf-polars: Add chunked Parquet sink support using the libcudf writer #18969

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Matt711 opened this issue May 27, 2025 · 0 comments · May be fixed by #19015
Open

[FEA] cudf-polars: Add chunked Parquet sink support using the libcudf writer #18969

Matt711 opened this issue May 27, 2025 · 0 comments · May be fixed by #19015
Assignees
Labels
cudf.polars Issues specific to cudf.polars feature request New feature or request

Comments

@Matt711
Copy link
Contributor

Matt711 commented May 27, 2025

Is your feature request related to a problem? Please describe.
cudf-polars supports chunked parquet scans using the libcudf reader. This feature request proposes doing the same thing for parquet sinks.

Describe the solution you'd like
Enable libcudf’s chunked Parquet writer as the default sinking strategy in cudf-polars. The GPU engine config should allow enabling/disabling chunked writing;. And we could reuse the same config option currently used for chunked parquet scans.

Additional context
xref #18567

@Matt711 Matt711 self-assigned this May 27, 2025
@Matt711 Matt711 added feature request New feature or request cudf.polars Issues specific to cudf.polars labels May 27, 2025
@Matt711 Matt711 linked a pull request May 28, 2025 that will close this issue
3 tasks
@GPUtester GPUtester moved this from Todo to In Progress in cuDF Python May 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cudf.polars Issues specific to cudf.polars feature request New feature or request
Projects
Status: In Progress
Development

Successfully merging a pull request may close this issue.

1 participant