Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add ParquetObjectReader::with_runtime #6248

Closed
tustvold opened this issue Aug 14, 2024 · 3 comments · Fixed by #6612
Closed

Add ParquetObjectReader::with_runtime #6248

tustvold opened this issue Aug 14, 2024 · 3 comments · Fixed by #6612
Assignees
@wiedld @itsjunetime
Labels
enhancement Any new improvement worthy of a entry in the changelog help wanted

Comments

@tustvold
Copy link
Contributor

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Tokio is a cooperative threadpool and has strong assumptions that tasks yield frequently. Running CPU-bound work on the same threadpool results in poor throughput and can lead to connections getting dropped.

Describe the solution you'd like

We should add a with_runtime option to ParquetObjectReader that takes a tokio handle and spawns all work to the provided runtime. This encourages decoupling CPU-bound query execution / parquet decoding, from cooperatively scheduled IO.

Describe alternatives you've considered

I'm aware of a number of codebases, such as InfluxDB IOx, that implement this spawning logic at the ObjectStore boundary instead.

Additional context

#4040 attempted to add something similar to object_store, but concluded this really belongs in the integrations

#5882 may be related to this
@tustvold tustvold added enhancement Any new improvement worthy of a entry in the changelog help wanted labels Aug 14, 2024
@tustvold tustvold mentioned this issue Aug 14, 2024
Open
tustvold added a commit to tustvold/arrow-rs that referenced this issue Aug 14, 2024
@tustvold
Add ParquetObjectReader::with_runtime (apache#6248)
cad06ea
@tustvold tustvold mentioned this issue Aug 14, 2024
Closed
@alamb
Copy link
Contributor

alamb commented Oct 2, 2024

I believe @wiedld plans to pick this one up

@wiedld
Copy link
Contributor

wiedld commented Oct 2, 2024

take

@itsjunetime
Copy link
Contributor

take

itsjunetime pushed a commit to itsjunetime/arrow-rs that referenced this issue Oct 21, 2024
@tustvold @itsjunetime
Add ParquetObjectReader::with_runtime (apache#6248)
62cb6be
@itsjunetime itsjunetime mentioned this issue Oct 21, 2024
Merged
alamb added a commit that referenced this issue Nov 2, 2024
@itsjunetime @tustvold @alamb
Add ParquetObjectReader::with_runtime (#6612)
22bc772 
* Add ParquetObjectReader::with_runtime (#6248)

* Add test for ParquetObjectReader::with_runtime and fix clippy complaints

* Switch ParquetObjectReader runtime tests to not depend on tokio_unstable anymore

* Add doc-comment for test_runtime_thread_id_different

Co-authored-by: Andrew Lamb <[email protected]>

* - Add comment about why we don't use spawn for metadata
- Remove outdated comment about target_has_atomic
- Add test to verify reader fails when spawned on a shutdown runtime

* Avoid use of Infallable and From conversion

---------

Co-authored-by: Raphael Taylor-Davies <[email protected]>
Co-authored-by: Andrew Lamb <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@wiedld wiedld

@itsjunetime itsjunetime

Labels
enhancement Any new improvement worthy of a entry in the changelog help wanted
Projects
None yet
Milestone
No milestone
4 participants
@alamb @tustvold @wiedld @itsjunetime