feat: compile and execute partitioned query #806

bjchambers · 2023-10-11T03:11:53Z

The main change here is having the sparrow-execution tests compile
from a logical plan, created within a session, and execute things.

Changes Include

Fix naming field_ref vs. fieldref (matches Python)
Tweak pipeline scheduling so that only transforms are grouped together.
Tweak pipelines so that each may output to multiple consumers
Introduces a SourceTasks so we can poll all inputs on a single
tokio task.
Fixes to how the scheduler detects "completion" (empty queue isn't
sufficient).
Add a "monitor" to detect failing threads ASAP
Add a "worker idle" so that workers with no tasks sleep.

crates/sparrow-backend/src/logical_to_physical.rs

crates/sparrow-backend/src/pipeline_schedule.rs

crates/sparrow-execution/src/lib.rs

crates/sparrow-execution/src/source_tasks.rs

crates/sparrow-execution/src/tests.rs

crates/sparrow-logical/src/typecheck.rs

crates/sparrow-physical/src/plan.rs

crates/sparrow-scheduler/src/pending.rs

crates/sparrow-scheduler/src/queue.rs

crates/sparrow-scheduler/src/sink.rs

crates/sparrow-scheduler/src/task.rs

crates/sparrow-scheduler/src/worker.rs

crates/sparrow-scheduler/src/worker_pool.rs

crates/sparrow-execution/src/tests.rs

crates/sparrow-execution/src/write_channel_pipeline.rs

crates/sparrow-scheduler/src/sink.rs

crates/sparrow-execution/src/lib.rs

crates/sparrow-interfaces/src/source.rs

crates/sparrow-execution/src/tests.rs

The main change here is having the `sparrow-execution` tests compile from a logical plan, created within a session, and execute things. Changes Include - Fix naming `field_ref` vs. `fieldref` (matches Python) - Tweak pipeline scheduling so that only transforms are grouped together. - Tweak pipelines so that each may output to multiple consumers - Introduces a `SourceTasks` so we can poll all inputs on a single tokio task. - Fixes to how the scheduler detects "completion" (empty queue isn't sufficient). TODO - [x] Actually execute the compiled query - [x] Connect to sources - [x] Move compilation / execution into session. - [x] Look at compiling out `tracing::trace` level statements. - [x] Look at running the query in the sessions tokio runtime - [ ] Move logical expr building to session (out of sparrow-logical) - [ ] Replace `ScalarValue` with arrows scalar? Literal too? - [ ] Look at suspending workers when queue is empty / waking on push - [ ] Look at a trait for logical functions (type-checking, compilation)

crates/sparrow-backend/src/logical_to_physical.rs

...snapshots/sparrow_backend__logical_to_physical__tests__logical_to_physical_arithmetic-3.snap

crates/sparrow-execution/src/lib.rs

crates/sparrow-backend/src/pipeline_schedule.rs

crates/sparrow-batch/src/batch.rs

crates/sparrow-execution/src/lib.rs

crates/sparrow-backend/src/logical_to_physical.rs

crates/sparrow-execution/src/lib.rs

crates/sparrow-physical/src/step.rs

crates/sparrow-scheduler/src/worker.rs

crates/sparrow-scheduler/src/worker_pool.rs

bjchambers requested a review from jordanrfrazier October 11, 2023 03:11

cla-bot bot added the cla-signed Set when all authors of a PR have signed our CLA label Oct 11, 2023

github-actions bot added the sparrow label Oct 11, 2023