Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: compile and execute partitioned query #806

Merged
merged 16 commits into from
Oct 17, 2023
Merged

Commits on Oct 12, 2023

  1. draft: feat: compile and execute partitioned query

    The main change here is having the `sparrow-execution` tests compile
    from a logical plan, created within a session, and execute things.
    
    Changes Include
    
    - Fix naming `field_ref` vs. `fieldref` (matches Python)
    - Tweak pipeline scheduling so that only transforms are grouped together.
    - Tweak pipelines so that each may output to multiple consumers
    - Introduces a `SourceTasks` so we can poll all inputs on a single
      tokio task.
    - Fixes to how the scheduler detects "completion" (empty queue isn't
      sufficient).
    
    TODO
    
    - [x] Actually execute the compiled query
    - [x] Connect to sources
    - [x] Move compilation / execution into session.
    - [x] Look at compiling out `tracing::trace` level statements.
    - [x] Look at running the query in the sessions tokio runtime
    - [ ] Move logical expr building to session (out of sparrow-logical)
    - [ ] Replace `ScalarValue` with arrows scalar? Literal too?
    - [ ] Look at suspending workers when queue is empty / waking on push
    - [ ] Look at a trait for logical functions (type-checking, compilation)
    bjchambers committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    d2d887c View commit details
    Browse the repository at this point in the history

Commits on Oct 13, 2023

  1. comments

    bjchambers committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    2faf0cc View commit details
    Browse the repository at this point in the history
  2. monitor thread panics

    bjchambers committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    b9797e2 View commit details
    Browse the repository at this point in the history
  3. line up output schemas

    bjchambers committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    5e65b9b View commit details
    Browse the repository at this point in the history
  4. some comments

    bjchambers committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    ced7840 View commit details
    Browse the repository at this point in the history

Commits on Oct 16, 2023

  1. Configuration menu
    Copy the full SHA
    9eb4afa View commit details
    Browse the repository at this point in the history
  2. lock changes

    bjchambers committed Oct 16, 2023
    Configuration menu
    Copy the full SHA
    76acd0e View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    f0cf075 View commit details
    Browse the repository at this point in the history

Commits on Oct 17, 2023

  1. comment

    bjchambers committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    c31dc1b View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    e6a85dd View commit details
    Browse the repository at this point in the history
  3. more fixes / renamings

    bjchambers committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    9a86aa4 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    9683be4 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    14e7641 View commit details
    Browse the repository at this point in the history
  6. comments

    bjchambers committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    648e7d8 View commit details
    Browse the repository at this point in the history
  7. comments

    bjchambers committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    297594f View commit details
    Browse the repository at this point in the history
  8. fix comment

    bjchambers committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    378bfe2 View commit details
    Browse the repository at this point in the history