Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ref: Simplify expressions in physical plan #782

Merged
merged 7 commits into from
Oct 2, 2023

Commits on Sep 29, 2023

  1. ref: Simplify expressions in physical plan

    Change expressions to always produce the result of the expression.
    This simplifies evaluation, without affecting the expressiviness:
    
    1. We can now produce a primitive result (such as a boolean or integer)
       if that is the type of the final expression.
    2. We can still produce records by adding a `record` expression to the
       end (which is the standard way of creating intermediate records).
    3. We can still reference parts of a record input using `FieldRef`.
    
    This eliminates the weird "some expressions are marked as output"
    pattern. It also means that we can operate on `ArrayRef` generally and
    use `DataType` as the type, instead of sometimes using `RecordBatch` and
    `SchemaRef`.
    
    Other smaller changes in this PR:
    - Move the `Expressions` out of the `StepKind`. This allows the
      `StepKind` to serve as an identifier of the step, regardless of the
      expressions it computes, simplifying some of the de-duplication work
      in the logical-to-physical compilation.
    - Rename the `column` expression to `input`, since it accesses the
      input data to the expression evaluation.
    - Addition of a method to "intern" expression names, replacing a `&str`
      with `&'static str` for expression operators available in the binary.
    bjchambers committed Sep 29, 2023
    Configuration menu
    Copy the full SHA
    cf673d3 View commit details
    Browse the repository at this point in the history
  2. clippy

    bjchambers committed Sep 29, 2023
    Configuration menu
    Copy the full SHA
    ca4284a View commit details
    Browse the repository at this point in the history
  3. fix

    bjchambers committed Sep 29, 2023
    Configuration menu
    Copy the full SHA
    b974cc1 View commit details
    Browse the repository at this point in the history
  4. unused imports

    bjchambers committed Sep 29, 2023
    Configuration menu
    Copy the full SHA
    79b781f View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2023

  1. comments

    bjchambers committed Oct 2, 2023
    Configuration menu
    Copy the full SHA
    5761d80 View commit details
    Browse the repository at this point in the history
  2. remove input_type

    bjchambers committed Oct 2, 2023
    Configuration menu
    Copy the full SHA
    87d5495 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    d06ba4c View commit details
    Browse the repository at this point in the history