2024-10-31 kernel meeting notes (spooky edition)

[zachschuermann]

summary

0.4.1 released! (had to disable optimization that causes bug - see changelog)

meeting items

• PR summaries
• blockers/releases/updates
• PR board review
• summarize in summary above.

pull requests/ongoing threads

write path in progress, overall tracking issue at #377

• [write] add Transaction with commit info and commit implementation #370
• [write] Transaction append data API #393

summary of PRs

• from @hntd187 Map access for expressions #352
• actions to be pub(crate) will break delta-rs - need to follow up there (zach)
  • for write path delta-rs needs to create those structs to write them!

attendees

@zachschuermann @OussamaSaoudi-db @scovich

notes stream

• need to fix cargo swallowing cbindgen warns/errors!
• 0.4.1 release! see changelog
• need eyes on column name PR: ColumnName tracks a path of field names instead of a simple string #445
• need eyes on write support data path PR: [write] Transaction append data API #393
• logging in Handle nested columns #86 gaps for nested columns
• need eyes on harmonizing predicate evaluation (waiting on col name PR): Harmonized predicate eval #420
  • today three ways to eval pred: (1) arrow default engine over row groups (read parquet), (2) data skipping expr to turn pred into another pred, (3) do parquet row group skipping - which does eval but does specialized bool eval on top of parquet stats
  • harmonization PR will unify these. harmonize delta data skipping and parquet data skipping. parquet opts now available in kernel. bugs fixed. and now scalar evaluation with only the need to turn col to scalar. take WHERE to data skip expr and eval with known min max vals and verify if it does/doesn't skip.
  • all data skipping UT for parquet: most now generalized stats-based skipping tests that cover kernel as well
  • data skipping log same (shared)
  • only col resolution changes and type of result
  • parquet data skipping/kernel data skipping
    • delta keeps stats in metadata (and applies at query level)
    • log replay we have add files (dataframe-like) with stats. and then we can run a query across it to filter
    • delta data skipping is basically query over relation of addfiles
    • parquet does bool where clause eval over footer stats
    • parquet skipping return option
    • delta data skipping return expr that resolves to option when it runs
    • skipping logic that constructs this is the same for both
    • learning: rust traits are awesome (and flexible)
• ryan reviewing zach pr
• ryan opened as_any PR (zach needs something similar)