2024-10-24 kernel meeting notes

[zachschuermann]

summary

0.4.0 released!!

meeting items

• PR summaries
• blockers/releases/updates
• PR board review
• summarize in summary above.

pull requests/ongoing threads

write path in progress, overall tracking issue at #377

• [write] add Transaction with commit info and commit implementation #370
• [write] Transaction append data API #393

summary of PRs

• expression visitor FFI getting close Add visitor for converting kernel expressions to engine expressions #363
• Map access for expressions #352
• actions to be pub(crate) will break delta-rs - need to follow up there (zach)
  • for write path delta-rs needs to create those structs to write them!
• ryan landed some awesome column name macros! Expression::Column references a ColumnName object instead of a String #400
• schema transformation trait Define and use a schema transformation trait #395

attendees

@nicklan @rtyler @roeap @zachschuermann @OussamaSaoudi-db @scovich

notes stream

• arrow discussion: not as nice as we thought..
• discussion on new datafusion/arrow/delta-rs/etc.
• friction on internal field names (item vs element) in e.g. MapType or ArrayType
• solution: use ordinals?
• would be nice if arrow had a 'user-visibly-equal' operator/trait.
• robert: need checkpointing rewrite. pulls in lots of deps, does stuff in its own way. code that can generate actions from parquet types and vice versa. since delta-rs hasn't adopted kernel scan they do this. but this isn't really needed anymore - was from a time without arrow. arrow has now displaced it.
• checkpointing: reading and writing checkpoints. delta-rs would like to get rid of checkpointing code.
• robert can pick up checkpointing in december!
• kernel/delta-rs checkpointing integration story: todo zach aim for december
• delta-rs: mini-expression lang to do skipping on partition values.
  • this expression system should be replaced with expression logic from kernel
• robert: profiling would be nice?
• zach to pick up robert PR
• tyler: delta-rs moving to delta-kernel-rs concerning since kernel is following protocol and the real world (read: delta-spark) does not.
• tyler: protocol needs to come out of delta-spark, DAT need more investment, delta-spark doesn't follow the protocol.
• ryan: is this kernel issue? tyler: delta-rs currently has hacks/real world tables
• zach: propose new approach: every test/feature/etc. should have real-world tests
• ryan: can we file bugs against delta-kernel-rs
• robert: entrypoint and exit point into delta-rs normalize? [what does this mean?] fix to protocol-compliant tables/impls
• nick: two things (1) future issues: having delta-rs on kernel means better prioritization/bugfix? (2) huge library of existing problems that delta-rs already deals with that kernel does not. kernel could cause regressions. what can we do to prevent that? maybe maintain two implementations? invest more in DAT?
• tyler: great ideas :) shouting more since group of people downstream of spark unless in CI for spark we have something check if is readable by kernel (for example). new commit stuff: experimental but enabled in current releases by default. some subset of tables in the world with _commits dir that delta-rs and kernel doesn't support. BAD: delta-spark is just enabling this by default and goes off writing tables.
• ryan: where does commits dir come from? tyler: delta-spark off the shelf pip install (not DBR) will do _commits. was this a release goof?
• more kernel = more testing/'make sure kernel doesn't break'/etc.
• moving towards kernel-first development!
• tyler: can we do kernel-rs injection and report some telemetry?
• PR board review