2024-03-21 kernel meeting notes

[zachschuermann]

summary

REPO IS NOW PUBLIC!!!!!! 🥳

action items

some PRs in flight:

• Fix parition columns + add support for binary and decimal scalars #146 get DAT working!
• start implementing FFI #91
• Use a proc macro for being able to derive schemas for our action structs #129 (shouldn't be closed)
• Switch to new data passing API #109
• Fixed file path resolution so tests pass on Windows #131
• remove arrow dependency from data skipping #132
• Initial work to use maturin+pyo3 to get a python interface #130
• use spawn_blocking to try and send back read results #141

attendees

@nicklan @zachschuermann @hntd187 @ryan-johnson-databricks @tdas

notes stream

zach: hackathon update!

• did delta-sharing hackathon with nick and others! [WIP] delta sharing hackathon 2024Q1 #144
• delta-sharing branch: https://github.com/delta-io/delta-sharing/tree/2024q1-hackathon
• TLDR: delta-shared table with DVs can be read via delta-kernel-rs
• 1. delta-sharing changes to export local delta_log with presigned URLs
• 2. delta-kernel-rs python bindings so delta-sharing client can leverage rust kernel
• 3. delta-kernel-rs changes to read local delta log and presigned URLs
     nick: Fix parition columns + add support for binary and decimal scalars #146
• discusson on: https://github.com/delta-incubator/delta-kernel-rs/pull/146/files#r1533878107
• add DAT case for microsecond/nanosecond read