2023-09-14 kernel meeting notes

[zachschuermann]

summary

Limited meeting with design discussion on sync/async and kernel interfaces. Note we now have the ability for multiple reviewer requests and CI should be running on all PRs!

action items:

• list of connectors to target (add Clickhouse and pytorch)

action items from last time:

• action item: merge @wjones127 PR? proposal: remove async api, keep async default client #29
• action item: how to request multiple reviewers (@MrPowers)
  • probably update to Github Teams [done]
  • @MrPowers is owner of incubator org now so we can get this stuff sorted

attendees

@roeap @ryan-johnson-databricks @nicklan @MrPowers @zachschuermann

notes stream

• catch-up Robert on sync/async work
• H-shaped connector: unavoidable and desirable (we don't want to enforce any opinion on execution)
  • data flows through the middle of the H
  • critical API piece that we expose as an iterator, etc. that can plug in to all the different models (async iterators (streams) vs iter)
  • more design work needed here
• ideally we want API where we call next() and you get an item and block until it's ready or None if exhausted (can plug in to async systems above)
  • but not sure if this is possible
• example: DuckDB doesn't want a notion of Pending. But other systems may want this (delta-rs?). Can we marry the two?
• if kernel knows nothing of Pending --> what are the implications?
• DVs/row indexes
  • kernel includes a row_index column for reading
  • can push to parquet reader
  • could require generation of row_index if the underlying reader doesn't expose
• remember goal: avoid pessimizing performance. ship something above a reasonable threshold.
• from @MrPowers
  • pytorch connector being marketed (plan to migrate on to delta-kernel-rs)
  • chatting with Clickhouse, we could use this as another C++ integration example
• from @roeap
  • consider VACUUM in kernel?
  • VACUUM in delta-rs: integrated logical planning with DataFusion
  • benefits from async
  • stuck: when to do what
  • we need to do async in logical planning
  • curious what VACUUM would look like from SQL to execution
  • what is the logical plan? (command node with info or something else) --> using abstraction in DataFusion - a GoDoVacuumPlan?
    • now you need physical plan that actually implements VACUUM
    • VACUUM is atypical sql command (LIST - expensive, lots of DELETE - expensive)
    • good case study