2023-12-14 kernel meeting notes

[zachschuermann]

summary

lightweight meeting before the holidays!

action items

from last time

• Nick design doc
• new hackathon/FFI code review
• kernel write draft review

attendees

@nicklan @roeap @zachschuermann @vkorukanti @ryan-johnson-databricks

notes stream

@nicklan discussion on #100

• new flow: leverage expression handling and then visitor
• visiting won't churn through tons of null rows
• P+M example: we want to not have visitor do all the work
• ryan: we can only do this at column batch level (all the columns at the same time) - do a predicate that applies to everything
• robert: we could ask for more functions? (ab)use an API that would do a filter? we need something to model DVs. but also want to have some selection vector API in general. we could require this filter function in expression system and we can leverage it wherever we want?
• given batch and selection vector -> engine give back valid rows. (this is the application of DV)
• consider: NULL_IF and IS_NULL. this is the arrow way of expressing a null-safe comparison.
• want to say 'this value = false' not NULL
• we can hoist up null-safe comparisons, make an API, then engine will implement (e.g. arrow does this differently than spark)
• IS_NULL(x) OR x is null-safe comparison (and need short-circuit semantics)
• nick - want to get first steps of data stuff done without solving all these problems
• multi-step process: skipping, filtering, extraction
• nice that we can do it in pieces (correctness stands) - build extraction then skipping/filtering as optimizations
• likely want to push on this code first since it will touch lots of the kernel
• need to review/merge roberts PR!

venki: expression tree stuff - need a way for engine to pass expressions through kernel (partition pruning pass-through)

• java kernel: make the expressions generic (just tree of expressions recursively defined)
• rust we restrict what expressions the connector can add
• robert: expression rewrite for file skipping - we need partition pruning? we want to enable the engine to do their own optimization
• kernel is willing to provide BASIC file skipping (not yet impl)
• if engine wants to do their own file skipping, they should be able to bypass kernel file skipping
  options
  (1) engine says here is where clause, give back file skip expression based on read schema
  (2) OR kernel give stats schema and i will give back expression to blindly evaluate (kernel doesn't need to know anything about expressions here)

if kernel needs to pass things to engine 'opaquely', why does kernel need to pass around this at all? can the engine just track itself?

• consider: for data skipping kernel has to do the log replay bit - requires round trip through kernel
• to explore more later
• we might be able to remove the 'pass around opaque stuff'

interesting observation here (similar to removing file context etc)

• do we really need to 'move around' engine-level things (opaque types etc.)
• to follow up on!!!

robert: parquet reads/file stats.

• we can push down projection to field level
• experiment with other parquet writers
• polars/pandas
• kernel has an opinion about what the schema of parquet files are
• spark schema doesn't have opinion on layout of columns internally
• parquet/arrow care about internal layout?
• create the same data in arrow and write to parquet via different writers: very different data is output
• nick: note we only need SPECIFIC implementations of this stuff
• a.b.c is a map and we just need to get something back that has a get keys OR just flatten the whole thing
• consider table properties and table features
• consider operations on an AddFile, need to inspect some specific tag (drill into map with visitor)
• if there is a map or array we need it in a usable form (ask engine to build a suitable data structure and give it back?)
• we could ask for two kinds of maps: materialized map OR probed map with few values as leaf entries in a row we are building (one item per key none if not exists)
• could say a.b[3] -> scalar
• let's build a DSL!
• we need some communication for 'what stuff i want'
• could pass a kernel expression? map_extract(..) etc.
• engine enum which encodes lifetimes of borrowed engine values
• either visit every row and do something or produce another column batch
• ryan: when do we need to borrow? anything in 64 bits we should just Copy
• even with strings, if we visit every row, we are doing something with that string first have to make an owned copy
• nick: concern is FFI
• if we are copying strings around
• engine passes over a slice (chars with leng) turn into rust string (owned) - original is only borrowed long enough to perform the copy

TODO to sync on @roeap PR

FFI: consideration - do C one or do C and C++?