Contributor sync notes - 2024-01-17

[tdikland]

These are rough notes of the biweekly contributor sync. Please contact @MrPowers if you want to be included.

@MrPowers:

• Would be great to graduate this project to delta-io this year.

@roeap:

• Isolation of core Delta-Sharing router logic. Split the admin/auth parts off the main protocol
• Expand authentication mechanisms to the delta sharing server (AAD, other IdP)
• Memory model should be aligned with delta-kernel-rs. This is already being done in the delta-rs repo and could be used in the current server implementation.
• Datafushion is very extensible, would like to see the same behaviour in delta sharing server.

@linzhou-db:

• Server is responsible for creating presigned urls for all the data needed in the client to read the table (this used to be parquet files only, newer delta reader features require the sharing of more data e.g. deletion vectors, column mappings, etc.)
• It may be a good idea to publish a client core library that is able to read/interpret the presigned urls (i.e. read parquet files, deletion vectors, mappings, etc. and combine them).

Actions:

• Create issues on the repo based on this discussion
• @r3stl355 can contribute his logic for self-assigning issues

[MrPowers]

Great meeting, thanks for typing up the notes!