Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support inserting in merge_insert with a subset of columns #2904

Open
Tracked by #3016
wjones127 opened this issue Sep 17, 2024 · 0 comments · May be fixed by #3100
Open
Tracked by #3016

Support inserting in merge_insert with a subset of columns #2904

wjones127 opened this issue Sep 17, 2024 · 0 comments · May be fixed by #3100
Assignees
Labels
enhancement New feature or request rust Rust related tasks

Comments

@wjones127
Copy link
Contributor

Right now we supporting updating a subset of columns with merge insert. However, we left when_not_matched_insert_all() as TODO. (see: #2639). Users should be able to provide a subset of columns in the merge insert query, and have the missing columns filled in with null. (Relatedly, we might also consider something similar with append.)

We left this as TODO in large part because we haven't seriously approached null values. When inserting a subset of columns, the missing columns should be filled with nulls. The default V1 format doesn't universally support nulls, so it would not have worked well. Now that V2 is the default, this will make more sense. We should decide again whether we should allow missing (null) fields in fragments, or require that there be all null columns in the data files. We have a vestigial comment in our proto files right now:

lance/protos/table.proto

Lines 182 to 183 in 65b32d4

// If column exists in the schema, but the related file does not exist,
// treat this column as nulls.

@wjones127 wjones127 added enhancement New feature or request rust Rust related tasks labels Sep 17, 2024
@wjones127 wjones127 self-assigned this Nov 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request rust Rust related tasks
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant