feat: implement support for append_dedup in iceberg dest #48731
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Skipped Deployment
|
octavia-squidington-iii
added
area/connectors
| @@ -57,16 +59,26 @@ public InternalRecordWrapper wrapper() { | |||
| return wrapper; | |||
| } | |||
|
|
|||
| public Record constructIdentifierRecord(Record row) { | |||
There was a problem hiding this comment.
Nit: does this need to be public?
one nit
| val identifierFields = mutableSetOf<Int>() | ||
| val identifierFieldNames = primaryKeys.flatten().toSet() | ||
| val icebergTypeConverter = AirbyteTypeToIcebergSchema() | ||
| this.properties.entries.forEach { (name, field) -> | ||
| val id = generatedSchemaFieldId() | ||
| mutableListOf.add( | ||
| val isPrimaryKey = identifierFieldNames.contains(name) | ||
| val isOptional = !isPrimaryKey && field.nullable |
There was a problem hiding this comment.
fyi we tried enforcing PK requiredness at one point and it caused a lot of problems (there's a surprising number of sources that emit null PKs). If iceberg supports null values in a partition key, we should do that instead
There was a problem hiding this comment.
It just wont allow you to mark a field as an identifier field if its nullable. Only non-nullable fields can be identifier fields
There was a problem hiding this comment.
ugh. I think we Officially Support composite PKs with nullable fields #31926 (see also slack #p0-primay-keys-cannot-be-null)
so that's going to be awkward
No description provided.