Can not read metaData from Delta Live Tables full-refreshed

[mats16]

DeltaKernel can not read a Delta table managed by Delta Live Tables (DLT) after full-refresh.
(can read before full-refresh)

error message

IO Error: Hit DeltaKernel FFI error (from: While trying to read from delta table: 's3://databricks-workspace-bucket/unity-catalog/__unitystorage/schemas/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/tables/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/'): Hit error: 2 (ArrowError) with message (Json error: whilst decoding field 'metaData': Encountered unmasked nulls in non-nullable StructArray child: Field { name: "schemaString", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} })

I checked _delta_log/ directory, metaData is exist.

Env

• Python 3.12
• DuckDB v1.1.2

[samansmink]

I think this is related? duckdb/duckdb-delta#96

[zachschuermann]

Hi @mats16 @samansmink is this still an issue?

[samansmink]

@zachschuermann I'm not really sure how to reproduce this tbh

[zachschuermann]

I vaguely remember running into this before: something like DLT doesn't write schemaString? But that should really be fixed upstream instead of us trying to cope with something that's required that isn't there..

[scovich]

I asked some DLT people, and it sounds like they know about the problem and have a fix in progress that should roll out "soon"