feat(python): add schema conversion of FixedSizeBinaryArray and FixedSizeListType

[balbok0]

Description

Map FixedSizeBinaryType to BinaryArray, since Delta does not support fixed arrays.

Related Issue(s)

None.

Documentation

N/A

Minimal Example

I've noticed this error when doing subsequent calls to like so:

import deltalake as dl
import pyarrow as pa

schema = pa.schema([
    ("field_a", pa.binary(4)),
    # To simulate fix, switch this line to: ("field_a", pa.binary()),
])
table = pa.Table.from_pylist(
    [
        {"field_a": val.to_bytes(4, "little")}
        for val in range(0, 100)
    ],
    schema=schema
)

# This works
dl.write_deltalake(
    "bad_table",
    data=table,
    mode="append",
)

# This fails
dl.write_deltalake(
    "bad_table",
    data=table,
    mode="append",
)

with error:

ValueError: Schema of data does not match table schema
Data schema:
field_a: fixed_size_binary[4]
Table Schema:
field_a: binary

[github-actions]

ACTION NEEDED

delta-rs follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

[ion-elgreco]

@balbok0 can you extend the tests also to include this fixedSizedBinary

[balbok0]

I changed one test case to include a check for fixed binary. I think it should be enough (unit test-wise), since it's a really small change, but let me know if you would like more checks, just in case.

For more-integration like, is there a test case with repeated write to a tmp_path or existing_table that I can reference? I tried looking for it, but couldn't find any.

[balbok0]

Actually, I just found the same exact issue with pa.FixedSizeListType, so bundled a fix with this PR.

[ion-elgreco]

You need to set the type hints properly in the functions and also define the stubs

[balbok0]

Hi, sorry for being slow on resolving issues. I've fixed a typo on one of the tests I added. Just to be sure:

• I've noticed a lot of currently existing tests failing/erroring. I assume these are not related?
• Is it ok to skip the integration tests in this case?

[ion-elgreco]

@balbok0 no the tests are related, I suggest you execute it locally to debug. Decimal is hit as a false positive while doing the schema change, so the logic needs fixing

[balbok0]

Ok, switched it to type instead of isinstance, which will not include DecimalType from pyarrow. As far as I tested locally tests should pass.

[ion-elgreco]

Thanks! @balbok0