Delta-rs not using latest object_store which fails the table optimize

[gprashmi]

Environment

Delta-rs version: 0.21.0

---

Bug

What happened: We write data to delta table using delta-rs with PyArrow engine with DayHour as partition column. Our Day

deltalake.write_deltalake(
            table_or_uri=delta_table_path,
            data=df,
            partition_by=[dayhour_partition_column],
            schema_mode='overwrite',
            mode="append",
            storage_options={"AWS_S3_ALLOW_UNSAFE_RENAME": "true"},
        )

I ran the optimize command using the spark sql query below on the delta table

optimize_query = f"""
OPTIMIZE delta.`s3_table_path`
ZORDER BY (col1, col2)
"""
spark.sql(optimize_query)

After optimize, it creates partitions with spaces and does not properly encode the partition urls as shown in the below image i.e; it creates new partitions url with spaces (.zstd.parquet).

The issue #2978 suggests delta-rs needs to be updated to have the recent object_store version 0.10.2 for this to be fixed.

@ion-elgreco @rtyler Could you please let me know if this is going to be fixed anytime soon?

[gprashmi]

@thomasfrederikhoeck

[ion-elgreco]

This is not a PR though

[gprashmi]

@ion-elgreco I would need to create a pull request to make this change from my end? This is not planned as part of the next release?

[thomasfrederikhoeck]

I have created a PR #2994

[thomasfrederikhoeck]

@gprashmi should be closed in #2994