You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your idea related to a problem? Please describe.
I am trying to create glue tables for data in S3 written by AWS DMS as partitioned parquet files. The problem is that AWS DMS writes the partitions in the format "2023/05/01/" and not in the Hive standard like "year=2023/month=05/day=01".
Now when I try to create the glue tables using the Wrangler method "store_parquet_metadata", the partitions are not recognized because in the internal method "_extract_partitions_metadata_from_paths" is filtered for "=".
Describe the solution you'd like
Currently only hive conform partitioning seems to be supported. It would be better if you could pass the partition keys when calling the method.
The text was updated successfully, but these errors were encountered:
Is your idea related to a problem? Please describe.
I am trying to create glue tables for data in S3 written by AWS DMS as partitioned parquet files. The problem is that AWS DMS writes the partitions in the format "2023/05/01/" and not in the Hive standard like "year=2023/month=05/day=01".
Now when I try to create the glue tables using the Wrangler method "store_parquet_metadata", the partitions are not recognized because in the internal method "_extract_partitions_metadata_from_paths" is filtered for "=".
Describe the solution you'd like
Currently only hive conform partitioning seems to be supported. It would be better if you could pass the partition keys when calling the method.
The text was updated successfully, but these errors were encountered: