Upserting records to s3: .s3.merge_upsert_table vs overwrite partition parameter in s3 write functions

[AdrianoNicolucci]

I would like to upsert records in a parquet table without overwriting the entire partition.

Would the s3.merge_upsert_table function achieve this or is the s3.merge_upsert_table method just the same as using s3.to_parquet method with the parameter overwrite_partitions (Partition Upsert) enabled?

[jaidisido]

Thank you for discussing this. The current s3.merge_upsert_table method is very naive and lacks numerous features. As you have pointed out in your raised issues, loading the entire table into a data frame is inefficient and the method does not support partitioning.

I have already covered this in this issue but to summarise once more:

• AWS Glue Governed Tables is a new feature (currently in preview) which allows multiple users to concurrently insert, delete, and modify rows across tables within ACID transactions
• We are currently working on a PR to integrate it as soon as it's available
• You can find examples of what it will allow you to do here. But it should cover all of the elements that you have raised (inefficiency, partitioning...)
• Once released, the s3.merge_upsert_table method will be deprecated in Wrangler

[AdrianoNicolucci]

great thanks for the reply! Good to know that merge_upsert_table is being deprecated. I'll wait for governed tables to be released.