-
Notifications
You must be signed in to change notification settings - Fork 824
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add copy_if_not_exists
support for AmazonS3 via DynamoDB Lock Support
#4880
Comments
FWIW, the DynamoDB lock protocol hasn't become the preferred protocol for Delta Lake or Lance, mostly because of the dangers of lease-based locks. Spark Delta Lake went with a slightly different protocol, described in this doc. In Lance we did something similar. delta-rs still uses dynamodb-lock, but will eventually move to match the protocol used by Spark Delta Lake. That being said, offering something decent out-of-the-box for S3 might be nice. |
Hmm... Yeah, I spent some time bashing at this, and it isn't easy to provide strong guarantees with this approach 😅 As iceberg also is doing something similar using DynamoDB to store the latest metadata file, I think lets not do this for now. |
copy_if_not_exists
support for AmazonS3 via DynamoDB Lock Support
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
#4239 added copy_if_not_exists support for S3-compatible stores that support it, i.e. everything other than S3 😅.
Having native support for this, and potentially other conditional operations in the future (#4879), would reduce user friction and I think unlock a number of exciting use-cases by making support consistent across the backends.
Describe the solution you'd like
I would like to be able to configure a DynamoDB to use in a manner compatible with the AWS DynamoDB Lock Client.
Describe alternatives you've considered
These is a Rust port of the AWS lock client, but it depends on rusoto, and doesn't appear to have very strong fencing guarantees.
Additional context
FYI @wjones127
The text was updated successfully, but these errors were encountered: