Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove ObjectStore::append #5016

Merged
merged 1 commit into from
Nov 1, 2023

Conversation

tustvold
Copy link
Contributor

@tustvold tustvold commented Nov 1, 2023

Which issue does this PR close?

Closes #.

Rationale for this change

ObjectStore::append has always been somewhat at odds with the design goals of the crate, as it is not an object store API, and has rather unfortunate characteristics with regards to atomicity. It was largely added to accommodate streaming workloads in DataFusion that are overloading the file IO abstractions.

apache/datafusion#7994 tracks addressing this upstream, and is an effort I intend to drive to completion. Whilst removing it is slightly jumping the gun, I want this to make the next release #5010 which will still give plenty of time to prepare DataFusion for this change

This functionality was only implemented by InMemory and LocalFileSystem and so applications can easily reproduce this functionality should they so desire.

What changes are included in this PR?

Are there any user-facing changes?

@github-actions github-actions bot added the object-store Object Store Interface label Nov 1, 2023
@@ -94,8 +94,7 @@
//!
//! This provides some compelling advantages:
//!
//! * Except where explicitly stated otherwise, operations are atomic, and readers
//! cannot observe partial and/or failed writes
//! * All operations are atomic, and readers cannot observe partial and/or failed writes
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉

@tustvold tustvold added the api-change Changes to the arrow API label Nov 1, 2023
@tustvold tustvold mentioned this pull request Nov 1, 2023
2 tasks
@tustvold tustvold merged commit 94fe6bb into apache:master Nov 1, 2023
15 checks passed
@alamb
Copy link
Contributor

alamb commented Nov 2, 2023

cc @metesynnada

@alamb
Copy link
Contributor

alamb commented Nov 2, 2023

So as I understand t, the idea here that if people want to read from object storage / local files, but append to local files, they need to implement append as a special case (without the object store API)

@tustvold
Copy link
Contributor Author

tustvold commented Nov 2, 2023

I would phrase it as, if you have a use-case that involves reading and writing local files, including appending to them, or mmaping them, or some other filesystem specific functionality, then you should use the filesystem directly.

apache/datafusion#8021 contains an example of doing this for FIFO files in the context of DataFusion

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api-change Changes to the arrow API object-store Object Store Interface
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants