Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add BufWriter::with_attributes and ::with_tags in object_store #5693

Merged
merged 2 commits into from
Apr 26, 2024

Conversation

netthier
Copy link
Contributor

@netthier netthier commented Apr 25, 2024

Which issue does this PR close?

Closes #5692.

What changes are included in this PR?

This PR adds the method with_attributes to BufWriter, which allows users to set attributes on uploaded objects.

I originally intended to add with_tags as well, as both put_opts and put_multipart_opts support them, but I did not know how to retrieve tags after setting them, leaving me unable to test that method. Therefore I guess that this only closes the linked issue partially.
It also adds with_tags which does the same but for tags.

Are there any user-facing changes?

Yes, this PR introduces two new methods on BufWriter. There are no breaking changes though.

@github-actions github-actions bot added the object-store Object Store Interface label Apr 25, 2024
@tustvold
Copy link
Contributor

how to retrieve tags after setting them, leaving me unable to test that method.

We don't expose it publicly, as there is no consistency between stores, but there is an integration test of object tagging that you could hook into

@netthier
Copy link
Contributor Author

netthier commented Apr 25, 2024

Could you maybe elaborate on how that hooking into could look like? If it's not too difficult I'd like to include it in this PR for completeness.

@tustvold
Copy link
Contributor

https://github.com/apache/arrow-rs/blob/master/object_store/src/lib.rs#L2362 is an integration test called by both the AWS and Azure test suites, and is provided with a method for retrieving the tags for a given path.

It should just be a case of extending this integration test with your new BufWriter functionality and that should do the trick

@netthier
Copy link
Contributor Author

netthier commented Apr 25, 2024

I'm trying to add BufWriter to tagging, but I can't figure out how to get the Arc<dyn ObjectStore> required. tagging takes a &dyn ObjectStore, and changing the method signature so that the caller needs to pass an Arc leads to more issues...

Any idea what I could do?

EDIT: As both AmazonS3 and MicrosoftAzure have an Arc inside, I could just construct an entirely new client to wrap in an Arc. Though that's not very elegant.
EDIT 2: This is what I now pushed.

@netthier netthier changed the title Add BufWriter::with_attributes Add BufWriter::with_attributes and ::with_tags Apr 25, 2024
@alamb alamb changed the title Add BufWriter::with_attributes and ::with_tags Add BufWriter::with_attributes and ::with_tags in object_store Apr 26, 2024
@tustvold tustvold merged commit 08af471 into apache:master Apr 26, 2024
14 checks passed
@netthier netthier deleted the bufwriter-opts branch April 26, 2024 15:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
object-store Object Store Interface
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Allow specifying PUT options when using BufWriter
2 participants