Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add negative metadata cache ttl #1246

Merged
merged 9 commits into from
Feb 11, 2025
Merged

Add negative metadata cache ttl #1246

merged 9 commits into from
Feb 11, 2025

Conversation

notoriaga
Copy link
Contributor

Adds a new CLI argument --negative-cache-ttl that lets you set the TTL for negative metadata entries separately from --metadata-ttl. My use case is a write once read many bucket. Objects do not get deleted from this bucket, and new objects are added every few minutes. I'd like to be able to set --metadata-ttl indefinite and --negative-cache-ttl 60 to effectively utilize the caching while still being able to pick up new objects. There is an open issue for this here - #831

Does this change impact existing behavior?

No, if --negative-cache-ttl is omitted the existing behavior is maintained (use --metadata-ttl or the default file_ttl).

Does this change need a changelog entry? Does it require a version change?

Because this is a new feature I believe it would require both.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and I agree to the terms of the Developer Certificate of Origin (DCO).

@notoriaga notoriaga temporarily deployed to PR integration tests January 25, 2025 00:19 — with GitHub Actions Inactive
@notoriaga notoriaga had a problem deploying to PR integration tests January 25, 2025 00:19 — with GitHub Actions Failure
@notoriaga notoriaga had a problem deploying to PR integration tests January 25, 2025 00:19 — with GitHub Actions Failure
@notoriaga notoriaga had a problem deploying to PR integration tests January 25, 2025 00:19 — with GitHub Actions Failure
@notoriaga notoriaga had a problem deploying to PR integration tests January 25, 2025 00:19 — with GitHub Actions Failure
@notoriaga notoriaga had a problem deploying to PR integration tests January 25, 2025 00:19 — with GitHub Actions Failure
@notoriaga notoriaga had a problem deploying to PR integration tests January 25, 2025 00:19 — with GitHub Actions Failure
@notoriaga notoriaga temporarily deployed to PR integration tests January 27, 2025 21:13 — with GitHub Actions Inactive
@notoriaga notoriaga had a problem deploying to PR integration tests January 27, 2025 21:13 — with GitHub Actions Failure
@notoriaga notoriaga had a problem deploying to PR integration tests January 27, 2025 21:13 — with GitHub Actions Failure
@notoriaga notoriaga had a problem deploying to PR integration tests January 27, 2025 21:13 — with GitHub Actions Failure
@notoriaga notoriaga had a problem deploying to PR integration tests January 27, 2025 21:13 — with GitHub Actions Failure
@notoriaga notoriaga had a problem deploying to PR integration tests January 27, 2025 21:13 — with GitHub Actions Failure
@notoriaga notoriaga had a problem deploying to PR integration tests January 27, 2025 21:13 — with GitHub Actions Failure
passaro
passaro reviewed Jan 28, 2025
View reviewed changes
Copy link
Contributor

@passaro passaro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @notoriaga, thank you for opening this pull request!

I've got a few suggestions, but the next step would be to add a couple of tests. See existing tests for the negative cache, e.g. test_negative_lookup_with_caching in superblock.rs and lookup_with_negative_cache in lookup_test.rs. Happy to help if you need more guidance.

@notoriaga @passaro
Update mountpoint-s3/src/fs/config.rs
52774f1 
Co-authored-by: Alessandro Passaro <[email protected]>
Signed-off-by: Steven Meyer <[email protected]>
@notoriaga notoriaga requested a deployment to PR integration tests January 31, 2025 23:32 — with GitHub Actions Waiting
@notoriaga notoriaga requested a deployment to PR integration tests January 31, 2025 23:32 — with GitHub Actions Waiting
@notoriaga notoriaga requested a deployment to PR integration tests January 31, 2025 23:32 — with GitHub Actions Waiting
@notoriaga notoriaga requested a deployment to PR integration tests January 31, 2025 23:32 — with GitHub Actions Waiting
@notoriaga notoriaga requested a deployment to PR integration tests January 31, 2025 23:32 — with GitHub Actions Waiting
@notoriaga notoriaga requested a deployment to PR integration tests January 31, 2025 23:32 — with GitHub Actions Waiting
@notoriaga notoriaga requested a deployment to PR integration tests January 31, 2025 23:32 — with GitHub Actions Waiting
@notoriaga @passaro
Update mountpoint-s3/tests/fuse_tests/lookup_test.rs
3bd3836 
Co-authored-by: Alessandro Passaro <[email protected]>
Signed-off-by: Steven Meyer <[email protected]>
@notoriaga notoriaga requested a deployment to PR integration tests January 31, 2025 23:32 — with GitHub Actions Waiting
@notoriaga notoriaga requested a deployment to PR integration tests January 31, 2025 23:32 — with GitHub Actions Waiting
@notoriaga notoriaga requested a deployment to PR integration tests January 31, 2025 23:32 — with GitHub Actions Waiting
@notoriaga notoriaga requested a deployment to PR integration tests January 31, 2025 23:32 — with GitHub Actions Waiting
@notoriaga notoriaga temporarily deployed to PR integration tests February 1, 2025 22:51 — with GitHub Actions Inactive
@notoriaga notoriaga temporarily deployed to PR integration tests February 1, 2025 22:51 — with GitHub Actions Inactive
@notoriaga notoriaga temporarily deployed to PR integration tests February 1, 2025 22:51 — with GitHub Actions Inactive
@notoriaga notoriaga temporarily deployed to PR integration tests February 1, 2025 22:51 — with GitHub Actions Inactive
@notoriaga notoriaga temporarily deployed to PR integration tests February 1, 2025 22:51 — with GitHub Actions Inactive
passaro
passaro reviewed Feb 3, 2025
View reviewed changes
Copy link
Contributor

@passaro passaro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. I have only one remaining suggestion.

@notoriaga notoriaga requested a deployment to PR integration tests February 11, 2025 00:59 — with GitHub Actions Waiting
@notoriaga notoriaga requested a deployment to PR integration tests February 11, 2025 00:59 — with GitHub Actions Waiting
@notoriaga notoriaga requested a deployment to PR integration tests February 11, 2025 00:59 — with GitHub Actions Waiting
@notoriaga notoriaga requested a deployment to PR integration tests February 11, 2025 00:59 — with GitHub Actions Waiting
@notoriaga notoriaga requested a deployment to PR integration tests February 11, 2025 00:59 — with GitHub Actions Waiting
@notoriaga notoriaga requested a deployment to PR integration tests February 11, 2025 00:59 — with GitHub Actions Waiting
@notoriaga notoriaga requested a deployment to PR integration tests February 11, 2025 00:59 — with GitHub Actions Waiting
@notoriaga notoriaga force-pushed the notoriaga/negative-cache-ttl branch from 61ac01b to 9aa0934 Compare February 11, 2025 01:00
@notoriaga notoriaga temporarily deployed to PR integration tests February 11, 2025 01:00 — with GitHub Actions Inactive
@notoriaga notoriaga temporarily deployed to PR integration tests February 11, 2025 01:00 — with GitHub Actions Inactive
@notoriaga notoriaga temporarily deployed to PR integration tests February 11, 2025 01:00 — with GitHub Actions Inactive
@notoriaga notoriaga temporarily deployed to PR integration tests February 11, 2025 01:00 — with GitHub Actions Inactive
@notoriaga notoriaga temporarily deployed to PR integration tests February 11, 2025 01:00 — with GitHub Actions Inactive
@notoriaga notoriaga temporarily deployed to PR integration tests February 11, 2025 01:00 — with GitHub Actions Inactive
@notoriaga notoriaga temporarily deployed to PR integration tests February 11, 2025 01:00 — with GitHub Actions Inactive
@notoriaga
Copy link
Contributor Author

Nice. I have only one remaining suggestion.

Sorry for the delay! I think I got what you mean. One thing I'm not sure about is I also set use_negative_cache = true when calling with_negative_metadata_ttl with something other than minimal. I guess that would let you enable negative metadata caching but not the normal caching. Not sure if that makes sense to do.

passaro
passaro approved these changes Feb 11, 2025
View reviewed changes
Copy link
Contributor

@passaro passaro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Thank you for your contribution!

mountpoint-s3/tests/fs.rs
negative_cache_ttl: Duration::from_secs(600),
..Default::default()
},
cache_config: CacheConfig::new(TimeToLive::Duration(Duration::from_secs(600))),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's better! Thanks

@passaro passaro added this pull request to the merge queue Feb 11, 2025
Merged via the queue into awslabs:main with commit 5e580a8 Feb 11, 2025
24 checks passed
@passaro passaro mentioned this pull request Feb 11, 2025
Merged
github-merge-queue bot pushed a commit that referenced this pull request Feb 11, 2025
@passaro
Increase version to 1.15 and update CHANGELOG after adding new metada…
981a3e1 
…ta TTL flag (#1265)

The change in #1246 requires a minor version increase and a new entry in
the changelog.

---

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license and I agree to the terms of
the [Developer Certificate of Origin
(DCO)](https://developercertificate.org/).

Signed-off-by: Alessandro Passaro <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@passaro passaro passaro approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@notoriaga @passaro