Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Perform cache maintenance in a separate thread #2239

Open
jmazanec15 opened this issue Oct 25, 2024 · 3 comments · May be fixed by #2308
Open

Perform cache maintenance in a separate thread #2239

jmazanec15 opened this issue Oct 25, 2024 · 3 comments · May be fixed by #2308
Assignees
Labels
Enhancements Increases software capabilities beyond original client specifications Roadmap:Vector Database/GenAI Project-wide roadmap label v2.19.0

Comments

@jmazanec15
Copy link
Member

Description

For the NativeMemoryCacheManager, we allow users to enable an "expirary" time setting, so that cache entries that arent touched for some duration, get marked as expired and are then evicted.

However, the problem is that the guava cache does not maintain a separate thread that can perform this maintenance. Instead it will typically perform this maintenance on writes and sometimes on reads. See https://github.com/google/guava/wiki/CachesExplained#timed-eviction. Thus, if you are not touching the cache for awhile, entries may be expired but not get evicted. Thus, resources are not freed.

The Guava team recommends creating a separate thread that will call cache.cleanUp() to explicitly perform maintenance. This will ensure that the time-based evictions happen as one would expect.

@jmazanec15 jmazanec15 added Enhancements Increases software capabilities beyond original client specifications v2.19.0 labels Oct 25, 2024
@jmazanec15 jmazanec15 moved this from Backlog to Backlog (Hot) in Vector Search RoadMap Oct 25, 2024
@luyuncheng
Copy link
Collaborator

LGTM

@navneet1v
Copy link
Collaborator

@jmazanec15 we should do the cache cleanup for Quantization state cache too.

@vamshin vamshin added the Roadmap:Vector Database/GenAI Project-wide roadmap label label Nov 1, 2024
@vamshin vamshin moved this from Backlog (Hot) to 2.19.0 in Vector Search RoadMap Nov 1, 2024
@owenhalpert
Copy link

Hey, I'll pick this one up. Thank you.

@navneet1v navneet1v assigned owenhalpert and unassigned vamshin Nov 25, 2024
@owenhalpert owenhalpert linked a pull request Dec 3, 2024 that will close this issue
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Enhancements Increases software capabilities beyond original client specifications Roadmap:Vector Database/GenAI Project-wide roadmap label v2.19.0
Projects
Status: New
Status: 2.19.0
Development

Successfully merging a pull request may close this issue.

5 participants