Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement GC algorithm for values in LMDB store #3574

Closed
kenwenzel opened this issue Jan 6, 2022 · 0 comments · Fixed by #3577
Closed

Implement GC algorithm for values in LMDB store #3574

kenwenzel opened this issue Jan 6, 2022 · 0 comments · Fixed by #3577
Assignees
Labels
📶 enhancement issue is a new feature or improvement M2 Fixed in milestone 2
Milestone

Comments

@kenwenzel
Copy link
Contributor

Problem description

This is a follow up issue of #3416.

While the LMDB store is feature complete in comparison to memory and native store it still misses the functionality to evict unused values.

Preferred solution

  • track subject, predicate, object and context IDs when statements are removed from the triple indexes
  • before finally committing a transaction:
    • test tracked IDs against the triple indexes and remove IDs which are still in use
    • add remaining IDs to some kind of "recycle" list within the value store and remove the associated data

Are you interested in contributing a solution yourself?

Yes

Alternatives you've considered

Implement the GC algorithm on a second thread to impact update performance as few as possible.
While this may be the better option it is harder to implement and I would go with a synchronous implementation for the first PoC.

Anything else?

No response

@kenwenzel kenwenzel added the 📶 enhancement issue is a new feature or improvement label Jan 6, 2022
@abrokenjester abrokenjester linked a pull request Jan 7, 2022 that will close this issue
5 tasks
@abrokenjester abrokenjester added this to the 4.0.0 milestone Jan 7, 2022
@hmottestad hmottestad modified the milestones: 4.0.0, 4.1.0 Apr 22, 2022
@hmottestad hmottestad added M1 Fixed in milestone 1 and removed M1 Fixed in milestone 1 labels Jul 2, 2022
@hmottestad hmottestad modified the milestones: 4.1.0, 4.2.0 Jul 31, 2022
@hmottestad hmottestad modified the milestones: 4.2.0, 4.3.0 Sep 16, 2022
@github-project-automation github-project-automation bot moved this to 📋 Backlog in RDF4J Planning Jan 3, 2023
@abrokenjester abrokenjester moved this from 📋 Backlog to 🏗 In progress in RDF4J Planning Jan 3, 2023
@hmottestad hmottestad modified the milestones: 4.3.0, 5.0.0 May 19, 2023
kenwenzel added a commit to kenwenzel/rdf4j that referenced this issue Sep 11, 2023
- uses revision-based eviction to support lazy values
@hmottestad hmottestad added the M2 Fixed in milestone 2 label Oct 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
📶 enhancement issue is a new feature or improvement M2 Fixed in milestone 2
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants