Add persistent ephemeral node support to metadata store #772
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bryanlb
force-pushed
the
bburkholder/zk-reconnect
branch
13 times, most recently
from
f6f6e79
to
6bcba22
Compare
bryanlb
force-pushed
the
bburkholder/zk-reconnect
branch
2 times, most recently
from
b6af00f
to
d3ca51f
Compare
bryanlb
force-pushed
the
bburkholder/zk-reconnect
branch
from
d3ca51f
to
74ceef6
Compare
mansu
reviewed
Mar 1, 2024
| @@ -15,12 +16,14 @@ public class CacheSlotMetadataStore extends KaldbPartitioningMetadataStore<Cache | |||
| * Initializes a cache slot metadata store at the CACHE_SLOT_ZK_PATH. This should be used to | |||
| * create/update the cache slots, and for listening to all cache slot events. | |||
| */ | |||
| public CacheSlotMetadataStore(AsyncCuratorFramework curatorFramework) throws Exception { | |||
| public CacheSlotMetadataStore(AsyncCuratorFramework curatorFramework, MeterRegistry meterRegistry) | |||
There was a problem hiding this comment.
A large part of this diff if adding meterRegistry parameter. It would be far more easier to review the diff if meterRegistry change is done in a separate PR.
kaldb/src/main/java/com/slack/kaldb/metadata/core/KaldbMetadataStore.java
Show resolved
Hide resolved
|
This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 3 days. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This adds an optional opt-in mode to run the
KaldbMetadataStoreusing a combination ofPersistentNodeandPersistentWatcherfor ephemeral nodes. This system property flag will be removed once the logic is validated, at which point this will no longer be able to be opt-out.The goal of this PR is to support using ephemeral nodes across ZK session disconnects (#644 (comment)), and uses the Curator
PersistentNodein order to recreate nodes when they disappear, andPersistentWatcherto subscribe to updates to ephemeral nodes by other instances (ie - manager updating cache/recovery ephemerals).https://curator.apache.org/docs/recipes-persistent-node
https://curator.apache.org/docs/recipes-persistent-watcher
This PR also includes: