Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix previous shard info publishing when changing community shard #5762

Open
igor-sirotin opened this issue Aug 23, 2024 · 2 comments
Open

Fix previous shard info publishing when changing community shard #5762

igor-sirotin opened this issue Aug 23, 2024 · 2 comments
Labels
E:Community Sharding Tasks to get sharding working on communities

Comments

@igor-sirotin
Copy link
Collaborator

Ideally we should unsubscribe when we're no longer interested in receiving messages or publishing to the topic.
So in this flow you describe:

- Change community shard to 256
- At this point we unsubscribe from shard 128.
- And then send the new shard info. But we should send it on previous shard, so we do it on shard 128.
⚠️ And this is how we call Publish without being subscribed. And it seem to be working fine, 1 peer found.

We need to switch the order of unsubscribing and sending the previous shard info.
I do also see this flow prone to errors because it's possible that publishing the new shard might fail. So perhaps the way to go is: Publish the new shard info. Confirm that the message is received by a storenode (this is done with the message verification logic that @kaichaosun implemented, and only then, unsubscribe from the shard. (Maybe this verification and unsubscribe attempt could be done async?)

Originally posted by @richard-ramos in #5759 (comment)

@igor-sirotin igor-sirotin added the E:Community Sharding Tasks to get sharding working on communities label Aug 23, 2024
@kaichaosun
Copy link
Contributor

Not sure how change shard works for community, if we have a big community, changing from shard 128 to shard 256 (or any other shard), users in community get the changing shard message at different time, right?
We probably need to subscribe both shards for a certain period, for example 10 days. After that, only new shard will be supported, and messages through old shard will not be seen by others.

@igor-sirotin
Copy link
Collaborator Author

igor-sirotin commented Aug 26, 2024

We probably need to subscribe both shards for a certain period, for example 10 days

In theory this shouldn't be needed. Even when changing the shard immediately, members will be able to fetch the shard info from the storenodes on the old shard, then switch to the new shard and fetch everything here again from the point when the shard was changed.

Though such algorithm is not implemented as well 😄
There's probably only 1 timestamp of "last sync with store nodes" and it's not rewinded when shard change was detected.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
E:Community Sharding Tasks to get sharding working on communities
Projects
None yet
Development

No branches or pull requests

2 participants