Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data Loss After Restarting All Leaders and Followers in Redis Cluster #1164

Open
captainpro-eng opened this issue Dec 12, 2024 · 0 comments
Open
Labels
bug Something isn't working help wanted Extra attention is needed

Comments

@captainpro-eng
Copy link

captainpro-eng commented Dec 12, 2024

We are currently running a Redis cluster with the following versions:
Redis Operator Helm Chart: 18.0.5
Redis Operator Image: 18.0.1
Redis Image: 7.0.12

We have tested multiple Redis failover scenarios, and in most cases, the cluster state is marked as "OK" and data is preserved after restarts. However, we encountered a scenario where restarting all the masters and slaves results in the cluster state being "OK", but all data is lost. Below is a summary of the test cases:

Tested Scenarios:

  • Restarted 3 leaders only: After all leaders came back up, the cluster state became "OK" and all data was preserved.

  • Restarted all the leaders : After all leaders came back up, the cluster state became "OK" and all data was secure.

  • Restarted all the followers : After all followers came back up, the cluster state became "OK" and all data was secure.

  • Restarted 3 leaders and 3 followers : After all leaders and followers came back up, the cluster state became "OK" and all data was secure.

  • Restarted all the leaders and all the followers: After all masters and slaves came back up, the cluster state was "OK", but all data was lost.

@captainpro-eng captainpro-eng added the bug Something isn't working label Dec 12, 2024
@drivebyer drivebyer added the help wanted Extra attention is needed label Dec 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

2 participants