Data Loss in Redis Cluster After Deleting All Pods Due to IP Changes

[captainpro-eng]

Issue: When all Redis pods in a Kubernetes cluster are deleted, data saved in the Append Only File (AOF) is lost upon restart. After the pods come up, Redis successfully loads the AOF, and the cluster state initially shows as "ok." However, soon after, the cluster state changes to "fail" due to IP address changes, leading to the removal of cached data.

Steps to Reproduce:

Deploy a Redis cluster using multiple pods in a Kubernetes environment.
Save data in Redis using AOF.
Gracefully shut down and delete all Redis pods simultaneously.
Allow the pods to restart automatically.
Observed Behavior:

After restarting, each Redis pod receives a new IP address.
Redis successfully loads the AOF and reports the cluster state as "ok."
Shortly thereafter, the cluster state changes to "fail" due to communication issues between nodes caused by the new IP addresses.
As a result, all cached data is lost when the cluster fails, even though the AOF file was loaded successfully at startup.
Expected Behavior:

The Redis cluster should retain data integrity and successfully recover from AOF without data loss, even after pod restarts and IP changes.
Nodes should be able to communicate effectively, maintaining the cluster state as "ok" after loading the AOF.