[GCS FT] GCS FT misconfiguration

[kevin85421]

Search before asking

• I searched the issues and found no similar issues.

KubeRay Component

ray-operator

What happened + What you expected to happen

See this Slack thread for more details.

The user creates a RayService without setting the annotation ray.io/ft-enabled: "true", but they do set RAY_REDIS_ADDRESS and REDIS_PASSWORD. As a result, the RayCluster enables GCS FT and writes data to the external Redis, but KubeRay is unaware of this. KubeRay doesn't configure RAY_external_storage_namespace so that the RayCluster writes data to the key default in the Redis.

When the user triggers a zero-downtime upgrade, the new RayCluster also attempts to read metadata from the default key in Redis. Therefore, the new RayCluster will see some information from the old RayCluster.

Solution:

• Explicitly disable GCS FT if KubeRay determines that the RayCluster has not enabled GCS FT.

Reproduction script

TODO

Anything else

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[kevin85421]

/assign @rueian

[rueian]

👌