docs: update apiserver HACluster docs

-e
Signed-off-by: machichima <[email protected]>

Author: machichima

apiserver/Autoscaling.md

@@ -145,18 +145,13 @@ trigger a scale-up and launch a new worker pod.
 
 ### Validate that RayCluster is deployed correctly
 
-Run:
+Run following command to get list of pods running. You should see something like below:
 
-```shell
+```sh
 kubectl get pods
-```
-
-You should get something like this:
-
-```shell
-NAME                                READY   STATUS    RESTARTS   AGE
-kuberay-operator-545586d46c-f9grr   1/1     Running   0          49m
-test-cluster-head                   2/2     Running   0          3m1s
+# NAME                                READY   STATUS    RESTARTS   AGE
+# kuberay-operator-545586d46c-f9grr   1/1     Running   0          49m
+# test-cluster-head                   2/2     Running   0          3m1s
 ```
 
 Note that there is no worker for `test-cluster` as we set its initial replicas to 0. You

apiserver/HACluster.md

@@ -1,57 +1,101 @@
 # Creating HA cluster with API Server
 
-One of the issue for long-running Ray applications, for example, Ray Serve is that Ray Head node is a single
-point of failure, which means that if the Head node dies, complete cluster has to be restarted. Fortunately,
-KubeRay cluster provides an option to create
-[fault tolerance Ray cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
-The similar type of highly available Ray cluster can also be created using API server. The foundation of this
-approach is ensuring high availability Global Control Service (GCS) data. GCS manages cluster-level
-metadata. By default, the GCS lacks fault tolerance as it stores all data in-memory, and a failure can cause the
-entire Ray cluster to fail. To make the GCS fault tolerant, you must have a high-availability Redis. This way,
-in the event of a GCS restart, it retrieves all the data from the Redis instance and resumes its regular
-functioning.
-
-## Creating external Redis cluster
+One of the issue for long-running Ray application (e.g. RayServe) is that if Ray head node
+dies, the whole cluster has to be restarted. Fortunately, KubeRay cluster solved it by
+introducing [Fault Tolerance Ray Cluster](https://docs.ray.io/en/master/cluster/kubernetes/user-guides/kuberay-gcs-ft.html).
+
+The RayCluster with high availability can also be created in API server, which aims to
+ensure a high availability Global Control Service (GCS) data. The GCS manages
+cluster-level metadata by storing all data in memory, which is lack of fault tolerance. A
+single failure can cause the entire RayCluster to fail. To enable GCS's fault tolerance,
+we should have a highly available Redis so that when GCS restart, it can resume its
+status by retrieve previous data from the Redis instance.
+
+We will provide a detailed example on how to create this highly available API Server.
+
+## Setup
+
+### Setup Ray Operator and API Server
+
+Refer to [README](README.md) for setting up KubRay operator and API server.
+
+Alternatively, you could build and deploy the Operator and API server from local repo for
+development purpose.
+
+```shell
+make start-local-apiserver deploy
+```
+
+## Example
+
+Before going through the example, remove any running RayClusters to ensure a successful
+run through of the example below.
+
+```sh
+kubectl delete raycluster --all
+```
+
+### Create external Redis cluster
 
 A comprehensive documentation on creating Redis cluster on Kubernetes can be found
 [here]( https://www.dragonflydb.io/guides/redis-kubernetes). For this example we will use a rather simple
-[yaml file](test/cluster/redis/redis.yaml). To create Redis run:
+[RedisYAML]. Simply download this YAML file and run:
 
 ```sh
 kubectl create ns redis
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/redis/redis.yaml -n redis
+kubectl apply -f redis.yaml -n redis
 ```
 
-Note that here we are deploying redis to the `redis` namespace, that we are creating here.
+Note that we created a new `redis` namespace and deploy redis in it.
 
 Alternatively, if you run on the cloud you can use managed version of HA Redis, which will not require
 you to stand up, run, manage and monitor your own version of redis.
 
-## Creating Redis password secret
+Check if the redis is successfully set up with following command. You should see
+`redis-config` in the list:
+
+```sh
+kubectl get configmaps -n redis
 
-Before creating your cluster, you need to create [secret](test/cluster/redis/redis_passwrd.yaml) in the
+# NAME               DATA   AGE
+# redis-config       1      19s
+```
+
+### Create Redis password secret
+
+Before creating your cluster, you need to create secret in the
 namespace where you are planning to create your Ray cluster (remember, that secret is visible only within a given
-namespace). To create a secret for using external redis, run:
+namespace). To create a secret for using external redis, please download the [Secret] and
+run following command:
 
 ```sh
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/redis/redis_passwrd.yaml
+kubectl apply -f redis_passwrd.yaml
 ```
 
-## Ray Code for testing
+### Install ConfigMap
+
+We will use this [ConfigMap] which contains code for our example. For the real world
+cases, it is recommended to pack user code in an image.
 
-For both Ray Jobs and Ray Serve we recommend packaging user code in the image. For a simple testing here
-we will create a [config map](test/cluster/code_configmap.yaml), containing simple code, that we will use for
-testing. To deploy it run the following:
+Please download the config map and deploy it with following command:
 
 ```sh
-kubectl apply -f <your location>/kuberay/apiserver/test/cluster/code_configmap.yaml
+kubectl apply -f code_configmap.yaml
 ```
 
-## API server request
+### Create RayCluster
 
-To create a Ray cluster we can use the following curl command:
+Use following command to create a compute template and a RayCluster:
 
 ```sh
+curl -X POST 'localhost:31888/apis/v1/namespaces/default/compute_templates' \
+--header 'Content-Type: application/json' \
+--data '{
+  "name": "default-template",
+  "namespace": "default",
+  "cpu": 2,
+  "memory": 4
+}'
 curl -X POST 'localhost:31888/apis/v1/namespaces/default/clusters' \
 --header 'Content-Type: application/json' \
 --data '{
@@ -134,84 +178,124 @@ curl -X POST 'localhost:31888/apis/v1/namespaces/default/clusters' \
 }'
 ```
 
-Note that computeTemplate here has to be created using this [command](test/cluster//template/simple)
+To enable the RayCluster's GCS fault tolerance feature, we added the annotation:
 
-Lets discuss the important pieces here:
-You need to specify annotation, that tells Ray that this is cluster with GCS fault tolerance
-
-```sh
-ray.io/ft-enabled: "true"
+```json
+"annotations" : {
+    "ray.io/ft-enabled": "true"
+}
 ```
 
-For the `headGroupSpec` you need the following. In the `rayStartParams` you need to add information about Redis
-password.
+For connecting to Redis, we set the following content in `rayStartParams` of
+`headGroupSpec`: We also added following content in `rayStartParams` of `headGroupSpec`,
+which set the Redis password and the number of cpu. Setting `num-cpu` to 0 ensures that no
+application code runs on a head node.
 
-```sh
+```json
 "redis-password:: "$REDIS_PASSWORD"
 "num-cpu": "0"
 ```
 
-Where the value of `REDIS_PASSWORD` comes from environment variable (below). Additionally `num-cpus:
-0` ensures that no application code runs on a head node.
+Here, the `$REDIS_PASSWORD` is defined in `headGroupSpec`'s environment variable below:
 
-The following environment variable have to be added here:
+```json
+"environment": {
+    "values": {
+        "RAY_REDIS_ADDRESS": "redis.redis.svc.cluster.local:6379"
+    },
+    "valuesFrom": {
+        "REDIS_PASSWORD": {
+            "source": 1,
+            "name": "redis-password-secret",
+            "key": "password"
+        }
+    }
+},
+```
 
-```sh
-       "environment": {
-         "values": {
-            "RAY_REDIS_ADDRESS": "redis.redis.svc.cluster.local:6379"
-         },
-         "valuesFrom": {
-            "REDIS_PASSWORD": {
-                "source": 1,
-                "name": "redis-password-secret",
-                "key": "password"
-            }
-         }
-       },
+For the `workerGroupSpecs`, we set the `gcs_rpc_server_reconnect_timeout` environment
+variable, which controls the GCS heartbeat timeout (default 60 seconds). This controls how
+long after the head node dies do we kill the worker node. While it takes time to restart
+the head node, we want this values to be large enough to prevent the worker node being
+killed during the restarting period.
+
+```json
+"environment": {
+    "values": {
+        "RAY_gcs_rpc_server_reconnect_timeout_s": "300"
+    }
+},
 ```
 
-For the `workerGroupSpecs` you might want to increase `gcs_rpc_server_reconnect_timeout` by specifying the following
-environment variable:
+### Validate that RayCluster is deployed correctly
+
+Run following command to get list of pods running. You should see one head and worker node
+like below:
 
 ```sh
-        "environment": {
-           "values": {
-             "RAY_gcs_rpc_server_reconnect_timeout_s": "300"
-           }
-        },
+kubectl get pods
+# NAME                                READY   STATUS    RESTARTS   AGE
+# ha-cluster-head                     1/1     Running   0          2m36s
+# ha-cluster-small-wg-worker-22lbx    1/1     Running   0          2m36s
 ```
 
-This environment variable allows to increase GCS heartbeat timeout, which is 60 sec by default. The reason for
-increasing it is because restart of the head node can take some time, and we want to make sure that the worker node
-will not be killed during this time.
+### Create an Actor
+
+Before we try to trigger the restoration, we need to find a way to validate our GCS restore
+is working correctly. We will validate this by creating a detached actor. If it still
+exists and functions after the head node deletion and restoration, we can confirm that the
+GCS data is restored correctly.
+
+Run following command for creating a detached actor. Please change `ha-cluster-head` to
+your head node's name:
 
-## Testing resulting cluster
+```sh
+kubectl exec -it ha-cluster-head -- python3 /home/ray/samples/detached_actor.py
+```
 
-Once the cluster is created, we can validate that it is working correctly. To do this first create a detached actor.
-To do this, note the name of the head node and create a detached actor using the following command:
+Then, open a new termianl and use port-forward to enable accessing to the Ray dashboard.
+The dashboard can be accessed through `http://localhost:8265`:
 
 ```sh
-kubectl exec -it <head node pod name> -- python3 /home/ray/samples/detached_actor.py
+kubectl port-forward pod/ha-cluster-head 8265:8265
 ```
 
-Once this is done, open Ray dashboard (using port-forward). In the cluster tab you should see 2 nodes and in the
-Actor's pane you should see created actor.
+In the dashboard, You can see 2 nodes in the Cluster pane, which is head and worker:
+
+![hacluster-dashboard-cluster](img/hacluster-dashboard-cluster.png)
+
+If you go to the Actor pane, you can see the actor we created earlier:
 
-Now you can delete head node pod:
+![hacluster-dashboard-actor](img/hacluster-dashboard-actor.png)
+
+### Trigger the GCS restore
+
+To trigger the restoration, we can simply delete the head node with:
 
 ```sh
-kubectl delete pods <head node pod name>
+kubectl delete pods ha-cluster-head
 ```
 
-The operator will recreate it. Make sure that only head node is recreated (note that it now has a different name),
-while worker node stays as is. Now you can go to the dashboard and make sure that in the Cluster tab you still see
-2 nodes and in the Actor's pane you still see created actor.
+If you list the pods now, you can see a new head node is recreated
 
-For additional test run the following command:
+```sh
+kubectl get pods
+# NAME                                READY   STATUS    RESTARTS   AGE
+# ha-cluster-head                     0/1     Running   0          5s
+# ha-cluster-small-wg-worker-tpgqs    1/1     Running   0          9m19s
+```
+
+Note that only head node will be recreated, while the worker node stays as is.
+
+Port-forward again and access the dashboard through `http://localhost:8265`:
 
 ```sh
-kubectl exec -it <head node pod name> -- python3 /home/ray/samples/increment_counter.py
+kubectl port-forward pod/ha-cluster-head 8265:8265
 ```
 
-and make sure that it executes correctly. Note that the name of the head node here is different
+You can see one pod marked as "DEAD" in the Cluster pane and the actor in Actors pane
+still running.
+
+[RedisYAML]: test/cluster/redis/redis.yaml
+[Secret]: test/cluster/redis/redis_passwrd.yaml
+[ConfigMap]: test/cluster/code_configmap.yaml