doc(upgrade): add upgrade known issue for cert rotation

docs/upgrade/v1-2-1-to-v1-2-2.md

@@ -18,7 +18,76 @@ For air-gapped environments, see [Prepare an air-gapped upgrade](./automatic.md#
 
 ---
 
-### 1. The `harvester-seeder` add-on is not upgraded
+### 1. Upgrade stuck in the `Upgrading System Service` state
+
+Expired certificates of certain system services may cause an upgrade to become stuck in the `Upgrading System Service` state for an extended period. To resolve this issue, perform the following steps:
+
+1. Identify the name of the `apply-manifest` job.
+
+    ```
+    kubectl get jobs -n harvester-system -l harvesterhci.io/upgradeComponent=manifest
+    ```
+
+    Example output:
+    ```
+    NAME                                 COMPLETIONS   DURATION   AGE
+    hvst-upgrade-9gmg2-apply-manifests   0/1           46s        46s
+    ```
+
+1. Check the logs of the `apply-manifest` job.
+
+    ```
+    kubectl logs jobs/hvst-upgrade-9gmg2-apply-manifests -n harvester-system
+    ```
+
+    If the following messages appear in the log, continue to the next step:
+
+    ```
+    Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioning, current generation: 30259)...
+    Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioning, current generation: 30259)...
+    Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioning, current generation: 30259)...
+    Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioning, current generation: 30259)...
+    ```
+
+1. Check the state of the Cluster API (CAPI) cluster.
+
+    ```
+    kubectl get clusters.provisioning.cattle.io local -n fleet-local -o yaml
+    ```
+
+    If the output is similar to the following example, the issue likely exists in the cluster.
+    ```
+        - lastUpdateTime: "2023-01-17T16:26:48Z"
+          message: 'configuring bootstrap node(s) custom-24cb32ce8387: waiting for probes:
+            kube-controller-manager, kube-scheduler'
+          reason: Waiting
+          status: Unknown
+          type: Updated
+    ```
+
+1. Identify the hostname of the affected node, and implement the [workaround](https://github.com/harvester/harvester/issues/3863#issuecomment-1539681311) to see if service certificates expire.
+
+    ```
+    kubectl get machines.cluster.x-k8s.io -n fleet-local <machine_name> -o yaml | yq .status.nodeRef.name
+    ```
+
+    Replace `<machine_name>` with the name of the affected node (as stated in the output of the previous step).
+
+    :::note
+
+    If multiple nodes joined the cluster around the same time, you must implement the [workaround](https://github.com/harvester/harvester/issues/3863#issuecomment-1539681311) on all those nodes.
+
+    :::
+
+
+- Related issue:
+  - [[DOC/ENHANCEMENT] need to add cert-rotate feature, otherwise upgrade may stuck on Waiting for CAPI cluster fleet-local/local to be provisioned](https://github.com/harvester/harvester/issues/3863)
+- Workaround:
+  - https://github.com/harvester/harvester/issues/3863#issuecomment-1539681311
+
+---
+
+### 2. The `harvester-seeder` add-on is not upgraded
 
 When upgrading Harvester clusters from v1.2.1 to v1.2.2, the `harvester-seeder` add-on is not fully upgraded because the Helm chart versions do not match. The `harvester-seeder` chart bundled with Harvester v1.2.2 is of v0.1.1, while the `harvester-seeder` add-on uses the v0.1.0 chart.
 

versioned_docs/version-v1.2/upgrade/v1-2-1-to-v1-2-2.md

@@ -18,7 +18,76 @@ For air-gapped environments, see [Prepare an air-gapped upgrade](./automatic.md#
 
 ---
 
-### 1. The `harvester-seeder` add-on is not upgraded
+### 1. Upgrade stuck in the `Upgrading System Service` state
+
+Expired certificates of certain system services may cause an upgrade to become stuck in the `Upgrading System Service` state for an extended period. To resolve this issue, perform the following steps:
+
+1. Identify the name of the `apply-manifest` job.
+
+    ```
+    kubectl get jobs -n harvester-system -l harvesterhci.io/upgradeComponent=manifest
+    ```
+
+    Example output:
+    ```
+    NAME                                 COMPLETIONS   DURATION   AGE
+    hvst-upgrade-9gmg2-apply-manifests   0/1           46s        46s
+    ```
+
+1. Check the logs of the `apply-manifest` job.
+
+    ```
+    kubectl logs jobs/hvst-upgrade-9gmg2-apply-manifests -n harvester-system
+    ```
+
+    If the following messages appear in the log, continue to the next step:
+
+    ```
+    Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioning, current generation: 30259)...
+    Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioning, current generation: 30259)...
+    Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioning, current generation: 30259)...
+    Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioning, current generation: 30259)...
+    ```
+
+1. Check the state of the Cluster API (CAPI) cluster.
+
+    ```
+    kubectl get clusters.provisioning.cattle.io local -n fleet-local -o yaml
+    ```
+
+    If the output is similar to the following example, the issue likely exists in the cluster.
+    ```
+        - lastUpdateTime: "2023-01-17T16:26:48Z"
+          message: 'configuring bootstrap node(s) custom-24cb32ce8387: waiting for probes:
+            kube-controller-manager, kube-scheduler'
+          reason: Waiting
+          status: Unknown
+          type: Updated
+    ```
+
+1. Identify the hostname of the affected node, and implement the [workaround](https://github.com/harvester/harvester/issues/3863#issuecomment-1539681311) to see if service certificates expire.
+
+    ```
+    kubectl get machines.cluster.x-k8s.io -n fleet-local <machine_name> -o yaml | yq .status.nodeRef.name
+    ```
+
+    Replace `<machine_name>` with the name of the affected node (as stated in the output of the previous step).
+
+    :::note
+
+    If multiple nodes joined the cluster around the same time, you must implement the [workaround](https://github.com/harvester/harvester/issues/3863#issuecomment-1539681311) on all those nodes.
+
+    :::
+
+
+- Related issue:
+  - [[DOC/ENHANCEMENT] need to add cert-rotate feature, otherwise upgrade may stuck on Waiting for CAPI cluster fleet-local/local to be provisioned](https://github.com/harvester/harvester/issues/3863)
+- Workaround:
+  - https://github.com/harvester/harvester/issues/3863#issuecomment-1539681311
+
+---
+
+### 2. The `harvester-seeder` add-on is not upgraded
 
 When upgrading Harvester clusters from v1.2.1 to v1.2.2, the `harvester-seeder` add-on is not fully upgraded due to the chart version mismatch:
 

versioned_docs/version-v1.3/upgrade/v1-2-1-to-v1-2-2.md

@@ -18,7 +18,76 @@ For air-gapped environments, see [Prepare an air-gapped upgrade](./automatic.md#
 
 ---
 
-### 1. The `harvester-seeder` add-on is not upgraded
+### 1. Upgrade stuck in the `Upgrading System Service` state
+
+Expired certificates of certain system services may cause an upgrade to become stuck in the `Upgrading System Service` state for an extended period. To resolve this issue, perform the following steps:
+
+1. Identify the name of the `apply-manifest` job.
+
+    ```
+    kubectl get jobs -n harvester-system -l harvesterhci.io/upgradeComponent=manifest
+    ```
+
+    Example output:
+    ```
+    NAME                                 COMPLETIONS   DURATION   AGE
+    hvst-upgrade-9gmg2-apply-manifests   0/1           46s        46s
+    ```
+
+1. Check the logs of the `apply-manifest` job.
+
+    ```
+    kubectl logs jobs/hvst-upgrade-9gmg2-apply-manifests -n harvester-system
+    ```
+
+    If the following messages appear in the log, continue to the next step:
+
+    ```
+    Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioning, current generation: 30259)...
+    Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioning, current generation: 30259)...
+    Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioning, current generation: 30259)...
+    Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioning, current generation: 30259)...
+    ```
+
+1. Check the state of the Cluster API (CAPI) cluster.
+
+    ```
+    kubectl get clusters.provisioning.cattle.io local -n fleet-local -o yaml
+    ```
+
+    If the output is similar to the following example, the issue likely exists in the cluster.
+    ```
+        - lastUpdateTime: "2023-01-17T16:26:48Z"
+          message: 'configuring bootstrap node(s) custom-24cb32ce8387: waiting for probes:
+            kube-controller-manager, kube-scheduler'
+          reason: Waiting
+          status: Unknown
+          type: Updated
+    ```
+
+1. Identify the hostname of the affected node, and implement the [workaround](https://github.com/harvester/harvester/issues/3863#issuecomment-1539681311) to see if service certificates expire.
+
+    ```
+    kubectl get machines.cluster.x-k8s.io -n fleet-local <machine_name> -o yaml | yq .status.nodeRef.name
+    ```
+
+    Replace `<machine_name>` with the name of the affected node (as stated in the output of the previous step).
+
+    :::note
+
+    If multiple nodes joined the cluster around the same time, you must implement the [workaround](https://github.com/harvester/harvester/issues/3863#issuecomment-1539681311) on all those nodes.
+
+    :::
+
+
+- Related issue:
+  - [[DOC/ENHANCEMENT] need to add cert-rotate feature, otherwise upgrade may stuck on Waiting for CAPI cluster fleet-local/local to be provisioned](https://github.com/harvester/harvester/issues/3863)
+- Workaround:
+  - https://github.com/harvester/harvester/issues/3863#issuecomment-1539681311
+
+---
+
+### 2. The `harvester-seeder` add-on is not upgraded
 
 When upgrading Harvester clusters from v1.2.1 to v1.2.2, the `harvester-seeder` add-on is not fully upgraded due to the chart version mismatch: