feat: add image garbage collections

Signed-off-by: PoAn Yang <[email protected]>

docs/upgrade/troubleshooting.md

@@ -15,9 +15,9 @@ Here are some tips to troubleshoot a failed upgrade:
 - Check [version-specific upgrade notes](./automatic.md#upgrade-support-matrix). You can click the version in the support matrix table to see if there are any known issues.
 - Dive into the upgrade [design proposal](https://github.com/harvester/harvester/blob/master/enhancements/20220413-zero-downtime-upgrade.md). The following section briefly describes phases within an upgrade and possible diagnostic methods.
 
-## Diagnose the upgrade flow 
+## Diagnose the upgrade flow
 
-A Harvester upgrade process contains several phases. 
+A Harvester upgrade process contains several phases.
     ![](/img/v1.2/upgrade/ts_upgrade_phases.png)
 
 ### Phase 1: Provision upgrade repository VM.
@@ -200,3 +200,14 @@ deployment.apps/hvst-upgrade-xxxxx-upgradelog-downloader scaled
 ```
 
 :::
+
+### Cleanup unused images for enough disks to avoid image garbage collections
+
+The default value of `imageGCHighThresholdPercent` in [KubeletConfiguration](https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-KubeletConfiguration) is `85`. If kubelet detects disk usage is more than 85%, it tries to remove unused images.
+
+During Harvester upgrade, the system loads new images to each node. If disk usage is more than 85%, new images may be candidates for cleanup, because it's not used by any container.
+In an airgapped environment, this may break the upgrade, because new images can't be found in the cluster.
+
+If you get error message like 'Node xxx will reach xx.xx% storage space after loading new images. It's higher than kubelet image garbage collection threshold 85%.', you can run `crictl rmi --prune` to cleanup unused images first, before new upgrade starts.
+
+![Disk space not enough error message](/img/v1.4/upgrade/disk-space-not-enough-error-message.png)

versioned_docs/version-v1.3/upgrade/troubleshooting.md

@@ -15,9 +15,9 @@ Here are some tips to troubleshoot a failed upgrade:
 - Check [version-specific upgrade notes](./automatic.md#upgrade-support-matrix). You can click the version in the support matrix table to see if there are any known issues.
 - Dive into the upgrade [design proposal](https://github.com/harvester/harvester/blob/master/enhancements/20220413-zero-downtime-upgrade.md). The following section briefly describes phases within an upgrade and possible diagnostic methods.
 
-## Diagnose the upgrade flow 
+## Diagnose the upgrade flow
 
-A Harvester upgrade process contains several phases. 
+A Harvester upgrade process contains several phases.
     ![](/img/v1.2/upgrade/ts_upgrade_phases.png)
 
 ### Phase 1: Provision upgrade repository VM.
@@ -200,3 +200,14 @@ deployment.apps/hvst-upgrade-xxxxx-upgradelog-downloader scaled
 ```
 
 :::
+
+### Cleanup unused images for enough disks to avoid image garbage collections
+
+The default value of `imageGCHighThresholdPercent` in [KubeletConfiguration](https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-KubeletConfiguration) is `85`. If kubelet detects disk usage is more than 85%, it tries to remove unused images.
+
+During Harvester upgrade, the system loads new images to each node. If disk usage is more than 85%, new images may be candidates for cleanup, because it's not used by any container.
+In an airgapped environment, this may break the upgrade, because new images can't be found in the cluster.
+
+If you get error message like 'Node xxx will reach xx.xx% storage space after loading new images. It's higher than kubelet image garbage collection threshold 85%.', you can run `crictl rmi --prune` to cleanup unused images first, before new upgrade starts.
+
+![Disk space not enough error message](/img/v1.4/upgrade/disk-space-not-enough-error-message.png)