From d6a09ac6c7c41edf54e75d0002ce6d2559a87bd2 Mon Sep 17 00:00:00 2001 From: timfeirg Date: Mon, 17 Jul 2023 17:16:19 +0800 Subject: [PATCH 01/10] docs update on authentication-webhook --- docs/zh_cn/administration/going-production.md | 39 +++++++++++++++++++ .../administration/troubleshooting-cases.md | 9 ++++- 2 files changed, 47 insertions(+), 1 deletion(-) diff --git a/docs/zh_cn/administration/going-production.md b/docs/zh_cn/administration/going-production.md index c3fd1f23a5..baaac06327 100644 --- a/docs/zh_cn/administration/going-production.md +++ b/docs/zh_cn/administration/going-production.md @@ -210,3 +210,42 @@ spec: - --leader-election-lease-duration=15s # Leader 的间隔，默认为 15s ... ``` + +## 启用 Kubelet authentication webhook {#authentication-webhook} + +如果 kubelet 没有启用 Authentication webhook，会导致 CSI Node 获取 Pod 列表时报错（该报错本身已经修复，见后续描述）： + +``` +kubelet_client.go:99] GetNodeRunningPods err: Unauthorized +reconciler.go:70] doReconcile GetNodeRunningPods: invalid character 'U' looking for beginning of value +``` + +面对这种情况，我们建议[启用 Kubelet authentication webhook](../administration/going-production.md#authentication-webhook)。 + +```yaml {5,8} title="/var/lib/kubelet/config.yaml" +apiVersion: kubelet.config.k8s.io/v1beta1 +authentication: + webhook: + cacheTTL: 0s + enabled: true + ... +authorization: + mode: Webhook + ... +``` + +在 v0.21.0 及其后版本，就算未启用上方的 Authentication webhook，CSI Node 也不再会出现异常、而是绕过 kubelet，直接访问 APIServer 去获取信息（比如 `ListPod`）。考虑到「节点上有哪些 Pod」这个信息，本身就存在于 kubelet 空间，绕远去访问 APIServer，势必会产生少量额外的性能开销。因此在生产集群，我们仍推荐启用 Authentication webhook。 + +需要注意，就算使用了 v0.21.0 及之后的版本，CSI 驱动需要配置 `podInfoOnMount: true`，上边提到的避免报错的特性才会真正生效。如果你采用 [Helm 安装方式](../getting_started.md#helm)，该问题并不存在，因为 `podInfoOnMount` 已经写死安装文件里，随着升级自动启用。而如果你使用 kubectl 直接安装，你需要为 `k8s.yaml` 添加如下配置： + +```yaml {6} title="k8s.yaml" +... +apiVersion: storage.k8s.io/v1 +kind: CSIDriver +... +spec: + podInfoOnMount: true + ... +``` + +这也是为什么在生产环境，我们推荐用 Helm 安装 CSI 驱动，避免手动维护的 `k8s.yaml`，在升级时带来额外的心智负担。 diff --git a/docs/zh_cn/administration/troubleshooting-cases.md b/docs/zh_cn/administration/troubleshooting-cases.md index 4d99c49bfd..cc21d0d3af 100644 --- a/docs/zh_cn/administration/troubleshooting-cases.md +++ b/docs/zh_cn/administration/troubleshooting-cases.md @@ -29,7 +29,14 @@ kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: dri /var/lib/kubelet/csi-plugins/csi.juicefs.com/csi.sock: connect: no such file or directory ``` -此时需要[检查 CSI Node](./troubleshooting.md#check-csi-node)，确认其异常原因，并排查修复。 +此时需要[检查 CSI Node](./troubleshooting.md#check-csi-node)，确认其异常原因，并排查修复。常见的问题比如 kubelet 没有启用 Authentication webhook，导致获取 Pod 列表时报错： + +``` +kubelet_client.go:99] GetNodeRunningPods err: Unauthorized +reconciler.go:70] doReconcile GetNodeRunningPods: invalid character 'U' looking for beginning of value +``` + +面对这种情况，我们建议[启用 Kubelet authentication webhook](../administration/going-production.md#authentication-webhook)。 ## Mount Pod 异常 {#mount-pod-error} From a87b52425eb263f37fd586543eb13269a285d58f Mon Sep 17 00:00:00 2001 From: timfeirg Date: Tue, 18 Jul 2023 10:13:32 +0800 Subject: [PATCH 02/10] more --- docs/zh_cn/administration/going-production.md | 2 +- docs/zh_cn/guide/pv.md | 18 ++++++++---------- 2 files changed, 9 insertions(+), 11 deletions(-) diff --git a/docs/zh_cn/administration/going-production.md b/docs/zh_cn/administration/going-production.md index baaac06327..e5f6c6849b 100644 --- a/docs/zh_cn/administration/going-production.md +++ b/docs/zh_cn/administration/going-production.md @@ -234,7 +234,7 @@ authorization: ... ``` -在 v0.21.0 及其后版本，就算未启用上方的 Authentication webhook，CSI Node 也不再会出现异常、而是绕过 kubelet，直接访问 APIServer 去获取信息（比如 `ListPod`）。考虑到「节点上有哪些 Pod」这个信息，本身就存在于 kubelet 空间，绕远去访问 APIServer，势必会产生少量额外的性能开销。因此在生产集群，我们仍推荐启用 Authentication webhook。 +在 v0.21.0 及其后版本，就算未启用上方的 Authentication webhook，CSI Node 也不再会出现异常、而是绕过 kubelet，直接访问 APIServer 去获取信息（比如 `ListPod`），这种情况会产生少量额外的性能开销。因此在生产集群，我们仍推荐启用 Authentication webhook。 需要注意，就算使用了 v0.21.0 及之后的版本，CSI 驱动需要配置 `podInfoOnMount: true`，上边提到的避免报错的特性才会真正生效。如果你采用 [Helm 安装方式](../getting_started.md#helm)，该问题并不存在，因为 `podInfoOnMount` 已经写死安装文件里，随着升级自动启用。而如果你使用 kubectl 直接安装，你需要为 `k8s.yaml` 添加如下配置： diff --git a/docs/zh_cn/guide/pv.md b/docs/zh_cn/guide/pv.md index 182f7ab3b8..9b6bf843d2 100644 --- a/docs/zh_cn/guide/pv.md +++ b/docs/zh_cn/guide/pv.md @@ -711,17 +711,19 @@ spec: ### PV 容量分配 {#storage-capacity} -目前而言，JuiceFS CSI 驱动仅支持为动态 PersistentVolume 设置存储容量。在静态 PersistentVolume 与其 PersistentVolumeClaim 中指定的容量会被忽略，填写任意有效值即可，例如 `100Gi`： +从 v0.19.3 开始，JuiceFS CSI 驱动支持在动态配置设置存储容量（要注意，仅支持动态配置）。 + +在静态配置中，PVC 中指定的容量会被忽略，填写任意有效值即可，建议填写一个较大的数值，避免未来版本如果带来该功能支持时，因为容量超限导致问题。 ```yaml ... storageClassName: "" resources: requests: - storage: 100Gi + storage: 10Ti ``` -而在使用 StorageClass 的 PersistentVolumeClaim 中指定存储容量是有效的： +而在动态配置中，可以在 PVC 中指定存储容量，这个容量限制将会被翻译成 `juicefs quota` 命令，在 CSI Controller 中执行，为该 PV 所对应的子目录添加容量限制。关于 `juicefs quota` 命令，可以参考[社区版文档](https://juicefs.com/docs/zh/community/command_reference/#quota)，商业版文档待补充。 ```yaml ... @@ -731,18 +733,14 @@ resources: storage: 100Gi ``` -:::note 注意 -存储容量只对该 PersistentVolumeClaim 所使用的子目录有效，不会影响整个 JuiceFS volume 的存储配额。 -::: - -我们可以在应用 Pod 中使用 `df` 查看存储容量： +创建并挂载好 PV 后，可以进入容器用 `df -h` 验证容量生效： -```bash +```shell $ df -h Filesystem Size Used Avail Use% Mounted on overlay 84G 66G 18G 80% / tmpfs 64M 0 64M 0% /dev -JuiceFS:ce-secret 100G 0 100G 0% /data-0 +JuiceFS:myjfs 100G 0 100G 0% /data-0 ``` ### PV 扩容 {#pv-resize} From 50a213b4eae59caafd5dc40916dd2b96e147ebec Mon Sep 17 00:00:00 2001 From: timfeirg Date: Tue, 18 Jul 2023 17:26:03 +0800 Subject: [PATCH 03/10] Update docs/zh_cn/administration/going-production.md Co-authored-by: xixi --- docs/zh_cn/administration/going-production.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/zh_cn/administration/going-production.md b/docs/zh_cn/administration/going-production.md index e5f6c6849b..5601a99be7 100644 --- a/docs/zh_cn/administration/going-production.md +++ b/docs/zh_cn/administration/going-production.md @@ -236,7 +236,7 @@ authorization: 在 v0.21.0 及其后版本，就算未启用上方的 Authentication webhook，CSI Node 也不再会出现异常、而是绕过 kubelet，直接访问 APIServer 去获取信息（比如 `ListPod`），这种情况会产生少量额外的性能开销。因此在生产集群，我们仍推荐启用 Authentication webhook。 -需要注意，就算使用了 v0.21.0 及之后的版本，CSI 驱动需要配置 `podInfoOnMount: true`，上边提到的避免报错的特性才会真正生效。如果你采用 [Helm 安装方式](../getting_started.md#helm)，该问题并不存在，因为 `podInfoOnMount` 已经写死安装文件里，随着升级自动启用。而如果你使用 kubectl 直接安装，你需要为 `k8s.yaml` 添加如下配置： +需要注意，就算使用了 v0.21.0 及之后的版本，CSI 驱动需要配置 `podInfoOnMount: true`，上边提到的避免报错的特性才会真正生效。如果你采用 [Helm 安装方式](../getting_started.md#helm)，则 `podInfoOnMount` 默认开启无需配置，该特性会随着升级自动启用。而如果你使用 kubectl 直接安装，你需要为 `k8s.yaml` 添加如下配置： ```yaml {6} title="k8s.yaml" ... From 6bc0a884eddd6b714ece9993e2ef392af40260d6 Mon Sep 17 00:00:00 2001 From: timfeirg Date: Tue, 18 Jul 2023 20:30:47 +0800 Subject: [PATCH 04/10] translation --- docs/en/administration/going-production.md | 39 +++++++++++++++++++ docs/zh_cn/administration/going-production.md | 2 +- 2 files changed, 40 insertions(+), 1 deletion(-) diff --git a/docs/en/administration/going-production.md b/docs/en/administration/going-production.md index 8da9a17be2..e6963e68d1 100644 --- a/docs/en/administration/going-production.md +++ b/docs/en/administration/going-production.md @@ -210,3 +210,42 @@ spec: - --leader-election-lease-duration=15s # Interval between replicas competing for Leader, default to 15s ... ``` + +## Enable kubelet authentication webhook {#authentication-webhook} + +If authentication webhook isn't enabled, CSI Node will run into error when listing pods (this is however, a issue fixed in newer versions, continue reading for more): + +``` +kubelet_client.go:99] GetNodeRunningPods err: Unauthorized +reconciler.go:70] doReconcile GetNodeRunningPods: invalid character 'U' looking for beginning of value +``` + +When this happens, we recommend that you enable authentication webhook: + +```yaml {5,8} title="/var/lib/kubelet/config.yaml" +apiVersion: kubelet.config.k8s.io/v1beta1 +authentication: + webhook: + cacheTTL: 0s + enabled: true + ... +authorization: + mode: Webhook + ... +``` + +From v0.21.0, even if kubelet authentication webhook wasn't enabled, CSI Node will not run into errors. Instead it'll simply bypass kubelet, and obtain information directly from APIServer (like `ListPod`). Doing this adds a minor extra overhead to APIServer, thus authentication webhook is still recommended in production environments. + +Notice that CSI Driver must be configured `podInfoOnMount: true` for the above behavior to take effect. This problem doesn't exist however with Helm installations, because `podInfoOnMount` is hard-coded into template files and automatically applied between upgrades. So with kubectl installations, ensure these settings are put into `k8s.yaml`: + +```yaml {6} title="k8s.yaml" +... +apiVersion: storage.k8s.io/v1 +kind: CSIDriver +... +spec: + podInfoOnMount: true + ... +``` + +As is demonstrated above, we recommend using Helm to install CSI Driver, as this avoids the toil of maintaining & reviewing `k8s.yaml`. diff --git a/docs/zh_cn/administration/going-production.md b/docs/zh_cn/administration/going-production.md index 5601a99be7..a2bb11241a 100644 --- a/docs/zh_cn/administration/going-production.md +++ b/docs/zh_cn/administration/going-production.md @@ -220,7 +220,7 @@ kubelet_client.go:99] GetNodeRunningPods err: Unauthorized reconciler.go:70] doReconcile GetNodeRunningPods: invalid character 'U' looking for beginning of value ``` -面对这种情况，我们建议[启用 Kubelet authentication webhook](../administration/going-production.md#authentication-webhook)。 +面对这种情况，我们建议启用 authentication webhook： ```yaml {5,8} title="/var/lib/kubelet/config.yaml" apiVersion: kubelet.config.k8s.io/v1beta1 From 2bc9224fa958a470841801b522013eb529e132ea Mon Sep 17 00:00:00 2001 From: timfeirg Date: Wed, 19 Jul 2023 11:12:39 +0800 Subject: [PATCH 05/10] restart kubelet --- docs/en/administration/going-production.md | 2 +- docs/en/administration/troubleshooting-cases.md | 9 ++++++++- docs/zh_cn/administration/going-production.md | 6 +++--- 3 files changed, 12 insertions(+), 5 deletions(-) diff --git a/docs/en/administration/going-production.md b/docs/en/administration/going-production.md index e6963e68d1..3b46c14779 100644 --- a/docs/en/administration/going-production.md +++ b/docs/en/administration/going-production.md @@ -220,7 +220,7 @@ kubelet_client.go:99] GetNodeRunningPods err: Unauthorized reconciler.go:70] doReconcile GetNodeRunningPods: invalid character 'U' looking for beginning of value ``` -When this happens, we recommend that you enable authentication webhook: +When this happens, we recommend that you enable authentication webhook and restart kubelet: ```yaml {5,8} title="/var/lib/kubelet/config.yaml" apiVersion: kubelet.config.k8s.io/v1beta1 diff --git a/docs/en/administration/troubleshooting-cases.md b/docs/en/administration/troubleshooting-cases.md index 8d6184af4d..0c749d8611 100644 --- a/docs/en/administration/troubleshooting-cases.md +++ b/docs/en/administration/troubleshooting-cases.md @@ -31,7 +31,14 @@ If CSI Node pod is not properly running, and the socket file used to communicate /var/lib/kubelet/csi-plugins/csi.juicefs.com/csi.sock: connect: no such file or directory ``` -[Check CSI Node](./troubleshooting.md#check-csi-node) to debug and troubleshoot. +[Check CSI Node](./troubleshooting.md#check-csi-node) to debug and troubleshoot. A commonly encountered problem is kubelet being started without authentication webhook, which results in error when getting pod list: + +``` +kubelet_client.go:99] GetNodeRunningPods err: Unauthorized +reconciler.go:70] doReconcile GetNodeRunningPods: invalid character 'U' looking for beginning of value +``` + +In such case, we recommend [enabling kubelet authentication webhook](../administration/going-production.md#authentication-webhook). ## Mount Pod failure {#mount-pod-error} diff --git a/docs/zh_cn/administration/going-production.md b/docs/zh_cn/administration/going-production.md index a2bb11241a..b2abe977f3 100644 --- a/docs/zh_cn/administration/going-production.md +++ b/docs/zh_cn/administration/going-production.md @@ -213,14 +213,14 @@ spec: ## 启用 Kubelet authentication webhook {#authentication-webhook} -如果 kubelet 没有启用 Authentication webhook，会导致 CSI Node 获取 Pod 列表时报错（该报错本身已经修复，见后续描述）： +如果 Kubelet 没有启用 Authentication webhook，会导致 CSI Node 获取 Pod 列表时报错（该报错本身已经修复，见后续描述）： ``` kubelet_client.go:99] GetNodeRunningPods err: Unauthorized reconciler.go:70] doReconcile GetNodeRunningPods: invalid character 'U' looking for beginning of value ``` -面对这种情况，我们建议启用 authentication webhook： +面对这种情况，我们建议启用 authentication webhook 并重启 Kubelet： ```yaml {5,8} title="/var/lib/kubelet/config.yaml" apiVersion: kubelet.config.k8s.io/v1beta1 @@ -234,7 +234,7 @@ authorization: ... ``` -在 v0.21.0 及其后版本，就算未启用上方的 Authentication webhook，CSI Node 也不再会出现异常、而是绕过 kubelet，直接访问 APIServer 去获取信息（比如 `ListPod`），这种情况会产生少量额外的性能开销。因此在生产集群，我们仍推荐启用 Authentication webhook。 +在 v0.21.0 及其后版本，就算未启用上方的 Authentication webhook，CSI Node 也不再会出现异常、而是绕过 Kubelet，直接访问 APIServer 去获取信息（比如 `ListPod`），这种情况会产生少量额外的性能开销。因此在生产集群，我们仍推荐启用 Authentication webhook。 需要注意，就算使用了 v0.21.0 及之后的版本，CSI 驱动需要配置 `podInfoOnMount: true`，上边提到的避免报错的特性才会真正生效。如果你采用 [Helm 安装方式](../getting_started.md#helm)，则 `podInfoOnMount` 默认开启无需配置，该特性会随着升级自动启用。而如果你使用 kubectl 直接安装，你需要为 `k8s.yaml` 添加如下配置： From 9a992921d3d65697bbe5601634b915928b33abcd Mon Sep 17 00:00:00 2001 From: zwwhdls Date: Wed, 19 Jul 2023 11:26:30 +0800 Subject: [PATCH 06/10] update docs for kubelet authn & authz --- docs/zh_cn/administration/going-production.md | 25 ++++++++----------- 1 file changed, 11 insertions(+), 14 deletions(-) diff --git a/docs/zh_cn/administration/going-production.md b/docs/zh_cn/administration/going-production.md index 5601a99be7..dbbae73987 100644 --- a/docs/zh_cn/administration/going-production.md +++ b/docs/zh_cn/administration/going-production.md @@ -211,30 +211,27 @@ spec: ... ``` -## 启用 Kubelet authentication webhook {#authentication-webhook} +## 启用 Kubelet 认证鉴权 {kubelet-authn-authz} -如果 kubelet 没有启用 Authentication webhook，会导致 CSI Node 获取 Pod 列表时报错（该报错本身已经修复，见后续描述）： +[Kubelet 的认证鉴权](https://kubernetes.io/docs/reference/access-authn-authz/kubelet-authn-authz/)分为很多种，默认允许所有请求。但若 kubelet 关闭了匿名访问，会导致 CSI Node 获取 Pod 列表时报错（该报错本身已经修复，见后续描述）： ``` kubelet_client.go:99] GetNodeRunningPods err: Unauthorized reconciler.go:70] doReconcile GetNodeRunningPods: invalid character 'U' looking for beginning of value ``` -面对这种情况，我们建议[启用 Kubelet authentication webhook](../administration/going-production.md#authentication-webhook)。 +面对这种情况，我们建议两种解决方法： -```yaml {5,8} title="/var/lib/kubelet/config.yaml" -apiVersion: kubelet.config.k8s.io/v1beta1 -authentication: - webhook: - cacheTTL: 0s - enabled: true - ... -authorization: - mode: Webhook - ... +1. [对 Kubelet 启用 X509 客户端证书认证](https://kubernetes.io/docs/reference/access-authn-authz/kubelet-authn-authz/#kubelet-authentication) ，并配置 CSI Node 对 Kubelet 的认证信息，以保证 CSI Node 能够正常访问 Kubelet。具体配置方式为将 Kubelet 的认证证书路径以环境变量的方式配置在 CSI Node 中，如下： + +```shell +# 将 和 替换为实际的路径 +kubectl -n kube-system set env daemonset/juicefs-csi-node -c juicefs-plugin KUBELET_CLIENT_CERT= KUBELET_CLIENT_KEY= ``` -在 v0.21.0 及其后版本，就算未启用上方的 Authentication webhook，CSI Node 也不再会出现异常、而是绕过 kubelet，直接访问 APIServer 去获取信息（比如 `ListPod`），这种情况会产生少量额外的性能开销。因此在生产集群，我们仍推荐启用 Authentication webhook。 +2. 将 Kubelet 鉴权委派给 APIServer，具体请参考[官方文档](https://kubernetes.io/docs/reference/access-authn-authz/kubelet-authn-authz/#kubelet-authorization)。 + +在 v0.21.0 及其后版本，即使采取以上任意一种措施，CSI Node 也不再会出现异常，不会直接连接 Kubelet，而是 watch APIServer 去获取信息，由于 watch list 机制在启动时会对 APIServer 进行一次 `ListPod` 请求（有 `labelSelector`），在集群负载较大的情况下，会对 APIServer 造成额外的压力。因此在生产集群，我们仍推荐配置 CSI Node 对 Kubelet 的认证。 需要注意，就算使用了 v0.21.0 及之后的版本，CSI 驱动需要配置 `podInfoOnMount: true`，上边提到的避免报错的特性才会真正生效。如果你采用 [Helm 安装方式](../getting_started.md#helm)，则 `podInfoOnMount` 默认开启无需配置，该特性会随着升级自动启用。而如果你使用 kubectl 直接安装，你需要为 `k8s.yaml` 添加如下配置： From 213d58e70507b581052613c2c16e9a253e84ecc7 Mon Sep 17 00:00:00 2001 From: zwwhdls Date: Wed, 19 Jul 2023 11:47:43 +0800 Subject: [PATCH 07/10] fix md lint --- docs/zh_cn/administration/going-production.md | 12 ++++++------ docs/zh_cn/administration/troubleshooting-cases.md | 2 +- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/zh_cn/administration/going-production.md b/docs/zh_cn/administration/going-production.md index dbbae73987..cbba890ddf 100644 --- a/docs/zh_cn/administration/going-production.md +++ b/docs/zh_cn/administration/going-production.md @@ -211,9 +211,9 @@ spec: ... ``` -## 启用 Kubelet 认证鉴权 {kubelet-authn-authz} +## 启用 Kubelet 认证鉴权 {#kubelet-authn-authz} -[Kubelet 的认证鉴权](https://kubernetes.io/docs/reference/access-authn-authz/kubelet-authn-authz/)分为很多种，默认允许所有请求。但若 kubelet 关闭了匿名访问，会导致 CSI Node 获取 Pod 列表时报错（该报错本身已经修复，见后续描述）： +[Kubelet 的认证鉴权](https://kubernetes.io/docs/reference/access-authn-authz/kubelet-authn-authz)分为很多种，默认允许所有请求。但若 kubelet 关闭了匿名访问，会导致 CSI Node 获取 Pod 列表时报错（该报错本身已经修复，见后续描述）： ``` kubelet_client.go:99] GetNodeRunningPods err: Unauthorized @@ -224,10 +224,10 @@ reconciler.go:70] doReconcile GetNodeRunningPods: invalid character 'U' looking 1. [对 Kubelet 启用 X509 客户端证书认证](https://kubernetes.io/docs/reference/access-authn-authz/kubelet-authn-authz/#kubelet-authentication) ，并配置 CSI Node 对 Kubelet 的认证信息，以保证 CSI Node 能够正常访问 Kubelet。具体配置方式为将 Kubelet 的认证证书路径以环境变量的方式配置在 CSI Node 中，如下： -```shell -# 将 和 替换为实际的路径 -kubectl -n kube-system set env daemonset/juicefs-csi-node -c juicefs-plugin KUBELET_CLIENT_CERT= KUBELET_CLIENT_KEY= -``` + ```shell + # 将 和 替换为实际的路径 + kubectl -n kube-system set env daemonset/juicefs-csi-node -c juicefs-plugin KUBELET_CLIENT_CERT= KUBELET_CLIENT_KEY= + ``` 2. 将 Kubelet 鉴权委派给 APIServer，具体请参考[官方文档](https://kubernetes.io/docs/reference/access-authn-authz/kubelet-authn-authz/#kubelet-authorization)。 diff --git a/docs/zh_cn/administration/troubleshooting-cases.md b/docs/zh_cn/administration/troubleshooting-cases.md index cc21d0d3af..326fa1fa17 100644 --- a/docs/zh_cn/administration/troubleshooting-cases.md +++ b/docs/zh_cn/administration/troubleshooting-cases.md @@ -36,7 +36,7 @@ kubelet_client.go:99] GetNodeRunningPods err: Unauthorized reconciler.go:70] doReconcile GetNodeRunningPods: invalid character 'U' looking for beginning of value ``` -面对这种情况，我们建议[启用 Kubelet authentication webhook](../administration/going-production.md#authentication-webhook)。 +面对这种情况，我们建议[启用 Kubelet 认证鉴权](../administration/going-production.md#kubelet-authn-authz)。 ## Mount Pod 异常 {#mount-pod-error} From 344a1db8b548cd932f0673f050fc365a49043293 Mon Sep 17 00:00:00 2001 From: timfeirg Date: Wed, 19 Jul 2023 16:43:38 +0800 Subject: [PATCH 08/10] translation --- docs/en/administration/going-production.md | 27 +++++++++---------- .../administration/troubleshooting-cases.md | 2 +- docs/en/guide/pv.md | 19 ++++++------- docs/zh_cn/administration/going-production.md | 4 +-- .../administration/troubleshooting-cases.md | 2 +- 5 files changed, 24 insertions(+), 30 deletions(-) diff --git a/docs/en/administration/going-production.md b/docs/en/administration/going-production.md index 3b46c14779..64de367102 100644 --- a/docs/en/administration/going-production.md +++ b/docs/en/administration/going-production.md @@ -211,30 +211,27 @@ spec: ... ``` -## Enable kubelet authentication webhook {#authentication-webhook} +## Enable kubelet authentication {#kubelet-authn-authz} -If authentication webhook isn't enabled, CSI Node will run into error when listing pods (this is however, a issue fixed in newer versions, continue reading for more): +Kubelet comes with [different authentication modes](https://kubernetes.io/zh-cn/docs/reference/access-authn-authz/kubelet-authn-authz/), and default `AlwaysAllow` mode effectively disables authentication. But if kubelet uses other authentication modes, CSI Node will run into error when listing pods (this is however, a issue fixed in newer versions, continue reading for more): ``` kubelet_client.go:99] GetNodeRunningPods err: Unauthorized reconciler.go:70] doReconcile GetNodeRunningPods: invalid character 'U' looking for beginning of value ``` -When this happens, we recommend that you enable authentication webhook and restart kubelet: +This can be resolved using one of below methods: -```yaml {5,8} title="/var/lib/kubelet/config.yaml" -apiVersion: kubelet.config.k8s.io/v1beta1 -authentication: - webhook: - cacheTTL: 0s - enabled: true - ... -authorization: - mode: Webhook - ... -``` +1. [Enable X509 client certificate authentication for kubelet](https://kubernetes.io/docs/reference/access-authn-authz/kubelet-authn-authz/#kubelet-authentication), and configure these credentials into CSI Node so that it gains access to kubelet. To do this, pass these certificates to the CSI Node startup command: + + ```shell + # Replace KUBELET_CLIENT_CERT> and to actual certificate path + kubectl -n kube-system set env daemonset/juicefs-csi-node -c juicefs-plugin KUBELET_CLIENT_CERT= KUBELET_CLIENT_KEY= + ``` + +1. Delegate kubelet authentication to APIServer, refer to [Kubernetes documentation](https://kubernetes.io/docs/reference/access-authn-authz/kubelet-authn-authz/#kubelet-authorization) for more. -From v0.21.0, even if kubelet authentication webhook wasn't enabled, CSI Node will not run into errors. Instead it'll simply bypass kubelet, and obtain information directly from APIServer (like `ListPod`). Doing this adds a minor extra overhead to APIServer, thus authentication webhook is still recommended in production environments. +From v0.21.0, any of the above measures can fix the CSI Node error, because CSI Node will connect to APIServer and watch for changes. However, this watch process initiates with a `ListPod` request (with `labelSelector`), this adds a minor extra overhead to APIServer, thus authentication webhook is still recommended in production environments. Notice that CSI Driver must be configured `podInfoOnMount: true` for the above behavior to take effect. This problem doesn't exist however with Helm installations, because `podInfoOnMount` is hard-coded into template files and automatically applied between upgrades. So with kubectl installations, ensure these settings are put into `k8s.yaml`: diff --git a/docs/en/administration/troubleshooting-cases.md b/docs/en/administration/troubleshooting-cases.md index 0c749d8611..1dbac7ca7d 100644 --- a/docs/en/administration/troubleshooting-cases.md +++ b/docs/en/administration/troubleshooting-cases.md @@ -38,7 +38,7 @@ kubelet_client.go:99] GetNodeRunningPods err: Unauthorized reconciler.go:70] doReconcile GetNodeRunningPods: invalid character 'U' looking for beginning of value ``` -In such case, we recommend [enabling kubelet authentication webhook](../administration/going-production.md#authentication-webhook). +Read our docs on [enabling kubelet authentication](../administration/going-production.md#kubelet-authn-authz) to fix this. ## Mount Pod failure {#mount-pod-error} diff --git a/docs/en/guide/pv.md b/docs/en/guide/pv.md index 1060c785db..f431efb450 100644 --- a/docs/en/guide/pv.md +++ b/docs/en/guide/pv.md @@ -713,16 +713,18 @@ You can also use tools provided by a community developer to automatically add `m ### PV storage capacity {#storage-capacity} -For now, JuiceFS CSI Driver doesn't support setting storage capacity for static PersistentVolume. the storage specified under static PersistentVolume and its PersistentVolumeClaim is simply ignored, just use a reasonable size as placeholder (e.g. `100Gi`). +From v0.19.3, JuiceFS CSI Driver supports setting storage capacity under dynamic provisioning (and dynamic provisioning only, static provisioning isn't supported). + +In static provisioning, the storage specified in PVC/PV is simply ignored, fill in any a reasonably large size for future-proofing. ```yaml storageClassName: "" resources: requests: - storage: 100Gi + storage: 10Ti ``` -However, setting storage capacity works on dynamic PersistentVolumeClaim referring StorageClass: +Under dynamic provisioning, you can specify storage capacity in PVC definition, and it'll be translated into a `juicefs quota` command, which will be executed within CSI Controller, to properly apply the specified capacity quota upon the corresponding subdir. To learn more about `juicefs quota`, check [Community Edition docs](https://juicefs.com/docs/community/command_reference/#quota) and Cloud Service docs (work in progress). ```yaml ... @@ -732,12 +734,7 @@ resources: storage: 100Gi ``` -:::note -The storage capacity only takes effects on the subpath used by this PersistentVolumeClaim, -it does not affect the quota of the whole JuiceFS volume -::: - -We can check the storage capacity by executing `df` command in the Pod which uses this PVC: +After PV is created and mounted, verify by executing `df -h` command within the application pod: ```bash $ df -h @@ -760,8 +757,8 @@ parameters: csi.storage.k8s.io/node-publish-secret-namespace: default csi.storage.k8s.io/provisioner-secret-name: juicefs-secret csi.storage.k8s.io/provisioner-secret-namespace: default - csi.storage.k8s.io/controller-expand-secret-name: juicefs-secret # same as provisioner-secret-name - csi.storage.k8s.io/controller-expand-secret-namespace: default # same as provisioner-secret-namespace + csi.storage.k8s.io/controller-expand-secret-name: juicefs-secret # same as provisioner-secret-name + csi.storage.k8s.io/controller-expand-secret-namespace: default # same as provisioner-secret-namespace allowVolumeExpansion: true # indicates support for expansion ``` diff --git a/docs/zh_cn/administration/going-production.md b/docs/zh_cn/administration/going-production.md index cbba890ddf..5fb0ecd836 100644 --- a/docs/zh_cn/administration/going-production.md +++ b/docs/zh_cn/administration/going-production.md @@ -213,7 +213,7 @@ spec: ## 启用 Kubelet 认证鉴权 {#kubelet-authn-authz} -[Kubelet 的认证鉴权](https://kubernetes.io/docs/reference/access-authn-authz/kubelet-authn-authz)分为很多种，默认允许所有请求。但若 kubelet 关闭了匿名访问，会导致 CSI Node 获取 Pod 列表时报错（该报错本身已经修复，见后续描述）： +[Kubelet 的认证鉴权](https://kubernetes.io/zh-cn/docs/reference/access-authn-authz/kubelet-authn-authz/)分为很多种，默认的 `AlwaysAllow` 模式允许所有请求。但若 kubelet 关闭了匿名访问，会导致 CSI Node 获取 Pod 列表时报错（该报错本身已经修复，见后续描述）： ``` kubelet_client.go:99] GetNodeRunningPods err: Unauthorized @@ -229,7 +229,7 @@ reconciler.go:70] doReconcile GetNodeRunningPods: invalid character 'U' looking kubectl -n kube-system set env daemonset/juicefs-csi-node -c juicefs-plugin KUBELET_CLIENT_CERT= KUBELET_CLIENT_KEY= ``` -2. 将 Kubelet 鉴权委派给 APIServer，具体请参考[官方文档](https://kubernetes.io/docs/reference/access-authn-authz/kubelet-authn-authz/#kubelet-authorization)。 +1. 将 Kubelet 鉴权委派给 APIServer，具体请参考[官方文档](https://kubernetes.io/docs/reference/access-authn-authz/kubelet-authn-authz/#kubelet-authorization)。 在 v0.21.0 及其后版本，即使采取以上任意一种措施，CSI Node 也不再会出现异常，不会直接连接 Kubelet，而是 watch APIServer 去获取信息，由于 watch list 机制在启动时会对 APIServer 进行一次 `ListPod` 请求（有 `labelSelector`），在集群负载较大的情况下，会对 APIServer 造成额外的压力。因此在生产集群，我们仍推荐配置 CSI Node 对 Kubelet 的认证。 diff --git a/docs/zh_cn/administration/troubleshooting-cases.md b/docs/zh_cn/administration/troubleshooting-cases.md index 326fa1fa17..68fdac4ecb 100644 --- a/docs/zh_cn/administration/troubleshooting-cases.md +++ b/docs/zh_cn/administration/troubleshooting-cases.md @@ -36,7 +36,7 @@ kubelet_client.go:99] GetNodeRunningPods err: Unauthorized reconciler.go:70] doReconcile GetNodeRunningPods: invalid character 'U' looking for beginning of value ``` -面对这种情况，我们建议[启用 Kubelet 认证鉴权](../administration/going-production.md#kubelet-authn-authz)。 +面对这种情况，阅读[启用 Kubelet 认证鉴权](../administration/going-production.md#kubelet-authn-authz)了解如何修复该问题。 ## Mount Pod 异常 {#mount-pod-error} From ed5d01d356704eb2b0f329377f85bbd0062d5116 Mon Sep 17 00:00:00 2001 From: timfeirg Date: Wed, 19 Jul 2023 16:55:39 +0800 Subject: [PATCH 09/10] lint --- docs/en/administration/going-production.md | 2 +- docs/zh_cn/administration/going-production.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/administration/going-production.md b/docs/en/administration/going-production.md index 64de367102..15212147f5 100644 --- a/docs/en/administration/going-production.md +++ b/docs/en/administration/going-production.md @@ -213,7 +213,7 @@ spec: ## Enable kubelet authentication {#kubelet-authn-authz} -Kubelet comes with [different authentication modes](https://kubernetes.io/zh-cn/docs/reference/access-authn-authz/kubelet-authn-authz/), and default `AlwaysAllow` mode effectively disables authentication. But if kubelet uses other authentication modes, CSI Node will run into error when listing pods (this is however, a issue fixed in newer versions, continue reading for more): +Kubelet comes with [different authentication modes](https://kubernetes.io/zh-cn/docs/reference/access-authn-authz/kubelet-authn-authz), and default `AlwaysAllow` mode effectively disables authentication. But if kubelet uses other authentication modes, CSI Node will run into error when listing pods (this is however, a issue fixed in newer versions, continue reading for more): ``` kubelet_client.go:99] GetNodeRunningPods err: Unauthorized diff --git a/docs/zh_cn/administration/going-production.md b/docs/zh_cn/administration/going-production.md index 5fb0ecd836..c53f601d1b 100644 --- a/docs/zh_cn/administration/going-production.md +++ b/docs/zh_cn/administration/going-production.md @@ -213,7 +213,7 @@ spec: ## 启用 Kubelet 认证鉴权 {#kubelet-authn-authz} -[Kubelet 的认证鉴权](https://kubernetes.io/zh-cn/docs/reference/access-authn-authz/kubelet-authn-authz/)分为很多种，默认的 `AlwaysAllow` 模式允许所有请求。但若 kubelet 关闭了匿名访问，会导致 CSI Node 获取 Pod 列表时报错（该报错本身已经修复，见后续描述）： +[Kubelet 的认证鉴权](https://kubernetes.io/zh-cn/docs/reference/access-authn-authz/kubelet-authn-authz)分为很多种，默认的 `AlwaysAllow` 模式允许所有请求。但若 kubelet 关闭了匿名访问，会导致 CSI Node 获取 Pod 列表时报错（该报错本身已经修复，见后续描述）： ``` kubelet_client.go:99] GetNodeRunningPods err: Unauthorized From 089c0d9a729f703531be4b207ab95d37bdaafb0c Mon Sep 17 00:00:00 2001 From: timfeirg Date: Wed, 19 Jul 2023 17:17:07 +0800 Subject: [PATCH 10/10] minor fix --- docs/en/administration/going-production.md | 2 +- docs/zh_cn/administration/going-production.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/administration/going-production.md b/docs/en/administration/going-production.md index 15212147f5..5254915a07 100644 --- a/docs/en/administration/going-production.md +++ b/docs/en/administration/going-production.md @@ -231,7 +231,7 @@ This can be resolved using one of below methods: 1. Delegate kubelet authentication to APIServer, refer to [Kubernetes documentation](https://kubernetes.io/docs/reference/access-authn-authz/kubelet-authn-authz/#kubelet-authorization) for more. -From v0.21.0, any of the above measures can fix the CSI Node error, because CSI Node will connect to APIServer and watch for changes. However, this watch process initiates with a `ListPod` request (with `labelSelector`), this adds a minor extra overhead to APIServer, thus authentication webhook is still recommended in production environments. +From v0.21.0, even without carrying out any of the above measures, CSI Node will continue to work normally, in the face of authentication error, CSI Node will bypass kubelet and connect to APIServer and watch for changes. However, this watch process initiates with a `ListPod` request (with `labelSelector` to minimize performance impact), this adds a minor extra overhead to APIServer, thus authentication webhook is still recommended in production environments. Notice that CSI Driver must be configured `podInfoOnMount: true` for the above behavior to take effect. This problem doesn't exist however with Helm installations, because `podInfoOnMount` is hard-coded into template files and automatically applied between upgrades. So with kubectl installations, ensure these settings are put into `k8s.yaml`: diff --git a/docs/zh_cn/administration/going-production.md b/docs/zh_cn/administration/going-production.md index c53f601d1b..1ad57aee70 100644 --- a/docs/zh_cn/administration/going-production.md +++ b/docs/zh_cn/administration/going-production.md @@ -231,7 +231,7 @@ reconciler.go:70] doReconcile GetNodeRunningPods: invalid character 'U' looking 1. 将 Kubelet 鉴权委派给 APIServer，具体请参考[官方文档](https://kubernetes.io/docs/reference/access-authn-authz/kubelet-authn-authz/#kubelet-authorization)。 -在 v0.21.0 及其后版本，即使采取以上任意一种措施，CSI Node 也不再会出现异常，不会直接连接 Kubelet，而是 watch APIServer 去获取信息，由于 watch list 机制在启动时会对 APIServer 进行一次 `ListPod` 请求（有 `labelSelector`），在集群负载较大的情况下，会对 APIServer 造成额外的压力。因此在生产集群，我们仍推荐配置 CSI Node 对 Kubelet 的认证。 +在 v0.21.0 及其后版本，即使未采取以上措施，CSI Node 也能继续正常工作：如果遭遇鉴权错误，CSI Node 就不再直连 Kubelet，而是 watch APIServer 去获取信息，由于 watch list 机制在启动时会对 APIServer 进行一次 `ListPod` 请求（携带了 `labelSelector`，最大程度减少开销），在集群负载较大的情况下，会对 APIServer 造成额外的压力。因此在生产集群，我们仍推荐配置 CSI Node 对 Kubelet 的认证。 需要注意，就算使用了 v0.21.0 及之后的版本，CSI 驱动需要配置 `podInfoOnMount: true`，上边提到的避免报错的特性才会真正生效。如果你采用 [Helm 安装方式](../getting_started.md#helm)，则 `podInfoOnMount` 默认开启无需配置，该特性会随着升级自动启用。而如果你使用 kubectl 直接安装，你需要为 `k8s.yaml` 添加如下配置：