`manager.go` "x509: certificate signed by unknown authority, unable to fully scrape metrics from source"

[agilgur5]

I've exhausted all the options and tried A TON of different possible fixes (been at it for more than a full day) but none worked, so I am thinking this may be a bug here and writing here

What happened:

I deployed metrics-server onto a kops cluster on AWS using the Helm chart with no changes. metrics-server errors with the full logs listed below. Summarized by the title and a short snippet below:

1 manager.go:111] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:ip-172-20-38-5.us-west-1.compute.internal: unable to fetch metrics from Kubelet ip-172-20-38-5.us-west-1.compute.internal (ip-172-20-38-5.us-west-1.compute.internal): Get https://ip-172-20-38-5.us-west-1.compute.internal:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority, unable to fully scrape metrics from source .....

Per the FAQ, I tried to set --kubelet-certificate-authority to the kops CA (hostPath /srv/kubernetes/ca.crt), but still got this error. I tried setting --client-ca-file to that as well and still got this error.

I tried changing --requestheader-client-ca-file (which seems to only be for kubeadm and unrelated) to this CA cert as well as to the proxy-client CA but then got the authentication.go error that that was supposed to fix per the FAQ

The only thing that worked for me was setting --kubelet-insecure-tls which made the errors disappear and made kubectl top nodes start working. But this is, of course, insecure and a MITM attack vector, but I couldn't get any other flag to work 😕
Does the --kubelet-certificate-authority flag not work? I looked at various issues and none of them really resolve it, they all say to set --kubelet-insecure-tls. Many are DNS issues which I'm not having.

What you expected to happen:

Helm installation to either work without configuration (validating against the cluster CA) or to work once --kubelet-certificate-authority is specified.

Anything else we need to know?:

I have set --authentication-token-webhook=true and --authorization-mode=Webhook as you can see below via kops config. (without these, and without performing a rolling-update after setting them, I get 401 Unauthorized instead, which is expected. Once they are set, this error happens instead).

My error and config look very similar to #362 (comment) who also asked if the --kubelet-certificate-authority flag was ignored.

I'm getting this error in multiple clusters, on multiple k8s and kops versions, all of them have the same error.

Environment:

• Kubernetes distribution (GKE, EKS, Kubeadm, the hard way, etc.): kops on AWS. v1.15 and v1.17, errors in both
• Container Network Setup (flannel, calico, etc.): Cilium in some, Calico in others, errors in both
• Kubernetes version (use kubectl version): v1.15.11 in some, v1.18.3 in others, errors in both
• Metrics Server manifest: helm upgrade --install metrics-server with the defaults. Some changes tried as listed above
• Kubelet config:

admin@ip-172-20-58-86:~$ ps -aux | grep kubelet
root      1975  3.0  1.3 649748 111596 ?       Ssl  Jun10  41:06 /usr/local/bin/kubelet --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-root=/ --client-ca-file=/srv/kubernetes/ca.crt --cloud-provider=aws --cluster-dns=100.64.0.10 --cluster-domain=cluster.local --enable-debugging-handlers=true --eviction-hard=memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%,imagefs.available<10%,imagefs.inodesFree<5% --feature-gates=ExperimentalCriticalPodAnnotation=true --hostname-override=ip-172-20-58-86.us-west-1.compute.internal --kubeconfig=/var/lib/kubelet/kubeconfig --network-plugin=cni --node-labels=kops.k8s.io/instancegroup=nodes,kubernetes.io/role=node,node-role.kubernetes.io/node= --non-masquerade-cidr=100.64.0.0/10 --pod-infra-container-image=k8s.gcr.io/pause-amd64:3.0 --pod-manifest-path=/etc/kubernetes/manifests --register-schedulable=true --v=2 --volume-plugin-dir=/usr/libexec/kubernetes/kubelet-plugins/volume/exec/ --cni-bin-dir=/opt/cni/bin/ --cni-conf-dir=/etc/cni/net.d/

• Metrics Server logs:

Open spoiler for logs:

Most relevant snippet:

E0610 20:12:15.220770       1 manager.go:111] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:ip-172-20-38-5.us-west-1.compute.internal: unable to fetch metrics from Kubelet ip-172-20-38-5.us-west-1.compute.internal (ip-172-20-38-5.us-west-1.compute.internal): Get https://ip-172-20-38-5.us-west-1.compute.internal:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:ip-172-20-86-41.us-west-1.compute.internal: unable to fetch metrics from Kubelet ip-172-20-86-41.us-west-1.compute.internal (ip-172-20-86-41.us-west-1.compute.internal): Get https://ip-172-20-86-41.us-west-1.compute.internal:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:ip-172-20-36-220.us-west-1.compute.internal: unable to fetch metrics from Kubelet ip-172-20-36-220.us-west-1.compute.internal (ip-172-20-36-220.us-west-1.compute.internal): Get https://ip-172-20-36-220.us-west-1.compute.internal:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:ip-172-20-82-244.us-west-1.compute.internal: unable to fetch metrics from Kubelet ip-172-20-82-244.us-west-1.compute.internal (ip-172-20-82-244.us-west-1.compute.internal): Get https://ip-172-20-82-244.us-west-1.compute.internal:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:ip-172-20-70-141.us-west-1.compute.internal: unable to fetch metrics from Kubelet ip-172-20-70-141.us-west-1.compute.internal (ip-172-20-70-141.us-west-1.compute.internal): Get https://ip-172-20-70-141.us-west-1.compute.internal:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:ip-172-20-45-12.us-west-1.compute.internal: unable to fetch metrics from Kubelet ip-172-20-45-12.us-west-1.compute.internal (ip-172-20-45-12.us-west-1.compute.internal): Get https://ip-172-20-45-12.us-west-1.compute.internal:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:ip-172-20-52-222.us-west-1.compute.internal: unable to fetch metrics from Kubelet ip-172-20-52-222.us-west-1.compute.internal (ip-172-20-52-222.us-west-1.compute.internal): Get https://ip-172-20-52-222.us-west-1.compute.internal:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:ip-172-20-72-191.us-west-1.compute.internal: unable to fetch metrics from Kubelet ip-172-20-72-191.us-west-1.compute.internal (ip-172-20-72-191.us-west-1.compute.internal): Get https://ip-172-20-72-191.us-west-1.compute.internal:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority]

Full logs:

$ kubectl -n kube-system logs metrics-server-8555869558-kvqdq 
I0610 20:11:15.159707       1 secure_serving.go:116] Serving securely on [::]:8443
E0610 20:11:44.938984       1 reststorage.go:135] unable to fetch node metrics for node "ip-172-20-86-41.us-west-1.compute.internal": no metrics known for node
E0610 20:11:44.939013       1 reststorage.go:135] unable to fetch node metrics for node "ip-172-20-45-12.us-west-1.compute.internal": no metrics known for node
E0610 20:11:44.939020       1 reststorage.go:135] unable to fetch node metrics for node "ip-172-20-70-141.us-west-1.compute.internal": no metrics known for node
E0610 20:11:44.939026       1 reststorage.go:135] unable to fetch node metrics for node "ip-172-20-38-5.us-west-1.compute.internal": no metrics known for node
E0610 20:11:44.939039       1 reststorage.go:135] unable to fetch node metrics for node "ip-172-20-72-191.us-west-1.compute.internal": no metrics known for node
E0610 20:11:44.939045       1 reststorage.go:135] unable to fetch node metrics for node "ip-172-20-36-220.us-west-1.compute.internal": no metrics known for node
E0610 20:11:44.939050       1 reststorage.go:135] unable to fetch node metrics for node "ip-172-20-52-222.us-west-1.compute.internal": no metrics known for node
E0610 20:11:44.939056       1 reststorage.go:135] unable to fetch node metrics for node "ip-172-20-82-244.us-west-1.compute.internal": no metrics known for node
E0610 20:11:44.945326       1 reststorage.go:160] unable to fetch pod metrics for pod logging/fluentbit-cloudwatch-fluent-bit-5pfbd: no metrics known for pod
E0610 20:11:44.945347       1 reststorage.go:160] unable to fetch pod metrics for pod istio-system/istiod-c7757dcf7-vz8cn: no metrics known for pod
E0610 20:11:44.945353       1 reststorage.go:160] unable to fetch pod metrics for pod monitoring/prometheus-operator-prometheus-node-exporter-kwpbp: no metrics known for pod
E0610 20:11:44.945358       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/coredns-56676bcc8f-vpqk8: no metrics known for pod
E0610 20:11:44.945363       1 reststorage.go:160] unable to fetch pod metrics for pod monitoring/alertmanager-prometheus-operator-alertmanager-0: no metrics known for pod
E0610 20:11:44.945397       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/cilium-6j4gr: no metrics known for pod
E0610 20:11:44.945417       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-controller-manager-ip-172-20-38-5.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945422       1 reststorage.go:160] unable to fetch pod metrics for pod monitoring/prometheus-operator-operator-5dd8f8f568-24x48: no metrics known for pod
E0610 20:11:44.945428       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/cilium-9glt8: no metrics known for pod
E0610 20:11:44.945433       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/metrics-server-8555869558-kvqdq: no metrics known for pod
E0610 20:11:44.945438       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube2iam-9zwtv: no metrics known for pod
E0610 20:11:44.945444       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/etcd-manager-events-ip-172-20-70-141.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945449       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kops-controller-2c7bp: no metrics known for pod
E0610 20:11:44.945454       1 reststorage.go:160] unable to fetch pod metrics for pod monitoring/prometheus-operator-prometheus-node-exporter-5pjrf: no metrics known for pod
E0610 20:11:44.945459       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-apiserver-ip-172-20-38-5.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945464       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-scheduler-ip-172-20-45-12.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945470       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-proxy-ip-172-20-36-220.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945475       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-scheduler-ip-172-20-70-141.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945480       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/cilium-74z29: no metrics known for pod
E0610 20:11:44.945485       1 reststorage.go:160] unable to fetch pod metrics for pod logging/fluentbit-cloudwatch-fluent-bit-gfgzt: no metrics known for pod
E0610 20:11:44.945491       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kops-controller-v2spk: no metrics known for pod
E0610 20:11:44.945496       1 reststorage.go:160] unable to fetch pod metrics for pod monitoring/prometheus-operator-prometheus-node-exporter-fjzm2: no metrics known for pod
E0610 20:11:44.945504       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/dns-controller-6598994b65-8fsts: no metrics known for pod
E0610 20:11:44.945510       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/coredns-56676bcc8f-vxmzh: no metrics known for pod
E0610 20:11:44.945515       1 reststorage.go:160] unable to fetch pod metrics for pod monitoring/prometheus-prometheus-operator-prometheus-0: no metrics known for pod
E0610 20:11:44.945520       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-controller-manager-ip-172-20-45-12.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945526       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-proxy-ip-172-20-86-41.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945559       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/cilium-27wk8: no metrics known for pod
E0610 20:11:44.945566       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-apiserver-ip-172-20-70-141.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945571       1 reststorage.go:160] unable to fetch pod metrics for pod istio-operator/istio-operator-54fc967ff8-f9jm5: no metrics known for pod
E0610 20:11:44.945576       1 reststorage.go:160] unable to fetch pod metrics for pod istio-system/istiocoredns-774588b997-thb66: no metrics known for pod
E0610 20:11:44.945581       1 reststorage.go:160] unable to fetch pod metrics for pod external-secrets/external-secrets-kubernetes-external-secrets-5459f65756-7ncdf: no metrics known for pod
E0610 20:11:44.945587       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/cilium-n58ld: no metrics known for pod
E0610 20:11:44.945592       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-proxy-ip-172-20-82-244.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945597       1 reststorage.go:160] unable to fetch pod metrics for pod kubernetes-dashboard/kubernetes-dashboard-74b4487bfc-pxfkw: no metrics known for pod
E0610 20:11:44.945602       1 reststorage.go:160] unable to fetch pod metrics for pod logging/fluentbit-cloudwatch-fluent-bit-jblrb: no metrics known for pod
E0610 20:11:44.945608       1 reststorage.go:160] unable to fetch pod metrics for pod kubernetes-dashboard/dashboard-metrics-scraper-66b49655d4-qgdj8: no metrics known for pod
E0610 20:11:44.945613       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-proxy-ip-172-20-72-191.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945618       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/etcd-manager-events-ip-172-20-45-12.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945623       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube2iam-bh9hb: no metrics known for pod
E0610 20:11:44.945629       1 reststorage.go:160] unable to fetch pod metrics for pod monitoring/prometheus-operator-prometheus-node-exporter-6nqhh: no metrics known for pod
E0610 20:11:44.945651       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-proxy-ip-172-20-45-12.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945658       1 reststorage.go:160] unable to fetch pod metrics for pod logging/fluentbit-cloudwatch-fluent-bit-bkssn: no metrics known for pod
E0610 20:11:44.945663       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kops-controller-khqfh: no metrics known for pod
E0610 20:11:44.945668       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-scheduler-ip-172-20-38-5.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945674       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/cilium-operator-987865868-vjhrv: no metrics known for pod
E0610 20:11:44.945679       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/etcd-manager-main-ip-172-20-38-5.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945684       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-proxy-ip-172-20-52-222.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945689       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/etcd-manager-main-ip-172-20-70-141.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945720       1 reststorage.go:160] unable to fetch pod metrics for pod monitoring/prometheus-operator-prometheus-node-exporter-jw7nb: no metrics known for pod
E0610 20:11:44.945728       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/etcd-manager-events-ip-172-20-38-5.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945733       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-controller-manager-ip-172-20-70-141.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945739       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/cilium-vmjfb: no metrics known for pod
E0610 20:11:44.945744       1 reststorage.go:160] unable to fetch pod metrics for pod istio-system/grafana-5f6f8cbf75-h5kl5: no metrics known for pod
E0610 20:11:44.945749       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-apiserver-ip-172-20-45-12.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945755       1 reststorage.go:160] unable to fetch pod metrics for pod monitoring/prometheus-operator-kube-state-metrics-5fdcd78bc-8ldlf: no metrics known for pod
E0610 20:11:44.945760       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/cilium-dn7qb: no metrics known for pod
E0610 20:11:44.945765       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/guard-76b48854ff-59df8: no metrics known for pod
E0610 20:11:44.945770       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/coredns-autoscaler-cd5745894-sz59s: no metrics known for pod
E0610 20:11:44.945775       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube2iam-ndckw: no metrics known for pod
E0610 20:11:44.945780       1 reststorage.go:160] unable to fetch pod metrics for pod logging/fluentbit-cloudwatch-fluent-bit-wmx2q: no metrics known for pod
E0610 20:11:44.945786       1 reststorage.go:160] unable to fetch pod metrics for pod external-secrets/external-secrets-kubernetes-external-secrets-5459f65756-rfh8q: no metrics known for pod
E0610 20:11:44.945790       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/cilium-dnn2k: no metrics known for pod
E0610 20:11:44.945796       1 reststorage.go:160] unable to fetch pod metrics for pod monitoring/prometheus-operator-prometheus-node-exporter-nlmm4: no metrics known for pod
E0610 20:11:44.945801       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-proxy-ip-172-20-70-141.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945806       1 reststorage.go:160] unable to fetch pod metrics for pod istio-system/istio-ingressgateway-6cf56d9598-4xw9n: no metrics known for pod
E0610 20:11:44.945811       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube2iam-knwr6: no metrics known for pod
E0610 20:11:44.945816       1 reststorage.go:160] unable to fetch pod metrics for pod monitoring/prometheus-operator-prometheus-node-exporter-gmwvg: no metrics known for pod
E0610 20:11:44.945821       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-proxy-ip-172-20-38-5.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945826       1 reststorage.go:160] unable to fetch pod metrics for pod monitoring/prometheus-operator-prometheus-node-exporter-5mjnv: no metrics known for pod
E0610 20:11:44.945832       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/etcd-manager-main-ip-172-20-45-12.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945837       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube2iam-sprw4: no metrics known for pod
E0610 20:11:44.945842       1 reststorage.go:160] unable to fetch pod metrics for pod istio-system/prometheus-dfd976959-rkjw9: no metrics known for pod
E0610 20:11:46.238622       1 reststorage.go:160] unable to fetch pod metrics for pod istio-system/istio-ingressgateway-6cf56d9598-4xw9n: no metrics known for pod
E0610 20:11:46.250946       1 reststorage.go:160] unable to fetch pod metrics for pod istio-system/istiod-c7757dcf7-vz8cn: no metrics known for pod
E0610 20:12:01.255135       1 reststorage.go:160] unable to fetch pod metrics for pod istio-system/istio-ingressgateway-6cf56d9598-4xw9n: no metrics known for pod
E0610 20:12:01.261228       1 reststorage.go:160] unable to fetch pod metrics for pod istio-system/istiod-c7757dcf7-vz8cn: no metrics known for pod
E0610 20:12:15.220770       1 manager.go:111] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:ip-172-20-38-5.us-west-1.compute.internal: unable to fetch metrics from Kubelet ip-172-20-38-5.us-west-1.compute.internal (ip-172-20-38-5.us-west-1.compute.internal): Get https://ip-172-20-38-5.us-west-1.compute.internal:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:ip-172-20-86-41.us-west-1.compute.internal: unable to fetch metrics from Kubelet ip-172-20-86-41.us-west-1.compute.internal (ip-172-20-86-41.us-west-1.compute.internal): Get https://ip-172-20-86-41.us-west-1.compute.internal:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:ip-172-20-36-220.us-west-1.compute.internal: unable to fetch metrics from Kubelet ip-172-20-36-220.us-west-1.compute.internal (ip-172-20-36-220.us-west-1.compute.internal): Get https://ip-172-20-36-220.us-west-1.compute.internal:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:ip-172-20-82-244.us-west-1.compute.internal: unable to fetch metrics from Kubelet ip-172-20-82-244.us-west-1.compute.internal (ip-172-20-82-244.us-west-1.compute.internal): Get https://ip-172-20-82-244.us-west-1.compute.internal:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:ip-172-20-70-141.us-west-1.compute.internal: unable to fetch metrics from Kubelet ip-172-20-70-141.us-west-1.compute.internal (ip-172-20-70-141.us-west-1.compute.internal): Get https://ip-172-20-70-141.us-west-1.compute.internal:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:ip-172-20-45-12.us-west-1.compute.internal: unable to fetch metrics from Kubelet ip-172-20-45-12.us-west-1.compute.internal (ip-172-20-45-12.us-west-1.compute.internal): Get https://ip-172-20-45-12.us-west-1.compute.internal:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:ip-172-20-52-222.us-west-1.compute.internal: unable to fetch metrics from Kubelet ip-172-20-52-222.us-west-1.compute.internal (ip-172-20-52-222.us-west-1.compute.internal): Get https://ip-172-20-52-222.us-west-1.compute.internal:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:ip-172-20-72-191.us-west-1.compute.internal: unable to fetch metrics from Kubelet ip-172-20-72-191.us-west-1.compute.internal (ip-172-20-72-191.us-west-1.compute.internal): Get https://ip-172-20-72-191.us-west-1.compute.internal:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority]

/king bug

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[agilgur5]

/remove-lifecycle stale

This issue has never been responded to...

There was a response in kubernetes/kops#6879 (comment) , but that revealed that there's a missing capability required for this to work 😕

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[serathius]

I think problem was resolved by kubernetes/kops#10022 on kops side and #589 on Metrics Server side.

Feel free to reopen if problem still persists.