Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

microk8s.kubectl should be used for internal health checking as using the system kubectl may fail #3175

Closed
nobuto-m opened this issue May 30, 2022 · 7 comments · Fixed by #3176
Closed

microk8s.kubectl should be used for internal health checking as using the system kubectl may fail #3175

nobuto-m opened this issue May 30, 2022 · 7 comments · Fixed by #3176

Comments

@nobuto-m
Copy link
Contributor

Summary

microk8s enable <addon> may fail especially when microk8s' Kubernetes version and the system's kubectl version are different.

What Should Happen Instead?

microk8s enable <addon> works without an error.

Reproduction Steps

  1. Install microk8s explicitly with 1.21 (for Kubeflow on microk8s)

     sudo snap install microk8s --classic --channel=1.21

  2. Install kubectl with the latest version (for other Kubernetes)

    sudo snap install kubectl --classic

  3. Enable addons

    microk8s enable dns storage ingress metallb:10.64.140.43-10.64.140.49

$ microk8s enable dns storage ingress metallb:10.64.140.43-10.64.140.49
Traceback (most recent call last):
  File "/snap/microk8s/3202/scripts/wrappers/enable.py", line 43, in <module>
    enable(prog_name="microk8s enable")
  File "/snap/microk8s/3202/usr/lib/python3/dist-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/snap/microk8s/3202/usr/lib/python3/dist-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/snap/microk8s/3202/usr/lib/python3/dist-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/snap/microk8s/3202/usr/lib/python3/dist-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/snap/microk8s/3202/scripts/wrappers/enable.py", line 36, in enable
    enabled_addons, _ = get_status(get_available_addons(get_current_arch()), True)
  File "/snap/microk8s/3202/scripts/wrappers/status.py", line 157, in get_status
    kube_output = kubectl_get("all")
  File "/snap/microk8s/3202/scripts/wrappers/common/utils.py", line 169, in kubectl_get
    return run("kubectl", kubeconfig, "get", cmd, "--all-namespaces", die=False)
  File "/snap/microk8s/3202/scripts/wrappers/common/utils.py", line 39, in run
    result.check_returncode()
  File "/snap/microk8s/3202/usr/lib/python3.6/subprocess.py", line 389, in check_returncode
    self.stderr)
subprocess.CalledProcessError: Command '('kubectl', '--kubeconfig=/var/snap/microk8s/3202/credentials/client.config', 'get', 'all', '--all-namespaces')' returned non-zero exit status 1.

$ kubectl --kubeconfig=/var/snap/microk8s/3202/credentials/client.config get all --all-namespaces
NAMESPACE     NAME                                          READY   STATUS    RESTARTS   AGE
kube-system   pod/calico-node-t7wtm                         1/1     Running   0          17m
kube-system   pod/calico-kube-controllers-f7868dd95-9klz2   1/1     Running   0          17m

NAMESPACE   NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
default     service/kubernetes   ClusterIP   10.152.183.1   <none>        443/TCP   17m

NAMESPACE     NAME                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-system   daemonset.apps/calico-node   1         1         1       1            1           kubernetes.io/os=linux   17m

NAMESPACE     NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/calico-kube-controllers   1/1     1            1           17m

NAMESPACE     NAME                                                DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/calico-kube-controllers-f7868dd95   1         1         1       17m
Error from server (NotFound): Unable to list "autoscaling/v2, Resource=horizontalpodautoscalers": the server could not find the requested resource

The error must be from the difference between the client version and the server version.
Error from server (NotFound): Unable to list "autoscaling/v2, Resource=horizontalpodautoscalers": the server could not find the requested resource

Introspection Report

Can you suggest a fix?

Isn't this section supposed to use microk8s.kubectl instead of the system kubectl?

microk8s/scripts/wrappers/common/utils.py

Lines 189 to 199 in eb95dc1

def kubectl_get(cmd, namespace="--all-namespaces"):
if namespace == "--all-namespaces":
return run("kubectl", kubeconfig, "get", cmd, "--all-namespaces", die=False)
else:
return run("kubectl", kubeconfig, "get", cmd, "-n", namespace, die=False)
def kubectl_get_clusterroles():
return run(
"kubectl", kubeconfig, "get", "clusterroles", "--show-kind", "--no-headers", die=False
)

Are you interested in contributing with a fix?

Not immediately. Would be nice to be picked up by somebody else.
nobuto-m added a commit to nobuto-m/microk8s that referenced this issue May 31, 2022
@nobuto-m
Make sure to use microk8s.kubectl
cfa4e04 
If we call kubectl simply, the system installed kubectl has precedence.
That may lead to some weird issues like an error due to a discrepancy
between kubectl cli and kubernetes API versions.

Closes: canonical#3175
@nobuto-m nobuto-m mentioned this issue May 31, 2022
Merged
3 tasks
nobuto-m added a commit to nobuto-m/microk8s that referenced this issue May 31, 2022
@nobuto-m
Make sure to use microk8s.kubectl
5d4c567 
If we call kubectl simply, the system installed kubectl has precedence.
That may lead to some weird issues like an error due to a discrepancy
between kubectl cli and kubernetes API versions.

Closes: canonical#3175
@ktsakalozos
Copy link
Member

Thank you for reporting this @nobuto-m and for offering a patch. We will have news on this soon.

nobuto-m added a commit to nobuto-m/microk8s that referenced this issue Jun 3, 2022
@nobuto-m
Make sure to use microk8s-kubectl.wrapper
0c27125 
If we call kubectl simply, the system installed kubectl has precedence.
That may lead to some weird issues like an error due to a discrepancy
between kubectl cli and kubernetes API versions.

Closes: canonical#3175
neoaggelos pushed a commit that referenced this issue Jun 6, 2022
@nobuto-m
Make sure to use microk8s-kubectl.wrapper (#3176)
90c02bb 
If we call kubectl simply, the system installed kubectl has precedence.
That may lead to some weird issues like an error due to a discrepancy
between kubectl cli and kubernetes API versions.

Closes: #3175
neoaggelos pushed a commit that referenced this issue Jun 6, 2022
@nobuto-m @neoaggelos
Make sure to use microk8s-kubectl.wrapper (#3176)
fdd551c 
If we call kubectl simply, the system installed kubectl has precedence.
That may lead to some weird issues like an error due to a discrepancy
between kubectl cli and kubernetes API versions.

Closes: #3175
neoaggelos pushed a commit that referenced this issue Jun 6, 2022
@nobuto-m @neoaggelos
Make sure to use microk8s-kubectl.wrapper (#3176)
5a8ac2b 
If we call kubectl simply, the system installed kubectl has precedence.
That may lead to some weird issues like an error due to a discrepancy
between kubectl cli and kubernetes API versions.

Closes: #3175
neoaggelos pushed a commit that referenced this issue Jun 6, 2022
@nobuto-m @neoaggelos
Make sure to use microk8s-kubectl.wrapper (#3176)
97ae582 
If we call kubectl simply, the system installed kubectl has precedence.
That may lead to some weird issues like an error due to a discrepancy
between kubectl cli and kubernetes API versions.

Closes: #3175
neoaggelos pushed a commit that referenced this issue Jun 6, 2022
@nobuto-m @neoaggelos
Make sure to use microk8s-kubectl.wrapper (#3176)
4593a8f 
If we call kubectl simply, the system installed kubectl has precedence.
That may lead to some weird issues like an error due to a discrepancy
between kubectl cli and kubernetes API versions.

Closes: #3175
neoaggelos pushed a commit that referenced this issue Jun 6, 2022
@nobuto-m @neoaggelos
Make sure to use microk8s-kubectl.wrapper (#3176)
7ed7af4 
If we call kubectl simply, the system installed kubectl has precedence.
That may lead to some weird issues like an error due to a discrepancy
between kubectl cli and kubernetes API versions.

Closes: #3175
ktsakalozos pushed a commit that referenced this issue Jun 8, 2022
@neoaggelos @nobuto-m
Make sure to use microk8s-kubectl.wrapper (#3176) (#3209)
53ee677 
If we call kubectl simply, the system installed kubectl has precedence.
That may lead to some weird issues like an error due to a discrepancy
between kubectl cli and kubernetes API versions.

Closes: #3175

Co-authored-by: Nobuto Murata <[email protected]>
ktsakalozos pushed a commit that referenced this issue Jun 8, 2022
@neoaggelos @nobuto-m
Make sure to use microk8s-kubectl.wrapper (#3176) (#3207)
f7a5f7d 
If we call kubectl simply, the system installed kubectl has precedence.
That may lead to some weird issues like an error due to a discrepancy
between kubectl cli and kubernetes API versions.

Closes: #3175

Co-authored-by: Nobuto Murata <[email protected]>
ktsakalozos pushed a commit that referenced this issue Jun 8, 2022
@neoaggelos @nobuto-m
Make sure to use microk8s-kubectl.wrapper (#3176) (#3206)
aa44bac 
If we call kubectl simply, the system installed kubectl has precedence.
That may lead to some weird issues like an error due to a discrepancy
between kubectl cli and kubernetes API versions.

Closes: #3175

Co-authored-by: Nobuto Murata <[email protected]>
ktsakalozos pushed a commit that referenced this issue Jun 8, 2022
@neoaggelos @nobuto-m
Make sure to use microk8s-kubectl.wrapper (#3176) (#3208)
a794c53 
If we call kubectl simply, the system installed kubectl has precedence.
That may lead to some weird issues like an error due to a discrepancy
between kubectl cli and kubernetes API versions.

Closes: #3175

Co-authored-by: Nobuto Murata <[email protected]>
@ycliuhw
Copy link
Member

ycliuhw commented Jul 6, 2022 •
edited
Loading

Hi guys,
It seems I still got issues with the latest edge.

$ microk8s enable dns
Traceback (most recent call last):
  File "/snap/microk8s/3543/scripts/wrappers/enable.py", line 41, in <module>
    enable(prog_name="microk8s enable")
  File "/snap/microk8s/3543/usr/lib/python3/dist-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/snap/microk8s/3543/usr/lib/python3/dist-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/snap/microk8s/3543/usr/lib/python3/dist-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/snap/microk8s/3543/usr/lib/python3/dist-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/snap/microk8s/3543/scripts/wrappers/enable.py", line 37, in enable
    xable("enable", addons)
  File "/snap/microk8s/3543/scripts/wrappers/common/utils.py", line 438, in xable
    enabled_addons_info, disabled_addons_info = get_status(available_addons_info, True)
  File "/snap/microk8s/3543/scripts/wrappers/common/utils.py", line 484, in get_status
    kube_output = kubectl_get("all,ingress")
  File "/snap/microk8s/3543/scripts/wrappers/common/utils.py", line 209, in kubectl_get
    return run(KUBECTL, "get", cmd, "--all-namespaces", die=False)
  File "/snap/microk8s/3543/scripts/wrappers/common/utils.py", line 67, in run
    result.check_returncode()
  File "/snap/microk8s/3543/usr/lib/python3.6/subprocess.py", line 389, in check_returncode
    self.stderr)
subprocess.CalledProcessError: Command '('/snap/microk8s/3543/microk8s-kubectl.wrapper', 'get', 'all,ingress', '--all-namespaces')' returned non-zero exit status 1.

@nobuto-m
Copy link
Contributor Author

nobuto-m commented Jul 6, 2022 via email

@ycliuhw
Copy link
Member

ycliuhw commented Jul 6, 2022

hmm, after remove with --purge and then re-install it again, I can't reproduce it now.

@emrul
Copy link

emrul commented Aug 28, 2022

@ycliuhw - same thing happened to me and I didn't think to try remove with purge until I read your comment. Worked fine after retrying.

@bayurzxsmtp
Copy link

hmm, after remove with --purge and then re-install it again, I can't reproduce it now.

What do youo mean?

@a-yiorgos
Copy link

hmm, after remove with --purge and then re-install it again, I can't reproduce it now.

What do youo mean?

sudo snap remove --purge microk8s and reinstall

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Make sure to use microk8s.kubectl nobuto-m/microk8s
6 participants
@emrul @a-yiorgos @nobuto-m @ycliuhw @ktsakalozos @bayurzxsmtp