microk8s is not running. microk8s.inspect showing no error #886
Comments
|
Your apiserver is complaining about an invalid bearer token. Was this a fresh installation? |
|
@balchua no it's not. I rebooted the machine after it's been running for a while |
|
Thanks @ibigbug can you try to restart microk8s? |
|
@balchua not seem working wtf@k8s-master:~$ microk8s.stop
[sudo] password for wtf:
Stopped.
wtf@k8s-master:~$ microk8s.start
Started.
Enabling pod scheduling
wtf@k8s-master:~$ microk8s.status
microk8s is not running. Use microk8s.inspect for a deeper inspection.
wtf@k8s-master:~$ |
|
pod status |
|
Are you running multi nodes? |
|
yes 1 master + 1 follower |
|
Can you go to the worker/follower node and do a |
|
it doesn't actually allow me: |
|
Is it possible to make it a single node cluster to see if it is still running? I think you may need to do |
|
still not working. maybe I'll just reinstall.. |
|
You may want to pin it to a particular channel ex. 1.16 stable. |
|
@ibigbug I see that the kubelets cannot register with the apiserver. The last time they registered with the API server was on the 22nd of Dec. The error you have looks like this: Any idea what might have changed around then? |
|
if it's saying node not found, might be due to the reboot of VM? |
|
I also had the same issue. It happens with v1.17 only (other versions like v1.16, v1.15, v1.14) are ok. Also, it happens with v1.17 after I try to install kubeflow using I reproduced this behaviour in another machine as well. |
|
Same error. |
|
Either removing and re-installing fixed the issue or the version: |
|
After several delete and re-installs, I narrowed it down to microk8s dying the moment I try to change the context to use. I enabled DNS, then created two namespaces, then two contexts, I checked the status of microk8s after each command and it was running. After I did: |
|
@TribalNightOwl thanks for the info. When you added the context, did u add it to the file |
|
@TribalNightOwl your context |
No, I just used these commands:
yes: |
I will try again and report back. Although I would argue that microk8s shouldn't stop running (and refuse to start) due to this. |
BINGO! it didn't die this time. |
|
@TribalNightOwl microk8s is not actually dying. The The message can be misleading though. |
Manually deleted this section: |
|
Hold on, I got it: My previous was still pointing to something non-existent. I did: That completely proves your previous comment, thanks! |
|
How about changing the message: Currently: New: Or: Something that would make the user think about having a misconfigured client and not necessarily microk8s dying. |
We could also detect such problems and suggest a fix in |
|
On my system I found that the problem went away after I updated the rather too old kubectl installed in /usr/local/bin on my system. I had assumed that microk8s would exclusively use it's own kubectl, but apparently not. |
|
Solved it for me @gavinB-orange - had to remove my previously installed kubectl and then it microk8s started working!
|
|
@alexgleason you said the server was rebooted, was it a graceful reboot or was it due to some power failure? Could you share the output of |
|
I have an open issue for it in raft canonical/raft#192, but haven't been able to reproduce yet. The solution in most of the cases is indeed to remove the offending segment. |
|
Hey guys, thank you for the help, I appreciate it. I deleted the file and now I see this in the logs when starting: inspection-report-20210810_132505.tar.gz
It was indeed a power failure. 😕 |
|
You'll have to delete Remember to always backup your data before you start deleting things ;-) |
|
Wow that worked, thank you so much! I'm able to see my nodes and resources now. I really appreciate your help. |
|
Glad it worked, you will have lost a couple of minutes of data it seems. |
|
@MathieuBordere I am facing error which can't start microk8s .. I tried all posible ways from this issue. but nothing can fix my errors. Can you please check my mk8s inspect tarball ? |
|
Hi, my local kubernetes installation suddenly stopped working. I got the same problem where microk8s status shows |
|
snap refresh microk8s --classic --channel=1.20/stable
…On Tue, 28 Sep 2021 at 5:07 PM Vin678 ***@***.***> wrote:
Hi, my local kubernetes installation suddenly stopped working. I got the
same problem where microk8s status shows
microk8s is not running. Use microk8s inspect for a deeper inspection.
And microk8s inspect shows no error. Kubectl get pods is showing a
connection refused error:
The connection to the server 127.0.0.1:16443 was refused - did you
specify the right host or port?
Does someone have any clue why microk8s could suddenly stop working. I
didn't do any configuration changes or system updates when this started
happening.
inspection-report-20210927_113253.tar.gz
<https://github.com/ubuntu/microk8s/files/7243140/inspection-report-20210927_113253.tar.gz>
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#886 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AU4J4C6OLMHYSWA3OIN6C3LUEGLFZANCNFSM4KB5UPDA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
|
|
It doesn't look like updating to version 1.20 changed anything. Still get the same error. |
|
@Vin678 something is force killing the apiserver. I couldn't find anything in the logs. |
|
I run into the same issue today. apparently for me it came down to the x509certificate, eg: If I move the clock on the computer 1 hour forward everything works fine then. I run it on an Ubuntu 20.04 laptop, set to the Automatic Date/Time and GMT timezone. Not sure what caused this, but hope this is useful. UPD: |
|
@balchua enviroment root@ajinlong:/var/snap/microk8s/3597# uname -a
Linux ajinlong 5.15.0-47-generic #51-Ubuntu SMP Thu Aug 11 07:51:15 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
root@ajinlong:/var/snap/microk8s/3597# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.1 LTS
Release: 22.04
Codename: jammy
After But i found processes referring microk8s root@ajinlong:/home/ajinlong# ps -ef | grep microk8s
root 13376 1 17 15:40 ? 00:01:16 /snap/microk8s/3597/kubelite --scheduler-args-file=/var/snap/microk8s/3597/args/kube-scheduler --controller-manager-args-file=/var/snap/microk8s/3597/args/kube-controller-manager --proxy-args-file=/var/snap/microk8s/3597/args/kube-proxy --kubelet-args-file=/var/snap/microk8s/3597/args/kubelet --apiserver-args-file=/var/snap/microk8s/3597/args/kube-apiserver --kubeconfig-file=/var/snap/microk8s/3597/credentials/client.config --start-control-plane=true
root 13397 1 0 15:40 ? 00:00:00 /bin/bash /snap/microk8s/3597/apiservice-kicker
root 13438 1 5 15:40 ? 00:00:23 /snap/microk8s/3597/bin/k8s-dqlite --storage-dir=/var/snap/microk8s/3597/var/kubernetes/backend/ --listen=unix:///var/snap/microk8s/3597/var/kubernetes/backend/kine.sock:12379
root 13544 1 0 15:40 ? 00:00:00 /bin/bash /snap/microk8s/3597/run-cluster-agent-with-args
root 13556 1 0 15:40 ? 00:00:04 /snap/microk8s/3597/bin/containerd --config /var/snap/microk8s/3597/args/containerd.toml --root /var/snap/microk8s/common/var/lib/containerd --state /var/snap/microk8s/common/run/containerd --address /var/snap/microk8s/common/run/containerd.sock
root 13745 13544 0 15:40 ? 00:00:01 /snap/microk8s/3597/bin/cluster-agent --bind 0.0.0.0:25000 --keyfile /var/snap/microk8s/3597/certs/server.key --certfile /var/snap/microk8s/3597/certs/server.crt --timeout 240
root 17820 15171 0 15:48 pts/2 00:00:00 grep --color=auto microk8s
|
|
Hi @codecoron In the containerd logs, I see Can you try to see if https://microk8s.io/docs/registry-private#configure-registry-mirrors-7 solves your issue? |
|
It doesn't work root@ajinlong:/home# cat /var/snap/microk8s/current/args/certs.d/k8s.gcr.io/hosts.toml
server = "https://k8s.gcr.io"
[host."https://registry.cn-hangzhou.aliyuncs.com/google_containers"]
capabilities = ["pull", "resolve"]
Any ideas? @neoaggelos More related info root@ajinlong:/home# microk8s kubectl get nodes
NAME STATUS ROLES AGE VERSION
ajinlong NotReady <none> 31h v1.24.4-2+2f38f78fa07274
root@ajinlong:/home# microk8s kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 31h |
|
same error, but I worked in a wsl system: Building the report tarball inspection-report-20220914_024117.tar.gz Thank you |
|
|
|
As Chinese user, it may caused by GFW. As |
|
I'm in the same boat. I have 3 boxes, all with the same hardware and 1 will not run microk8s, even with a fresh install of Ubuntu 22.04.1 LTS. I've uninstalled, re-installed, reset microk8s, wiped the HD, and re-installed the OS. After all this microk8s extremely unstable. Run describe pods 3 times in a row. I'm not sure what to be looking for in the logs but a few things looks suspicious. and |
|
I get the same issue if installing Microk8s version 1.27 and then downgrading to version 1.26. |
|
I have the same issue with MicroK8s v1.26.4 revision 5219. microk8s stop microk8s start microk8s status microk8s is not running. Use microk8s inspect for a deeper inspection. microk8s inspect Building the report tarball |
|
Hi @tiansiyuan I see the following repeated in the containerd logs: Where is this? If a firewall is blocking access to registry.k8s.io, check if the following helps you https://microk8s.io/docs/registry-private#configure-registry-mirrors-7 Thanks! |
My microk8s also stopped working when i changed the time and I was getting similar erros. now is running perfect after fixing the time. |
I confirm this solves it. After applying a valid mirror, and |
|
I am just going to append to this issue, I seem to have the same problem but no idea on how to read the tarball (is there documentation for that?) |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
and others
Please run
microk8s.inspectand attach the generated tarball to this issue.wtf@k8s-master:~$ microk8s.inspect
Inspecting services
Service snap.microk8s.daemon-cluster-agent is running
Service snap.microk8s.daemon-flanneld is running
Service snap.microk8s.daemon-containerd is running
Service snap.microk8s.daemon-apiserver is running
Service snap.microk8s.daemon-apiserver-kicker is running
Service snap.microk8s.daemon-proxy is running
Service snap.microk8s.daemon-kubelet is running
Service snap.microk8s.daemon-scheduler is running
Service snap.microk8s.daemon-controller-manager is running
Service snap.microk8s.daemon-etcd is running
Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
Copy processes list to the final report tarball
Copy snap list to the final report tarball
Copy VM name (or none) to the final report tarball
Copy disk usage information to the final report tarball
Copy memory usage information to the final report tarball
Copy server uptime to the final report tarball
Copy current linux distribution to the final report tarball
Copy openSSL information to the final report tarball
Copy network configuration to the final report tarball
Inspecting kubernetes cluster
Inspect kubernetes cluster
Building the report tarball
Report tarball is at /var/snap/microk8s/1107/inspection-report-20200102_011315.tar.gz
inspection-report-20200102_011315.tar.gz
wtf@k8s-master:~$ microk8s.status
microk8s is not running. Use microk8s.inspect for a deeper inspection.
We appreciate your feedback. Thank you for using microk8s.
The text was updated successfully, but these errors were encountered: