-
Notifications
You must be signed in to change notification settings - Fork 782
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
High CPU usage by gvfs-udisks2-vo which is caused by microk8s #500
Comments
Hi @khteh I do not see this over here. Can you help me reproduce the issue you are seeing? Is it possible for you to share all the manifests you apply and the addons you enable? I see that some hours ago you deployed a redis cluster. Could that be related with the spike in the disk utilization? |
Hi @ktsakalozos these are the add-ons that I have enabled:
I delete the statefulsets of redis cluster and now |
I am also seeing this occur in my cluster. Interestingly I have been using the same set of applications, so this must either be caused by a change in microk8s or a change in ubuntu. I'll see if I can get a simplified set of applications - right now I'm running a number of private applications. |
Maybe this helps: disabling "istio" lead to significantly less cpu usage in my case for gvfs-udisks2…. |
I have this as well and don't have a stateful set for redis, but do for things like kafka, zookeeper, elasticsearch.
The loop2 mount is the snap mount for microk8s.
Reading up about Either way - losing half to one CPU on seemingly regular file ops containers are doing for a monitor that seems to not do much except (in my naive understanding of it) expose volumes/filesystem to GNOME is a heavy price to pay? |
@tellisnz-shift just ask on the snapcraft forum [1] about the 'x-gvfs-hide' option. Perhaps the good people at snapcraft will be able to add this option when mounting MicroK8s. I see however that the same issue is reported for K3s [2] and the investigation over there pointed to outdated gnome packages. What distribution are you on? It would be great if we had a script that would reproduce this behavior. [1] #500 (comment) |
Thanks for asking on there @ktsakalozos. This is me currently:
I'm fully up to date as far as I am aware. As far as scripting, I will try and find some time over the weekend. |
I'm seeing the same behavior, on Ubuntu 19.04.
A few data points:
It makes sense that it's related to the storage plugin. The fact I saw high CPU usage after removing my services, and saw it resume to low CPU usage after enable/disabling the plugin, makes me think that there's something with the storage plugin that might be at fault. |
I'm seeing the same, Ubuntu 18.04 LTS.
This issue locks up my high-spec development machine for a few minutes. I/O becomes saturated. This is with a very basic cluster deployment with no "traffic". Interestingly, this only happens if I change network (i.e. connect to a wifi network) or resume from suspend. (Resuming from suspend may be triggering a "network change"). Disabling and re-enabling the storage plugin appears to fix it for a while. (As well as completely purging microK8s and reinstalling it) |
We're also seeing the same issue with Ubuntu. |
on an older 4 core i7 cpu the freezes are almost unbearable |
I faced the similar issue after leaving the system idle. It froze almost every Monday. Switched to Minikube and it's better now. |
Experience the same, and there is seem to be no way to disable it |
I'm getting this issue when deploying Kubeflow, which has lots of pods that require storage. Since I'm not running the cluster full-time, I was able to work around this by running Also, I tried adding the
|
FYI I have this on a ubuntu 20.04 system with kubernetes installed using kubespray. I did use microk8s at one point, but had already stopped it. Interestingly if I run nautilus on this machine, it freezes. Same with all browser file save dialogs. Running
Fixes both the 100% CPU issue, and makes nautilus stop from freezing |
Same here to on Fedora 33... gvfs-udisks2-volume-monitor uses between 50 and 75% cpu constantly, and gsd-housekeeping is constantly between 25-50% stopping monitor stopped it and its cpu usage, but I have no idea what side-effects that have? |
This seems to help in reducing the CPU usage of gvfs-udisk2-volume-monitor:
For me, the gvfs-udisk2-volume-monitor was constantly spiking to over 10% CPU consumption. After making the above change and rebooting, the spikes are 1-2% of CPU. This solution was inspired by https://github.com/moby/moby/blob/b96a0909f0ebc683de817665ff090d57ced6f981/contrib/udev/80-docker.rules#L3 |
It seems that the correct fix for this issue would be to make snap add |
I got here from searching "high cpu udisks2 ubuntu". Although my problem is in another environment, maybe my research can help here (?) I am running kubernetes installed with (Please feel free to hide this comment if this is not related or a kind of help to this issue) |
I've met the same problem as described @Richard87 , However stopping monitor with
Here is my system configuration $ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"9f2892aab98fe339f3bd70e3c470144299398ace", GitTreeState:"clean", BuildDate:"2020-08-13T16:12:48Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"9f2892aab98fe339f3bd70e3c470144299398ace", GitTreeState:"clean", BuildDate:"2020-08-13T16:04:18Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"} and further more I use
|
It has been 2 and half years since my first post on this issue. I take another look to see if I see the same CPU load issue with the latest version of everything.
|
@khteh does the workaround described in #500 (comment) work for you? |
I don't understand that "fix" and therefore, haven't tried. |
I tried the fix but no effect :-( |
Is there any progress here? It's annoying seeing an entire core on my laptop stuck at 100% usage for hours on end and it creates a huge amount of waste heat and kills battery life if I don't notice it. |
Any news about this? tried both:
and
with no luck I'm on ubuntu 22.04:
and installed microk8s with:
|
can't believe this issue is created in 2019 and still not solved. please anyone has found any robust solution please let me know . thanks . |
I don't see this error in the past year using microk8s in both localhost and production workload. Closing. |
This still happens on my system. It is Ubuntu 22.04 with microk8s 1.21. It happens any time I open a file browser while microk8s is running. Disabling the service doesn't fix the problem, the fix from the above comment doesn't fix it. |
The current version is |
Yeah, I definitely should be moving the microk8s version. I'll post something once I upgrade to a newer version. I'm running on a Lenovo Thinkbook 14 G3 with an AMD Ryzen 7 5700U. Kernel 6.2.0-37-generic. |
I contributed an improvement to snapcore as mentioned in #500 (comment) . However, there was a util-linux bug which is fixed by util-linux/util-linux@d85f45d since util-linux 2.38 . Ubuntu 22.04 has util-linux 2.37 . There's now a bug report in gvfs at https://bugs.launchpad.net/ubuntu/+source/gvfs/+bug/2047356 too |
I've been using this workaround successfully for years:
Reboot to ensure that it takes effect. |
@lhotari 's workaround worked instantly for me I'm not using microk8s on my cluster (It's a compose cluster), and I think it's probably a snap problem because of the prevailing loopbacks (I only skimmed this thread) but it still worked. |
Please run
microk8s.inspect
and attach the generated tarball to this issue.inspection-report-20190610_165704.tar.gz
We appreciate your feedback. Thank you for using microk8s.
top
shows:udisksctl monitor
shows it is mainly by microk8s. What happen?The text was updated successfully, but these errors were encountered: