Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

High CPU usage / ZFSNode syncing error #420

Closed
hrenard opened this issue Jun 18, 2022 · 4 comments
Closed

High CPU usage / ZFSNode syncing error #420

hrenard opened this issue Jun 18, 2022 · 4 comments
Assignees
Labels
backlog Will be picked up as a roadmap item. need more info More information is needed from user, need reproduction steps
Milestone

Comments

@hrenard
Copy link
Contributor

hrenard commented Jun 18, 2022

What steps did you take and what happened:
The zfs-localpv-node is permanetly using 1 CPU.

What did you expect to happen:
Not using as much resources.

The output of the following commands will help us better understand what's going on:
Gist

Anything else you would like to add:
The process using the CPU:

/sbin/zfs list -s name -o name,guid,available -H -p

I suspect this error is related:

openebs-zfs-plugin E0618 16:31:28.546617       1 zfsnode.go:279] error syncing 'kube-system/lyriste': update zfs node kube-system/lyriste: Operation cannot be fulfilled on zfsnodes.zfs.openebs.io "lyriste": the object has been modified; please apply your changes to the latest version and try again, requeuing

Environment:

  • ZFS-LocalPV version: 2.1.0
  • Kubernetes version: v1.23.3
  • Kubernetes installer & version: v1.23.3+k0s
  • Cloud provider or hardware configuration: homelab
  • OS: Ubuntu 20.04.4 LTS
@Abhinandan-Purkait Abhinandan-Purkait added backlog Will be picked up as a roadmap item. need more info More information is needed from user, need reproduction steps labels Jun 6, 2024
@Abhinandan-Purkait
Copy link
Member

@hrenard We will test this out and post our findings here. Thanks

@Abhinandan-Purkait Abhinandan-Purkait self-assigned this Sep 26, 2024
@Abhinandan-Purkait Abhinandan-Purkait added this to the v4.2 milestone Sep 26, 2024
@Abhinandan-Purkait
Copy link
Member

Scoping this as investigative item for v4.2

@Abhinandan-Purkait
Copy link
Member

@hrenard I created nearly 200 pools, and the CPU consumption went as high as 1m, which is expected. Can you provide more details around how to reproduce this. Having said that it's not reproducible under normal circumstances.

Btw, you can set limits to the resources via the helm chart which is commented block, if you don't want resources to exceed certain threshold. The pod would either get killed or stay up hanged if the limit is reached.

@hrenard
Copy link
Contributor Author

hrenard commented Jan 3, 2025

Thanks for investing this ! Sorry, I didn't follow up.
I think the syncing error used to trigger an unusual amount of zfs list which was hurting the cpu.
That being said, I'm not seeing the issue anymore. But my environment changed :

  • ZFS-LocalPV version: 2.6.2
  • Kubernetes version: v1.28.9
  • Kubernetes installer & version: v1.28.9+k0s
  • Cloud provider or hardware configuration: homelab
  • OS: NixOS 24.05

I'm closing the issue as no one seems to have had this issue, nor reproduce it.

@hrenard hrenard closed this as completed Jan 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backlog Will be picked up as a roadmap item. need more info More information is needed from user, need reproduction steps
Projects
None yet
Development

No branches or pull requests

2 participants