windmill worker cpu load?

[helmut72]

Hi, interesting product.

I've installed windmill, changed password, created an app with one button (just wanted to see the interface builder) and that's all. It's now idling more or less, but workers cpu load is always at about 6%. I also see no log output from the worker. What is the worker doing all the time?

cpu load before 20:00 was without windmill, cpu load after 20:30 is with windmill idling (logged out, browser tab closed):

I ask, because it nearly doubles the cpu load of my server. Other process are only go up when they do something.

Thank you!

[rubenfiszel]

It's very likely coming from the workers + db. The way queuing work on the postgresql implementation (there is a redis one that would consume less) is that worker poll every 50ms the database for new jobs. That is a lot of polling which creates load on both the db and workers. 50ms is low but guarantees super low latency with sufficient number of free workers.
In most cases, that wouldn't be an issue since:

• 6% is pretty low on what I assume to be a non-beefy server
• you do not pay that price when the worker is actually running the job since it's not polling anymore.
• in production settings, you would want to setup dedicated resources for windmill

You can adjust the frequency if you'd like by setting SLEEP_QUEUE as env variable to the workers to anything higher than 50

[helmut72]

Thanks for the info. I set SLEEP_QUEUE to 500 and now the worker process is ok, but there are still 20% peaks in Postgres. How can I enable Redis? I use your docker-compose example (with my own Caddy instance).

Now I'm testing Windmill on a Raspberry Pi 4 and idea is to run shell scripts with some parameters from a web interface. It's easier than SSH from a phone. Therefore I care about idling because it sleeps most of the time.

Maybe Windmill is oversized for my purpose.

[rubenfiszel]

The fact that it works at all on a rpi4 with low cpu is actually pretty cool. Not sure what are causing the peaks, maybe the token cleanup every 30s. Indeed windmill is not meant to be the lowest consumption framework :D

[helmut72]

Ok, let me play with it next days. Thank you! :)

[FrancisVarga]

Why is there a poll to db, cant use rabbitmq or celery or simple redis for job queue? The db is high load since you store logs etc there. and connection pooling is not working well, ill set it to 1000 and only 60 connections with 100 workers