PG k8s node "no space left on device"

[Elexy]

I am running a PoC with Windmill and deployed the standard helm chart. Everything is working like a charm but the other day the postgres pod stopped working with the above error. I know what it means and we were able to fix it, but I am looking for some guidance on what is stored in the DB and by that measure how to take that into account building things in Windmill.

Overall: great tool!

[rubenfiszel]

The result and logs are stored in the db. To avoid the db blowing up, in latest version you can configure the retention period in the superadmin settings.

How many jobs are you running and what is the average size of those jobs in terms of logs or results ?

We're thinking of adding better monitoring baked in, would appreciate your feedback if you'd be up for it

[Elexy]

I'll have a look at that retention setting.

We were running 4 flows with ~6 scripts on schedules between every 5 minutes and every 24h.

• with "result" you mean the result of every script?
• with "logs" you mean audit logs or the logs emitted (i.e console.log()) by scripts?

[Girgitt]

I am not sure this is related but apart from database some temporary data like log files in txt format are stored on disk e.g. in /tmp/windmill/logs (at least on worker nodes) and from what I can tell, in version 1.298 at least, the content of this directory is not truncated. That means if the retention period is set to let's say 3 days in the UI, some of the tables are trucated (but not all - e.g. these tables have entries from before retention period: audit, job_logs, token) and files from /tmp/windmill/logs are not deleted causing uncontrolled disk utilization on the docker host. As a workaround I run a simple script every day to delete files older than X days on each container and some additional jobs in admin workspace to truncate old data from some of the tables in DB.