Replies: 3 comments 10 replies
-
|
1.282.0 works fine and your browser logs are not useful. Your server/worker logs might be more useful as well as sharing the actual error |
Beta Was this translation helpful? Give feedback.
-
|
I tried to find any useful error log in worker-1,2,3 and server-1 but I failed. Is there any suggestion to debug? |
Beta Was this translation helpful? Give feedback.
-
|
I also tried to pull another version of image v1.281.3 and got the same problem. |
Beta Was this translation helpful? Give feedback.
-
|
Finally, I reset my instance and reproduce the problem again. It seems to be related to socketio or my usage. After running several times, all the python script execution will be blocked forever. Even if I has restart the docker container. |
Beta Was this translation helpful? Give feedback.
-
|
I was just hit by a similar issue in v1.298.0. python scripts never start to execute - as background runnables or when directly run from script editing window. restarting containers does not change anything, restarting whole docker service also. everything worked fine for weeks. This is not a proper report by any means - just a comment such things happen with windmill. I need to get it back to work and maybe then will have time to describe the issue. |
Beta Was this translation helpful? Give feedback.
-
|
During the event docker was utilizing 4 cores while windmill stayed at 2-5% reported in lazydocker by each container. On this host docker runs only single Windmill instance via docker-compose. Docker daemon logged to system journal: msg="got error while decoding json" error="invalid character 'm' looking for beginning of value" retries=1131 Then nothing for the last ~hour when issue stopped. A quick search suggests (docker/for-linux#1350) broken log file of one of containers. |
Beta Was this translation helpful? Give feedback.
-
|
Today there was a similar short outage and again docker logged issues with parsing logs: Jul 18 21:16:13 app_main_1.bmscare.cloud dockerd[2210886]: time="2024-07-18T21:16:13.128824148+02:00" level=warning msg="got error while decoding json" error="invalid character 'b' looking for beginning of value" retries=1141 For about 13 seconds - but journal also shows similar case around 21:16 yesterday so issue happens every couple hours |
Beta Was this translation helpful? Give feedback.
-
|
for now I changed json-file logger to syslog and see if it helps |
Beta Was this translation helpful? Give feedback.
-
|
No docker hiccups for the last 3 days since logging changed to syslog. Windmill runs smoothly all the time. |
Beta Was this translation helpful? Give feedback.
-
|
I can confirm in my case the issue with python background jobs' execution was with Docker logging and not with Windmill. The json-file driver occasionally fails. The default syslog TCP can also fail when the syslog server fails. This happened to me once but with promtail forwarder being the syslog TCP target for containers. When a syslog server is unresponsive, and syslog TCP is used, workers can suddenly stop executing jobs and Windmill's App debug window shows long running jobs ending with error similar to "workers tagged with python are not available"; or maybe this error appears after restart attempt and before restart the problematic python jobs end with a timeout. If you try to start workers with unresponsive syslog server, containers defined with docker-compose stay in "created" state and only dockerd's log mentions syslog connection failure which is not obvious reason for containers not being able to start. A proper fix was to switch to syslog UDP which doesn't care about the syslog server (but at least containers start normally and there should be no logging-related issues freezing worker containers). |
Beta Was this translation helpful? Give feedback.
-
After updating my self-hosted server to [v1.282.0], all the python script can not be run. I don't know why and how to fix it.
Here is the log in web console:
getWorker: workerId: workerMain.js label: editorWorkerService
Editor.9bI3LOeY.js:409 started client
Editor.9bI3LOeY.js:409 Workspace settings: null
Editor.9bI3LOeY.js:409 Global settings: {}
Editor.9bI3LOeY.js:409 started client
Editor.9bI3LOeY.js:409 Using interpreter executable: /usr/local/bin/ruff
Editor.9bI3LOeY.js:409 Inferred version 0.2.2 for: /usr/local/bin/ruff
Editor.9bI3LOeY.js:409 Found ruff 0.2.2 at /usr/local/bin/ruff
Editor.9bI3LOeY.js:409 Running Ruff with: /usr/local/bin/ruff ['--force-exclude', '--no-cache', '--no-fix', '--quiet', '--output-format', 'json', '-', '--stdin-filename', '/tmp/monaco/iuamkiue0w.py']
Editor.9bI3LOeY.js:409 [Info - 13:16:13] Pyright language server 1.1.351 starting
Editor.9bI3LOeY.js:409 [Info - 13:16:13] Server root directory: file:///usr/local/share/.config/yarn/global/node_modules/pyright/dist
Editor.9bI3LOeY.js:409 [Info - 13:16:13] Starting service instance "windmill"
Editor.9bI3LOeY.js:409 started client
Editor.9bI3LOeY.js:409 [Error - 13:16:14] File or directory "/tmp/monaco/iuamkiue0w.py" does not exist.
Editor.9bI3LOeY.js:409 [Info - 13:16:14] No source files found.
EditorTheme.D7nUzZ3v.js:16 ERR MainThreadFileSystem#$watch(): failed to stat a resource for file watching (extension: nullExtensionDescription, path: file:///usr/local/lib/python3.11, recursive: true, session: 0.847132592515452): EntryNotFound (FileSystemError): file not found
EditorTheme.D7nUzZ3v.js:16 ERR MainThreadFileSystem#$watch(): failed to stat a resource for file watching (extension: nullExtensionDescription, path: file:///root/.local/share/virtualenvs/-x-v5uFv0/lib/python3.11/site-packages, recursive: true, session: 0.18656932385241443): EntryNotFound (FileSystemError): file not found
AppPreview.Cq7lmGoz.js:18 started 018df871-0d4d-8b3a-cc11-fc94188079bd
Editor.9bI3LOeY.js:409 [Error - 13:16:15] Stopping server timed out
Editor.9bI3LOeY.js:409 Stopping client early because of mismatch
Editor.9bI3LOeY.js:408 WebSocket is already in CLOSING or CLOSED state.
Beta Was this translation helpful? Give feedback.
All reactions