Issues after Restart Server from UI

[vladmandic]

Note that Restart Server from UI is best-effort function and can lead to different random issues:

• Duplicate entries visible in UI
• KeyErrors logged in console
• Lists in menus contain duplicate items

Reason is that restart server must reload scripts/extensions (otherwise its pointless), but many scripts/extensions do not check if they are already loaded, so they register themselves as duplicates leading to symptoms described above.

Proper way would be for each script/extension to check if its already registered, but until that is done for every one of them, Restart Server from UI is a best-effort feature.

if you experience issues after Restart Server from UI, but they go away if you actually restart server from command line, report on this thread but do not create new Issue

[tacediak]

Brought over here instead of adding to the closed issue elsewhere as I think this is the right place for it. I've never issued an issue report as I'm new to all this so if it needs to go somewhere else I can move it.

Yes, I expect that as well. But if it happens after Restart via UI, that is a known issue and tracked separately. There is no easy way to clear Gradio in-memory states when restarting server from UI - that is going to be a long pending item. And I wrote several times about that. Any issue that happens after "Restart via UI" should be reproduced with clean restart or its not something I can look into individually. Even issue template states that.

Closing the issue unless this can be reproduced on clean server start - if yes, I'll reopen.

Hey there! I just installed Vladomatic last night for the very first time(I'm eastern US)(ETA: I think it would have been the last commit of the night so 0e39aa7?), so mine is completely mostly fresh and should be the newest version(as of last night) and I'm having the same issue as here #360 (comment). If I change any settings or reload ui for any reason and try to bring an image into the process tab after to then send its settings/prompt/etc to t2i, as soon as I load the image into the process tab the whole thing freezes on me and I have to close out entirely, terminate the process in the cmd window, and restart with webui.bat.

This is the error I get in the cmd window when I hit restart ui in settings.

Traceback (most recent call last):
File "X:\Testing\automatic\venv\lib\site-packages\gradio\routes.py", line 394, in run_predict
output = await app.get_blocks().process_api(
File "X:\Testing\automatic\venv\lib\site-packages\gradio\blocks.py", line 1075, in process_api
result = await self.call_function(
File "X:\Testing\automatic\venv\lib\site-packages\gradio\blocks.py", line 884, in call_function
prediction = await anyio.to_thread.run_sync(
File "X:\Testing\automatic\venv\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "X:\Testing\automatic\venv\lib\site-packages\anyio_backends_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
asyncio.exceptions.CancelledError
Server shutdown
Server restart

Then it finishes restarting and I pull an image into process tab(as pictured):

And while it recognizes when I hover over buttons, it doesn't acknowledge when I click and I can't click anywhere else either.
This is the error that comes up in the cmd window when I bring in the image, after the readout of the prompt/settings info from the image:

Traceback (most recent call last):
File "X:\Testing\automatic\venv\lib\site-packages\gradio\routes.py", line 394, in run_predict
output = await app.get_blocks().process_api(
File "X:\Testing\automatic\venv\lib\site-packages\gradio\blocks.py", line 1078, in process_api
data = self.postprocess_data(fn_index, result["prediction"], state)
File "X:\Testing\automatic\venv\lib\site-packages\gradio\blocks.py", line 994, in postprocess_data
block = self.blocks[output_id]
KeyError: 331

I'm running on windows 10, Chrome browser, nvidia RTX 3080, only cmd arguments are --listen --port, no extensions other than the ones that come with it installed, xformers enabled and up to date(though this happened before and after getting that up and running and sorted), only settings I've changed are which theme I'm using and which things are in quick settings at the top(though again, this happened before and after those were changed). If you need any other specs I can try to find them all. Currently working on getting another completely fresh with zero changes installation running just to triple check that it isn't any of the things I listed that are doing it. Will post if that comes back differently.

[vladmandic]

yup, thats exactly the bad behavior that is expected in some circumstances when you use Server restart from UI.
i'll work on that, but fix is anything but trivial and its in my backlog.
all those issues should go away after you actually restart webui (not from UI, but from command prompt).

[tacediak]

yup, thats exactly the bad behavior that is expected in some circumstances when you use Server restart from UI. i'll work on that, but fix is anything but trivial and its in my backlog. all those issues should go away after you actually restart webui (not from UI, but from command prompt).

Saw this reply after I'd already typed this stuff out but I'll drop here anyway just in case there's something that might be of use. Thanks so much for all your hard work! This fork(I think that's the right term?) is really nice!

Update: Completely fresh install(ec532ee) does the same thing. Same errors etc.
Pulled an image into the process tab before hitting restart ui and was able to send info over to t2i with no issue. Went into settings and hit restart ui, tried again, and it froze. I repeated all these steps 3 times just to be sure to the same result each time. Terminating from cmd and restarting from webui.bat fixed it until I hit restart ui again. Grabbed this info from cmd with specs:

13:03:07-925004 INFO Python 3.10.6 on Windows
13:03:07-988999 INFO Version: ec532ee Sat Apr 22 12:28:34 2023 -0400
13:03:08-306832 INFO Updating Wiki
13:03:09-386495 INFO nVidia toolkit detected
13:03:12-168495 INFO Torch 2.0.0+cu118
13:03:12-198495 INFO Torch backend: nVidia CUDA 11.8 cuDNN 8700
13:03:12-202495 INFO Torch detected GPU: NVIDIA GeForce RTX 2080 VRAM 8192 Arch (7, 5) Cores 46
13:03:12-206489 INFO Verifying requirements
13:03:12-244496 WARNING Package wrong version: timm 0.4.12 required 0.6.13
13:03:12-247504 INFO Installing package: timm==0.6.13
13:03:18-526032 INFO Running setup
13:03:18-529034 INFO Installing packages
13:03:18-531034 INFO Installing repositories
13:03:18-759033 INFO Installing submodules
13:03:21-330666 INFO Updating submodules
13:03:29-948499 INFO Extensions enabled: ['a1111-sd-webui-lycoris', 'clip-interrogator-ext', 'LDSR', 'Lora',
'multidiffusion-upscaler-for-automatic1111', 'prompt-bracket-checker', 'ScuNET',
'sd-dynamic-thresholding', 'sd-extension-aesthetic-scorer', 'sd-extension-steps-animation',
'sd-extension-system-info', 'sd-webui-controlnet', 'sd-webui-model-converter', 'seed_travel',
'stable-diffusion-webui-images-browser', 'stable-diffusion-webui-rembg', 'SwinIR']
13:03:52-387282 INFO Extensions enabled: []
13:03:52-390288 INFO Setting environment tuning
13:03:52-393287 INFO Running extension preloading
13:03:52-401304 INFO Server arguments: []
No module 'xformers'. Proceeding without it.
Available models: X:\Testing\automatic\models\Stable-diffusion 1

-Error readout when restarting ui-

Traceback (most recent call last):
File "X:\Testing\automatic\venv\lib\site-packages\gradio\routes.py", line 394, in run_predict
output = await app.get_blocks().process_api(
File "X:\Testing\automatic\venv\lib\site-packages\gradio\blocks.py", line 1075, in process_api
result = await self.call_function(
File "X:\Testing\automatic\venv\lib\site-packages\gradio\blocks.py", line 884, in call_function
prediction = await anyio.to_thread.run_sync(
File "X:\Testing\automatic\venv\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "X:\Testing\automatic\venv\lib\site-packages\anyio_backends_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
asyncio.exceptions.CancelledError
Server shutdown
Server restart
Available models: X:\Testing\automatic\models\Stable-diffusion 1

-Error readout when bringing in an image to process-

Traceback (most recent call last):
File "X:\Testing\automatic\venv\lib\site-packages\gradio\routes.py", line 394, in run_predict
output = await app.get_blocks().process_api(
File "X:\Testing\automatic\venv\lib\site-packages\gradio\blocks.py", line 1078, in process_api
data = self.postprocess_data(fn_index, result["prediction"], state)
File "X:\Testing\automatic\venv\lib\site-packages\gradio\blocks.py", line 994, in postprocess_data
block = self.blocks[output_id]
KeyError: 331

Hope any of this is helpful. If you need anymore info I can try to provide. :)

[some9000]

@vladmandic "Proper way would be for each script/extension to check if its already registered"

Keeping mind that I am basically stumbling through Python how should that be done? Those errors are popping up here on each restart, as described. Although, in this case, code is changing on each restart as well so that is probably not what I want. But for a final release version...

[meishild]

When i reaload my ui used to last Generation Info button

Generation Info
mini\(ttp\), (8k, RAW photo, best quality, masterpiece:1.2), cyberpunk city, miniature, landscape, isometric, <lyco:miniatureWorldStyle_v10:1> Negative prompt: (((blur))), (EasyNegative:1.2), ng_deepnegative_v1_75t, paintings, sketches, (worst quality:2), (low quality:2), (normal quality:2), lowres, normal quality, ((monochrome)), ((grayscale)), bad anatomy,(long hair:1.4),DeepNegative,(fat:1.2),facing away, looking away,tilted head,lowres,bad anatomy,bad hands, text, error, missing fingers,extra digit, fewer digits, cropped, worstquality, low quality, normal quality,jpegartifacts,signature, watermark, username,blurry,bad feet,cropped,poorly drawn hands,poorly drawn face,mutation,deformed,worst quality,low quality,normal quality,jpeg artifacts,signature,watermark,extra fingers,fewer digits,extra limbs,extra arms,extra legs,malformed limbs,fused fingers,too many fingers,long neck,cross-eyed,mutated hands,polar lowres,bad body,bad proportions,gross proportions,text,error,missing fingers,missing arms,missing legs,extra digi Steps: 40, Sampler: DPM++ 2M Karras, CFG scale: 7, Seed: 248372777, Size: 768x768, Model hash: 68c0a27380, Model: cetusMix_Coda2, Denoising strength: 0.5, ENSD: 31337, Hires upscale: 2, Hires steps: 20, Hires upscaler: R-ESRGAN 4x+ Anime6B

chrome console 1 error.

and my termial

Versions
`
commit 5dcaaba (HEAD -> master, origin/master, origin/diffusers, origin/HEAD)
Author: Vladimir Mandic [email protected]
Date: Thu Apr 27 09:42:41 2023 -0400

reduce extra network exception
`

[mark-wd]

Following up from #552 that this is indeed the same issue related to server restarts.

[vladmandic]

New server restart functionality is published as of today.
Yes, there are possible scenarios where some things are maybe not working, but now I'm no longer saying "fails-as-designed" - any issues, please report and i'll try to address them.