ExtraNetworks: Performance Updates and Improvements

[Sj-Si]

⚠️ THE FOLLOWING COMMENT IS THE MOST UP-TO-DATE STATUS OF THIS PR. START THERE: ⚠️

#15530 (comment)

⚠️ DESCRIPTION BELOW THIS POINT IS OUT OF DATE ⚠️

Description

This PR is a complete overhaul of the code that I previously wrote for the ExtraNetworksTreeView PR (#14588) in response to the negative feedback such as in #15164 and various issues that have been posted.

The goal of this PR is to improve performance and reduce resource use across ExtraNetworks tabs.

I have tested these changes with up to 100,000 models at a time. With this many models, the initial load is slow but as soon as the server fully generates the dataset it performs very well. Before, my browser would just crash if I even tried to load the page with more than 50k models. The real limiting factor here is the server (though it takes quite a bit to reach the limit). Right now, ui_extra_networks.py is generating the HTML for all of the files it discovers in any of the models directories. It then waits for requests from the client and sends only the requested items HTML back to the client. This way the client doesn't have massive amounts of DOM elements constantly loaded. So as long as the device running the server can load the generated HTML for every single item in the models folders then it should be fine. I don't see this being an issue until some psycho comes along with 10 million models but I don't really think we should be pandering to such a niche audience. Even then, I think other things will crash before this new code does.

Sorry ahead of time... this is a big one.

The following has changed:

• Implemented a modified version of Clusterize.js to allow for an asynchronous data loader.
  • This allows us to offload generating and storing HTML strings to the server instead of the client.
  • HTML strings for tree/card elements are generated in the python server and only fetched by the client via JavaScript when they are needed. Fetching is done in smaller chunks to reduce overhead.
  • Set up a pseudo paging system to use as a cache for fetched HTML elements. This way we can reduce the number of times we fetch data from the server.
• Reduced DOM complexity by minimizing the amount of nested elements in the tree/cards views.
• Created all new SVG icons for the ExtraNetworks tab controls.
  • Accidentally went on a tangent and ended up learning how SVGs work.
  • I designed all of these SVGs so we don't have to worry about licensing crap.
    • I'm not a graphics artist in any way so feel free to change these designs as needed or draw up a sample and I'll try to update it.
• Created a utils.js file and added a lot of helpful utility functions that could probably be used elsewhere if needed.
• Tab now allows use of both the tree and directory views at same time.
  • Selecting a filter button in one view selects the corresponding button in the other view.
• Fixed various bugs that went undiscovered in my previous implementation.
• Added a custom event that fires whenever a resizeHandle.js handle is double clicked.
  • This allowed me to implement an auto-resize function on the tree view whenever the resize handle is double clicked. This automatically sets the width of the tree view to the width of the widest row in the list.
• Added an option to only show directories in the tree view per some user's request.
• Updated CSS to use variables instead of hardcoded values.
  • This way these changes should be way more compatible with custom themes.

UI Changes

Below is an example of what the new format looks like:

Controls

The controls have been grouped with a header to hopefully make their intent more obvious.

Directory View

Directory view buttons have been styled a bit better and now the selected directory will be highlighted.

Tree View

Slightly modified the appearance of the tree view and simplified the code that generates it. Also directories in the tree view are no longer expanded when the item itself is clicked. Now the user must click the chevron on a directory in order to expand it.

Checklist:

• I have read contributing wiki page
• I have performed a self-review of my own code
• My code follows the style guidelines
• My code passes tests

  ⚠️ Can't run tests with my environment however the tests in CI did pass.

Additional Notes

Someone suggested that we use emojis for the controls in the extra networks tabs. Personally I hate emojis and they look disgusting. On top of that, I couldn't find any that really fit the purpose of each of the controls. If someone has a theme (like lobo or whatever) then they can handle replacing these SVGs if they want. The SVGs are all easy to query so they can handle a find/replace if they want.

One thing that I couldn't figure out is how to manually trigger the gradio loading icon within the extra networks pane (see below):

This loading page appears whenever the page is reloaded or refreshed but I would have liked to have been able to manually trigger it to show up and stay there until the data is fully loaded but I couldn't figure out where this loading pane was even coming from. I'd really like to be able to put one in the Tree View and one in the Cards View. Then I would trigger each manually whenever I need to load more data.

Also at some point someone requested that clicking one of the directory view buttons to filter cards should only show direct children of that directory and not any of the nested directory content. What was the consensus on this? Should that be the behavior or do we prefer to show all nested items? The current behavior is the same as it always has been and it is a pretty simple implementation where clicking a button simply adds that path to the search box and updates the filter based on the search box text. Changing the behavior of clicking directory buttons would definitely require a bit of redesign and I think this change has already had enough scope creep as is.

Needs Testing

I would like to have this feature undergo a bit more testing than the last time in order to avoid having so many people get upset at the same time. I'd really appreciate feedback for this PR so that we can catch as many problems as possible before merging.

I have really only tested in the following environment:

• Server running on WSL Ubuntu
• Client connecting via Chrome on Windows 10 (localhost:7860)

[Sj-Si]

When I enable Card Detail View the pane on the right does not appear. I have it enabled by default with default size. Previously when it was supposed to be empty it did appear but with one of the recent changes it doesn't show.

Edit: Just noticed it does work for Checkpoints.

Weird. It is working fine for me still. Using chrome on windows10 and server running in WSL Ubuntu.

Can you check your browser console and the terminal in which you started the server and see if it is throwing any errors anywhere? I suspect the server wouldn't throw any since you should have gotten a popup message saying that it failed to show the model details. Also I just want to make sure but are you pressing and holding the model for it to show the details? Model details won't show up on a normal click. Actually... do you think it would make more sense to show model details whenever the model is clicked to add it to the prompt?

Also which browser and OS are you using?

[Foolishkun]

Are you referring to this browser console? If so then nothing shows up when pressing and holding.

This is my terminal. Nothing new comes up when trying to get the details to show.

I'm using Chrome and Brave on Windows 11.

As for how to show the details I'd be in favor of if you have the option enabled it should show on click.

[Foolishkun]

So 5f32eb1 kind of fixed it for me! With left click selected it opens and displays as it should, however if I have it on long press then it still doesn't work.

[MrKuenning]

With the card details, hold does not work for me, left click does. However left click also inserts the model and keywords into the prompt. What about making it right click instead?

[Sj-Si]

@MrKuenning

With the card details, hold does not work for me, left click does. However left click also inserts the model and keywords into the prompt. What about making it right click instead?

Yeah I just changed the default to left click. My thought process was that users with the details view enabled would probably find it useful to show details for the model they just added to the prompt. If this doesn't make sense from a user perspective then let me know.

@Foolishkun

So 5f32eb1 kind of fixed it for me! With left click selected it opens and displays as it should, however if I have it on long press then it still doesn't work.

Could you enable the Long Press setting for this then go to javascript/extraNetworks.js and add some logging in there to see what is happening?

stable-diffusion-webui/javascript/extraNetworks.js

Lines 1483 to 1494 in 5b8ed54

	function extraNetworksCardOnLongPress(event) {
	const left_click_opt = opts.extra_networks_card_details_click_behavior.toLowerCase().trim();
	// Only execute this event if user setting is "long press".
	if (left_click_opt !== "long press") {
	return;
	}
	const btn = event.target.closest(".card");
	const pane = btn.closest(".extra-network-pane");
	const tab = extra_networks_tabs[pane.dataset.tabnameFull];
	tab.showDetsView(btn);
	}

So just add something like the following

function extraNetworksCardOnLongPress(event) {
+   console.log("extraNetworksCardOnLongPress: enter");
    const left_click_opt = opts.extra_networks_card_details_click_behavior.toLowerCase().trim();
    // Only execute this event if user setting is "long press".
    if (left_click_opt !== "long press") {
+       console.log("extraNetworksCardOnLongPress: long press not enabled");
        return;
    }
    const btn = event.target.closest(".card");
    const pane = btn.closest(".extra-network-pane");
    const tab = extra_networks_tabs[pane.dataset.tabnameFull];
+   console.log("extraNetworksCardOnLongPress:", btn);
    tab.showDetsView(btn);
+   console.log("extraNetworksCardOnLongPress: leave");
}

Also add a log statement here

stable-diffusion-webui/javascript/extraNetworks.js

Lines 925 to 965 in 5b8ed54

	showDetsView(source_elem) {
	const div_dets = this.container_elem.querySelector(".extra-network-content--dets-view");
	const _popup = (msg) => {
	const elem = document.createElement("pre");
	elem.classList.add("popup-metadata");
	elem.textContent = msg;
	popup(elem);
	};
	const _clear_details = () => {
	div_dets.innerHTML = "";
	};
	const _show_details = (response) => {
	if (!isObject(response) || !isString(response.html)) {
	console.warn("Error parsing model details.");
	div_dets.innerHTML = "Error parsing model details.";
	return;
	}
	div_dets.innerHTML = response.html;
	convertElementShadowDOM(div_dets, "[data-parse-as-shadow-dom]");
	};
	_clear_details();
	requestGet(
	"./sd_extra_networks/get-model-details",
	{
	extra_networks_tabname: this.extra_networks_tabname,
	model_name: source_elem.dataset.name,
	},
	(response) => _show_details(response),
	() => _popup("Error fetching model details."),
	);
	if (this.dets_view_en) {
	this.resize_grid.toggle({elem: div_dets, override: true});
	}
	}

        const _show_details = (response) => {
            if (!isObject(response) || !isString(response.html)) {
                console.warn("Error parsing model details.");
                div_dets.innerHTML = "Error parsing model details.";
                return;
            }

            div_dets.innerHTML = response.html;
+           console.log("_show_details: response:", response.html);
            convertElementShadowDOM(div_dets, "[data-parse-as-shadow-dom]");
        };

Let me know what you see in the browser console when you try to show model details after adding these logging statements.

EDIT: I'm also interested if you get different behavior if you long press a model in the tree view (this should also show the model details just like long pressing the cards).

[Foolishkun]

@Sj-Si

Assuming these changes are correct, nothing shows in the browser console even still when long pressing on the cards.
Long pressing in the tree Does work, the detail view pops up and displays correctly.
Both left click and long press in tree give feedback in the browser console.
This was tested before 26f77b9

[Sj-Si]

@Foolishkun Inspect one of those card elements in your browser's developer tools. Should be something like <div class="card"> at the top level. When you click and hold on one of the cards, do you see the class pressed get added to the card when your mouse is down (<div class="card pressed">)?

[Foolishkun]

@Foolishkun Inspect one of those card elements in your browser's developer tools. Should be something like <div class="card"> at the top level. When you click and hold on one of the cards, do you see the class pressed get added to the card when your mouse is down (<div class="card pressed">)?

Yup it says "card pressed" when pressed until the loading bar reaches wherever your cursor is on the card, "card short-pressed" when clicked, "card short-pressed pressed" when clicked then pressed

[Sj-Si]

@Foolishkun

stable-diffusion-webui/javascript/extraNetworks.js

Lines 2040 to 2049 in 26f77b9

	const long_press_res = _get_long_press_event_elem(event);
	if (!isNullOrUndefined(long_press_res)) {
	event.stopPropagation();
	long_press_res.target.classList.add("pressed");
	extra_networks_event_long_press_timer = setTimeout(() => {
	long_press_res.target.classList.remove("pressed");
	extra_networks_event_long_press_timer = null;
	_on_long_press(event, long_press_res.target, long_press_res.handler, long_press_res.modify_classes);
	}, long_press_time_ms);
	}

Add these logging statement and let me know what it prints to the browser console when you long press a card.

        const long_press_res = _get_long_press_event_elem(event);
        if (!isNullOrUndefined(long_press_res)) {
+           console.log("setup longpress timer:", long_press_res.target);
            event.stopPropagation();
            long_press_res.target.classList.add("pressed");
            extra_networks_event_long_press_timer = setTimeout(() => {
+               console.log("inside longpress timer:", event.target);
                long_press_res.target.classList.remove("pressed");
                extra_networks_event_long_press_timer = null;
                _on_long_press(event, long_press_res.target, long_press_res.handler, long_press_res.modify_classes);
            }, long_press_time_ms);
        }

The setup longpress timer message should print as soon as you mousedown on the card. The other should print as soon as the card animation ends. In chome (idk about other browsers), you should be able to hover over the element that it prints in the console and it should highlight the card that you long pressed (if it prints the correct element). If it doesn't do this, then check to see if they have the same data attributes (like data-div-id="1").

This is what I get when I long press a card.

[Foolishkun]

@Sj-Si

The setup longpress timer message should print as soon as you mousedown on the card. The other should print as soon as the card animation ends. In chome (idk about other browsers), you should be able to hover over the element that it prints in the console and it should highlight the card that you long pressed (if it prints the correct element). If it doesn't do this, then check to see if they have the same data attributes (like data-div-id="1").

The first setup timer occurs as you said however the inside timer never popped up

[Vesperindustrial]

Minor(?) issue i'm running into is that I can't open the details pane (the one that lets you see the description, set default positive/negative prompt words, etc.) after opening it once. It seems to work one more time after switching folders and then back, and then not working again until i switch folders again and come back.

EDIT: Nevermind, seems like its an issue with the CivitAI Browser plugin adding an extra icon to get a formatted information pane from civitai. That icon only appears after clicking on the above pane icon (edit metadata) however. Disabling the civitai browser plugin makes the issue go away

[MrKuenning]

I have been getting an error saying "error fetching model details" quite often when I select various models.

This pops up even when I have card detail view turned off.

The model still adds to the prompt, and it still works to generate images. It just throws the error each time they are selected.

In the console I get the following dump:

*** API error: GET: http://10.1.1.5:7799/sd_extra_networks/get-model-details?extra_networks_tabname=lora&model_name=Style%20Of%20Peter%20Bagge%20224 {'error': 'ZeroDivisionError', 'detail': '', 'body': '', 'errors': 'division by zero'}
    Traceback (most recent call last):
      File "E:\AI\A1111-Portable Dev - Copy\webui\venv\lib\site-packages\anyio\streams\memory.py", line 98, in receive
        return self.receive_nowait()
      File "E:\AI\A1111-Portable Dev - Copy\webui\venv\lib\site-packages\anyio\streams\memory.py", line 93, in receive_nowait
        raise WouldBlock
    anyio.WouldBlock

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "E:\AI\A1111-Portable Dev - Copy\webui\venv\lib\site-packages\starlette\middleware\base.py", line 78, in call_next
        message = await recv_stream.receive()
      File "E:\AI\A1111-Portable Dev - Copy\webui\venv\lib\site-packages\anyio\streams\memory.py", line 118, in receive
        raise EndOfStream
    anyio.EndOfStream

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "E:\AI\A1111-Portable Dev - Copy\webui\modules\api\api.py", line 186, in exception_handling
        return await call_next(request)
      File "E:\AI\A1111-Portable Dev - Copy\webui\venv\lib\site-packages\starlette\middleware\base.py", line 84, in call_next
        raise app_exc
      File "E:\AI\A1111-Portable Dev - Copy\webui\venv\lib\site-packages\starlette\middleware\base.py", line 70, in coro
        await self.app(scope, receive_or_disconnect, send_no_error)
      File "E:\AI\A1111-Portable Dev - Copy\webui\venv\lib\site-packages\starlette\middleware\base.py", line 108, in __call__
        response = await self.dispatch_func(request, call_next)
      File "E:\AI\A1111-Portable Dev - Copy\webui\modules\api\api.py", line 150, in log_and_time
        res: Response = await call_next(req)
      File "E:\AI\A1111-Portable Dev - Copy\webui\venv\lib\site-packages\starlette\middleware\base.py", line 84, in call_next
        raise app_exc
      File "E:\AI\A1111-Portable Dev - Copy\webui\venv\lib\site-packages\starlette\middleware\base.py", line 70, in coro
        await self.app(scope, receive_or_disconnect, send_no_error)
      File "E:\AI\A1111-Portable Dev - Copy\webui\venv\lib\site-packages\starlette\middleware\cors.py", line 84, in __call__
        await self.app(scope, receive, send)
      File "E:\AI\A1111-Portable Dev - Copy\webui\venv\lib\site-packages\starlette\middleware\gzip.py", line 24, in __call__
        await responder(scope, receive, send)
      File "E:\AI\A1111-Portable Dev - Copy\webui\venv\lib\site-packages\starlette\middleware\gzip.py", line 44, in __call__
        await self.app(scope, receive, self.send_with_gzip)
      File "E:\AI\A1111-Portable Dev - Copy\webui\venv\lib\site-packages\starlette\middleware\exceptions.py", line 79, in __call__
        raise exc
      File "E:\AI\A1111-Portable Dev - Copy\webui\venv\lib\site-packages\starlette\middleware\exceptions.py", line 68, in __call__
        await self.app(scope, receive, sender)
      File "E:\AI\A1111-Portable Dev - Copy\webui\venv\lib\site-packages\fastapi\middleware\asyncexitstack.py", line 21, in __call__
        raise e
      File "E:\AI\A1111-Portable Dev - Copy\webui\venv\lib\site-packages\fastapi\middleware\asyncexitstack.py", line 18, in __call__
        await self.app(scope, receive, send)
      File "E:\AI\A1111-Portable Dev - Copy\webui\venv\lib\site-packages\starlette\routing.py", line 718, in __call__
        await route.handle(scope, receive, send)
      File "E:\AI\A1111-Portable Dev - Copy\webui\venv\lib\site-packages\starlette\routing.py", line 276, in handle
        await self.app(scope, receive, send)
      File "E:\AI\A1111-Portable Dev - Copy\webui\venv\lib\site-packages\starlette\routing.py", line 66, in app
        response = await func(request)
      File "E:\AI\A1111-Portable Dev - Copy\webui\venv\lib\site-packages\fastapi\routing.py", line 237, in app
        raw_response = await run_endpoint_function(
      File "E:\AI\A1111-Portable Dev - Copy\webui\venv\lib\site-packages\fastapi\routing.py", line 165, in run_endpoint_function
        return await run_in_threadpool(dependant.call, **values)
      File "E:\AI\A1111-Portable Dev - Copy\webui\venv\lib\site-packages\starlette\concurrency.py", line 41, in run_in_threadpool
        return await anyio.to_thread.run_sync(func, *args)
      File "E:\AI\A1111-Portable Dev - Copy\webui\venv\lib\site-packages\anyio\to_thread.py", line 33, in run_sync
        return await get_asynclib().run_sync_in_worker_thread(
      File "E:\AI\A1111-Portable Dev - Copy\webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 877, in run_sync_in_worker_thread
        return await future
      File "E:\AI\A1111-Portable Dev - Copy\webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 807, in run
        result = context.run(func, *args)
      File "E:\AI\A1111-Portable Dev - Copy\webui\modules\ui_extra_networks.py", line 1278, in get_model_details
        return JSONResponse({"html": page.gen_model_details_html(model_name)})
      File "E:\AI\A1111-Portable Dev - Copy\webui\modules\ui_extra_networks.py", line 1066, in gen_model_details_html
        model_specific = self.get_model_detail_extra_html(model_name)
      File "E:\AI\A1111-Portable Dev - Copy\webui\extensions-builtin\Lora\ui_extra_networks_lora.py", line 205, in get_model_detail_extra_html
        cmap_idx = math.floor((tag_count - min_tag) / (max_tag - min_tag) * (cmap.N - 1))
    ZeroDivisionError: division by zero

---

[Sj-Si]

@MrKuenning I think I've fixed your issues

[MrKuenning]

Seems to work now.

[freecoderwaifu]

Performance is still great and everything seems good. Kinda stopped testing because everything looked ok on my side and bugfixing seemed to be shifting to the mobile UI, which I couldn't really provide feedback on, so was just waiting for the merge.

I did find a couple of very small issues though.

Embeddings tab seems to have infinite scrolling, it goes back to the top after scrolling all the way down. Only seems to happen on that tab.

Recording.2024-06-22.124119.mp4

On the checkpoints tab, there's a weird issue that prevents scrolling all the way down. Only seems to happen when there are two full rows at the bottom, say:

XXXXX
XXXXX gets stuck

XXXXX
X doesn't get stuck

Untitled.mp4

Seems to be fixed by making the very bottom row uneven by disabling dir view, or also by resizing the Extra Networks frame from the bottom.

Untitled2.mp4

Only seems to happen on the Checkpoints tab, couldn't reproduce on the Lora tab. No visible errors on any console.

[Sj-Si]

@freecoderwaifu Go ahead and test it out again. I fixed a few bugs with scrolling. I think both your issues stemmed from the same problem so I'm hoping they're both fixed now. I don't think your issues were necessarily related to the specific tabs, I think it was just that you had just the right number of models in each of those individual tabs for the problem to show itself.

[freecoderwaifu]

@Sj-Si Thank you, it's fixed, everything else still looking good.

[Vesperindustrial]

Unsure how recent it is as I only noticed it yesterday but the tree view seems to...collapse its previous state when you go back to it from the txt2img tab (i.e. if you have folder A and A-A expanded and come back, it will all be collapsed down to just the root A). I didnt see any obvious settings that might cause it. I actually see the folders compact themselves back up when i flip back to the lora tab. No console errors appear and the actual card display view doesnt change from the previous selection, its only on the tree pane.

[Sj-Si]

@Vesperindustrial

Unsure how recent it is as I only noticed it yesterday but the tree view seems to...collapse its previous state when you go back to it from the txt2img tab (i.e. if you have folder A and A-A expanded and come back, it will all be collapsed down to just the root A). I didnt see any obvious settings that might cause it. I actually see the folders compact themselves back up when i flip back to the lora tab. No console errors appear and the actual card display view doesnt change from the previous selection, its only on the tree pane.

Fixed. I moved a variable around and missed it when updating references to it.

[Vesperindustrial]

@Vesperindustrial

Unsure how recent it is as I only noticed it yesterday but the tree view seems to...collapse its previous state when you go back to it from the txt2img tab (i.e. if you have folder A and A-A expanded and come back, it will all be collapsed down to just the root A). I didnt see any obvious settings that might cause it. I actually see the folders compact themselves back up when i flip back to the lora tab. No console errors appear and the actual card display view doesnt change from the previous selection, its only on the tree pane.

Fixed. I moved a variable around and missed it when updating references to it.

Seems to be fixed now, thanks!