Disk Usage at 100% / Long model load time

[lyonstyle001]

it seems after pulling the latest commit [e0543e4]
loading models is taking too long compared to before pulling this commit today

Calculating model hash: D:\AI\automatic\models\Stable-diffusion\1.5\FaeTastic.safetensors ---------- 2.1/2.1 GB 0:00:00
Loading weights: D:\AI\automatic\models\VAE\kl-f8-anime2.vae.pt -------------------------------- 404.7/404.7 MB 0:00:00
17:43:24-580155 INFO     Applying scaled dot product cross attention optimization
No Image data blocks found.
17:43:25-447901 INFO     Model loaded in 246.6s (load=227.6s config=0.5s create=0.4s hash=11.7s apply=0.4s vae=4.6s
                         move=0.6s embeddings=0.9s)

I was having a different error message that is why i did a fresh clone. just want to post this here it might be something. not really sure

[vladmandic]

new method of loading that I've added yesterday is 50% faster when loading models from storage that supports memory-mapping.
in your case, you may want to enable Settings -> Stable Diffusion -> When loading models attempt stream loading optimized for slow or network storage.

[lyonstyle001]

hi @vladmandic
even with the setting enabled im still experiencing long model load times

19:01:12-775541 INFO     Model loaded in 408.2s (load=397.7s config=0.1s create=3.0s apply=0.5s vae=4.4s
                         embeddings=2.4s)
19:01:12-922668 INFO     Startup time: 488.8s (torch=38.5s gradio=10.4s libraries=8.1s models=2.5s codeformer=0.8s
                         gfpgan=0.1s scripts=14.8s upscalers=0.1s opts onchange=0.2s ui=4.5s start=0.4s scripts
                         app_started_callback=0.1s checkpoint=408.4s)
19:01:12-927672 INFO     Memory used: 3.14 total: 15.91
19:05:05-971100 INFO     Available models: D:\AI\automatic\models\Stable-diffusion 41
No Image data blocks found.
Loading weights: D:\AI\automatic\models\Stable-diffusion\1.5\epiHyperPhotogodess-v1.safetensors ------- 2.1/2.1 0:00:00
                                                                                                        GB
19:16:15-343857 INFO     Creating model from config: D:\AI\automatic\configs\v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Calculating model hash: D:\AI\automatic\models\Stable-diffusion\1.5\epiHyperPhotogodess-v1.safetensors ---- 2.1… 0:00:…
                                                                                                            GB
Loading weights: D:\AI\automatic\models\VAE\kl-f8-anime2.vae.pt -------------------------------- 404.7/404.7 MB 0:00:00
19:16:34-746502 INFO     Applying scaled dot product cross attention optimization
No Image data blocks found.
19:16:36-021096 INFO     Model loaded in 675.8s (load=650.9s config=4.3s create=1.2s hash=14.0s apply=0.4s vae=3.7s
                         embeddings=1.3s)

i thought the model loading was stuck so i shutdown the server and then this INFO message popped up
19:16:36-021096 INFO Model loaded in 675.8s (load=650.9s config=4.3s create=1.2s hash=14.0s apply=0.4s vae=3.7s embeddings=1.3s)
675 seconds or 11 minutes is a long time loading a 2GB model.

just here to share my findings with the current version of the repo [c470f39]

[vladmandic]

there is something terribly wrong there.

• just looking at load times vs hash times (hash only occurs once for each model and then its stored), they both read entire file
  and hash was created in ~14sec (which is ok) while model load took 650sec
• config is normally so fast it doesn't even show up (events that take less than 0.1sec are not shown in timing summary)

what is your hardware platform?
any information about storage you use?
can you check windows event logs for anything suspicious?
what is the cpu usage and disk usage (screenshot task manager -> performance for both)?

[lyonstyle001]

hi apologize for interrupting your MiamiGP 😁
but good news is, I think i managed to fix it

20:20:34-982551 INFO     Available models: D:\AI\automatic\models\Stable-diffusion 41
No Image data blocks found.
Loading weights: D:\AI\automatic\models\Stable-diffusion\1.5\Deliberate-v2.safetensors ------------- 0.0/2.1 GB -:--:--
20:21:21-635073 INFO     Creating model from config: D:\AI\automatic\configs\v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Calculating model hash: D:\AI\automatic\models\Stable-diffusion\1.5\Deliberate-v2.safetensors -------- 2.1/2.1  0:00:00
                                                                                                       GB
Loading weights: D:\AI\automatic\models\VAE\kl-f8-anime2.vae.pt ---------------------- --------- 283.0/404.7 MB 0:00:00
20:21:28-590040 INFO     Applying scaled dot product cross attention optimization
No Image data blocks found.
20:21:29-332177 INFO     Model loaded in 50.5s (load=2.5s config=40.3s create=0.6s hash=1.5s apply=0.4s vae=4.3s
                         embeddings=0.7s)
Progress 4.48it/s ---------------------------------------- 100% 0:00:00 0:00:04
Progress 5.06it/s ---------------------------------------- 100% 0:00:00 0:00:04
Progress 5.04it/s ---------------------------------------- 100% 0:00:00 0:00:04
Progress 4.95it/s ---------------------------------------- 100% 0:00:00 0:00:05
Progress 0.93it/s ---------------------------------------- 100% 0:00:00 0:00:26
Progress 5.02it/s ---------------------------------------- 100% 0:00:00 0:00:04
Progress 0.94it/s ---------------------------------------- 100% 0:00:00 0:00:26
20:25:43-291049 INFO     Available models: D:\AI\automatic\models\Stable-diffusion 41
No Image data blocks found.
Loading weights: D:\AI\automatic\models\Stable-diffusion\1.5\Niji3DStyle-v10.safetensors ----------- 0.0/4.3 GB -:--:--
20:26:19-399025 INFO     Creating model from config: D:\AI\automatic\configs\v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Calculating model hash: D:\AI\automatic\models\Stable-diffusion\1.5\Niji3DStyle-v10.safetensors ------- 4.3/4.3 0:00:00
                                                                                                        GB
Loading weights: D:\AI\automatic\models\VAE\kl-f8-anime2.vae.pt ---------------------- --------- 283.0/404.7 MB 0:00:00
20:28:21-415204 INFO     Applying scaled dot product cross attention optimization
No Image data blocks found.
20:28:22-181361 INFO     Model loaded in 130.5s (load=2.7s config=5.0s create=0.4s hash=23.8s apply=93.5s vae=4.2s
                         embeddings=0.8s)

what i did was i deleted the venv folder and have it reinstall everything using webui.bat, did a gitpull (which is up to date) for safe measure.
definitely a weird occurance. but then i should atleast let you know aswell.

enjoy your vacay @vladmandic