how to run it on low v-ram systems ? #269

usernamex1x23 · 2023-04-20T06:39:25Z

usernamex1x23
Apr 20, 2023

on the other webui i just add the line
set COMMANDLINE_ARGS= --opt-split-attention --precision full --no-half --lowvram --always-batch-cond-uncond
in the webui-user.bat file
after which it manages to run on low v-ram its slow but it runs.
any beginners guide to achieve this on this system ?

hollowstrawberry · 2023-04-20T07:07:51Z

hollowstrawberry
Apr 20, 2023

--lowvram should be enough, and --precision full --no-half only if you're on AMD graphics card. I think your have to set the other options from within the webui settings.

4 replies

iDeNoh Apr 20, 2023

--precision-full and --no-half are only necessary on pre rdna 2 cards.

pc-v2 Apr 30, 2023

--precision-full and --no-half are only necessary on pre rdna 2 cards.

I can't run it with that args, I am using 5600xt

pc-v2 Apr 30, 2023

--precision-full and --no-half are only necessary on pre rdna 2 cards.

I can't run it with that args, I am using 5600xt

edit: delete --precision-full

here's my webui-user.bat copied and edited for this version

@echo off

set PYTHON=
set GIT=
set VENV_DIR=
set COMMANDLINE_ARGS= --lowvram --no-half --autolaunch

call webui.bat

and the console output

Using VENV: D:\Portable Apps\stable-diffusion-vladmandic\automatic\venv
20:40:06-618560 INFO     Python 3.10.6 on Windows
20:40:06-929612 INFO     Version: b23b6a6e Sun Apr 30 08:55:44 2023 -0400
20:40:07-445527 INFO     Latest published version: 917ecad43c4da4f3aad54048284ad0ee44a04634 2023-04-30T13:31:38Z
20:40:07-446528 INFO     Setting environment tuning
20:40:07-460541 INFO     Using DirectML Backend
20:40:07-462543 INFO     Installing package: torch==2.0.0 torchvision torch-directml
20:40:10-955742 INFO     Torch 2.0.0+cpu
20:40:10-966752 INFO     Torch backend: DirectML (torch-directml 0.2.0.dev230426)
20:40:11-025571 INFO     Torch detected GPU: AMD Radeon RX 5600 XT
20:40:11-027574 INFO     DirectML default device: AMD Radeon RX 5600 XT
20:40:11-028575 INFO     Verifying requirements
20:40:11-087628 INFO     No changes detected: Quick launch active
20:40:11-093634 INFO     Running extension preloading
20:40:11-094635 INFO     Server arguments: ['--lowvram', '--no-half', '--autolaunch']
No module 'xformers'. Proceeding without it.
usage: launch.py [-h] [--medvram] [--lowvram] [--lowram] [--ckpt CKPT] [--vae VAE] [--data-dir DATA_DIR] [--models-dir MODELS_DIR] [--allow-code] [--share] [--enable-insecure]
                 [--use-cpu USE_CPU [USE_CPU ...]] [--listen] [--port PORT] [--hide-ui-dir-config] [--freeze-settings] [--gradio-auth GRADIO_AUTH] [--gradio-auth-path GRADIO_AUTH_PATH] [--autolaunch]
                 [--disable-console-progressbars] [--disable-safe-unpickle] [--api-auth API_AUTH] [--api-log] [--device-id DEVICE_ID] [--cors-origins CORS_ORIGINS] [--cors-regex CORS_REGEX]
                 [--tls-keyfile TLS_KEYFILE] [--tls-certfile TLS_CERTFILE] [--server-name SERVER_NAME] [--no-hashing] [--no-download-sd-model] [--profile] [--disable-queue] [--reset] [--upgrade]
                 [--noupdate] [--nodirectml] [--skip-requirements] [--skip-extensions] [--skip-git] [--experimental] [--test] [--lyco-dir LYCO_DIR] [--controlnet-dir CONTROLNET_DIR]
                 [--controlnet-annotator-models-path CONTROLNET_ANNOTATOR_MODELS_PATH] [--no-half-controlnet] [--debug]
launch.py: error: argument --no-half: expected one argument
Press any key to continue . . .

vladmandic Apr 30, 2023
Maintainer

--precision-full and --no-half are no longer cmd flags, most of those have been moved to UI -> Settings -> CUDA

vladmandic · 2023-04-20T11:21:12Z

vladmandic
Apr 20, 2023
Maintainer

re: webui-user.bat, there are 100s different ways to write it, just create your own.
creating one with a single line like this is enough, really no need to complicate in that file.
@webui.bat --lowvram
but yes, you can still use COMMANDLINE_ARGS as well if you want.

2 replies

pc-v2 Apr 30, 2023

how to make it run with more than one args?

20:41:30-797057 INFO     Python 3.10.6 on Windows
20:41:31-104699 INFO     Version: b23b6a6e Sun Apr 30 08:55:44 2023 -0400
20:41:31-684661 INFO     Latest published version: 917ecad43c4da4f3aad54048284ad0ee44a04634 2023-04-30T13:31:38Z
20:41:31-685662 INFO     Setting environment tuning
20:41:31-698674 INFO     Using DirectML Backend
20:41:31-700675 INFO     Installing package: torch==2.0.0 torchvision torch-directml
20:41:35-091312 INFO     Torch 2.0.0+cpu
20:41:35-102322 INFO     Torch backend: DirectML (torch-directml 0.2.0.dev230426)
20:41:35-178102 INFO     Torch detected GPU: AMD Radeon RX 5600 XT
20:41:35-179103 INFO     DirectML default device: AMD Radeon RX 5600 XT
20:41:35-180104 INFO     Verifying requirements
20:41:35-244163 INFO     No changes detected: Quick launch active
20:41:35-250168 INFO     Running extension preloading
20:41:35-252170 INFO     Server arguments: ['--lowvram', '--no-half']
No module 'xformers'. Proceeding without it.
usage: launch.py [-h] [--medvram] [--lowvram] [--lowram] [--ckpt CKPT] [--vae VAE] [--data-dir DATA_DIR] [--models-dir MODELS_DIR] [--allow-code] [--share] [--enable-insecure]
                 [--use-cpu USE_CPU [USE_CPU ...]] [--listen] [--port PORT] [--hide-ui-dir-config] [--freeze-settings] [--gradio-auth GRADIO_AUTH] [--gradio-auth-path GRADIO_AUTH_PATH] [--autolaunch]
                 [--disable-console-progressbars] [--disable-safe-unpickle] [--api-auth API_AUTH] [--api-log] [--device-id DEVICE_ID] [--cors-origins CORS_ORIGINS] [--cors-regex CORS_REGEX]
                 [--tls-keyfile TLS_KEYFILE] [--tls-certfile TLS_CERTFILE] [--server-name SERVER_NAME] [--no-hashing] [--no-download-sd-model] [--profile] [--disable-queue] [--reset] [--upgrade]
                 [--noupdate] [--nodirectml] [--skip-requirements] [--skip-extensions] [--skip-git] [--experimental] [--test] [--lyco-dir LYCO_DIR] [--controlnet-dir CONTROLNET_DIR]
                 [--controlnet-annotator-models-path CONTROLNET_ANNOTATOR_MODELS_PATH] [--no-half-controlnet] [--debug]
launch.py: error: argument --no-half: expected one argument
Press any key to continue . . .

pc-v2 Apr 30, 2023

never mind, --lowvram is enough

qpl23 · 2023-04-20T17:45:48Z

qpl23
Apr 20, 2023

Running on an AMD RX Vega 64 (8GB vram) I found initially I couldn't upscale anything much beyond the default 512x512 before getting OOM errors.

I've done the following things, which have allowed me to upscale to 1200x720 in img-to-img:

• followed info I saw in a discussion somewhere here, to manually install pytorch 2.1 nightly + rocm in venv with

source venv/bin/activate
pip install --upgrade --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/rocm5.4.2

(may have manually uninstalled existing pytorch previously)

• manually removed xformers in venv with pip uninstall

• commented out code that tries to install it

• commented out other code that might try installing things at startup, like the setup.run_setup() and setup.check_torch() invocations in launch.py (only after an initial run which did those setup() calls, of course)

• added export PYTORCH_HIP_ALLOC_CONF=garbage_collection_threshold:0.6,max_split_size_mb:32 at the top of webui.sh

• configured "upcast to float 32", "sub-quadratic" cross-attention optimization and "SDP disable memory attention" in Stable Diffusion settings section

Interestingly, I tried enabling the tiled VAE option in "from image" tab, and without the "sub-quadratic" setting this gave an improvement.

After enabling sub-quadratic, checking tiled VAE produces a stacktrace ending with:

│ /tile_utils/attn.py:185 in sub_quad_attnblock_forward                                          │
│                                                                                                │
│   184 │   v = v.contiguous()                                                                   │
│ ❱ 185 │   out = sub_quad_attention(q, k, v, q_chunk_size=shared.cmd_opts.sub_quad_q_chunk_size │
│   186 │   out = rearrange(out, 'b (h w) c -> b c h w', h=h) 

AttributeError: 'Namespace' object has no attribute 'sub_quad_q_chunk_size'

Since sub-quadratic gave the bigger improvement, I've gone with that.

• got used to using the "unload checkpoint" and "reload checkpoint" buttons in settings tab before doing any upscaling - I think this might be down to pytorch 2.1 nightly sometimes leaving the checkpoint loaded when it shouldn't and so wasting VRAM.

Please note, this is all voodoo - I have no real idea what I'm doing. I don't know enough about python, pytorch, diffusion models or graphics cards to do better. Like this, I can scale an image up to 1200x720 on "from image" tab which was impossible for me previously, so maybe it helps someone. Alternatively, maybe someone can help me :)

I can well imagine the horror a developer might feel seeing advice to trample all over their code like this, so I don't advise or recommend it, I just mention what I've done. Also, I have yet to get around to trying the latest install options provided which might well avoid some manual steps here.

I should mention this is a lot better than results I got from the parent repo and from comfy, both of which consistently caused an immediate instant hard reset on reaching the VAE stage once I coaxed them into using the GPU at all. I much appreciate the work by the developer!

5 replies

vladmandic Apr 20, 2023
Maintainer

this is a pretty good start.
and generating images works without issues?
anyone else with amd can confirm this settings?

qpl23 Apr 20, 2023

generating images works without issues?

Apart from user issues, as far as I can tell, yes :)

vladmandic Apr 20, 2023
Maintainer

ok, one unrelated question - what is the amd cli tool that amd users can execute to get info and can you provide examples?
for nvidia, its nvidia-smi
i cannot rely on torch to tell me anything since i want info to determine which torch to install in the first place :)

qpl23 Apr 20, 2023

I guess rocm-smi?

There's a lot of command options, as you can see from the link. If I run it with no args after unloading/reloading checkpoint via settings tab buttons:

======================= ROCm System Management Interface =======================
================================= Concise Info =================================
GPU  Temp (DieEdge)  AvgPwr  SCLK    MCLK    Fan    Perf  PwrCap  VRAM%  GPU%  
0    33.0c           8.0W    852Mhz  167Mhz  9.41%  auto  220.0W   30%   0%    
================================================================================
============================= End of ROCm SMI Log ==============================

While generating 640x480 image, UniPC, 10 steps:

======================= ROCm System Management Interface =======================
================================= Concise Info =================================
GPU  Temp (DieEdge)  AvgPwr  SCLK     MCLK    Fan    Perf  PwrCap  VRAM%  GPU%  
0    39.0c           220.0W  1630Mhz  167Mhz  10.2%  auto  220.0W   40%   83%   
================================================================================
============================= End of ROCm SMI Log ==============================

...

======================= ROCm System Management Interface =======================
================================= Concise Info =================================
GPU  Temp (DieEdge)  AvgPwr  SCLK    MCLK    Fan     Perf  PwrCap  VRAM%  GPU%  
0    42.0c           30.0W   852Mhz  167Mhz  13.73%  auto  220.0W   50%   0%    
================================================================================
============================= End of ROCm SMI Log ==============================

After generation:

======================= ROCm System Management Interface =======================
================================= Concise Info =================================
GPU  Temp (DieEdge)  AvgPwr  SCLK    MCLK    Fan     Perf  PwrCap  VRAM%  GPU%  
0    39.0c           9.0W    852Mhz  167Mhz  15.29%  auto  220.0W   34%   0%    
================================================================================
============================= End of ROCm SMI Log ==============================

Note: VRAM at 34%, not quite back to initial 30%!

There's also rocminfo giving gpu model info etc.

If you want some further detailed testing on this hardware, I'm up for it, but have to leave it here for now.

iDeNoh Apr 21, 2023

I'm unable to run rocm-smi, I wonder if that may be platform specific, or specific to a version of rocm

xDepredadorxD · 2023-05-01T09:24:00Z

xDepredadorxD
May 1, 2023

create a file namex.bat

@echo off

set PYTHON="C:\Users\Admin\AppData\Local\Programs\Python\Python310\python.exe"
set GIT=
set VENV_DIR=
set COMMANDLINE_ARGS=--lowvram --no-half-controlnet
git pull
call webui.bat

1 reply

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

how to run it on low v-ram systems ? #269

{{title}}

Replies: 4 comments 12 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

how to run it on low v-ram systems ? #269

Replies: 4 comments · 12 replies

vladmandic Apr 30, 2023 Maintainer

vladmandic Apr 20, 2023 Maintainer

vladmandic Apr 20, 2023 Maintainer

vladmandic Apr 20, 2023 Maintainer

Replies: 4 comments 12 replies

vladmandic Apr 30, 2023
Maintainer

vladmandic
Apr 20, 2023
Maintainer

vladmandic Apr 20, 2023
Maintainer

vladmandic Apr 20, 2023
Maintainer