Open
Description
Expected Behavior
WanImageToVideo should run consistently without oom (or consistently ooming I guess, but that isn't required).
Actual Behavior
I'm able to run a wan2.1_i2v_480p_14B_fp8_scaled workflow successfully on my 7900GRE setup.
However, if I change the prompt (or any pre-WanImageToVideo node) I get an persistent OOM from WanImageToVideo.
I can "fix" the issue by
- Restarting comfyui, as it won't OOM on the first run.
OR -
- Set
Load Clip/device = defaultand run (OOMs) - Set back to
Load Clip/device = cpuand run (works)
- Set
This makes me think this is a bug and the node should be able to maybe automatically recover from the OOM or avoid it somehow.
Steps to Reproduce
Run a wan2.1_i2v_480p_14B workflow, then again with a different prompt.
Debug Logs
!!! Exception during processing !!! HIP error: out of memory
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.
Traceback (most recent call last):
File "/home/alex/project/ComfyUI/execution.py", line 347, in execute
output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/alex/project/ComfyUI/execution.py", line 222, in get_output_data
return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/alex/project/ComfyUI/execution.py", line 194, in _map_node_over_list
process_inputs(input_dict, i)
File "/home/alex/project/ComfyUI/execution.py", line 183, in process_inputs
results.append(getattr(obj, func)(**inputs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/alex/project/ComfyUI/comfy_extras/nodes_wan.py", line 38, in encode
concat_latent_image = vae.encode(image[:, :, :, :3])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/alex/project/ComfyUI/comfy/sd.py", line 591, in encode
out = self.first_stage_model.encode(pixels_in).to(self.output_device).float()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/alex/project/ComfyUI/comfy/ldm/wan/vae.py", line 517, in encode
out_ = self.encoder(
^^^^^^^^^^^^^
File "/home/alex/project/ComfyUI/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/alex/project/ComfyUI/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/alex/project/ComfyUI/comfy/ldm/wan/vae.py", line 325, in forward
x = layer(x, feat_cache, feat_idx)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/alex/project/ComfyUI/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/alex/project/ComfyUI/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/alex/project/ComfyUI/comfy/ldm/wan/vae.py", line 213, in forward
x = layer(x, feat_cache[idx])
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/alex/project/ComfyUI/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/alex/project/ComfyUI/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/alex/project/ComfyUI/comfy/ldm/wan/vae.py", line 35, in forward
return super().forward(x)
^^^^^^^^^^^^^^^^^^
File "/home/alex/project/ComfyUI/comfy/ops.py", line 116, in forward
return super().forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/alex/project/ComfyUI/venv/lib/python3.12/site-packages/torch/nn/modules/conv.py", line 725, in forward
return self._conv_forward(input, self.weight, self.bias)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/alex/project/ComfyUI/venv/lib/python3.12/site-packages/torch/nn/modules/conv.py", line 720, in _conv_forward
return F.conv3d(
^^^^^^^^^
RuntimeError: HIP error: out of memory
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.Other
No response