User allocator error allocating 86114304000-byte buffer failure of TensorRT 10.6 when running demo_img2vid. py on GPU rtx4090

[kolyh]

Description

I tried to convert onnx to an engine file using demo_img2vid. py on rtx4090，but received the following error message：

[E] [defaultAllocator.cpp::allocate::31] Error Code 1: Cuda Runtime (out of memory)

[E] Error Code: 9: Skipping tactic 0x0000000000000000 due to exception [tunable_graph.cpp:create:117] autotuning: User allocator error allocating 86114304000-byte buffer

[E] [virtualMemoryBuffer.cpp::resizePhysical::154] Error Code 2: OutOfMemory (Requested size was 16581132288 bytes.)

[E] [virtualMemoryBuffer.cpp::resizePhysical::154] Error Code 2: OutOfMemory (Requested size was 16157507584 bytes.)

[E] [defaultAllocator.cpp::allocate::31] Error Code 1: Cuda Runtime (out of memory)

[E] Error Code: 9: Skipping tactic 0x0000000000000000 due to exception [tunable_graph.cpp:create:117] autotuning: User allocator error allocating 86114304000-byte buffer

[E] [virtualMemoryBuffer.cpp::resizePhysical::154] Error Code 2: OutOfMemory (Requested size was 16286482432 bytes.)

[E] [virtualMemoryBuffer.cpp::resizePhysical::154] Error Code 2: OutOfMemory (Requested size was 15862857728 bytes.)

[E] [defaultAllocator.cpp::allocate::31] Error Code 1: Cuda Runtime (out of memory)

[E] Error Code: 9: Skipping tactic 0x0000000000000000 due to exception [tunable_graph.cpp:create:117] autotuning: User allocator error allocating 11206656000-byte buffer

[E] [defaultAllocator.cpp::allocate::31] Error Code 1: Cuda Runtime (out of memory)

[E] Error Code: 9: Skipping tactic 0x0000000000000000 due to exception [tunable_graph.cpp:create:117] autotuning: User allocator error allocating 11206656000-byte buffer

[E] [virtualMemoryBuffer.cpp::resizePhysical::154] Error Code 2: OutOfMemory (Requested size was 12238979072 bytes.)

[E] [virtualMemoryBuffer.cpp::resizePhysical::154] Error Code 2: OutOfMemory (Requested size was 12091129856 bytes.)

[E] [virtualMemoryBuffer.cpp::resizePhysical::154] Error Code 2: OutOfMemory (Requested size was 12681478144 bytes.)

[E] [virtualMemoryBuffer.cpp::resizePhysical::154] Error Code 2: OutOfMemory (Requested size was 12460228608 bytes.)

[E] [virtualMemoryBuffer.cpp::resizePhysical::154] Error Code 2: OutOfMemory (Requested size was 12091129856 bytes.)

[E] [defaultAllocator.cpp::allocate::31] Error Code 1: Cuda Runtime (out of memory)

[E] Error Code: 9: Skipping tactic 0x0000000000000000 due to exception [tunable_graph.cpp:create:117] autotuning: User allocator error allocating 11206656000-byte buffer

[E] [defaultAllocator.cpp::allocate::31] Error Code 1: Cuda Runtime (out of memory)

[E] Error Code: 9: Skipping tactic 0x0000000000000000 due to exception [tunable_graph.cpp:create:117] autotuning: User allocator error allocating 11206656000-byte buffer

[E] [defaultAllocator.cpp::allocate::31] Error Code 1: Cuda Runtime (out of memory)

[E] Error Code: 9: Skipping tactic 0x0000000000000000 due to exception [tunable_graph.cpp:create:117] autotuning: User allocator error allocating 11206656000-byte buffer

[E] [defaultAllocator.cpp::allocate::31] Error Code 1: Cuda Runtime (out of memory)

[E] Error Code: 9: Skipping tactic 0x0000000000000000 due to exception [tunable_graph.cpp:create:117] autotuning: User allocator error allocating 86114304000-byte buffer

[E] [virtualMemoryBuffer.cpp::resizePhysical::154] Error Code 2: OutOfMemory (Requested size was 17466130432 bytes.)

[E] [virtualMemoryBuffer.cpp::resizePhysical::154] Error Code 2: OutOfMemory (Requested size was 17042505728 bytes.)

[E] [defaultAllocator.cpp::allocate::31] Error Code 1: Cuda Runtime (out of memory)

[E] Error Code: 9: Skipping tactic 0x0000000000000000 due to exception [tunable_graph.cpp:create:117] autotuning: User allocator error allocating 86114304000-byte buffer

[E] [virtualMemoryBuffer.cpp::resizePhysical::154] Error Code 2: OutOfMemory (Requested size was 17466130432 bytes.)

[E] [virtualMemoryBuffer.cpp::resizePhysical::154] Error Code 2: OutOfMemory (Requested size was 17042505728 bytes.)

[E] [defaultAllocator.cpp::allocate::31] Error Code 1: Cuda Runtime (out of memory)

[E] Error Code: 9: Skipping tactic 0x0000000000000000 due to exception [tunable_graph.cpp:create:117] autotuning: User allocator error allocating 86114304000-byte buffer

[E] [virtualMemoryBuffer.cpp::resizePhysical::154] Error Code 2: OutOfMemory (Requested size was 16286482432 bytes.)

How should I fix this error？

Environment

docker：nvcr.io/nvidia/pytorch:24.07-py3

TensorRT Version: 10.6

NVIDIA GPU: rtx4090

NVIDIA Driver Version:NVIDIA-SMI 535.161.07

CUDA Version:12.2

CUDNN Version:

Operating System:

Python Version (if applicable): 3.10

Tensorflow Version (if applicable):

PyTorch Version (if applicable):2.4.0a0+3bcc3cddb5.nv24.7

Baremetal or Container (if so, version):

Steps To Reproduce

docker run --rm -it --gpus all -v $PWD:/workspace nvcr.io/nvidia/pytorch:24.07-py3 /bin/bash
python3 -m pip install --upgrade pip
pip install --pre tensorrt-cu12
export TRT_OSSPATH=/workspace
cd $TRT_OSSPATH/demo/Diffusion
pip3 install -r requirements.txt
python3 demo_img2vid.py --version svd-xt-1.1 --onnx-dir onnx-svd-xt-1-1 --engine-dir engine-svd-xt-1-1 --hf-token=$HF_TOKEN

[lix19937]

86114304000-byte means 80.2GB, overrange of rtx4090 gpu memory( 24GB ). You can try to assign --height and --width with a smaller value.

[kolyh]

86114304000-byte means 80.2GB, overrange of rtx4090 gpu memory( 24GB ). You can try to assign --height and --width with a smaller value.

I made the same attempt on A100, using a resolution of 1280 * 1280.
But the same error occurred
Is 1280 * 1280 a large resolution?

[lix19937]

When you load engine, ou can watch nvidia-smi to see gpu mem state.

[poweiw]

Without knowing the size of your model it's hard to estimate how much vram you might need to use, but as @lix19937 said you can try to use a smaller frame size or also try --fp8 or --int8 for a smaller precision. cc @asfiyab-nvidia

[tppqt]

Without knowing the size of your model it's hard to estimate how much vram you might need to use, but as @lix19937 said you can try to use a smaller frame size or also try --fp8 or --int8 for a smaller precision. cc @asfiyab-nvidia

I've run into this problem too, it's a Tensorrt model for SVD-XT-1-1, over 2GB, its ontology is a small .onnx and then there's a data that's supposed to be an onnx, running the onnx to tensorrt code reports insufficient GPU allocation, and it's modeled as follows
https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt-1-1-tensorrt/tree/main/unet-temp.opt

[tppqt]

it's in demo->Diffusion->demo_img2vid.py
I run:
python3 demo_img2vid.py --version svd-xt-1.1 --onnx-dir onnx-svd-xt-1-1 --engine-dir engine-svd-xt-1-1 --input-image https://www.hdcarwallpapers.com/walls/2018_chevrolet_camaro_zl1_nascar_race_car_2-HD.jpg
and I got this:
[E] [defaultAllocator.cpp::allocate::19] Error code 1: cuda Runtime (out of memory)TE] Error code: 9: skipping tactic 0x0000000000000000 due to exception [tunablegraph.cpp:create:116] autotuning: user allocator error allocating 86114305024-byte buffer
[E] [virtualMemoryBuffer.cpp::resizePhysical::140] Error code 2: out of Memory (Requested size was 22942842880 bytes.)[E] [virtualMemoryBuffer.cpp::resizePhysical::140] Error code 2: out of Memory (Requested size was 23532142592 bytes.)[E] [defaultAllocator.cpp::allocate::19] Error code 1: cuda Runtime (out of memory)TE] Error code: 9: skipping tactic 0x0000000000000000 due to exception [tunablegraph.cpp:create:116] autotuning: User allocator error allocating 86114305024-byte buffer
[E][virtualMemoryBuffer.cpp::resizePhysical::140] Error code 2: 0utofMemory (Requested size was 22942842880 bytes.)

what can I do?

[lix19937]

see https://github.com/NVIDIA/TensorRT/blob/release/10.7/demo/Diffusion/demo_img2vid.py#L34-L35 and https://github.com/NVIDIA/TensorRT/blob/release/10.7/demo/Diffusion/demo_img2vid.py#L66 user can assign some para.

[praveenperfecto]

[E] [defaultAllocator.cpp::allocate::28] Error Code 1: Cuda Runtime (out of memory)
[E] Error Code: 9: Skipping tactic 0x0000000000000000 due to exception [tunable_graph.cpp:create:123] autotuning: User allocator error allocating 86114304000-byte buffer
[E] [defaultAllocator.cpp::allocate::28] Error Code 1: Cuda Runtime (out of memory)
[E] Error Code: 9: Skipping tactic 0x0000000000000000 due to exception [tunable_graph.cpp:create:123] autotuning: User allocator error allocating 86114304000-byte buffer

the issue persist?