Add support for TensorRT with future Diffusers?

[neurogen-dev]

I take it this PR adds support for TensotRT to the diffusers? Will it be possible to add TensorRT support to Automatic, after the new diffusers with this PR are released?

huggingface/diffusers#2974

[vladmandic]

possibly, will know more after some studying. but note that tensorrt doesn't work with standard models - model itself needs to be recompiled which does take a while.

[myndxero]

On my 4090 I can saturate it with TensorRT and have gotten over 90 it/s with batchsize=1. About .25 seconds to generate an image.
I should note that I have to push my i9-13900K to its limit by killing my chrome browsers. That let me go from 5.5 GHz to over 5.8.

Jealous of your 4090, only 3090 here, but I too have the i9 13900!

[DuckersMcQuack]

possibly, will know more after some studying. but note that tensorrt doesn't work with standard models - model itself needs to be recompiled which does take a while.

I understand, I was testing a TensorRT based on Lsmith. But the performance gain is very decent. 50-55 it/s vs 28 it/s on wsl2 with RTX 4090. 512x512, batch size 1, euler a

What environment did you get it to work in? I could not for the life of me get it to even install/launch. Nor sadly did nerdyrodent nor aitrepenour have any video on it, and would have loved to try it with my 3090.

[DuckersMcQuack]

possibly, will know more after some studying. but note that tensorrt doesn't work with standard models - model itself needs to be recompiled which does take a while.

I do not mind converting my favorite models to a tensorRT optimized format. If say there was a way to queue models, i could just have it convert overnight for X amount of models.

[Tillerz]

Does that conversion only work on Linux where you can 'compile' models? If so, is it worth it? I thought about running a virtual linux box on my PC just for automatic1111.

[mobilehacker]

I have successfully used auto-like interface called Radiata, with tensor rt on rtx3060 on windows 11. Performance was super great, and models recompile for tensors in about 1 hour. I really hope to get tensor rt support soon!