Tensor RT

[YJonmo]

Hi there,

Thanks for this great work.

Is there any tensor RT version of the weights available?

Regards,
Jacob

[jatentaki]

Hi, thanks for the kind words. I have not created any such conversion. If I were to, how would I start? I know TensorRT is a DNN acceleration framework, but do I convert it somehow automatically or?

[YJonmo]

It could be a few lines of code depending on the model design. Sometimes it might not be straightforward, for example when using operations such as torch.einsum or have if statements in the model.

You could directly create the TensorRT version using this library.
Alternatively, you could convert the torch weights to ONNX and then to TRT format as described here.