Which export format is fastest for inferencing on Nvidia GPUs?

[haimat]

I have just started out with Anomalib, thanks for the great work on this guys. Our main task is to run fast inferencing on Nvidia GPUs. I have seen there are three export options for trainied models - OpenVino, ONNX, Torch. In the context of inference times on the GPU, which of these three peforms best?

[samet-akcay]

Hi @haimat, if you are using NVIDIA GPU, you may not even need to export the model and use Engine directly, which also has predict method for inference tasks. Something like Engine.predict(..) after the train and test. This entrypoint uses a Lightning model under the hood. If you want a raw Torch model without Lightning or any other dependencies, you could use the TorchInferencer.

OpenVINO inferencer becomes handy especially when you have an edge device or CPU for inference. It accelerates the inference speed on these hardware.

Hope this makes sense. If not, feel free to shoot some other questions :)

[haimat]

@samet-akcay Thanks for your response.
What if I don't want to infer not directly after training, but in another application?
Would it be faster to export the trained model to ONNX and run it via onnx-gpu -
or is it faster to export to Torch and use the TorrchInferencer with the "GPU" flag?