TensorRT C++ API Tutorial

How to use TensorRT C++ API for high performance GPU inference.
A Venice Computer Vision Presentation

Video Presentation . Presentation Slides · Venice Computer Vision

TensorRT C++ Tutorial

This project demonstrates how to use the TensorRT C++ API for high performance GPU inference. It covers how to do the following:

• How to install TensorRT on Ubuntu 20.04
• How to generate a TRT engine file optimized for your GPU
• How to specify a simple optimization profile
• How to read / write data from / into GPU memory and work with GPU images.
• How to use cuda stream to run async inference and later synchronize.
• How to work with models with static and dynamic batch sizes.
• New: Supports models with multiple outputs (and even works with batching!).
• New: Supports models with multiple inputs.
• The code can be used as a base for many models, including Insightface ArcFace, YoloV7, SCRFD face detection, and many other single / multiple input - single / multiple output models. You will just need to implement the appropriate post-processing code.

Getting Started

The following instructions assume you are using Ubuntu 20.04. You will need to supply your own onnx model for this sample code, or you can download the sample model (see Sanity Check section below). Ensure to specify a dynamic batch size when exporting the onnx model if you would like to use batching. If not, you will need to set Options.doesSupportDynamicBatchSize to false.

Prerequisites

• Install OpenCV with cuda support. Instructions can be found here.
• sudo apt install build-essential
• sudo apt install python3-pip
• pip3 install cmake
• Download TensorRT from here.
• Extract, and then navigate to the CMakeLists.txt file and replace the TODO with the path to your TensorRT installation.

Building the library

• mkdir build && cd build
• cmake ..
• make -j$(nproc)

Sanity check

• To perform a sanity check, download the following ArcFace model from here and place it in the ./models directory.
• Make sure Options.doesSupportDynamicBatchSize is set to false before passing it the Options to the Engine constructor on this line.
• Uncomment the code for printing out the feature vector at the bottom of ./src/main.cpp.
• Running inference using said model and the image located in inputs/face_chip.jpg should produce the following feature vector:

-0.0548096 -0.0994873 0.176514 0.161377 0.226807 0.215942 -0.296143 -0.0601807 0.240112 -0.18457 ...

How to debug

• If you have having issues creating the TensorRT engine file from the onnx model, I would advise using the trtexec command line tool (comes packaged in the TensorRT download bundle in the /bin directory). It will provide you with more debug information.

Changelog

V2.1

• Added support for models with multiple inputs. Implementation now supports models with single inputs, multiple inputs, single outputs, multiple outputs, and batching.

V2.0

• Requires OpenCV cuda to be installed. To install, follow instructions here.
• Options.optBatchSizes has been removed, replaced by Options.optBatchSize.
• Support models with more than a single output (ex. SCRFD).
• Added support for models which do not support batch inference (first input dimension is fixed).
• More error checking.
• Fixed a bunch of common issues people were running into with the original V1.0 version.
• Remove whitespace from GPU device name