new mmdetection env converted trt engine inference error

[lucasjinreal]

Hi, I have a model converted last month, it was works ok, recently update mmdeteciton, the inference were broken, I got some error like this:

plugins/src/plugin/batchedNMSPlugin/batchedNMSPlugin.cpp,127
[1]    636007 IOT instruction (core dumped) 

Any clue on why this thing happen?

[grimoire]

Can you provide more details about the enviroment? You can use this tool to collect the enviroment.

[lucasjinreal]

@grimoire thanks, here is the env output:

Collecting environment information...
PyTorch version: 1.11.0.dev20220124+cu115
Is debug build: False
CUDA used to build PyTorch: 11.5
OS: Ubuntu 21.10 (x86_64)
GCC version: (Ubuntu 11.2.0-7ubuntu2) 11.2.0
Clang version: Could not collect
CMake version: version 3.22.2
Libc version: glibc-2.34
Python version: 3.9.7 (default, Sep 16 2021, 13:09:58)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.13.0-28-generic-x86_64-with-glibc2.34
Is CUDA available: True
CUDA runtime version: 11.5.119
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 2060 SUPER
Nvidia driver version: 470.103.01
cuDNN version: Probably one of the following:
/usr/local/cuda-11.5/targets/x86_64-linux/lib/libcudnn.so.8
/usr/local/cuda-11.5/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8
/usr/local/cuda-11.5/targets/x86_64-linux/lib/libcudnn_adv_train.so.8
/usr/local/cuda-11.5/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8
/usr/local/cuda-11.5/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8
/usr/local/cuda-11.5/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8
/usr/local/cuda-11.5/targets/x86_64-linux/lib/libcudnn_ops_train.so.8
Versions of relevant libraries:
[pip3] mmcv==1.4.4
[pip3] mmcv-full==1.4.4
[pip3] mmdet==2.21.0
[pip3] mmdet2trt==0.5.0
[pip3] pytorch-lightning==1.5.10
[pip3] pytorch3d==0.6.1
[pip3] tensorrt==8.2.3.0
[pip3] torch==1.11.0.dev20220124+cu115
[pip3] torch2trt-dynamic==0.5.0
[pip3] torchaudio==0.11.0.dev20220124+cu115
[pip3] torchfile==0.1.0
[pip3] torchmetrics==0.7.2
[pip3] torchvision==0.12.0.dev20220124+cu115
[conda] Could not collect

[grimoire]

Model conversion works fine on latest mmdetection on my device.
I am trying to build a similar enviroment that you provided. It might take some time. What have you change since last time successful conversion?