performance regression with TensorRT can anybody help understand nsight?

[ninono12345]

Hello everyone

I am working on a pytorch object tracking model to convert it to tensorrt for faster inference

When inferencing tensorrt with a single batch the model is about 2x faster, but when adding batches, it becomes SLOWER

batch of 1 inference time:
pytorch - 40ms
tensorrt - 20ms

batch of 8 inference time:
pytorch - 50ms
tensorrt - 85ms

When adding batches the inference time on pytorch pretty much doesn't increase, but inferencing time with tensorrt engine increases significantly!!!

Can anybody help, why is the speed regression happening?

I have exported the model with batch size 1 and batch size 4 to nsight systems:

1 batch inference test: https://drive.google.com/file/d/1achvISpSc1pvlV2RLfSNLxCLlRsZHcnT/view?usp=sharing
4 batch inference test: https://drive.google.com/file/d/1ZuHsO28LIlETNIcWk6lh7miv2Lovco9D/view?usp=sharing

Can anybody with experience in Nvidia NSight help me understand these graphs and compare them to know where and why the performance reggression is happening?

Thank you