Troubleshoot mistral model

[krishnanpooja]

System Info

GPU-A100
TensorRT-LLM: 0.15.0

Who can help?

@ncomly-nvidia @byshiue

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Run the steps mentioned under https://nvidia.github.io/TensorRT-LLM/reference/troubleshooting.html#debug-on-e2e-models

to debug the output from the following model - https://github.com/huggingface/transformers/blob/5d7739f15a6e50de416977fe2cc9cb516d67edda/src/transformers/models/mistral/modeling_mistral.py#L1015

Expected behavior

easy debugging of new models

actual behavior

no debugging setup available

additional notes

I need to debug the reason for reduction in accuracy of Biomistral model (https://github.com/huggingface/transformers/blob/5d7739f15a6e50de416977fe2cc9cb516d67edda/src/transformers/models/mistral/modeling_mistral.py#L1015) on TensorRT-Engine compared to vLLM.
Looking at the instructions in https://nvidia.github.io/TensorRT-LLM/reference/troubleshooting.html#debug-on-e2e-models, I was wondering if there is any model closely related to my model above that I can use to troubleshoot? (I hope some nearly similar model exists because Im able to draw inference using TensorRT-LLM, its just the accuracy seems to be low.
Can you please help me?

[zhangts20]

Maybe you can run it with python runtime, then debug it following https://nvidia.github.io/TensorRT-LLM/reference/troubleshooting.html#debug-on-e2e-models