Update on the development branch

[kaiyux]

Hi,

The TensorRT-LLM team is pleased to announce that we are pushing an update to the development branch (and the Triton backend) this April 24th, 2024.

This update includes:

• Model Support
  • Support Qwen2
• Features
  • Add W4A(fp)8 CUTLASS kernels for the NVIDIA Ada Lovelace architecture
  • Add debug options (--visualize_network and --dry_run) to trtllm-build command to visualize the TensorRT network before engine build, thanks for the contribution from @Lokiiiiii in Adding debug options to trtllm-build to visualize the TRT Network before Engine build #1238
  • Add FP8 quantization support for Mixtral
  • Integrate new NVIDIA Hopper XQA kernels for LLaMA 2 70B model
• API
  • [BREAKING CHANGE] Rename GptModelConfig to ModelConfig
• Bug fixes
  • Fix error when converting the models weights of Qwen 72B INT4-GPTQ Qwen-72B-chat-GPTQ TP=4 ERROR #1344
  • Fix early stopping and failures on in-flight batching cases of Medusa Fail to run Medusa IFB with triton inference server #1449
  • Support more NVLink version for auto parallelism KeyError: 6 when getting nvlink_bandwidth #1467
  • Fix the assert failure caused by default values of sampling config [TensorRT-LLM][ERROR] Assertion failed: hasValues == configValue.has_value() (/app/tensorrt_llm/cpp/include/tensorrt_llm/runtime/samplingConfig.h:46 #1447
• Documentation
  • Add deprecation disclaimer for GptSession

Thanks,
The TensorRT-LLM Engineering Team