Update on the development branch

[kaiyux]

Hi,

The TensorRT-LLM team is pleased to announce that we are pushing an update to the development branch (also includes the Triton backend) this November 24th, 2023.

This update includes:

• Features
  • Support OOTB int8 weight-only quantization for LLaMA
  • Support OOTB fp8 KV cache
  • Support Python binding of C++ runtime on Windows
• Bug fixes
  • Add warm up to fix 1st token latency issue Doubts on 1st token latency decay #154
  • Fix Build error in codellama model's word embedding layer Build error in codellama model's word embedding layer (32016 not divisible by 64) #301, Build AWQ format codallama-13b-base engine failed because of Padding length too large #373
  • Fix contextFMHA flag check for cross attention and relative position bias Will the setting "context_fmha" support "cross_attention" ? #461
  • Fix docker build failure make -C docker release_build FAILED, REGRESSION?? #447

Thanks,
The TensorRT-LLM Engineering Team