[Performance issue] Unable to Reproduce the Throughput of Llama 3.1 8B FP8 on H100

[skymizerFuji]

Hi, I attempted to reproduce the Total Output Throughput (tokens/sec) reported here:
https://github.com/NVIDIA/TensorRT-LLM/blob/v0.20.0/docs/source/performance/perf-overview.md#llama-31-8b-fp8

For ISL/OSL 128/128, the expected throughput is 27,688.36 tokens/sec, but I only achieved 7099.5345 tokens/sec.

Am I missing something?

Environment:
Container launched on Runpod
1 x H100 PCIe
80G VRAM
16 vCPU 251 GB RAM
tensorrt_llm==0.20.0

extra-llm-api-config.yml

cuda_graph_config:
  batch_sizes:
  - 128
enable_attention_dp: true

The way I installed trtllm

apt-get -y install libopenmpi-dev
pip3 install --upgrade pip setuptools && pip3 install tensorrt_llm

Full output log:

root@7f4c828c65a8:/workspace/trtllm-benchmark# trtllm-bench --model nvidia/Llama-3.1-8B-Instruct-FP8 throughput --dataset /workspace/datasets/nvidia/Llama-3.1-8B-Instruct-FP8/128_128_30000.jsonl --backend pytorch --extra_llm_api_options extra-llm-api-config.yml --kv_cache_free_gpu_mem_fraction 0.8
<frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cuda module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.driver module instead.
<frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
2025-07-23 09:13:59,432 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  warnings.warn(
[TensorRT-LLM] TensorRT-LLM version: 0.20.0
[07/23/2025-09:14:01] [TRT-LLM] [I] Preparing to run throughput benchmark...
Parse safetensors files: 100%|█████████████████████████████████████| 2/2 [00:04<00:00,  2.26s/it]
[07/23/2025-09:14:15] [TRT-LLM] [I]
===========================================================
= DATASET DETAILS
===========================================================
Dataset Path:         /workspace/datasets/nvidia/Llama-3.1-8B-Instruct-FP8/128_128_30000.jsonl
Number of Sequences:  30000

-- Percentiles statistics ---------------------------------

        Input              Output           Seq. Length
-----------------------------------------------------------
MIN:   128.0000            28.0000           156.0000
MAX:   128.0000            28.0000           156.0000
AVG:   128.0000            28.0000           156.0000
P50:   128.0000            28.0000           156.0000
P90:   128.0000            28.0000           156.0000
P95:   128.0000            28.0000           156.0000
P99:   128.0000            28.0000           156.0000
===========================================================

Fetching 12 files: 100%|████████████████████████████████████████| 12/12 [00:00<00:00, 208.34it/s]
Parse safetensors files: 100%|█████████████████████████████████████| 2/2 [00:00<00:00,  8.31it/s]
[07/23/2025-09:14:16] [TRT-LLM] [I] Validating KV Cache config against kv_cache_dtype="auto"
[07/23/2025-09:14:16] [TRT-LLM] [I] KV cache quantization set to "auto". Using checkpoint KV quantization.
[07/23/2025-09:14:16] [TRT-LLM] [I] Estimated engine size: 7.48 GB
[07/23/2025-09:14:16] [TRT-LLM] [I] Estimated total available memory for KV cache: 72.17 GB
[07/23/2025-09:14:16] [TRT-LLM] [I] Estimated total KV cache memory: 68.56 GB
[07/23/2025-09:14:16] [TRT-LLM] [I] Estimated max number of requests in KV cache memory: 7200.59
[07/23/2025-09:14:16] [TRT-LLM] [I] Estimated max batch size (after fine-tune): 4096
[07/23/2025-09:14:16] [TRT-LLM] [I] Estimated max num tokens (after fine-tune): 23552
[07/23/2025-09:14:16] [TRT-LLM] [I] Max batch size and max num tokens not provided. Using heuristics or pre-defined settings: max_batch_size=4096, max_num_tokens=23552.
[07/23/2025-09:14:16] [TRT-LLM] [I] Setting PyTorch max sequence length to 156
[07/23/2025-09:14:16] [TRT-LLM] [I] Setting up throughput benchmark.
[07/23/2025-09:14:16] [TRT-LLM] [W] Using default gpus_per_node: 1
[07/23/2025-09:14:16] [TRT-LLM] [I] Set nccl_plugin to None.
[07/23/2025-09:14:17] [TRT-LLM] [I] PyTorchConfig(extra_resource_managers={}, use_cuda_graph=True, cuda_graph_batch_sizes=[1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 256, 512, 1024, 2048, 4096], cuda_graph_max_batch_size=4096, cuda_graph_padding_enabled=True, disable_overlap_scheduler=False, moe_max_num_tokens=None, attn_backend='TRTLLM', moe_backend='CUTLASS', mixed_sampler=False, enable_trtllm_sampler=False, kv_cache_dtype='auto', use_kv_cache=True, enable_iter_perf_stats=False, enable_iter_req_stats=False, print_iter_log=False, torch_compile_enabled=False, torch_compile_fullgraph=True, torch_compile_inductor_enabled=False, torch_compile_piecewise_cuda_graph=False, torch_compile_enable_userbuffers=True, autotuner_enabled=True, enable_layerwise_nvtx_marker=False, load_format=<LoadFormat.AUTO: 0>)
rank 0 using MpiPoolSession to spawn MPI processes
[07/23/2025-09:14:17] [TRT-LLM] [I] Generating a new HMAC key for server proxy_request_queue
[07/23/2025-09:14:17] [TRT-LLM] [I] Generating a new HMAC key for server proxy_request_error_queue
[07/23/2025-09:14:17] [TRT-LLM] [I] Generating a new HMAC key for server proxy_result_queue
[07/23/2025-09:14:17] [TRT-LLM] [I] Generating a new HMAC key for server proxy_stats_queue
[07/23/2025-09:14:17] [TRT-LLM] [I] Generating a new HMAC key for server proxy_kv_cache_events_queue
<frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cuda module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.driver module instead.
<frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
Multiple distributions found for package optimum. Picked distribution: optimum
2025-07-23 09:14:27,490 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  warnings.warn(
[TensorRT-LLM] TensorRT-LLM version: 0.20.0
[TensorRT-LLM][INFO] Refreshed the MPI local session
[07/23/2025-09:14:30] [TRT-LLM] [I] Validating KV Cache config against kv_cache_dtype="auto"
[07/23/2025-09:14:30] [TRT-LLM] [I] KV cache quantization set to "auto". Using checkpoint KV quantization.
[07/23/2025-09:14:31] [TRT-LLM] [I] Rank 0 uses 8.46 GB for model weights.
[07/23/2025-09:14:31] [TRT-LLM] [I] Rank 0 prefetching /workspace/.huggingface/hub/models--nvidia--Llama-3.1-8B-Instruct-FP8/snapshots/026c7c29fdd02d53f17c125d2dec8cb1a2251c23/model-00001-of-00002.safetensors to memory...
[07/23/2025-09:14:31] [TRT-LLM] [I] Rank 0 prefetching /workspace/.huggingface/hub/models--nvidia--Llama-3.1-8B-Instruct-FP8/snapshots/026c7c29fdd02d53f17c125d2dec8cb1a2251c23/model-00002-of-00002.safetensors to memory...
[07/23/2025-09:15:43] [TRT-LLM] [I] Rank 0 finished prefetching /workspace/.huggingface/hub/models--nvidia--Llama-3.1-8B-Instruct-FP8/snapshots/026c7c29fdd02d53f17c125d2dec8cb1a2251c23/model-00002-of-00002.safetensors.
[07/23/2025-09:15:56] [TRT-LLM] [I] Rank 0 finished prefetching /workspace/.huggingface/hub/models--nvidia--Llama-3.1-8B-Instruct-FP8/snapshots/026c7c29fdd02d53f17c125d2dec8cb1a2251c23/model-00001-of-00002.safetensors.
[07/23/2025-09:15:56] [TRT-LLM] [I] Loading /workspace/.huggingface/hub/models--nvidia--Llama-3.1-8B-Instruct-FP8/snapshots/026c7c29fdd02d53f17c125d2dec8cb1a2251c23/model-00001-of-00002.safetensors
[07/23/2025-09:15:56] [TRT-LLM] [I] Loading /workspace/.huggingface/hub/models--nvidia--Llama-3.1-8B-Instruct-FP8/snapshots/026c7c29fdd02d53f17c125d2dec8cb1a2251c23/model-00002-of-00002.safetensors
Loading weights: 100%|██████████| 617/617 [00:02<00:00, 256.10it/s]
Model init total -- 88.37s
[07/23/2025-09:15:59] [TRT-LLM] [W] Both free_gpu_memory_fraction and max_tokens are set (to 0.800000011920929 and 28384, respectively). The smaller value will be used.
[TensorRT-LLM][INFO] Max KV cache pages per sequence: 5 [window size=157]
[TensorRT-LLM][INFO] Number of tokens per block: 32.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 1.73 GiB for max tokens in paged KV cache (28384).
[07/23/2025-09:15:59] [TRT-LLM] [I] max_seq_len=157, max_num_requests=4096, max_num_tokens=23552
[07/23/2025-09:15:59] [TRT-LLM] [I] [Autotuner]: Autotuning process starts ...
[07/23/2025-09:15:59] [TRT-LLM] [I] Run autotuning warmup for batch size=1
2025-07-23 09:15:59,759 - INFO - flashinfer.jit: Loading JIT ops: norm
/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  warnings.warn(
/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  warnings.warn(
2025-07-23 09:15:59,903 - INFO - flashinfer.jit: Finished loading JIT ops: norm
2025-07-23 09:16:00,896 - INFO - flashinfer.jit: Loading JIT ops: silu_and_mul
/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  warnings.warn(
/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  warnings.warn(
2025-07-23 09:16:01,321 - INFO - flashinfer.jit: Finished loading JIT ops: silu_and_mul
[07/23/2025-09:16:01] [TRT-LLM] [I] Autotuner Cache size after warmup 0
[07/23/2025-09:16:01] [TRT-LLM] [I] [Autotuner]: Autotuning process ends
[07/23/2025-09:16:01] [TRT-LLM] [I] Creating CUDA graph instances for 24 batch sizes.
[07/23/2025-09:16:02] [TRT-LLM] [I] Memory used after loading model weights (inside torch) in memory usage profiling: 8.55 GiB
[07/23/2025-09:16:02] [TRT-LLM] [I] Memory used after loading model weights (outside torch) in memory usage profiling: 3.14 GiB
[07/23/2025-09:16:03] [TRT-LLM] [I] Memory dynamically allocated during inference (inside torch) in memory usage profiling: 2.95 GiB
[07/23/2025-09:16:03] [TRT-LLM] [I] Memory used outside torch (e.g., NCCL and CUDA graphs) in memory usage profiling: 3.78 GiB
[07/23/2025-09:16:03] [TRT-LLM] [I] Peak memory during memory usage profiling (torch + non-torch): 15.29 GiB, available KV cache memory when calculating max tokens: 52.51 GiB, fraction is set 0.800000011920929, kv size is 65536, device total memory 79.19 GiB, , tmp kv_mem 1.73 GiB
[07/23/2025-09:16:03] [TRT-LLM] [I] Estimated max tokens in KV cache : 860308
[07/23/2025-09:16:03] [TRT-LLM] [W] Both free_gpu_memory_fraction and max_tokens are set (to 0.800000011920929 and 860308, respectively). The smaller value will be used.
[TensorRT-LLM][INFO] Max KV cache pages per sequence: 5 [window size=157]
[TensorRT-LLM][INFO] Number of tokens per block: 32.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 52.51 GiB for max tokens in paged KV cache (860320).
[07/23/2025-09:16:03] [TRT-LLM] [I] max_seq_len=157, max_num_requests=4096, max_num_tokens=23552
[07/23/2025-09:16:03] [TRT-LLM] [I] [Autotuner]: Autotuning process starts ...
[07/23/2025-09:16:03] [TRT-LLM] [I] Run autotuning warmup for batch size=1
[07/23/2025-09:16:03] [TRT-LLM] [I] Autotuner Cache size after warmup 0
[07/23/2025-09:16:03] [TRT-LLM] [I] [Autotuner]: Autotuning process ends
[07/23/2025-09:16:03] [TRT-LLM] [I] Creating CUDA graph instances for 24 batch sizes.
[07/23/2025-09:16:04] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=4096
[07/23/2025-09:16:05] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=2048
[07/23/2025-09:16:05] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=1024
[07/23/2025-09:16:06] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=512
[07/23/2025-09:16:06] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=256
[07/23/2025-09:16:06] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=128
[07/23/2025-09:16:07] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=120
[07/23/2025-09:16:07] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=112
[07/23/2025-09:16:08] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=104
[07/23/2025-09:16:08] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=96
[07/23/2025-09:16:08] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=88
[07/23/2025-09:16:09] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=80
[07/23/2025-09:16:09] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=72
[07/23/2025-09:16:10] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=64
[07/23/2025-09:16:10] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=56
[07/23/2025-09:16:10] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=48
[07/23/2025-09:16:11] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=40
[07/23/2025-09:16:11] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=32
[07/23/2025-09:16:11] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=24
[07/23/2025-09:16:12] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=16
[07/23/2025-09:16:12] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=8
[07/23/2025-09:16:12] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=4
[07/23/2025-09:16:13] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=2
[07/23/2025-09:16:13] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=1
[07/23/2025-09:16:13] [TRT-LLM] [I] Setting up for warmup...
[07/23/2025-09:16:13] [TRT-LLM] [I] Running warmup.
[07/23/2025-09:16:13] [TRT-LLM] [I] Starting benchmarking async task.
[07/23/2025-09:16:13] [TRT-LLM] [I] Starting benchmark...
[07/23/2025-09:16:13] [TRT-LLM] [I] Request submission complete. [count=2, time=0.0001s, rate=30122.30 req/s]
[07/23/2025-09:16:14] [TRT-LLM] [I] Benchmark complete.
[07/23/2025-09:16:14] [TRT-LLM] [I] Stopping LLM backend.
[07/23/2025-09:16:14] [TRT-LLM] [I] Cancelling all 0 tasks to complete.
[07/23/2025-09:16:14] [TRT-LLM] [I] All tasks cancelled.
[07/23/2025-09:16:14] [TRT-LLM] [I] LLM Backend stopped.
[07/23/2025-09:16:14] [TRT-LLM] [I] Worker task cancelled.
[07/23/2025-09:16:14] [TRT-LLM] [I] Warmup done.
[07/23/2025-09:16:14] [TRT-LLM] [I] No log path provided, skipping logging.
[07/23/2025-09:16:14] [TRT-LLM] [I] Starting benchmarking async task.
[07/23/2025-09:16:14] [TRT-LLM] [I] Starting benchmark...
[07/23/2025-09:16:14] [TRT-LLM] [I] Request submission complete. [count=30000, time=0.1026s, rate=292499.08 req/s]
[07/23/2025-09:18:13] [TRT-LLM] [I] Benchmark complete.
[07/23/2025-09:18:13] [TRT-LLM] [I] Stopping LLM backend.
[07/23/2025-09:18:13] [TRT-LLM] [I] Cancelling all 0 tasks to complete.
[07/23/2025-09:18:13] [TRT-LLM] [I] All tasks cancelled.
[07/23/2025-09:18:13] [TRT-LLM] [I] LLM Backend stopped.
[07/23/2025-09:18:13] [TRT-LLM] [I] Worker task cancelled.
[07/23/2025-09:18:13] [TRT-LLM] [I] Benchmark done. Reporting results...
[07/23/2025-09:18:13] [TRT-LLM] [I] Validating KV Cache config against kv_cache_dtype="auto"
[07/23/2025-09:18:13] [TRT-LLM] [I] KV cache quantization set to "auto". Using checkpoint KV quantization.
[07/23/2025-09:18:13] [TRT-LLM] [I]

===========================================================
= PYTORCH BACKEND
===========================================================
Model:			nvidia/Llama-3.1-8B-Instruct-FP8
Model Path:		None
TensorRT-LLM Version:	0.20.0
Dtype:			bfloat16
KV Cache Dtype:		FP8
Quantization:		FP8

===========================================================
= REQUEST DETAILS
===========================================================
Number of requests:             30000
Number of concurrent requests:  14114.5246
Average Input Length (tokens):  128.0000
Average Output Length (tokens): 28.0000
===========================================================
= WORLD + RUNTIME INFORMATION
===========================================================
TP Size:                1
PP Size:                1
EP Size:                None
Max Runtime Batch Size: 4096
Max Runtime Tokens:     23552
Scheduling Policy:      GUARANTEED_NO_EVICT
KV Memory Percentage:   80.00%
Issue Rate (req/sec):   3.4376E+15

===========================================================
= PERFORMANCE OVERVIEW
===========================================================
Request Throughput (req/sec):                     253.5548
Total Output Throughput (tokens/sec):             7099.5345
Total Token Throughput (tokens/sec):              39554.5494
Total Latency (ms):                               118317.6163
Average request latency (ms):                     55666.5636
Per User Output Throughput [w/ ctx] (tps/user):   0.6702
Per GPU Output Throughput (tps/gpu):              7099.5345

-- Request Latency Breakdown (ms) -----------------------

[Latency] P50    : 54422.3568
[Latency] P90    : 94512.0713
[Latency] P95    : 98713.1150
[Latency] P99    : 102475.5310
[Latency] MINIMUM: 15187.1518
[Latency] MAXIMUM: 103317.3451
[Latency] AVERAGE: 55666.5636

===========================================================
= DATASET DETAILS
===========================================================
Dataset Path:         /workspace/datasets/nvidia/Llama-3.1-8B-Instruct-FP8/128_128_30000.jsonl
Number of Sequences:  30000

-- Percentiles statistics ---------------------------------

        Input              Output           Seq. Length
-----------------------------------------------------------
MIN:   128.0000            28.0000           156.0000
MAX:   128.0000            28.0000           156.0000
AVG:   128.0000            28.0000           156.0000
P50:   128.0000            28.0000           156.0000
P90:   128.0000            28.0000           156.0000
P95:   128.0000            28.0000           156.0000
P99:   128.0000            28.0000           156.0000
===========================================================

[07/23/2025-09:18:13] [TRT-LLM] [I] Thread proxy_dispatch_result_thread stopped.
[07/23/2025-09:18:13] [TRT-LLM] [I] Thread proxy_dispatch_stats_thread stopped.
[07/23/2025-09:18:13] [TRT-LLM] [I] Thread proxy_dispatch_kv_cache_events_thread stopped.
[07/23/2025-09:18:13] [TRT-LLM] [I] Thread await_response_thread stopped.
[07/23/2025-09:18:14] [TRT-LLM] [I] Thread dispatch_stats_thread stopped.
[07/23/2025-09:18:14] [TRT-LLM] [I] Thread dispatch_kv_cache_events_thread stopped.

OOM If I set kv_cache_free_gpu_mem_fraction to 0.9

[zbpatel]

@skymizerFuji can you try running with the following extra-llm-api-config.yml

use_cuda_graph: true
cuda_graph_padding_enabled: true
cuda_graph_batch_sizes:
  - 1
  - 2
  - 4
  - 8
  - 16
  - 32
  - 64
  - 128
  - 256
  - 384
  - 512
  - 1024
  - 2048
  - 4096
  - 8192

This is the file that we used in the perf-overview.md. Your config only has 128 for the cuda graph batch sizes, which is going to limit your performance gains from this feature - if you check the trtllm-bench output, you can see the runtime max batch size is auto-tuned to 4096, so I would recommend trying at least this high on the cuda graph batch sizes.

Max Runtime Batch Size: 4096

[skymizerFuji]

@zbpatel
Hi, thank you for your quick response. I have tried the original setting and your settings. The performance is still bad.

Original settings:

llm_options.yml

pytorch_backend_config:
  use_cuda_graph: true
  cuda_graph_padding_enabled: true
  cuda_graph_batch_sizes:
  - 1
  - 2
  - 4
  - 8
  - 16
  - 32
  - 64
  - 128
  - 256
  - 384
  - 512
  - 1024
  - 2048
  - 4096
  - 8192

Output 1 (Click to expand)

root@16eea827d55a:/workspace/trtllm-benchmark# trtllm-bench --model nvidia/Llama-3.1-8B-Instruct-FP8 throughput --dataset /workspace/datasets/nvidia/Llama-3.1-8B-Instruct-FP8/128_128_30000.jsonl --backend pytorch --extra_llm_api_options llm_options.yml --kv_cache_free_gpu_mem_fraction 0.8
<frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cuda module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.driver module instead.
<frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
2025-07-24 08:12:45,793 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  warnings.warn(
[TensorRT-LLM] TensorRT-LLM version: 0.20.0
[07/24/2025-08:12:47] [TRT-LLM] [I] Preparing to run throughput benchmark...
Parse safetensors files: 100%|█████████████████████████████████████| 2/2 [00:00<00:00,  9.17it/s]
[07/24/2025-08:12:49] [TRT-LLM] [I]
===========================================================
= DATASET DETAILS
===========================================================
Dataset Path:         /workspace/datasets/nvidia/Llama-3.1-8B-Instruct-FP8/128_128_30000.jsonl
Number of Sequences:  30000

-- Percentiles statistics ---------------------------------

        Input              Output           Seq. Length
-----------------------------------------------------------
MIN:   128.0000            28.0000           156.0000
MAX:   128.0000            28.0000           156.0000
AVG:   128.0000            28.0000           156.0000
P50:   128.0000            28.0000           156.0000
P90:   128.0000            28.0000           156.0000
P95:   128.0000            28.0000           156.0000
P99:   128.0000            28.0000           156.0000
===========================================================

Fetching 12 files: 100%|███████████████████████████████████████| 12/12 [00:00<00:00, 2977.50it/s]
Parse safetensors files: 100%|█████████████████████████████████████| 2/2 [00:00<00:00, 15.72it/s]
[07/24/2025-08:12:50] [TRT-LLM] [I] Validating KV Cache config against kv_cache_dtype="auto"
[07/24/2025-08:12:50] [TRT-LLM] [I] KV cache quantization set to "auto". Using checkpoint KV quantization.
[07/24/2025-08:12:50] [TRT-LLM] [I] Estimated engine size: 7.48 GB
[07/24/2025-08:12:50] [TRT-LLM] [I] Estimated total available memory for KV cache: 72.17 GB
[07/24/2025-08:12:50] [TRT-LLM] [I] Estimated total KV cache memory: 68.56 GB
[07/24/2025-08:12:50] [TRT-LLM] [I] Estimated max number of requests in KV cache memory: 7200.59
[07/24/2025-08:12:50] [TRT-LLM] [I] Estimated max batch size (after fine-tune): 4096
[07/24/2025-08:12:50] [TRT-LLM] [I] Estimated max num tokens (after fine-tune): 23552
[07/24/2025-08:12:50] [TRT-LLM] [I] Max batch size and max num tokens not provided. Using heuristics or pre-defined settings: max_batch_size=4096, max_num_tokens=23552.
[07/24/2025-08:12:50] [TRT-LLM] [I] Setting PyTorch max sequence length to 156
[07/24/2025-08:12:50] [TRT-LLM] [I] Setting up throughput benchmark.
[07/24/2025-08:12:50] [TRT-LLM] [W] Overriding pytorch_backend_config because it's specified in llm_options.yml
[07/24/2025-08:12:50] [TRT-LLM] [W] Using default gpus_per_node: 1
[07/24/2025-08:12:50] [TRT-LLM] [I] Set nccl_plugin to None.
[07/24/2025-08:12:50] [TRT-LLM] [I] PyTorchConfig(extra_resource_managers={}, use_cuda_graph=True, cuda_graph_batch_sizes=[1, 2, 4, 8, 16, 32, 64, 128, 256, 384, 512, 1024, 2048, 4096, 8192], cuda_graph_max_batch_size=0, cuda_graph_padding_enabled=True, disable_overlap_scheduler=False, moe_max_num_tokens=None, attn_backend='TRTLLM', moe_backend='CUTLASS', mixed_sampler=False, enable_trtllm_sampler=False, kv_cache_dtype='auto', use_kv_cache=True, enable_iter_perf_stats=False, enable_iter_req_stats=False, print_iter_log=False, torch_compile_enabled=False, torch_compile_fullgraph=True, torch_compile_inductor_enabled=False, torch_compile_piecewise_cuda_graph=False, torch_compile_enable_userbuffers=True, autotuner_enabled=True, enable_layerwise_nvtx_marker=False, load_format=<LoadFormat.AUTO: 0>)
rank 0 using MpiPoolSession to spawn MPI processes
[07/24/2025-08:12:50] [TRT-LLM] [I] Generating a new HMAC key for server proxy_request_queue
[07/24/2025-08:12:50] [TRT-LLM] [I] Generating a new HMAC key for server proxy_request_error_queue
[07/24/2025-08:12:50] [TRT-LLM] [I] Generating a new HMAC key for server proxy_result_queue
[07/24/2025-08:12:50] [TRT-LLM] [I] Generating a new HMAC key for server proxy_stats_queue
[07/24/2025-08:12:50] [TRT-LLM] [I] Generating a new HMAC key for server proxy_kv_cache_events_queue
<frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cuda module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.driver module instead.
<frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
Multiple distributions found for package optimum. Picked distribution: optimum
2025-07-24 08:13:00,746 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  warnings.warn(
[TensorRT-LLM] TensorRT-LLM version: 0.20.0
[TensorRT-LLM][INFO] Refreshed the MPI local session
[07/24/2025-08:13:02] [TRT-LLM] [I] Validating KV Cache config against kv_cache_dtype="auto"
[07/24/2025-08:13:02] [TRT-LLM] [I] KV cache quantization set to "auto". Using checkpoint KV quantization.
[07/24/2025-08:13:02] [TRT-LLM] [I] Rank 0 uses 8.46 GB for model weights.
[07/24/2025-08:13:02] [TRT-LLM] [I] Rank 0 prefetching /root/.cache/huggingface/hub/models--nvidia--Llama-3.1-8B-Instruct-FP8/snapshots/026c7c29fdd02d53f17c125d2dec8cb1a2251c23/model-00001-of-00002.safetensors to memory...
[07/24/2025-08:13:02] [TRT-LLM] [I] Rank 0 prefetching /root/.cache/huggingface/hub/models--nvidia--Llama-3.1-8B-Instruct-FP8/snapshots/026c7c29fdd02d53f17c125d2dec8cb1a2251c23/model-00002-of-00002.safetensors to memory...
[07/24/2025-08:13:05] [TRT-LLM] [I] Rank 0 finished prefetching /root/.cache/huggingface/hub/models--nvidia--Llama-3.1-8B-Instruct-FP8/snapshots/026c7c29fdd02d53f17c125d2dec8cb1a2251c23/model-00002-of-00002.safetensors.
[07/24/2025-08:13:06] [TRT-LLM] [I] Rank 0 finished prefetching /root/.cache/huggingface/hub/models--nvidia--Llama-3.1-8B-Instruct-FP8/snapshots/026c7c29fdd02d53f17c125d2dec8cb1a2251c23/model-00001-of-00002.safetensors.
[07/24/2025-08:13:06] [TRT-LLM] [I] Loading /root/.cache/huggingface/hub/models--nvidia--Llama-3.1-8B-Instruct-FP8/snapshots/026c7c29fdd02d53f17c125d2dec8cb1a2251c23/model-00001-of-00002.safetensors
[07/24/2025-08:13:06] [TRT-LLM] [I] Loading /root/.cache/huggingface/hub/models--nvidia--Llama-3.1-8B-Instruct-FP8/snapshots/026c7c29fdd02d53f17c125d2dec8cb1a2251c23/model-00002-of-00002.safetensors
Loading weights: 100%|██████████| 617/617 [00:01<00:00, 317.42it/s]
Model init total -- 5.92s
[07/24/2025-08:13:08] [TRT-LLM] [W] Both free_gpu_memory_fraction and max_tokens are set (to 0.800000011920929 and 28384, respectively). The smaller value will be used.
[TensorRT-LLM][INFO] Max KV cache pages per sequence: 5 [window size=157]
[TensorRT-LLM][INFO] Number of tokens per block: 32.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 1.73 GiB for max tokens in paged KV cache (28384).
[07/24/2025-08:13:08] [TRT-LLM] [I] max_seq_len=157, max_num_requests=4096, max_num_tokens=23552
[07/24/2025-08:13:08] [TRT-LLM] [I] [Autotuner]: Autotuning process starts ...
[07/24/2025-08:13:08] [TRT-LLM] [I] Run autotuning warmup for batch size=1
2025-07-24 08:13:08,536 - INFO - flashinfer.jit: Loading JIT ops: norm
/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  warnings.warn(
/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  warnings.warn(
2025-07-24 08:13:08,617 - INFO - flashinfer.jit: Finished loading JIT ops: norm
2025-07-24 08:13:09,342 - INFO - flashinfer.jit: Loading JIT ops: silu_and_mul
/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  warnings.warn(
/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  warnings.warn(
2025-07-24 08:13:09,474 - INFO - flashinfer.jit: Finished loading JIT ops: silu_and_mul
[07/24/2025-08:13:10] [TRT-LLM] [I] Autotuner Cache size after warmup 0
[07/24/2025-08:13:10] [TRT-LLM] [I] [Autotuner]: Autotuning process ends
[07/24/2025-08:13:10] [TRT-LLM] [I] Creating CUDA graph instances for 14 batch sizes.
[07/24/2025-08:13:10] [TRT-LLM] [I] Memory used after loading model weights (inside torch) in memory usage profiling: 8.55 GiB
[07/24/2025-08:13:10] [TRT-LLM] [I] Memory used after loading model weights (outside torch) in memory usage profiling: 3.14 GiB
[07/24/2025-08:13:11] [TRT-LLM] [I] Memory dynamically allocated during inference (inside torch) in memory usage profiling: 2.95 GiB
[07/24/2025-08:13:11] [TRT-LLM] [I] Memory used outside torch (e.g., NCCL and CUDA graphs) in memory usage profiling: 3.78 GiB
[07/24/2025-08:13:11] [TRT-LLM] [I] Peak memory during memory usage profiling (torch + non-torch): 15.29 GiB, available KV cache memory when calculating max tokens: 52.51 GiB, fraction is set 0.800000011920929, kv size is 65536, device total memory 79.19 GiB, , tmp kv_mem 1.73 GiB
[07/24/2025-08:13:11] [TRT-LLM] [I] Estimated max tokens in KV cache : 860308
[07/24/2025-08:13:11] [TRT-LLM] [W] Both free_gpu_memory_fraction and max_tokens are set (to 0.800000011920929 and 860308, respectively). The smaller value will be used.
[TensorRT-LLM][INFO] Max KV cache pages per sequence: 5 [window size=157]
[TensorRT-LLM][INFO] Number of tokens per block: 32.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 52.51 GiB for max tokens in paged KV cache (860320).
[07/24/2025-08:13:11] [TRT-LLM] [I] max_seq_len=157, max_num_requests=4096, max_num_tokens=23552
[07/24/2025-08:13:11] [TRT-LLM] [I] [Autotuner]: Autotuning process starts ...
[07/24/2025-08:13:11] [TRT-LLM] [I] Run autotuning warmup for batch size=1
[07/24/2025-08:13:11] [TRT-LLM] [I] Autotuner Cache size after warmup 0
[07/24/2025-08:13:11] [TRT-LLM] [I] [Autotuner]: Autotuning process ends
[07/24/2025-08:13:11] [TRT-LLM] [I] Creating CUDA graph instances for 14 batch sizes.
[07/24/2025-08:13:12] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=4096
[07/24/2025-08:13:12] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=2048
[07/24/2025-08:13:13] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=1024
[07/24/2025-08:13:13] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=512
[07/24/2025-08:13:14] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=384
[07/24/2025-08:13:14] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=256
[07/24/2025-08:13:14] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=128
[07/24/2025-08:13:15] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=64
[07/24/2025-08:13:15] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=32
[07/24/2025-08:13:15] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=16
[07/24/2025-08:13:15] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=8
[07/24/2025-08:13:16] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=4
[07/24/2025-08:13:16] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=2
[07/24/2025-08:13:16] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=1
[07/24/2025-08:13:17] [TRT-LLM] [I] Setting up for warmup...
[07/24/2025-08:13:17] [TRT-LLM] [I] Running warmup.
[07/24/2025-08:13:17] [TRT-LLM] [I] Starting benchmarking async task.
[07/24/2025-08:13:17] [TRT-LLM] [I] Starting benchmark...
[07/24/2025-08:13:17] [TRT-LLM] [I] Request submission complete. [count=2, time=0.0000s, rate=47133.13 req/s]
[07/24/2025-08:13:17] [TRT-LLM] [I] Benchmark complete.
[07/24/2025-08:13:17] [TRT-LLM] [I] Stopping LLM backend.
[07/24/2025-08:13:17] [TRT-LLM] [I] Cancelling all 0 tasks to complete.
[07/24/2025-08:13:17] [TRT-LLM] [I] All tasks cancelled.
[07/24/2025-08:13:17] [TRT-LLM] [I] LLM Backend stopped.
[07/24/2025-08:13:17] [TRT-LLM] [I] Worker task cancelled.
[07/24/2025-08:13:17] [TRT-LLM] [I] Warmup done.
[07/24/2025-08:13:17] [TRT-LLM] [I] No log path provided, skipping logging.
[07/24/2025-08:13:17] [TRT-LLM] [I] Starting benchmarking async task.
[07/24/2025-08:13:17] [TRT-LLM] [I] Starting benchmark...
[07/24/2025-08:13:17] [TRT-LLM] [I] Request submission complete. [count=30000, time=0.0618s, rate=485694.03 req/s]
[07/24/2025-08:15:17] [TRT-LLM] [I] Benchmark complete.
[07/24/2025-08:15:17] [TRT-LLM] [I] Stopping LLM backend.
[07/24/2025-08:15:17] [TRT-LLM] [I] Cancelling all 0 tasks to complete.
[07/24/2025-08:15:17] [TRT-LLM] [I] All tasks cancelled.
[07/24/2025-08:15:17] [TRT-LLM] [I] LLM Backend stopped.
[07/24/2025-08:15:17] [TRT-LLM] [I] Worker task cancelled.
[07/24/2025-08:15:17] [TRT-LLM] [I] Benchmark done. Reporting results...
[07/24/2025-08:15:17] [TRT-LLM] [I] Validating KV Cache config against kv_cache_dtype="auto"
[07/24/2025-08:15:17] [TRT-LLM] [I] KV cache quantization set to "auto". Using checkpoint KV quantization.
[07/24/2025-08:15:17] [TRT-LLM] [I]

===========================================================
= PYTORCH BACKEND
===========================================================
Model:			nvidia/Llama-3.1-8B-Instruct-FP8
Model Path:		None
TensorRT-LLM Version:	0.20.0
Dtype:			bfloat16
KV Cache Dtype:		FP8
Quantization:		FP8

===========================================================
= REQUEST DETAILS
===========================================================
Number of requests:             30000
Number of concurrent requests:  14147.9588
Average Input Length (tokens):  128.0000
Average Output Length (tokens): 28.0000
===========================================================
= WORLD + RUNTIME INFORMATION
===========================================================
TP Size:                1
PP Size:                1
EP Size:                None
Max Runtime Batch Size: 4096
Max Runtime Tokens:     23552
Scheduling Policy:      GUARANTEED_NO_EVICT
KV Memory Percentage:   80.00%
Issue Rate (req/sec):   2.9025E+15

===========================================================
= PERFORMANCE OVERVIEW
===========================================================
Request Throughput (req/sec):                     252.2077
Total Output Throughput (tokens/sec):             7061.8149
Total Token Throughput (tokens/sec):              39344.3974
Total Latency (ms):                               118949.5915
Average request latency (ms):                     56096.4641
Per User Output Throughput [w/ ctx] (tps/user):   0.5492
Per GPU Output Throughput (tps/gpu):              7061.8149

-- Request Latency Breakdown (ms) -----------------------

[Latency] P50    : 51235.6937
[Latency] P90    : 81739.8520
[Latency] P95    : 83991.4126
[Latency] P99    : 86319.8220
[Latency] MINIMUM: 31873.4904
[Latency] MAXIMUM: 87167.6468
[Latency] AVERAGE: 56096.4641

===========================================================
= DATASET DETAILS
===========================================================
Dataset Path:         /workspace/datasets/nvidia/Llama-3.1-8B-Instruct-FP8/128_128_30000.jsonl
Number of Sequences:  30000

-- Percentiles statistics ---------------------------------

        Input              Output           Seq. Length
-----------------------------------------------------------
MIN:   128.0000            28.0000           156.0000
MAX:   128.0000            28.0000           156.0000
AVG:   128.0000            28.0000           156.0000
P50:   128.0000            28.0000           156.0000
P90:   128.0000            28.0000           156.0000
P95:   128.0000            28.0000           156.0000
P99:   128.0000            28.0000           156.0000
===========================================================

Your settings:

extra-llm-api-config.yml

use_cuda_graph: true
cuda_graph_padding_enabled: true
cuda_graph_batch_sizes:
  - 1
  - 2
  - 4
  - 8
  - 16
  - 32
  - 64
  - 128
  - 256
  - 384
  - 512
  - 1024
  - 2048
  - 4096
  - 8192

Output 2

root@16eea827d55a:/workspace/trtllm-benchmark# trtllm-bench --model nvidia/Llama-3.1-8B-Instruct-FP8 throughput --dataset /workspace/datasets/nvidia/Llama-3.1-8B-Instruct-FP8/128_128_30000.jsonl --backend pytorch --extra_llm_api_options extra-llm-api-config.yml --kv_cache_free_gpu_mem_fraction 0.8
<frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cuda module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.driver module instead.
<frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
2025-07-24 08:18:19,614 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  warnings.warn(
[TensorRT-LLM] TensorRT-LLM version: 0.20.0
[07/24/2025-08:18:20] [TRT-LLM] [I] Preparing to run throughput benchmark...
Parse safetensors files: 100%|█████████████████████████████████████| 2/2 [00:00<00:00,  6.70it/s]
[07/24/2025-08:18:23] [TRT-LLM] [I]
===========================================================
= DATASET DETAILS
===========================================================
Dataset Path:         /workspace/datasets/nvidia/Llama-3.1-8B-Instruct-FP8/128_128_30000.jsonl
Number of Sequences:  30000

-- Percentiles statistics ---------------------------------

        Input              Output           Seq. Length
-----------------------------------------------------------
MIN:   128.0000            28.0000           156.0000
MAX:   128.0000            28.0000           156.0000
AVG:   128.0000            28.0000           156.0000
P50:   128.0000            28.0000           156.0000
P90:   128.0000            28.0000           156.0000
P95:   128.0000            28.0000           156.0000
P99:   128.0000            28.0000           156.0000
===========================================================

Fetching 12 files: 100%|███████████████████████████████████████| 12/12 [00:00<00:00, 5851.16it/s]
Parse safetensors files: 100%|█████████████████████████████████████| 2/2 [00:00<00:00,  9.75it/s]
[07/24/2025-08:18:24] [TRT-LLM] [I] Validating KV Cache config against kv_cache_dtype="auto"
[07/24/2025-08:18:24] [TRT-LLM] [I] KV cache quantization set to "auto". Using checkpoint KV quantization.
[07/24/2025-08:18:24] [TRT-LLM] [I] Estimated engine size: 7.48 GB
[07/24/2025-08:18:24] [TRT-LLM] [I] Estimated total available memory for KV cache: 72.17 GB
[07/24/2025-08:18:24] [TRT-LLM] [I] Estimated total KV cache memory: 68.56 GB
[07/24/2025-08:18:24] [TRT-LLM] [I] Estimated max number of requests in KV cache memory: 7200.59
[07/24/2025-08:18:24] [TRT-LLM] [I] Estimated max batch size (after fine-tune): 4096
[07/24/2025-08:18:24] [TRT-LLM] [I] Estimated max num tokens (after fine-tune): 23552
[07/24/2025-08:18:24] [TRT-LLM] [I] Max batch size and max num tokens not provided. Using heuristics or pre-defined settings: max_batch_size=4096, max_num_tokens=23552.
[07/24/2025-08:18:24] [TRT-LLM] [I] Setting PyTorch max sequence length to 156
[07/24/2025-08:18:24] [TRT-LLM] [I] Setting up throughput benchmark.
[07/24/2025-08:18:24] [TRT-LLM] [W] Using default gpus_per_node: 1
[07/24/2025-08:18:24] [TRT-LLM] [I] Set nccl_plugin to None.
[07/24/2025-08:18:24] [TRT-LLM] [I] PyTorchConfig(extra_resource_managers={}, use_cuda_graph=True, cuda_graph_batch_sizes=[1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 256, 512, 1024, 2048, 4096], cuda_graph_max_batch_size=4096, cuda_graph_padding_enabled=True, disable_overlap_scheduler=False, moe_max_num_tokens=None, attn_backend='TRTLLM', moe_backend='CUTLASS', mixed_sampler=False, enable_trtllm_sampler=False, kv_cache_dtype='auto', use_kv_cache=True, enable_iter_perf_stats=False, enable_iter_req_stats=False, print_iter_log=False, torch_compile_enabled=False, torch_compile_fullgraph=True, torch_compile_inductor_enabled=False, torch_compile_piecewise_cuda_graph=False, torch_compile_enable_userbuffers=True, autotuner_enabled=True, enable_layerwise_nvtx_marker=False, load_format=<LoadFormat.AUTO: 0>)
rank 0 using MpiPoolSession to spawn MPI processes
[07/24/2025-08:18:24] [TRT-LLM] [I] Generating a new HMAC key for server proxy_request_queue
[07/24/2025-08:18:25] [TRT-LLM] [I] Generating a new HMAC key for server proxy_request_error_queue
[07/24/2025-08:18:25] [TRT-LLM] [I] Generating a new HMAC key for server proxy_result_queue
[07/24/2025-08:18:25] [TRT-LLM] [I] Generating a new HMAC key for server proxy_stats_queue
[07/24/2025-08:18:25] [TRT-LLM] [I] Generating a new HMAC key for server proxy_kv_cache_events_queue
<frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cuda module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.driver module instead.
<frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
Multiple distributions found for package optimum. Picked distribution: optimum
2025-07-24 08:18:35,553 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  warnings.warn(
[TensorRT-LLM] TensorRT-LLM version: 0.20.0
[TensorRT-LLM][INFO] Refreshed the MPI local session
[07/24/2025-08:18:37] [TRT-LLM] [I] Validating KV Cache config against kv_cache_dtype="auto"
[07/24/2025-08:18:37] [TRT-LLM] [I] KV cache quantization set to "auto". Using checkpoint KV quantization.
[07/24/2025-08:18:38] [TRT-LLM] [I] Rank 0 uses 8.46 GB for model weights.
[07/24/2025-08:18:38] [TRT-LLM] [I] Rank 0 prefetching /root/.cache/huggingface/hub/models--nvidia--Llama-3.1-8B-Instruct-FP8/snapshots/026c7c29fdd02d53f17c125d2dec8cb1a2251c23/model-00001-of-00002.safetensors to memory...
[07/24/2025-08:18:38] [TRT-LLM] [I] Rank 0 prefetching /root/.cache/huggingface/hub/models--nvidia--Llama-3.1-8B-Instruct-FP8/snapshots/026c7c29fdd02d53f17c125d2dec8cb1a2251c23/model-00002-of-00002.safetensors to memory...
[07/24/2025-08:18:41] [TRT-LLM] [I] Rank 0 finished prefetching /root/.cache/huggingface/hub/models--nvidia--Llama-3.1-8B-Instruct-FP8/snapshots/026c7c29fdd02d53f17c125d2dec8cb1a2251c23/model-00002-of-00002.safetensors.
[07/24/2025-08:18:41] [TRT-LLM] [I] Rank 0 finished prefetching /root/.cache/huggingface/hub/models--nvidia--Llama-3.1-8B-Instruct-FP8/snapshots/026c7c29fdd02d53f17c125d2dec8cb1a2251c23/model-00001-of-00002.safetensors.
[07/24/2025-08:18:41] [TRT-LLM] [I] Loading /root/.cache/huggingface/hub/models--nvidia--Llama-3.1-8B-Instruct-FP8/snapshots/026c7c29fdd02d53f17c125d2dec8cb1a2251c23/model-00001-of-00002.safetensors
[07/24/2025-08:18:41] [TRT-LLM] [I] Loading /root/.cache/huggingface/hub/models--nvidia--Llama-3.1-8B-Instruct-FP8/snapshots/026c7c29fdd02d53f17c125d2dec8cb1a2251c23/model-00002-of-00002.safetensors
Loading weights: 100%|██████████| 617/617 [00:01<00:00, 324.91it/s]
Model init total -- 6.44s
[07/24/2025-08:18:44] [TRT-LLM] [W] Both free_gpu_memory_fraction and max_tokens are set (to 0.800000011920929 and 28384, respectively). The smaller value will be used.
[TensorRT-LLM][INFO] Max KV cache pages per sequence: 5 [window size=157]
[TensorRT-LLM][INFO] Number of tokens per block: 32.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 1.73 GiB for max tokens in paged KV cache (28384).
[07/24/2025-08:18:44] [TRT-LLM] [I] max_seq_len=157, max_num_requests=4096, max_num_tokens=23552
[07/24/2025-08:18:44] [TRT-LLM] [I] [Autotuner]: Autotuning process starts ...
[07/24/2025-08:18:44] [TRT-LLM] [I] Run autotuning warmup for batch size=1
2025-07-24 08:18:44,189 - INFO - flashinfer.jit: Loading JIT ops: norm
/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  warnings.warn(
/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  warnings.warn(
2025-07-24 08:18:44,330 - INFO - flashinfer.jit: Finished loading JIT ops: norm
2025-07-24 08:18:45,198 - INFO - flashinfer.jit: Loading JIT ops: silu_and_mul
/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  warnings.warn(
/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  warnings.warn(
2025-07-24 08:18:45,368 - INFO - flashinfer.jit: Finished loading JIT ops: silu_and_mul
[07/24/2025-08:18:46] [TRT-LLM] [I] Autotuner Cache size after warmup 0
[07/24/2025-08:18:46] [TRT-LLM] [I] [Autotuner]: Autotuning process ends
[07/24/2025-08:18:46] [TRT-LLM] [I] Creating CUDA graph instances for 24 batch sizes.
[07/24/2025-08:18:46] [TRT-LLM] [I] Memory used after loading model weights (inside torch) in memory usage profiling: 8.55 GiB
[07/24/2025-08:18:46] [TRT-LLM] [I] Memory used after loading model weights (outside torch) in memory usage profiling: 3.14 GiB
[07/24/2025-08:18:47] [TRT-LLM] [I] Memory dynamically allocated during inference (inside torch) in memory usage profiling: 2.95 GiB
[07/24/2025-08:18:47] [TRT-LLM] [I] Memory used outside torch (e.g., NCCL and CUDA graphs) in memory usage profiling: 3.78 GiB
[07/24/2025-08:18:47] [TRT-LLM] [I] Peak memory during memory usage profiling (torch + non-torch): 15.29 GiB, available KV cache memory when calculating max tokens: 52.51 GiB, fraction is set 0.800000011920929, kv size is 65536, device total memory 79.19 GiB, , tmp kv_mem 1.73 GiB
[07/24/2025-08:18:47] [TRT-LLM] [I] Estimated max tokens in KV cache : 860308
[07/24/2025-08:18:47] [TRT-LLM] [W] Both free_gpu_memory_fraction and max_tokens are set (to 0.800000011920929 and 860308, respectively). The smaller value will be used.
[TensorRT-LLM][INFO] Max KV cache pages per sequence: 5 [window size=157]
[TensorRT-LLM][INFO] Number of tokens per block: 32.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 52.51 GiB for max tokens in paged KV cache (860320).
[07/24/2025-08:18:47] [TRT-LLM] [I] max_seq_len=157, max_num_requests=4096, max_num_tokens=23552
[07/24/2025-08:18:47] [TRT-LLM] [I] [Autotuner]: Autotuning process starts ...
[07/24/2025-08:18:47] [TRT-LLM] [I] Run autotuning warmup for batch size=1
[07/24/2025-08:18:47] [TRT-LLM] [I] Autotuner Cache size after warmup 0
[07/24/2025-08:18:47] [TRT-LLM] [I] [Autotuner]: Autotuning process ends
[07/24/2025-08:18:47] [TRT-LLM] [I] Creating CUDA graph instances for 24 batch sizes.
[07/24/2025-08:18:48] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=4096
[07/24/2025-08:18:48] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=2048
[07/24/2025-08:18:49] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=1024
[07/24/2025-08:18:49] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=512
[07/24/2025-08:18:50] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=256
[07/24/2025-08:18:50] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=128
[07/24/2025-08:18:50] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=120
[07/24/2025-08:18:51] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=112
[07/24/2025-08:18:51] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=104
[07/24/2025-08:18:51] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=96
[07/24/2025-08:18:52] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=88
[07/24/2025-08:18:52] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=80
[07/24/2025-08:18:52] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=72
[07/24/2025-08:18:52] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=64
[07/24/2025-08:18:53] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=56
[07/24/2025-08:18:53] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=48
[07/24/2025-08:18:53] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=40
[07/24/2025-08:18:54] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=32
[07/24/2025-08:18:54] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=24
[07/24/2025-08:18:54] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=16
[07/24/2025-08:18:55] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=8
[07/24/2025-08:18:55] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=4
[07/24/2025-08:18:55] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=2
[07/24/2025-08:18:55] [TRT-LLM] [I] Run generation only CUDA graph warmup for batch size=1
[07/24/2025-08:18:56] [TRT-LLM] [I] Setting up for warmup...
[07/24/2025-08:18:56] [TRT-LLM] [I] Running warmup.
[07/24/2025-08:18:56] [TRT-LLM] [I] Starting benchmarking async task.
[07/24/2025-08:18:56] [TRT-LLM] [I] Starting benchmark...
[07/24/2025-08:18:56] [TRT-LLM] [I] Request submission complete. [count=2, time=0.0001s, rate=39839.84 req/s]
[07/24/2025-08:18:56] [TRT-LLM] [I] Benchmark complete.
[07/24/2025-08:18:56] [TRT-LLM] [I] Stopping LLM backend.
[07/24/2025-08:18:56] [TRT-LLM] [I] Cancelling all 0 tasks to complete.
[07/24/2025-08:18:56] [TRT-LLM] [I] All tasks cancelled.
[07/24/2025-08:18:56] [TRT-LLM] [I] LLM Backend stopped.
[07/24/2025-08:18:56] [TRT-LLM] [I] Worker task cancelled.
[07/24/2025-08:18:56] [TRT-LLM] [I] Warmup done.
[07/24/2025-08:18:56] [TRT-LLM] [I] No log path provided, skipping logging.
[07/24/2025-08:18:56] [TRT-LLM] [I] Starting benchmarking async task.
[07/24/2025-08:18:56] [TRT-LLM] [I] Starting benchmark...
[07/24/2025-08:18:56] [TRT-LLM] [I] Request submission complete. [count=30000, time=0.0610s, rate=492182.63 req/s]
[07/24/2025-08:20:57] [TRT-LLM] [I] Benchmark complete.
[07/24/2025-08:20:57] [TRT-LLM] [I] Stopping LLM backend.
[07/24/2025-08:20:57] [TRT-LLM] [I] Cancelling all 0 tasks to complete.
[07/24/2025-08:20:57] [TRT-LLM] [I] All tasks cancelled.
[07/24/2025-08:20:57] [TRT-LLM] [I] LLM Backend stopped.
[07/24/2025-08:20:57] [TRT-LLM] [I] Worker task cancelled.
[07/24/2025-08:20:57] [TRT-LLM] [I] Benchmark done. Reporting results...
[07/24/2025-08:20:58] [TRT-LLM] [I] Validating KV Cache config against kv_cache_dtype="auto"
[07/24/2025-08:20:58] [TRT-LLM] [I] KV cache quantization set to "auto". Using checkpoint KV quantization.
[07/24/2025-08:20:58] [TRT-LLM] [I]

===========================================================
= PYTORCH BACKEND
===========================================================
Model:			nvidia/Llama-3.1-8B-Instruct-FP8
Model Path:		None
TensorRT-LLM Version:	0.20.0
Dtype:			bfloat16
KV Cache Dtype:		FP8
Quantization:		FP8

===========================================================
= REQUEST DETAILS
===========================================================
Number of requests:             30000
Number of concurrent requests:  13299.1087
Average Input Length (tokens):  128.0000
Average Output Length (tokens): 28.0000
===========================================================
= WORLD + RUNTIME INFORMATION
===========================================================
TP Size:                1
PP Size:                1
EP Size:                None
Max Runtime Batch Size: 4096
Max Runtime Tokens:     23552
Scheduling Policy:      GUARANTEED_NO_EVICT
KV Memory Percentage:   80.00%
Issue Rate (req/sec):   2.3129E+15

===========================================================
= PERFORMANCE OVERVIEW
===========================================================
Request Throughput (req/sec):                     248.4751
Total Output Throughput (tokens/sec):             6957.3030
Total Token Throughput (tokens/sec):              38762.1169
Total Latency (ms):                               120736.4401
Average request latency (ms):                     53522.9014
Per User Output Throughput [w/ ctx] (tps/user):   0.5350
Per GPU Output Throughput (tps/gpu):              6957.3030

-- Request Latency Breakdown (ms) -----------------------

[Latency] P50    : 54486.1063
[Latency] P90    : 64358.7218
[Latency] P95    : 66932.7024
[Latency] P99    : 68887.7569
[Latency] MINIMUM: 39903.1921
[Latency] MAXIMUM: 69473.9218
[Latency] AVERAGE: 53522.9014

===========================================================
= DATASET DETAILS
===========================================================
Dataset Path:         /workspace/datasets/nvidia/Llama-3.1-8B-Instruct-FP8/128_128_30000.jsonl
Number of Sequences:  30000

-- Percentiles statistics ---------------------------------

        Input              Output           Seq. Length
-----------------------------------------------------------
MIN:   128.0000            28.0000           156.0000
MAX:   128.0000            28.0000           156.0000
AVG:   128.0000            28.0000           156.0000
P50:   128.0000            28.0000           156.0000
P90:   128.0000            28.0000           156.0000
P95:   128.0000            28.0000           156.0000
P99:   128.0000            28.0000           156.0000
===========================================================

[07/24/2025-08:20:58] [TRT-LLM] [I] Thread proxy_dispatch_result_thread stopped.
[07/24/2025-08:20:58] [TRT-LLM] [I] Thread proxy_dispatch_kv_cache_events_thread stopped.
[07/24/2025-08:20:58] [TRT-LLM] [I] Thread proxy_dispatch_stats_thread stopped.
[07/24/2025-08:20:58] [TRT-LLM] [I] Thread await_response_thread stopped.
[07/24/2025-08:20:58] [TRT-LLM] [I] Thread dispatch_stats_thread stopped.
[07/24/2025-08:20:58] [TRT-LLM] [I] Thread dispatch_kv_cache_events_thread stopped.

[zbpatel]

Hey a couple things:
1.(sorry I was wrong about using an unquantized model, please ignore, I misread the log)
2. in both of the rerun tests it looks like you only ran 128 isl, 28 osl. guessing this was just a typo :)

        Input              Output           Seq. Length
-----------------------------------------------------------
MIN:   128.0000            28.0000           156.0000

[skymizerFuji]

Hi, sorry I did make some typos.
I re-ran the benchmark, and the TPS is now 15,896.59. It's getting better, but still doesn't meet the expected 27,688.36 tokens/sec.

extra-llm-api-config.yml

use_cuda_graph: true
cuda_graph_padding_enabled: true
cuda_graph_batch_sizes:
  - 1
  - 2
  - 4
  - 8
  - 16
  - 32
  - 64
  - 128
  - 256
  - 384
  - 512
  - 1024
  - 2048
  - 4096
  - 8192

Output (click to expand)

root@16eea827d55a:/workspace/trtllm-benchmark# trtllm-bench --model nvidia/Llama-3.1-8B-Instruct-FP8 throughput --dataset /workspace/datasets/nvidia/Llama-3.1-8B-Instruct-FP8/128_128_30000.jsonl --backend pytorch --extra_llm_api_options extra-llm-api-config.yml --kv_cache_free_gpu_mem_fraction 0.8
...
===========================================================
= PYTORCH BACKEND
===========================================================
Model:			nvidia/Llama-3.1-8B-Instruct-FP8
Model Path:		None
TensorRT-LLM Version:	0.20.0
Dtype:			bfloat16
KV Cache Dtype:		FP8
Quantization:		FP8

===========================================================
= REQUEST DETAILS
===========================================================
Number of requests:             30000
Number of concurrent requests:  12703.5221
Average Input Length (tokens):  128.0000
Average Output Length (tokens): 128.0000
===========================================================
= WORLD + RUNTIME INFORMATION
===========================================================
TP Size:                1
PP Size:                1
EP Size:                None
Max Runtime Batch Size: 4096
Max Runtime Tokens:     8192
Scheduling Policy:      GUARANTEED_NO_EVICT
KV Memory Percentage:   80.00%
Issue Rate (req/sec):   3.9445E+15

===========================================================
= PERFORMANCE OVERVIEW
===========================================================
Request Throughput (req/sec):                     124.1921
Total Output Throughput (tokens/sec):             15896.5928
Total Token Throughput (tokens/sec):              31793.1856
Total Latency (ms):                               241561.1979
Average request latency (ms):                     102289.2670
Per User Output Throughput [w/ ctx] (tps/user):   1.2838
Per GPU Output Throughput (tps/gpu):              15896.5928

-- Request Latency Breakdown (ms) -----------------------

[Latency] P50    : 101868.5987
[Latency] P90    : 124564.6661
[Latency] P95    : 125361.4326
[Latency] P99    : 128251.4591
[Latency] MINIMUM: 73431.8791
[Latency] MAXIMUM: 128958.9539
[Latency] AVERAGE: 102289.2670

===========================================================
= DATASET DETAILS
===========================================================
Dataset Path:         /workspace/datasets/nvidia/Llama-3.1-8B-Instruct-FP8/128_128_30000.jsonl
Number of Sequences:  30000

-- Percentiles statistics ---------------------------------

        Input              Output           Seq. Length
-----------------------------------------------------------
MIN:   128.0000           128.0000           256.0000
MAX:   128.0000           128.0000           256.0000
AVG:   128.0000           128.0000           256.0000
P50:   128.0000           128.0000           256.0000
P90:   128.0000           128.0000           256.0000
P95:   128.0000           128.0000           256.0000
P99:   128.0000           128.0000           256.0000
===========================================================

</details>

[zbpatel]

I think the remaining difference here is likely that the test we ran was using the H100 SXM 80GB variant which is a higher performance version than H100 PCIE variant your test used. The SXM variant has a significantly higher power budget, memory bandwidth and cuda core count which is going to lead to higher performance on this application vs PCIE. On the specifications tab of this page you can see more information about these differences.

Unfortunately, I was not able to find any recent benchmarks of the PCIE variant that are available to share to compare against. If I am able to get that, I will update this thread.

I'm also working on updating our performance document right now, I will add some more information about the GPU variants used as this is important information and should be made more clear on the page.

[zbpatel]

Ok, I was able to get some benchmark data for 0.20 on H100 PCIE - 17,500 output tokens / s, which is ~10% higher than what you have. You might be able to close the gap a little more by setting the --kv_cache_free_gpu_mem_fraction 0.8 to .85 or higher.

I think this is within a reasonable delta to your results since there might be other variables that are hard to control or slight differences on the system your provider is offering, I'm going to close this issue.

You can find the PR for 0.21 perf information here: #6270