[Kernel] Expand FP8 support to Ampere GPUs using FP8 Marlin

[mgoin]

This work expands FP8 support in vLLM from GPUs with hardware FP8 support (Hopper and Ada Lovelace) to GPUs without native support (currently Ampere) by introducing FP8 Marlin - a fast fused dequantization kernel for FP8 to BF16/FP16 conversion.

Key features:

• Enables FP8 quantization on a wider range of GPUs (SM 8.0 and 8.7, Ampere)
• Improves performance up to 2x in memory-bound scenarios
• Maintains accuracy comparable to FP16 baselines
• Reduces weight memory usage by 2x, allowing larger batches
• Simple to use - just specify quantization="fp8" at runtime or use pre-quantized FP8 checkpoints

Implementation details:

• Based on existing 8-bit integer support in GPTQ Marlin kernel
• Packs FP8 weights into int32 doublewords (GPTQ format) and then permutes weights into Marlin format
• Efficient 4xFP8 to 4xFP16/BF16 dequantization using bit arithmetic and SIMT operations

End-to-end performance and accuracy results:



Individual layer sweeps:

As shown in the graphs, FP8 Marlin can provide significant speedups with minimal accuracy impact. Performance gains are higher on GPUs with less memory bandwidth (A10, RTX 3090) and for larger models.

Notes:

• This weight-only approach differs slightly from the existing W8A8 FP8 quantization, offering higher accuracy because the activations have no need to be quantized
• Currently expanding scales to be channelwise; future work will revert to per-tensor scales
• This does not include support for MoE models.

Testing:

• Tested on H100, A100, and A10 GPUs

This enhancement enables more users to benefit from FP8 quantization without hardware restrictions, improving vLLM's performance and efficiency across a broader range of setups!

[robertgshaw2-neuralmagic]

This is an awesome feature!

[comaniac]

Overall LGTM. Thanks!