A high-throughput and memory-efficient inference and serving engine for LLMs

Intelligent Mixture-of-Models Router for Efficient LLM Inference

Cost-efficient and pluggable Infrastructure components for GenAI inference

Community maintained hardware plugin for vLLM on Ascend

Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs

Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment with vLLM

Community maintained hardware plugin for vLLM on Intel Gaudi

Community maintained hardware plugin for vLLM on Spyre

A unified library for building, evaluating, and storing speculative decoding algorithms for LLM inference in vLLM

This repo hosts code for vLLM CI & Performance Benchmark infrastructure.