diff --git a/README.md b/README.md index 186bc27..d28141f 100644 --- a/README.md +++ b/README.md @@ -1,15 +1,17 @@ # nm-vllm-certs -## Overview +## Overview The `nm-vllm` packages published in this repository are Neuralmagic Enterprise Edition of [vLLM](https://github.com/vllm-project/vllm). Packages are versioned Python wheels and docker images. These are released as "production level" official releases and "beta level" Nightly's. Official releases are made at the discretion of Neuralmagic, but typically track with `vllm` releases. These wheels are available via "public pypi" as well as ["nm-pypi"](https://pypi.neuralmagic.com). Nightly's are released every night given green runs in automation. The wheels are available at ["nm-pypi"](https://pypi.neuralmagic.com). + ## Installation + ### PyPI The [nm-vllm PyPi package](https://pypi.neuralmagic.com/simple/nm-vllm/index.html) includes pre-compiled binaries for CUDA (version 12.1) kernels. For other PyTorch or CUDA versions, please compile the package from source. @@ -23,9 +25,10 @@ To utilize the weight sparsity features, include the optional `sparse` dependenc pip install nm-vllm[sparse] --extra-index-url https://pypi.neuralmagic.com/simple ``` + ### Docker -The [`nm-vllm-ent` container registry](https://github.com/neuralmagic/nm-vllm-certs/pkgs/container/nm-vllm-ent) includes premade docker images. +The `nm-vllm-ent` [container registry](https://github.com/neuralmagic/nm-vllm-certs/pkgs/container/nm-vllm-ent) includes premade docker images. Launch the OpenAI-compatible server with: @@ -34,6 +37,12 @@ MODEL_ID=Qwen/Qwen2-0.5B-Instruct docker run --gpus all --shm-size 2g ghcr.io/neuralmagic/nm-vllm-ent:latest --model $MODEL_ID ``` + +## Benchmarks + +Please see our benchmarking results [here]( https://neuralmagic.github.io/nm-vllm-certs/dev/bench/). + + ## Models Neural Magic maintains a variety of optimized models on our Hugging Face organization profiles: