Skip to content

Commit

Permalink
add Benchmark link (#5)
Browse files Browse the repository at this point in the history
* add Benchmarking session

* adjust link scope

---------

Co-authored-by: andy-neuma <[email protected]>
  • Loading branch information
andy-neuma and andy-neuma authored Jul 24, 2024
1 parent 93bdf2f commit daa8fd8
Showing 1 changed file with 11 additions and 2 deletions.
13 changes: 11 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,17 @@
# nm-vllm-certs

## Overview

## Overview
The `nm-vllm` packages published in this repository are Neuralmagic Enterprise Edition of [vLLM](https://github.com/vllm-project/vllm). Packages are versioned Python wheels and docker images. These are released as "production level" official releases and "beta level" Nightly's.

Official releases are made at the discretion of Neuralmagic, but typically track with `vllm` releases. These wheels are available via "public pypi" as well as ["nm-pypi"](https://pypi.neuralmagic.com).

Nightly's are released every night given green runs in automation. The wheels are available at ["nm-pypi"](https://pypi.neuralmagic.com).


## Installation


### PyPI
The [nm-vllm PyPi package](https://pypi.neuralmagic.com/simple/nm-vllm/index.html) includes pre-compiled binaries for CUDA (version 12.1) kernels. For other PyTorch or CUDA versions, please compile the package from source.

Expand All @@ -23,9 +25,10 @@ To utilize the weight sparsity features, include the optional `sparse` dependenc
pip install nm-vllm[sparse] --extra-index-url https://pypi.neuralmagic.com/simple
```


### Docker

The [`nm-vllm-ent` container registry](https://github.com/neuralmagic/nm-vllm-certs/pkgs/container/nm-vllm-ent) includes premade docker images.
The `nm-vllm-ent` [container registry](https://github.com/neuralmagic/nm-vllm-certs/pkgs/container/nm-vllm-ent) includes premade docker images.

Launch the OpenAI-compatible server with:

Expand All @@ -34,6 +37,12 @@ MODEL_ID=Qwen/Qwen2-0.5B-Instruct
docker run --gpus all --shm-size 2g ghcr.io/neuralmagic/nm-vllm-ent:latest --model $MODEL_ID
```


## Benchmarks

Please see our benchmarking results [here]( https://neuralmagic.github.io/nm-vllm-certs/dev/bench/).


## Models

Neural Magic maintains a variety of optimized models on our Hugging Face organization profiles:
Expand Down

0 comments on commit daa8fd8

Please sign in to comment.