From 844e5b2c5a470317b4ed3caf666d50fbc4f47d84 Mon Sep 17 00:00:00 2001 From: Rafael Vasquez Date: Wed, 6 Nov 2024 16:48:56 -0500 Subject: [PATCH] Uses git mv to preserve history of benchmarks.rst Signed-off-by: Rafael Vasquez --- docs/source/performance/benchmarks.rst | 30 +++++++++----------------- 1 file changed, 10 insertions(+), 20 deletions(-) diff --git a/docs/source/performance/benchmarks.rst b/docs/source/performance/benchmarks.rst index 6d4d7b544cb5d..e5c8d6a55de63 100644 --- a/docs/source/performance/benchmarks.rst +++ b/docs/source/performance/benchmarks.rst @@ -1,33 +1,23 @@ .. _benchmarks: -================ -Benchmark Suites -================ +Benchmark suites of vLLM +======================== -vLLM contains two sets of benchmarks: - -+ :ref:`Performance benchmarks ` -+ :ref:`Nightly benchmarks ` -.. _performance_benchmarks: - -Performance Benchmarks ----------------------- +vLLM contains two sets of benchmarks: -The performance benchmarks are used for development to confirm whether new changes improve performance under various workloads. They are triggered on every commit with both the ``perf-benchmarks`` and ``ready`` labels, and when a PR is merged into vLLM. ++ **Performance benchmarks**: benchmark vLLM's performance under various workloads at a high frequency (when a pull request (PR for short) of vLLM is being merged). See `vLLM performance dashboard `_ for the latest performance results. -The latest performance results are hosted on the public `vLLM Performance Dashboard `_. ++ **Nightly benchmarks**: compare vLLM's performance against alternatives (tgi, trt-llm, and lmdeploy) when there are major updates of vLLM (e.g., bumping up to a new version). The latest results are available in the `vLLM GitHub README `_. -More information on the performance benchmarks and their parameters can be found `here `__. -.. _nightly_benchmarks: +Trigger a benchmark +------------------- -Nightly Benchmarks ------------------- +The performance benchmarks and nightly benchmarks can be triggered by submitting a PR to vLLM, and label the PR with `perf-benchmarks` and `nightly-benchmarks`. -These compare vLLM's performance against alternatives (``tgi``, ``trt-llm``, and ``lmdeploy``) when there are major updates of vLLM (e.g., bumping up to a new version). They are primarily intended for consumers to evaluate when to choose vLLM over other options and are triggered on every commit with both the ``perf-benchmarks`` and ``nightly-benchmarks`` labels. -The latest nightly benchmark results are shared in major release blog posts such as `vLLM v0.6.0 `_. +.. note:: -More information on the nightly benchmarks and their parameters can be found `here `__. \ No newline at end of file + Please refer to `vLLM performance benchmark descriptions `_ and `vLLM nightly benchmark descriptions `_ for detailed descriptions on benchmark environment, workload and metrics.