Add single node benchmarks for vLLM Rust engine. #253
Conversation
|
Build container with code. Execute in container with 2 GPUs: |
piotrm-nvidia
requested review from
tanmayv25,
ishandhanani,
alec-flowers,
nnshah1 and
nv-anants
as code owners
|
|
||
| echo "Running benchmark..." | ||
|
|
||
| CONFIG_PREFIX="prefill_tp1dp1_generate_t1d1" |
There was a problem hiding this comment.
looks like we are actually using dp8? also typo t1d1
|
|
||
| # Benchmark Scripts | ||
|
|
||
| This folder contains scripts and utilities for benchmarking disaggregated (and baseline) inference configurations with various GPU topologies and model sizes. The primary scripts are: |
There was a problem hiding this comment.
"Baseline" should be defined
|
|
||
| echo "Activating Triton environment..." | ||
|
|
||
| source /opt/triton/venv/bin/activate |
There was a problem hiding this comment.
is this line really necessary?
(applies to other files as well)
There was a problem hiding this comment.
virtualenv should be activated by default in the vllm container now, shouldn't need to manually activate
|
|
||
| def get_label_from_name(name): | ||
| """ | ||
| Parses out a human-friendly label from the directory name. |
There was a problem hiding this comment.
Maybe clearer would be:
Drops "_tpXdpY" parts from the given string.
| Parses out a human-friendly label from the directory name. | ||
| For example, 'purevllm_tp1dp1' -> 'purevllm' | ||
| 'rustvllm_tp2dp4' -> 'rustvllm' | ||
| 'context_tp2dp2' -> 'context' (you could replace 'context' with 'disagg' if desired) |
There was a problem hiding this comment.
I don't understand this talk of possible replacement
| LOGGER = logging.getLogger(__name__) | ||
|
|
||
|
|
||
| def parse_tp_dp(name): |
There was a problem hiding this comment.
Maybe count_gpus_from_tpdp_occurances
| @@ -0,0 +1,375 @@ | |||
| #!/usr/bin/env python3 | |||
There was a problem hiding this comment.
Some of the code looks like something that we might want to use again in the future. Maybe we could extract it as a library? If so, we should think of a clean API and make it clearer which functions are "public", which "private" and make type annotations at least in the public part.
|
|
||
| This folder contains scripts and utilities for benchmarking disaggregated (and baseline) inference configurations with various GPU topologies and model sizes. The primary scripts are: | ||
|
|
||
| 1. **`bench_2GPUs_8B.sh`** |
There was a problem hiding this comment.
Abbreviated name (bench instead of benchmark) is fine in a private context, but this is a script that we give to the users. Let's stay with a full name benchmark_2GPUs_8B.sh
| @@ -0,0 +1,162 @@ | |||
| #!/bin/bash | |||
There was a problem hiding this comment.
Large part of the scripts is repeated 3 times, but it's fine for now. I work on run_deployment.py script in integration tests and I hoped that it can be used here instead soon enough.
| LOGGER = logging.getLogger(__name__) | ||
|
|
||
|
|
||
| def wait_for_server(url, model, timeout=300): |
There was a problem hiding this comment.
Once integration tests are merged to Github a similar function could be imported.
piotrm-nvidia
requested review from
nv-blazejkubiak,
ryanolson,
grahamking,
paulhendricks,
biswapanda,
tmonty12 and
GuanLuo
as code owners
No description provided.