Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add real bs to profiling #600

Closed
wants to merge 917 commits into from
Closed

Add real bs to profiling #600

wants to merge 917 commits into from

Conversation

kamil-kaczor
Copy link

Adds real batch size and sequence length to description of a graph in the high-level profiling.

russellb and others added 30 commits November 19, 2024 09:48
@russellb
[Doc] fix link for page that was renamed (vllm-project#10455)
fd9f124 
Signed-off-by: Russell Bryant <[email protected]>
@youkaichao
[6/N] torch.compile rollout to users (vllm-project#10437)
803f37e 
Signed-off-by: youkaichao <[email protected]>
@russellb
[Core] Avoid metrics log noise when idle (vllm-project#8868)
efa9084 
Signed-off-by: Russell Bryant <[email protected]>
@ElizaWszola
[Model][Quantization] HQQ support through Marlin kernel expansion (vl…
b00b33d 
…lm-project#9766)

Signed-off-by: ElizaWszola <[email protected]>
@maxdebayser
Change granite chat template to keep json list formatting for tool ca…
a324d3a 
…lls (vllm-project#10452)

Signed-off-by: Max de Bayser <[email protected]>
@Alexei-V-Ivanov-AMD
[CI/Build] Update Dockerfile.rocm (vllm-project#10434)
d5b68ab 
Signed-off-by: Alexei V. Ivanov <[email protected]>
@LucasWilkinson
[Bugfix] Marlin 2:4 temp fix for large M dim (>256) (vllm-project#10464)
d200972 
Signed-off-by: Lucas Wilkinson <[email protected]>
@liuyanyi
[Misc] Add __setitem__ for LazyDict (vllm-project#10469)
9e05252 
Signed-off-by: Yanyi Liu <[email protected]>
@Isotr0py
[Bugfix] Fix Mamba model initialization and MLP Speculator weights lo…
ad44437 
…ading (vllm-project#10456)

Signed-off-by: Isotr0py <[email protected]>
@DarkLight1337
[Bugfix] Enforce no chunked prefill for embedding models (vllm-projec…
b4be5a8 
…t#10470)

Signed-off-by: DarkLight1337 <[email protected]>
@rafvasq
[CI/Build] Add sphinx/rst linter for docs (vllm-project#10366)
709c9f1
@wchen61
[CI/Build] Support compilation with local cutlass path (vllm-project#…
7629a9c 
…10423) (vllm-project#10424)
@khluu
[ci/build] Combine nightly and optional (vllm-project#10465)
ed701ca
@skylee-01
[model] Reduce medusa weight (vllm-project#10454)
343041c 
Signed-off-by: skylee-01 <[email protected]>
@DarkLight1337
[Bugfix] Handle conflicts between modern and legacy fields (vllm-proj…
09dbf9f 
…ect#10471)

Signed-off-by: DarkLight1337 <[email protected]>
@MengqingCao
[Platforms] Refactor xpu code (vllm-project#10468)
d5b2844 
Signed-off-by: MengqingCao <[email protected]>
@bigPYJ1151
[Hardware][CPU] Support chunked-prefill and prefix-caching on CPU (vl…
63f1fde 
…lm-project#10355)

Signed-off-by: jiang1.li <[email protected]>
@michalkuligowski
Update ray_hpu_executor.py (#522)
8c3f56a
@mfylcek
Random sampler warmup (#506)
6338608 
Random sampler warmup
@jkaniecki
Skip empty steps in multi step sheduling (#526)
efe0268 
This change allows to skip empty steps in multistep scenario. We are
currently wasting host time on launching n-2 empty steps.
This PR removes it. The gain will be visible after device time
optimizations, as we are currently limited by HPU calculations inside
multistep.
@youkaichao
[platforms] restore xpu check for parallel config (vllm-project#10479)
772a667 
Signed-off-by: youkaichao <[email protected]>
@simon-mo
[perf bench] H200 development (vllm-project#9768)
5f1d6af 
Signed-off-by: simon-mo <[email protected]>
@youkaichao
[7/N] torch.compile, reduce compilation time (vllm-project#10460)
0cd3d97 
Signed-off-by: youkaichao <[email protected]>
@gcalmettes
[Bugfix]: allow extra fields in requests to openai compatible server (v…
c68f7ed 
…llm-project#10463)

Signed-off-by: Guillaume Calmettes <[email protected]>
@WoosukKwon
[TPU] Implement prefix caching for TPUs (vllm-project#10307)
2f77b6c 
Signed-off-by: Woosuk Kwon <[email protected]>
@youkaichao
[torch.compile] limit inductor threads and lazy import quant (vllm-pr…
388ee3d 
…oject#10482)

Signed-off-by: youkaichao <[email protected]>
@pavanimajety
[Core] Add Sliding Window Support with Flashinfer (vllm-project#10462)
6c1208d 
Signed-off-by: Pavani Majety <[email protected]>
@MengqingCao
[Platforms] Add device_type in Platform (vllm-project#10508)
9d82717 
Signed-off-by: MengqingCao <[email protected]>
@ProExpertProg
[torch.compile] Inductor code caching fix (vllm-project#10273)
8b0fe06 
Signed-off-by: luka <[email protected]>
Signed-off-by: Luka Govedic <[email protected]>
@DarkLight1337
[Misc] Increase default video fetch timeout (vllm-project#10495)
3430857 
Signed-off-by: DarkLight1337 <[email protected]>
xuechendi and others added 28 commits December 4, 2024 16:28
@xuechendi @mgoin
Benchmark serving structured output (vllm-project#10880)
82eb5ea 
Signed-off-by: Chendi Xue <[email protected]>
Co-authored-by: Michael Goin <[email protected]>
@dtrifiro @youkaichao
[CI/Build] improve python-only dev setup (vllm-project#9621)
e4c34c2 
Signed-off-by: Daniele Trifirò <[email protected]>
Signed-off-by: youkaichao <[email protected]>
Co-authored-by: youkaichao <[email protected]>
@WoosukKwon
[V1] Fix when max_model_len is not divisible by block_size (vllm-proj…
2a56e12 
…ect#10903)

Signed-off-by: Woosuk Kwon <[email protected]>
@khluu
[benchmark] Make H100 benchmark optional (vllm-project#10908)
7883c2b
@mgoin
[Bugfix] Fallback to outlines for complex json schemas (vllm-project#…
8d370e9 
…10899)

Signed-off-by: mgoin <[email protected]>
@DarkLight1337
[Doc] Create a new "Usage" section (vllm-project#10827)
aa39a8e 
Signed-off-by: DarkLight1337 <[email protected]>
@jeejeelee
[Bugfix] Fix BNB loader target_modules (vllm-project#10720)
1f958a7 
Signed-off-by: Jee Jee Li <[email protected]>
@tjohnson31415
[Misc] Update llama 3.2 template to support system prompt with images (…
39c89e7 
…vllm-project#10901)

Signed-off-by: Travis Johnson <[email protected]>
@xuechendi
[CI/BUILD] Spec decode ci (#524)
ad29332 
Add spec decode CI

---------

Signed-off-by: Chendi.Xue <[email protected]>
@jeejeelee
[Misc][LoRA] Clean up the function interface of Punica (vllm-project#…
571da8f 
…10917)

Signed-off-by: Jee Jee Li <[email protected]>
@Isotr0py @DarkLight1337
[CI/Build] Bump test transformers version (vllm-project#10106)
998eeaf 
Signed-off-by: Isotr0py <[email protected]>
Signed-off-by: DarkLight1337 <[email protected]>
Co-authored-by: DarkLight1337 <[email protected]>
@kzawora-intel
[Misc][Gaudi] Avoid torch.compile and enable lazy collectives (vllm-p…
a430652 
…roject#10897)

Signed-off-by: Konrad Zawora <[email protected]>
@youkaichao
[ci][build] add tests for python only compilation (vllm-project#10915)
9743d64 
Signed-off-by: youkaichao <[email protected]>
@youkaichao
[torch.compile] use size tuning for specific sizes (vllm-project#10933)
db87eb6 
Signed-off-by: youkaichao <[email protected]>
@youkaichao @WoosukKwon
[torch.compile] add logging for compilation time (vllm-project#10941)
b031a45 
Signed-off-by: youkaichao <[email protected]>
Co-authored-by: Woosuk Kwon <[email protected]>
@szutenberg
Add host traces to high-level profilings (#577)
a805205
@DarkLight1337
[CI/Build] Fix broken multimodal test (vllm-project#10950)
222f5b0
@youkaichao
[torch.compile] fix deprecated code (vllm-project#10948)
a1887f2 
Signed-off-by: youkaichao <[email protected]>
@nirda7
Enable patching Fused SDPA (#569)
e349f70
@xuechendi @michalkuligowski
revert INC fixed version installation in requirements-hpu.txt for 1.1…
6a4f673 
…9, add tmp one for 1.18 (#547)

Fix 1.19 INC incompatible issues

https://jira.habana-labs.com/browse/SW-209688

Signed-off-by: Chendi.Xue <[email protected]>
Co-authored-by: Michał Kuligowski <[email protected]>
@kzawora-intel
Add multiprocessing HPU executor (#559)
e0e47ed
@xuechendi
fix WorkerWrapperBase and spec_decode rebase (#582)
858e0a0 
Need to test with TORCH_COMPILE_DISABLE=true

Signed-off-by: Chendi.Xue <[email protected]>
@kzawora-intel
Merge remote-tracking branch 'origin/habana_main' into HEAD
21323ed
@kzawora-intel
Merge remote-tracking branch 'upstream/main' into HEAD
d8f395e
@kzawora-intel
fix mypy errors
48ab12b
@kzawora-intel
fix (hopefully) all linter errors
9204975
@kzawora-intel
Dec 06 rebase (#571)
ad8d5b7
@kamil-kaczor
Add real BS and seq_len to profiling
bcf66c5
@kamil-kaczor kamil-kaczor deleted the add_real_BS_to_profiling branch December 9, 2024 12:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.