Code

Upstream merge 24 12 16#330

Code

Merged

gshtras merged 102 commits intomainROCm/vllm:mainfrom upstream_merge_24_12_16ROCm/vllm:upstream_merge_24_12_16Copy head branch name to clipboardDec 16, 2024

+10,604-4,101

Commits

Commits on Dec 9, 2024

• Build tpu image in release pipeline (vllm-project#10936)

  Show description for 5ed5d5f
  

  richardsliu

  and

  khluu

  authored
  5ed5d5f

• [V1] Do not store None in self.generators (vllm-project#11038)

  Show description for 6faec54
  WoosukKwon

  authored
  6faec54

Commits on Dec 10, 2024

• [Docs] Add dedicated tool calling page to docs (vllm-project#10554)

  Show description for 6d52528
  

  mgoin

  and

  tlrmchlsmth

  authored
  6d52528

• [Model] Add has_weight to RMSNorm and re-enable weights loading tracker for Mamba (vllm-project#10739)

  Show description for d1f6d1c
  Isotr0py

  authored
  d1f6d1c

• [Bugfix] Fix usage of deprecated decorator (vllm-project#11025)

  Show description for 391d7b2
  DarkLight1337

  authored
  391d7b2

• [Frontend] Use request id from header (vllm-project#10968)

  Show description for 980ad39
  joerunde

  authored
  980ad39

• [Pixtral] Improve loading (vllm-project#11040)

  patrickvonplaten

  authored
  bc192a2

• [V1] Multiprocessing Tensor Parallel Support for v1 (vllm-project#9856)

  Show description for 28b3a1c
  tlrmchlsmth

  authored
  28b3a1c

• monitor metrics of tokens per step using cudagraph batchsizes (vllm-project#11031)

  Show description for ebf7780
  youkaichao

  authored
  ebf7780

• [Bugfix] Fix xgrammar failing to read a vocab_size from LlavaConfig on PixtralHF. (vllm-project#11043)

  sjuxax

  authored
  e35879c

• Update README.md (vllm-project#11034)

  dmoliveira

  authored
  bfd6104

• [Bugfix] cuda error running llama 3.2 (vllm-project#11047)

  GeneDer

  authored
  82c73fd

• Add example of helm chart for vllm deployment on k8s (vllm-project#9199)

  Show description for fe2e10c
  mfournioux

  authored
  fe2e10c

• [Bugfix] Handle <|tool_call|> token in granite tool parser (vllm-project#11039)

  Show description for beb16b2
  tjohnson31415

  authored
  beb16b2

• [Misc][LoRA] Add PEFTHelper for LoRA (vllm-project#11003)

  Show description for d05f886
  jeejeelee

  authored
  d05f886

• [Bugfix] Backport request id validation to v0 (vllm-project#11036)

  Show description for 9b9cef3
  joerunde

  authored
  9b9cef3

• [BUG] Remove token param vllm-project#10921 (vllm-project#11022)

  Show description for 250ee65
  flaviabeo

  authored
  250ee65

• [Core] Update to outlines >= 0.1.8 (vllm-project#10576)

  Show description for e739194
  russellb

  authored
  e739194

• [torch.compile] add a flag to track batchsize statistics (vllm-project#11059)

  Show description for 75f89dc
  youkaichao

  authored
  75f89dc

• [V1][Bugfix] Always set enable_chunked_prefill = True for V1 (vllm-project#11061)

  Show description for 134810b
  WoosukKwon

  authored
  134810b

Commits on Dec 11, 2024

• [Bugfix] Fix Mamba multistep (vllm-project#11071)

  Show description for 9a93973
  tlrmchlsmth

  authored
  9a93973

• [Misc] LoRA + Chunked Prefill (vllm-project#9057)

  aurickq

  authored
  d5c5154

• [Model] PP support for Mamba-like models (vllm-project#10992)

  Show description for ffa48c9
  mzusman

  authored
  ffa48c9

• Fix streaming for granite tool call when <|tool_call|> is present (vllm-project#11069)

  Show description for e39400a
  maxdebayser

  authored
  e39400a

• [CI/Build] Check transformers v4.47 (vllm-project#10991)

  Show description for 2e33fe4
  DarkLight1337

  authored
  2e33fe4

• [ci/build] Fix AMD CI dependencies (vllm-project#11087)

  khluu

  authored
  3fb4b4f

• [ci/build] Fix entrypoints test and pin outlines version (vllm-project#11088)

  khluu

  authored
  9974fca

• [Core] v1: Use atexit to handle engine core client shutdown (vllm-project#11076)

  Show description for 61b1d2f
  russellb

  authored
  61b1d2f

• [Bugfix] Fix Idefics3 fails during multi-image inference (vllm-project#11080)

  Show description for 2e32f5d
  B-201

  authored
  2e32f5d

• [Bugfix]: Clamp -inf logprob values in prompt_logprobs (vllm-project#11073)

  Show description for 40766ca
  rafvasq

  authored
  40766ca

• [Misc] Split up pooling tasks (vllm-project#10820)

  Show description for 8f10d5e
  DarkLight1337

  authored
  8f10d5e

• [Doc] Update docs to refer to pooling models (vllm-project#11093)

  Show description for cad5c0a
  DarkLight1337

  authored
  cad5c0a

• [CI/Build] Enable prefix caching test for AMD (vllm-project#11098)

  Show description for b2f7754
  hissu-hyvarinen

  authored
  b2f7754

• [Doc] Installed version of llmcompressor for int8/fp8 quantization (vllm-project#11103)

  Show description for fd22220
  bingps

  authored
  fd22220

• [torch.compile] use depyf to dump torch.compile internals (vllm-project#10972)

  Show description for 91642db
  youkaichao

  authored
  91642db

• [V1] Use input_ids as input for text-only models (vllm-project#11032)

  Show description for d643c2a
  WoosukKwon

  authored
  d643c2a

• [torch.compile] remove graph logging in ci (vllm-project#11110)

  Show description for 66aaa77
  youkaichao

  authored
  66aaa77

• [core] Bump ray to use _overlap_gpu_communication in compiled graph tests (vllm-project#10410)

  Show description for 72ff3a9
  

  ruisearch42

  and

  Rui Qiao

  authored
  72ff3a9

• [CI/Build] Split up VLM tests (vllm-project#11083)

  Show description for d1e21a9
  DarkLight1337

  authored
  d1e21a9

• [V1][Core] Remove should_shutdown to simplify core process termination (vllm-project#11113)

  Show description for 452a723
  tlrmchlsmth

  authored
  452a723

Commits on Dec 12, 2024

• [V1] VLM preprocessor hashing (vllm-project#11020)

  Show description for 4e11683
  

  3 peopleauthored
  4e11683

• [Bugfix] Multiple fixes to tool streaming with hermes and mistral (vllm-project#10979)

  Show description for 7439a8b
  cedonley

  authored
  7439a8b

• [Docs] Add media kit (vllm-project#11121)

  simon-mo

  authored
  8fb26da

• Update link to LlamaStack remote vLLM guide in serving_with_llamastack.rst (vllm-project#11112)

  Show description for 24a36d6
  terrytangyuan

  authored
  24a36d6

• [Core] cleanup zmq ipc sockets on exit (vllm-project#11115)

  Show description for ccede2b
  russellb

  authored
  ccede2b

• [Model] Add support for embedding model GritLM (vllm-project#10816)

  Show description for 1da8f0e
  pooyadavoodi

  authored
  1da8f0e

• [V1] Use more persistent buffers to optimize input preparation overheads (vllm-project#11111)

  Show description for f092153
  WoosukKwon

  authored
  f092153

• [Hardware][Intel-Gaudi] Enable LoRA support for Intel Gaudi (HPU) (vllm-project#10565)

  Show description for 8195824
  SanjuCSudhakaran

  authored
  8195824

• [core][distributed] initialization from StatelessProcessGroup (vllm-project#10986)

  Show description for 62de37a
  youkaichao

  authored
  62de37a

• [Misc][LoRA] Ensure Lora Adapter requests return adapter name (vllm-project#11094)

  Show description for 85362f0
  

  Jeffwan

  and

  jeejeelee

  authored
  85362f0

• [V1] Fix torch profiling for offline inference (vllm-project#11125)

  Show description for 4816d20
  ywang96

  authored
  4816d20

• fix(docs): typo in helm install instructions (vllm-project#11141)

  Show description for d4d5291
  ramonziai

  authored
  d4d5291

• [Bugfix] Quick fix to make Pixtral-HF load correctly again after 39e227c. (vllm-project#11024)

  sjuxax

  authored
  5d71257

• [Misc] Validate grammar and fail early (vllm-project#11119)

  comaniac

  authored
  2c97eca

• Fix logging of the vLLM Config (vllm-project#11143)

  JArnoldAMD

  authored
  9f3974a

• [Bugfix] Fix value unpack error of simple connector for KVCache transfer. (vllm-project#11058)

  Show description for db6c264
  ShangmingCai

  authored
  db6c264

Commits on Dec 13, 2024

• [Misc][V1] Fix type in v1 prefix caching (vllm-project#11151)

  comaniac

  authored
  78ed8f5

• [torch.compile] Dynamic fp8 + rms_norm fusion (vllm-project#10906)

  Show description for 30870b4
  

  ProExpertProg

  and

  Varun Sundar Rabindranath

  authored
  30870b4

• [Bugfix] Use runner_type instead of task in GritLM (vllm-project#11144)

  Show description for 1efce68
  pooyadavoodi

  authored
  1efce68

• [Bugfix] Update starcoder2 to remap k/v scale names for kv_cache quantization (vllm-project#11148)

  dsikka

  authored
  3989a79

• [ROCm][AMD] Disable auto enabling chunked prefill on ROCm (vllm-project#11146)

  Show description for 00c1bde
  gshtras

  authored
  00c1bde

• [Bugfix][V1] Fix 'NoneType' object has no attribute 'hash_value' (vllm-project#11157)

  Show description for 34f1a80
  comaniac

  authored
  34f1a80

• [core] clean up cudagraph batchsize padding logic (vllm-project#10996)

  Show description for be39e3c
  youkaichao

  authored
  be39e3c

• PaliGemma 2 support (vllm-project#11142)

  janimo

  authored
  7cd7409

• [Bugfix][CI][CPU] add missing datasets package to requirements-cpu.txt (vllm-project#11159)

  Show description for f93bf2b
  bigPYJ1151

  authored
  f93bf2b

• [Frontend] Separate pooling APIs in offline inference (vllm-project#11129)

  Show description for eeec9e3
  DarkLight1337

  authored
  eeec9e3

• [V1][VLM] Fix edge case bug for InternVL2 (vllm-project#11165)

  Show description for 969da7d
  ywang96

  authored
  969da7d

• [Refactor]A simple device-related refactor (vllm-project#11163)

  Show description for d1fa714
  

  noemotiovon

  and

  noemotiovon

  authored
  d1fa714

• [Core] support LoRA and prompt adapter in content-based hashing for Block Manager v2 prefix caching (vllm-project#8240)

  llsj14

  authored
  c31d4a5

• [Bugfix] using len(tokenizer) instead of tokenizer.vocab_size in AllowedTokenIdsLogitsProcessor (vllm-project#11156)

  zhangjf-nlp

  authored
  5b0ed83

• [Misc] Add tokenizer_mode param to benchmark_serving.py (vllm-project#11174)

  Show description for 238c0d9
  alexm-neuralmagic

  authored
  238c0d9

• [Doc] Reorganize online pooling APIs (vllm-project#11172)

  Show description for 0920ab9
  DarkLight1337

  authored
  0920ab9

• [Bugfix][Hardware][CPU] Enable Gemma2 with SDPA on CPU backend (vllm-project#11169)

  janimo

  authored
  0a56bcc

• [Distributed] Allow the placement group more time to wait for resources to be ready (vllm-project#11138)

  Show description for 0d8451c
  Jeffwan

  authored
  0d8451c

Commits on Dec 14, 2024

• [Core] V1: Use multiprocessing by default (vllm-project#11074)

  Show description for 4863e5f
  russellb

  authored
  4863e5f

• [V1][Bugfix] Fix EngineCoreProc profile (vllm-project#11185)

  Show description for 4b5b8a6
  tlrmchlsmth

  authored
  4b5b8a6

• [Bugfix][V1] Re-compute an entire block when fully cache hit (vllm-project#11186)

  Show description for 9855aea
  comaniac

  authored
  9855aea

• update compressed-tensors to latest version (vllm-project#11183)

  Show description for 24a3d12
  

  dhuangnm

  and

  dhuangnm

  authored
  24a3d12

• [Core] Update outlines and increase its threadpool size (vllm-project#11140)

  Show description for 4825926
  russellb

  authored
  4825926

• [V1][Bugfix] Fix V1 TP trust-remote-code (vllm-project#11182)

  Show description for ea7bd68
  tlrmchlsmth

  authored
  ea7bd68

• [Misc] Minor improvements to the readability of PunicaWrapperBase (vllm-project#11200)

  Show description for 3cb5769
  jeejeelee

  authored
  3cb5769

• [Frontend] Add logits_processors as an extra completion argument (vllm-project#11150)

  Show description for 9c3dadd
  bradhilton

  authored
  9c3dadd

• [VLM] Fully dynamic prompt replacement in merged input processor (vllm-project#11199)

  Show description for 93abf23
  DarkLight1337

  authored
  93abf23

• Enable mypy checking on V1 code (vllm-project#11105)

  Show description for 6d917d0
  markmc

  authored
  6d917d0

• [Performance][Core] Optimize the performance of evictor v1 and v2 by applying a priority queue and lazy deletion (vllm-project#7209)

  llsj14

  authored
  8869368

Commits on Dec 15, 2024

• [[Misc]Upgrade bitsandbytes to the latest version 0.45.0 (vllm-project#11201)

  jeejeelee

  authored
  15859f2

• [torch.compile] allow tracking forward time (vllm-project#11081)

  Show description for a1c0205
  youkaichao

  authored
  a1c0205

• [Misc] Clean up multi-modal processor (vllm-project#11207)

  Show description for b10609e
  DarkLight1337

  authored
  b10609e

• [Bugfix] Fix error handling of unsupported sliding window (vllm-project#11213)

  Show description for 96d673e
  DarkLight1337

  authored
  96d673e

• [Doc] add documentation for disaggregated prefilling (vllm-project#11197)

  Show description for 38e599d
  KuntaiDu

  authored
  38e599d

• [Core] Support disaggregated prefill with Mooncake Transfer Engine (vllm-project#10884)

  Show description for d263bd9
  ShangmingCai

  authored
  d263bd9

• [V1][Minor] Cache np arange to reduce input preparation overhead (vllm-project#11214)

  Show description for 25ebed2
  WoosukKwon

  authored
  25ebed2

Commits on Dec 16, 2024

• Update deploying_with_k8s.rst (vllm-project#10922)

  AlexHe99

  authored
  da6f409

• [Bugfix] Fix block size validation (vllm-project#10938)

  chenqianfzh

  authored
  69ba344

• [Bugfix] Fix the default value for temperature in ChatCompletionRequest (vllm-project#11219)

  yansh97

  authored
  17138af

• WIP: [CI/Build] simplify Dockerfile build for ARM64 / GH200 (vllm-project#11212)

  Show description for b3b1526
  

  cennn

  and

  drikster80

  authored
  b3b1526

• [Model] Support Cohere2ForCausalLM (Cohere R7B) (vllm-project#11203)

  janimo

  authored
  bddbbcb

• [Model] Refactor Ultravox to use merged input processor (vllm-project#11198)

  Show description for d927dbc
  

  Isotr0py

  and

  DarkLight1337

  authored
  d927dbc

• [Doc] Reorder vision language examples in alphabet order (vllm-project#11228)

  Show description for 2ca830d
  Isotr0py

  authored
  2ca830d

• Merge remote-tracking branch 'upstream/main'

  gshtras

  committed
  1a8e549

• Deprecating sync_openai

  gshtras

  committed
  78440dc

• Remove new irrelevant action

  gshtras

  committed
  ddec133