Code

Upstream merge 24 10 28#248

Code

Merged

gshtras merged 105 commits intomainROCm/vllm:mainfrom upstream_merge_24_10_28ROCm/vllm:upstream_merge_24_10_28Copy head branch name to clipboardOct 29, 2024

+9,696-3,186

Commits

Commits on Oct 21, 2024

• [Frontend] Reduce frequency of client cancellation checking (vllm-project#7959)

  njhill

  authored
  9d9186b

  Copy full SHA for 9d9186b

• [doc] fix format (vllm-project#9562)

  youkaichao

  authored
  d621c43

  Copy full SHA for d621c43

• [BugFix] Update draft model TP size check to allow matching target TP size (vllm-project#9394)

  Show description for 15713e3
  

  njhill

  and

  qibaoyuan

  authored
  15713e3

  Copy full SHA for 15713e3

• [Frontend] Don't log duplicate error stacktrace for every request in the batch (vllm-project#9023)

  Show description for 711f3a7
  wallashss

  authored
  711f3a7

  Copy full SHA for 711f3a7

• [CI] Make format checker error message more user-friendly by using emoji (vllm-project#9564)

  Show description for 575dceb
  KuntaiDu

  authored
  575dceb

  Copy full SHA for 575dceb

Commits on Oct 22, 2024

• 🐛 Fixup more test failures from memory profiling (vllm-project#9563)

  Show description for ef7faad
  joerunde

  authored
  ef7faad

  Copy full SHA for ef7faad

• [core] move parallel sampling out from vllm core (vllm-project#9302)

  youkaichao

  authored
  76a5e13

  Copy full SHA for 76a5e13

• [Bugfix]: serialize config by value for --trust-remote-code (vllm-project#6751)

  Show description for b729901
  

  tjohnson31415

  and

  DarkLight1337

  authored
  b729901

  Copy full SHA for b729901

• [CI/Build] Remove unnecessary fork_new_process (vllm-project#9484)

  DarkLight1337

  authored
  f085995

  Copy full SHA for f085995

• [Bugfix][OpenVINO] fix_dockerfile_openvino (vllm-project#9552)

  ngrozae

  authored
  29acd2c

  Copy full SHA for 29acd2c

• [Bugfix]: phi.py get rope_theta from config file (vllm-project#9503)

  Show description for 7469242
  

  Falko1

  and

  Isotr0py

  authored
  7469242

  Copy full SHA for 7469242

• [CI/Build] Replaced some models on tests for smaller ones (vllm-project#9570)

  Show description for c029221
  wallashss

  authored
  c029221

  Copy full SHA for c029221

• [Core] Remove evictor_v1 (vllm-project#9572)

  KuntaiDu

  authored
  ca30c3c

  Copy full SHA for ca30c3c

• [Doc] Use shell code-blocks and fix section headers (vllm-project#9508)

  Show description for f7db5f0
  rafvasq

  authored
  f7db5f0

  Copy full SHA for f7db5f0

• support TP in qwen2 bnb (vllm-project#9574)

  chenqianfzh

  authored
  0d02747

  Copy full SHA for 0d02747

• [Hardware][CPU] using current_platform.is_cpu (vllm-project#9536)

  wangshuai09

  authored
  3ddbe25

  Copy full SHA for 3ddbe25

• [V1] Implement vLLM V1 [1/N] (vllm-project#9289)

  WoosukKwon

  authored
  6c5af09

  Copy full SHA for 6c5af09

• [CI/Build][LoRA] Temporarily fix long context failure issue (vllm-project#9579)

  jeejeelee

  authored
  a48e3ec

  Copy full SHA for a48e3ec

• [Neuron] [Bugfix] Fix neuron startup (vllm-project#9374)

  Show description for 9dbcce8
  

  xendo

  and

  Jerzy Zagorski

  authored
  9dbcce8

  Copy full SHA for 9dbcce8

• [Model][VLM] Initialize support for Mono-InternVL model (vllm-project#9528)

  Isotr0py

  authored
  bb392ea

  Copy full SHA for bb392ea

• [Bugfix] Eagle: change config name for fc bias (vllm-project#9580)

  gopalsarda

  authored
  08075c3

  Copy full SHA for 08075c3

• [Hardware][Intel CPU][DOC] Update docs for CPU backend (vllm-project#6212)

  Show description for 32a1ee7
  

  3 peopleauthored
  32a1ee7

  Copy full SHA for 32a1ee7

• [Frontend] Support custom request_id from request (vllm-project#9550)

  Show description for 434984e
  guoyuhong

  authored
  434984e

  Copy full SHA for 434984e

• [BugFix] Prevent exporting duplicate OpenTelemetry spans (vllm-project#9017)

  ronensc

  authored
  cd5601a

  Copy full SHA for cd5601a

• [torch.compile] auto infer dynamic_arg_dims from type annotation (vllm-project#9589)

  youkaichao

  authored
  17c79f3

  Copy full SHA for 17c79f3

• [Bugfix] fix detokenizer shallow copy (vllm-project#5919)

  aurickq

  authored
  23b899a

  Copy full SHA for 23b899a

• [Misc] Make benchmarks use EngineArgs (vllm-project#9529)

  JArnoldAMD

  authored
  cb6fdaa

  Copy full SHA for cb6fdaa

• [Bugfix] Fix spurious "No compiled cutlass_scaled_mm ..." for W8A8 on Turing (vllm-project#9487)

  LucasWilkinson

  authored
  d1e8240

  Copy full SHA for d1e8240

• [BugFix] Fix metrics error for --num-scheduler-steps > 1 (vllm-project#8234)

  yuleil

  authored
  b17046e

  Copy full SHA for b17046e

• [Doc]: Update tensorizer docs to include vllm[tensorizer] (vllm-project#7889)

  Show description for 208cb34
  

  sethkimmel3

  and

  KaunilD

  authored
  208cb34

  Copy full SHA for 208cb34

Commits on Oct 23, 2024

• [Bugfix] Generate exactly input_len tokens in benchmark_throughput (vllm-project#9592)

  heheda12345

  authored
  65050a4

  Copy full SHA for 65050a4

• [Misc] Add an env var VLLM_LOGGING_PREFIX, if set, it will be prepend to all logging messages (vllm-project#9590)

  sfc-gh-zhwang

  authored
  29061ed

  Copy full SHA for 29061ed

• [Model] Support E5-V (vllm-project#9576)

  DarkLight1337

  authored
  831540c

  Copy full SHA for 831540c

• [Build] Fix FetchContent multiple build issue (vllm-project#9596)

  Show description for 51c24c9
  ProExpertProg

  authored
  51c24c9

  Copy full SHA for 51c24c9

• [Hardware][XPU] using current_platform.is_xpu (vllm-project#9605)

  MengqingCao

  authored
  2394962

  Copy full SHA for 2394962

• [Model] Initialize Florence-2 language backbone support (vllm-project#9555)

  Isotr0py

  authored
  3ff57eb

  Copy full SHA for 3ff57eb

• [VLM] Enable overriding whether post layernorm is used in vision encoder + fix quant args (vllm-project#9217)

  Show description for c18e1a3
  

  DarkLight1337

  and

  Isotr0py

  authored
  c18e1a3

  Copy full SHA for c18e1a3

• [Model] Add min_pixels / max_pixels to Qwen2VL as mm_processor_kwargs (vllm-project#9612)

  Show description for 31a08f5
  alex-jw-brooks

  authored
  31a08f5

  Copy full SHA for 31a08f5

• [Bugfix] Fix _init_vision_model in NVLM_D model (vllm-project#9611)

  Show description for e7116c0
  

  DarkLight1337

  and

  Isotr0py

  authored
  e7116c0

  Copy full SHA for e7116c0

• [misc] comment to avoid future confusion about baichuan (vllm-project#9620)

  Show description for dbdd3b5
  youkaichao

  authored
  dbdd3b5

  Copy full SHA for dbdd3b5

• [Bugfix] Fix divide by zero when serving Mamba models (vllm-project#9617)

  Show description for e5ac6a4
  tlrmchlsmth

  authored
  e5ac6a4

  Copy full SHA for e5ac6a4

• [Misc] Separate total and output tokens in benchmark_throughput.py (vllm-project#8914)

  mgoin

  authored
  fd0e2cf

  Copy full SHA for fd0e2cf

• [torch.compile] Adding torch compile annotations to some models (vllm-project#9614)

  CRZbulabula

  authored
  9013e24

  Copy full SHA for 9013e24

• [Frontend] Enable Online Multi-image Support for MLlama (vllm-project#9393)

  Show description for 150b779
  

  alex-jw-brooks

  and

  DarkLight1337

  authored
  150b779

  Copy full SHA for 150b779

• [Model] Add Qwen2-Audio model support (vllm-project#9248)

  Show description for fc6c274
  

  faychu

  and

  DarkLight1337

  authored
  fc6c274

  Copy full SHA for fc6c274

• [CI/Build] Add bot to close stale issues and PRs (vllm-project#9436)

  russellb

  authored
  b548d7a

  Copy full SHA for b548d7a

Commits on Oct 24, 2024

• [Bugfix][Model] Fix Mllama SDPA illegal memory access for batched multi-image (vllm-project#9626)

  Show description for bb01f29
  mgoin

  authored
  bb01f29

  Copy full SHA for bb01f29

• [Bugfix] Use "vision_model" prefix for MllamaVisionModel (vllm-project#9628)

  Show description for b7df53c
  mgoin

  authored
  b7df53c

  Copy full SHA for b7df53c

• [Bugfix]: Make chat content text allow type content (vllm-project#9358)

  Show description for 33bab41
  vrdn-23

  authored
  33bab41

  Copy full SHA for 33bab41

• [XPU] avoid triton import for xpu (vllm-project#9440)

  Show description for 056a68c
  

  yma11

  and

  DarkLight1337

  authored
  056a68c

  Copy full SHA for 056a68c

• [Bugfix] Fix PP for ChatGLM and Molmo (vllm-project#9422)

  DarkLight1337

  authored
  836e8ef

  Copy full SHA for 836e8ef

• [V1][Bugfix] Clean up requests when aborted (vllm-project#9629)

  Show description for 3770071
  WoosukKwon

  authored
  3770071

  Copy full SHA for 3770071

• [core] simplify seq group code (vllm-project#9569)

  Show description for 4fdc581
  

  youkaichao

  and

  zhuohan123

  authored
  4fdc581

  Copy full SHA for 4fdc581

• [torch.compile] Adding torch compile annotations to some models (vllm-project#9639)

  Show description for 8a02cd0
  

  CRZbulabula

  and

  youkaichao

  authored
  8a02cd0

  Copy full SHA for 8a02cd0

• [Kernel] add kernel for FATReLU (vllm-project#9610)

  Show description for 295a061
  jeejeelee

  authored
  295a061

  Copy full SHA for 295a061

• [torch.compile] expanding support and fix allgather compilation (vllm-project#9637)

  Show description for ad6f780
  

  CRZbulabula

  and

  youkaichao

  authored
  ad6f780

  Copy full SHA for ad6f780

• [Doc] Move additional tips/notes to the top (vllm-project#9647)

  DarkLight1337

  authored
  b979143

  Copy full SHA for b979143

• [Bugfix]Disable the post_norm layer of the vision encoder for LLaVA models (vllm-project#9653)

  litianjian

  authored
  f584549

  Copy full SHA for f584549

• Increase operation per run limit for "Close inactive issues and PRs" workflow (vllm-project#9661)

  Show description for de662d3
  hmellor

  authored
  de662d3

  Copy full SHA for de662d3

• [torch.compile] Adding torch compile annotations to some models (vllm-project#9641)

  Show description for d27cfbf
  

  CRZbulabula

  and

  youkaichao

  authored
  d27cfbf

  Copy full SHA for d27cfbf

• [CI/Build] Fix VLM test failures when using transformers v4.46 (vllm-project#9666)

  DarkLight1337

  authored
  c866e00

  Copy full SHA for c866e00

• [Model] Compute Llava Next Max Tokens / Dummy Data From Gridpoints (vllm-project#9650)

  Show description for 722d46e
  alex-jw-brooks

  authored
  722d46e

  Copy full SHA for 722d46e

• [Log][Bugfix] Fix default value check for image_url.detail (vllm-project#9663)

  mgoin

  authored
  e26d37a

  Copy full SHA for e26d37a

• [Performance][Kernel] Fused_moe Performance Improvement (vllm-project#9384)

  Show description for 5944909
  charlifu

  authored
  5944909

  Copy full SHA for 5944909

• [Bugfix] Remove xformers requirement for Pixtral (vllm-project#9597)

  Show description for c91ed47
  mgoin

  authored
  c91ed47

  Copy full SHA for c91ed47

Commits on Oct 25, 2024

• [ci/Build] Skip Chameleon for transformers 4.46.0 on broadcast test vllm-project#9675 (vllm-project#9676)

  khluu

  authored
  9f7b4ba

  Copy full SHA for 9f7b4ba

• [Model] add a lora module for granite 3.0 MoE models (vllm-project#9673)

  willmj

  authored
  a6f3721

  Copy full SHA for a6f3721

• [V1] Support sliding window attention (vllm-project#9679)

  Show description for 9645b9f
  WoosukKwon

  authored
  9645b9f

  Copy full SHA for 9645b9f

• [Bugfix] Fix compressed_tensors_moe bad config.strategy (vllm-project#9677)

  mgoin

  authored
  ca0d922

  Copy full SHA for ca0d922

• [Doc] Improve quickstart documentation (vllm-project#9256)

  Show description for 228cfbd
  rafvasq

  authored
  228cfbd

  Copy full SHA for 228cfbd

• [Bugfix] Fix crash with llama 3.2 vision models and guided decoding (vllm-project#9631)

  Show description for 6567e13
  

  3 peopleauthored
  6567e13

  Copy full SHA for 6567e13

Commits on Oct 26, 2024

• [Bugfix] Steaming continuous_usage_stats default to False (vllm-project#9709)

  Show description for 067e77f
  samos123

  authored
  067e77f

  Copy full SHA for 067e77f

• [Hardware][openvino] is_openvino --> current_platform.is_openvino (vllm-project#9716)

  MengqingCao

  authored
  5cbdccd

  Copy full SHA for 5cbdccd

• Fix: MI100 Support By Bypassing Custom Paged Attention (vllm-project#9560)

  MErkinSag

  authored
  55137e8

  Copy full SHA for 55137e8

• [Frontend] Bad words sampling parameter (vllm-project#9717)

  Show description for 07e981f
  Alvant

  authored
  07e981f

  Copy full SHA for 07e981f

• [Model] Add classification Task with Qwen2ForSequenceClassification (vllm-project#9704)

  Show description for 6650e6a
  

  kakao-kevin-us

  and

  jason9693

  authored
  6650e6a

  Copy full SHA for 6650e6a

Commits on Oct 27, 2024

• [Misc] SpecDecodeWorker supports profiling (vllm-project#9719)

  Show description for 67a6882
  Abatom

  authored
  67a6882

  Copy full SHA for 67a6882

• [core] cudagraph output with tensor weak reference (vllm-project#9724)

  Show description for 8549c82
  youkaichao

  authored
  8549c82

  Copy full SHA for 8549c82

• [Misc] Upgrade to pytorch 2.5 (vllm-project#9588)

  Show description for 3cb07a3
  

  bnellnm

  and

  youkaichao

  authored
  3cb07a3

  Copy full SHA for 3cb07a3

• Fix cache management in "Close inactive issues and PRs" actions workflow (vllm-project#9734)

  hmellor

  authored
  e130c40

  Copy full SHA for e130c40

• [Bugfix] Fix load config when using bools (vllm-project#9533)

  madt2709

  authored
  34a9941

  Copy full SHA for 34a9941

Commits on Oct 28, 2024

• [Hardware][ROCM] using current_platform.is_rocm (vllm-project#9642)

  Show description for 4e2d95e
  wangshuai09

  authored
  4e2d95e

  Copy full SHA for 4e2d95e

• [torch.compile] support moe models (vllm-project#9632)

  Show description for 32176fe
  youkaichao

  authored
  32176fe

  Copy full SHA for 32176fe

• Fix beam search eos (vllm-project#9627)

  robertgshaw2-neuralmagic

  authored
  feb92fb

  Copy full SHA for feb92fb

• [Bugfix] Fix ray instance detect issue (vllm-project#9439)

  yma11

  authored
  2adb440

  Copy full SHA for 2adb440

• [CI/Build] Adopt Mergify for auto-labeling PRs (vllm-project#9259)

  Show description for 8b0e4f2
  russellb

  authored
  8b0e4f2

  Copy full SHA for 8b0e4f2

• [Model][VLM] Add multi-video support for LLaVA-Onevision (vllm-project#8905)

  Show description for 5f8d807
  

  3 peopleauthored
  5f8d807

  Copy full SHA for 5f8d807

• Adding "torch compile" annotations to moe models (vllm-project#9758)

  CRZbulabula

  authored
  aa0addb

  Copy full SHA for aa0addb

• [misc] avoid circular import (vllm-project#9765)

  Show description for 97b61bf
  youkaichao

  authored
  97b61bf

  Copy full SHA for 97b61bf

• [torch.compile] add deepseek v2 compile (vllm-project#9775)

  Show description for 76ed534
  youkaichao

  authored
  76ed534

  Copy full SHA for 76ed534

• Merge remote-tracking branch 'upstream/main'

  gshtras

  committed
  b20aa29

  Copy full SHA for b20aa29

• Fix for dynamic quantization of the vision part of llama 3.2; Fix for _scaled_mm of 3 dimensional input tensor

  gshtras

  committed
  c0eb092

  Copy full SHA for c0eb092

Commits on Oct 29, 2024

• [Doc] fix third-party model example (vllm-project#9771)

  Show description for c5d7fb9
  russellb

  authored
  c5d7fb9

  Copy full SHA for c5d7fb9

• [Model][LoRA]LoRA support added for Qwen (vllm-project#9622)

  Show description for 7a4df5f
  jeejeelee

  authored
  7a4df5f

  Copy full SHA for 7a4df5f

• [Doc] Specify async engine args in docs (vllm-project#9726)

  DarkLight1337

  authored
  e74f2d4

  Copy full SHA for e74f2d4

• [Bugfix] Use temporary directory in registry (vllm-project#9721)

  DarkLight1337

  authored
  eae3d48

  Copy full SHA for eae3d48

• [Frontend] re-enable multi-modality input in the new beam search implementation (vllm-project#9427)

  Show description for ef7865b
  FerdinandZhong

  authored
  ef7865b

  Copy full SHA for ef7865b

• [Model] Add BNB quantization support for Mllama (vllm-project#9720)

  Isotr0py

  authored
  09500f7

  Copy full SHA for 09500f7

• Fix support for non quantized visual layers in otherwise quantized mllama model, including missing scaling factors

  Show description for 2454f4a
  gshtras

  committed
  2454f4a

  Copy full SHA for 2454f4a

• [Hardware] using current_platform.seed_everything (vllm-project#9785)

  Show description for 622b7ab
  wangshuai09

  authored
  622b7ab

  Copy full SHA for 622b7ab

• Reorganize imports; Restrict additional supported tensors in _scaled_mm to 3D; Use constant for default fp8 scale

  Show description for a23a23c
  gshtras

  committed
  a23a23c

  Copy full SHA for a23a23c

• Merge remote-tracking branch 'upstream/main' into upstream_merge_24_10_28

  gshtras

  committed
  b0a8c5d

  Copy full SHA for b0a8c5d

• Merge remote-tracking branch 'origin/partially_quantized_mllama_fix' into upstream_merge_24_10_28

  gshtras

  committed
  ab3f100

  Copy full SHA for ab3f100

• Merge remote-tracking branch 'origin/main' into upstream_merge_24_10_28

  gshtras

  committed
  64f51a5

  Copy full SHA for 64f51a5

• fix is_hip

  gshtras

  committed
  cfd7388

  Copy full SHA for cfd7388