Code

Upgrade the latest vLLM version 09/18#4

Code

Merged

Jeffwan merged 247 commits intoaibrix:mainaibrix/vllm:mainfrom vllm-project:mainvllm-project/vllm:mainCopy head branch name to clipboardSep 19, 2024

+39,289-10,397

Commits

Commits on Aug 26, 2024

• [Performance][BlockManagerV2] Mark prefix cache block as computed after schedule (#7822)

  comaniac

  authored
  2deb029

  Copy full SHA for 2deb029

• [Misc] Update qqq to use vLLMParameters (#7805)

  dsikka

  authored
  6653040

  Copy full SHA for 6653040

• [Misc] Update gptq_marlin_24 to use vLLMParameters (#7762)

  Show description for dd9857f
  

  dsikka

  and

  mgoin

  authored
  dd9857f

  Copy full SHA for dd9857f

• [misc] fix custom allreduce p2p cache file generation (#7853)

  youkaichao

  authored
  05826c8

  Copy full SHA for 05826c8

• [Bugfix] neuron: enable tensor parallelism (#7562)

  Show description for 760e9f7
  omrishiv

  authored
  760e9f7

  Copy full SHA for 760e9f7

Commits on Aug 27, 2024

• [Misc] Update compressed tensors lifecycle to remove prefix from create_weights (#7825)

  dsikka

  authored
  015e6cc

  Copy full SHA for 015e6cc

• [Core] Asynchronous Output Processor (#7049)

  Show description for 2eedede
  

  megha95

  and

  alexm-redhat

  authored
  2eedede

  Copy full SHA for 2eedede

• [Tests] Disable retries and use context manager for openai client (#7565)

  njhill

  authored
  39178c7

  Copy full SHA for 39178c7

• [core][torch.compile] discard the compile for profiling (#7796)

  youkaichao

  authored
  64cc644

  Copy full SHA for 64cc644

• Revert #7509 (#7887)

  comaniac

  authored
  9606c71

  Copy full SHA for 9606c71

• [Model] Add Mistral Tokenization to improve robustness and chat encoding (#7739)

  patrickvonplaten

  authored
  6fc4e6e

  Copy full SHA for 6fc4e6e

• [CI/Build][VLM] Cleanup multiple images inputs model test (#7897)

  Isotr0py

  authored
  9db6421

  Copy full SHA for 9db6421

• [Hardware][Intel GPU] Add intel GPU pipeline parallel support. (#7810)

  jikunshang

  authored
  076169f

  Copy full SHA for 076169f

• [CI/Build][ROCm] Enabling tensorizer tests for ROCm (#7237)

  alexeykondrat

  authored
  42e932c

  Copy full SHA for 42e932c

• [Bugfix] Fix phi3v incorrect image_idx when using async engine (#7916)

  Isotr0py

  authored
  b09c755

  Copy full SHA for b09c755

• [cuda][misc] error on empty CUDA_VISIBLE_DEVICES (#7924)

  youkaichao

  authored
  ed6f002

  Copy full SHA for ed6f002

• [Kernel] Expand MoE weight loading + Add Fused Marlin MoE Kernel (#7766)

  Show description for fc91188
  

  dsikka

  and

  ElizaWszola

  authored
  fc91188

  Copy full SHA for fc91188

• [benchmark] Update TGI version (#7917)

  philschmid

  authored
  345be0e

  Copy full SHA for 345be0e

• [Model] Add multi-image input support for LLaVA-Next offline inference (#7230)

  zifeitong

  authored
  5340a2d

  Copy full SHA for 5340a2d

• [mypy] Enable mypy type checking for vllm/core (#7229)

  jberkhahn

  authored
  9c71c97

  Copy full SHA for 9c71c97

Commits on Aug 28, 2024

• [Core][VLM] Stack multimodal tensors to represent multiple images within each prompt (#7902)

  petersalas

  authored
  fab5f53

  Copy full SHA for fab5f53

• [hardware][rocm] allow rocm to override default env var (#7926)

  youkaichao

  authored
  bc6e42a

  Copy full SHA for bc6e42a

• [Bugfix] Allow ScalarType to be compiled with pytorch 2.3 and add checks for registering FakeScalarType and dynamo support. (#7886)

  bnellnm

  authored
  c166e7e

  Copy full SHA for c166e7e

• [mypy][CI/Build] Fix mypy errors (#7929)

  DarkLight1337

  authored
  51f86bf

  Copy full SHA for 51f86bf

• [Core] Async_output_proc: Add virtual engine support (towards pipeline parallel) (#7911)

  alexm-redhat

  authored
  f508e03

  Copy full SHA for f508e03

• [Performance] Enable chunked prefill and prefix caching together (#7753)

  comaniac

  authored
  e358053

  Copy full SHA for e358053

• [ci][test] fix pp test failure (#7945)

  youkaichao

  authored
  f52a43a

  Copy full SHA for f52a43a

• [Doc] fix the autoAWQ example (#7937)

  stas00

  authored
  98c12cf

  Copy full SHA for 98c12cf

• [Bugfix][VLM] Fix incompatibility between #7902 and #7230 (#7948)

  DarkLight1337

  authored
  ef9baee

  Copy full SHA for ef9baee

• [Core][Kernels] Use FlashInfer backend for FP8 KV Cache when available. (#7798)

  Show description for b98cc28
  

  pavanimajety

  and

  simon-mo

  authored
  b98cc28

  Copy full SHA for b98cc28

• [Kernel] [Triton] [AMD] Adding Triton implementations awq_dequantize and awq_gemm to support AWQ (#7386)

  rasmith

  authored
  e5697d1

  Copy full SHA for e5697d1

• [TPU] Upgrade PyTorch XLA nightly (#7967)

  WoosukKwon

  authored
  eeffde1

  Copy full SHA for eeffde1

• [Doc] fix 404 link (#7966)

  stas00

  authored
  8c56e57

  Copy full SHA for 8c56e57

• [Kernel/Model] Migrate mamba_ssm and causal_conv1d kernels to vLLM (#7651)

  mzusman

  authored
  fdd9daa

  Copy full SHA for fdd9daa

• [Bugfix] Make torch registration of punica ops optional (#7970)

  bnellnm

  authored
  3cdfe1f

  Copy full SHA for 3cdfe1f

• [torch.compile] avoid Dynamo guard evaluation overhead (#7898)

  Show description for ce6bf3a
  

  youkaichao

  and

  WoosukKwon

  authored
  ce6bf3a

  Copy full SHA for ce6bf3a

• Remove faulty Meta-Llama-3-8B-Instruct-FP8.yaml lm-eval test (#7961)

  mgoin

  authored
  af59df0

  Copy full SHA for af59df0

Commits on Aug 29, 2024

• [Frontend] Minor optimizations to zmq decoupled front-end (#7957)

  Show description for 4289cad
  

  njhill

  and

  Robert Shaw

  authored
  4289cad

  Copy full SHA for 4289cad

• [torch.compile] remove reset (#7975)

  youkaichao

  authored
  a7f65c2

  Copy full SHA for a7f65c2

• [VLM][Core] Fix exceptions on ragged NestedTensors (#7974)

  petersalas

  authored
  74d5543

  Copy full SHA for 74d5543

• Revert "[Core][Kernels] Use FlashInfer backend for FP8 KV Cache when available." (#7982)

  youkaichao

  authored
  ef99a78

  Copy full SHA for ef99a78

• [Bugfix] Unify rank computation across regular decoding and speculative decoding (#7899)

  jmkuebler

  authored
  f205c09

  Copy full SHA for f205c09

• [Core] Combine async postprocessor and multi-step (#7921)

  alexm-redhat

  authored
  3f60f22

  Copy full SHA for 3f60f22

• [Core][Kernels] Enable FP8 KV Cache with Flashinfer backend. + BugFix for kv_cache_dtype=auto (#7985)

  Show description for 6b34215
  

  3 peopleauthored
  6b34215

  Copy full SHA for 6b34215

• extend cuda graph size for H200 (#7894)

  Show description for c334b18
  

  kushanam

  and

  youkaichao

  authored
  c334b18

  Copy full SHA for c334b18

• [Bugfix] Fix incorrect vocal embedding shards for GGUF model in tensor parallelism (#7954)

  Isotr0py

  authored
  d78789a

  Copy full SHA for d78789a

• [misc] update tpu int8 to use new vLLM Parameters (#7973)

  dsikka

  authored
  86a677d

  Copy full SHA for 86a677d

• [Neuron] Adding support for context-lenght, token-gen buckets. (#7885)

  Show description for 257afc3
  

  hbikki

  and

  Harsha Bikki

  authored
  257afc3

  Copy full SHA for 257afc3

• support bitsandbytes 8-bit and FP4 quantized models (#7445)

  chenqianfzh

  authored
  4664cea

  Copy full SHA for 4664cea

• Add more percentiles and latencies (#7759)

  wschin

  authored
  0c785d3

  Copy full SHA for 0c785d3

Commits on Aug 30, 2024

• [VLM] Disallow overflowing max_model_len for multimodal models (#7998)

  DarkLight1337

  authored
  4abed65

  Copy full SHA for 4abed65

• [Core] Logprobs support in Multi-step (#7652)

  afeldman-nm

  authored
  428dd14

  Copy full SHA for 428dd14

• [TPU] Async output processing for TPU (#8011)

  WoosukKwon

  authored
  80c7b08

  Copy full SHA for 80c7b08

• [Kernel] changing fused moe kernel chunk size default to 32k (#7995)

  avshalomman

  authored
  34a0e96

  Copy full SHA for 34a0e96

• [MODEL] add Exaone model support (#7819)

  nayohan

  authored
  dc13e99

  Copy full SHA for dc13e99

• [TPU] Support single and multi-host TPUs on GKE (#7613)

  richardsliu

  authored
  2148441

  Copy full SHA for 2148441

• [Bugfix] Fix import error in Exaone model (#8034)

  DarkLight1337

  authored
  afd39a4

  Copy full SHA for afd39a4

• [VLM][Model] TP support for ViTs (#7186)

  Show description for f97be32
  

  ChristopherCho

  and

  ywang96

  authored
  f97be32

  Copy full SHA for f97be32

• [Core] Increase default max_num_batched_tokens for multimodal models (#8028)

  DarkLight1337

  authored
  98cef6a

  Copy full SHA for 98cef6a

• [Frontend]-config-cli-args (#7737)

  Show description for 058344f
  

  3 peopleauthored
  058344f

  Copy full SHA for 058344f

• [TPU][Bugfix] Fix tpu type api (#8035)

  WoosukKwon

  authored
  2684efc

  Copy full SHA for 2684efc

• [Model] Adding support for MSFT Phi-3.5-MoE (#7729)

  Show description for 1248e85
  

  4 peopleauthored
  1248e85

  Copy full SHA for 1248e85

Commits on Aug 31, 2024

• [Bugfix] bugfix and add model test for flashinfer fp8 kv cache. (#8013)

  pavanimajety

  authored
  622f8ab

  Copy full SHA for 622f8ab

• [Bugfix] Fix import error in Phi-3.5-MoE (#8052)

  DarkLight1337

  authored
  d05f0a9

  Copy full SHA for d05f0a9

• [Bugfix] Fix ModelScope models in v0.5.5 (#8037)

  NickLucche

  authored
  4f5d844

  Copy full SHA for 4f5d844

• [BugFix][Core] Multistep Fix Crash on Request Cancellation (#8059)

  robertgshaw2-redhat

  authored
  8423aef

  Copy full SHA for 8423aef

• [Frontend][VLM] Add support for multiple multi-modal items (#8049)

  ywang96

  authored
  5231f08

  Copy full SHA for 5231f08

Commits on Sep 1, 2024

• [Misc] Optional installation of audio related packages (#8063)

  ywang96

  authored
  5b86b19

  Copy full SHA for 5b86b19

Commits on Sep 2, 2024

• [Model] Add Granite model (#7436)

  Show description for f8d6014
  

  shawntan

  and

  njhill

  authored
  f8d6014

  Copy full SHA for f8d6014

• [SpecDecode][Kernel] Flashinfer Rejection Sampling (#7244)

  LiuXiaoxuanPKU

  authored
  e6a26ed

  Copy full SHA for e6a26ed

• [TPU] Align worker index with node boundary (#7932)

  WoosukKwon

  authored
  e2b2aa5

  Copy full SHA for e2b2aa5

• [Core][Bugfix] Accept GGUF model without .gguf extension (#8056)

  Isotr0py

  authored
  4ca65a9

  Copy full SHA for 4ca65a9

• [Bugfix] Fix internlm2 tensor parallel inference (#8055)

  Isotr0py

  authored
  dd2a6a8

  Copy full SHA for dd2a6a8

• improve chunked prefill performance

  Show description for 6e36f4f
  noooop

  authored
  6e36f4f

  Copy full SHA for 6e36f4f

Commits on Sep 3, 2024

• [Bugfix] Fix single output condition in output processor (#7881)

  WoosukKwon

  authored
  0fbc669

  Copy full SHA for 0fbc669

• [Bugfix][VLM] Add fallback to SDPA for ViT model running on CPU backend (#8061)

  Isotr0py

  authored
  ec26653

  Copy full SHA for ec26653

• [Performance] Enable chunked prefill and prefix caching together (#8120)

  Show description for bd852f2
  

  3 peopleauthored
  bd852f2

  Copy full SHA for bd852f2

• [CI] Only PR reviewers/committers can trigger CI on PR (#8124)

  Show description for 95a178f
  khluu

  authored
  95a178f

  Copy full SHA for 95a178f

• [Core] Optimize Async + Multi-step (#8050)

  alexm-redhat

  authored
  6d646d0

  Copy full SHA for 6d646d0

• [Misc] Raise a more informative exception in add/remove_logger (#7750)

  Yard1

  authored
  652c83b

  Copy full SHA for 652c83b

• [CI/Build] make pip install vllm work in macos (for import only) (#8118)

  tomeras91

  authored
  c02638e

  Copy full SHA for c02638e

• [ci] Fix GHA workflow (#8129)

  Show description for f1575dc
  khluu

  authored
  f1575dc

  Copy full SHA for f1575dc

• [TPU][Bugfix] Fix next_token_ids shape (#8128)

  WoosukKwon

  authored
  0af3abe

  Copy full SHA for 0af3abe

• [CI] Change PR remainder to avoid at-mentions (#8134)

  simon-mo

  authored
  dc0b606

  Copy full SHA for dc0b606

• [Misc] Update GPTQ to use vLLMParameters (#7976)

  dsikka

  authored
  2188a60

  Copy full SHA for 2188a60

Commits on Sep 4, 2024

• [Benchmark] Add --async-engine option to benchmark_throughput.py (#7964)

  njhill

  authored
  d4db9f5

  Copy full SHA for d4db9f5

• [TPU][Bugfix] Use XLA rank for persistent cache path (#8137)

  WoosukKwon

  authored
  61f4a93

  Copy full SHA for 61f4a93

• [Misc] Update fbgemmfp8 to use vLLMParameters (#7972)

  Show description for e16fa99
  

  dsikka

  and

  mgoin

  authored
  e16fa99

  Copy full SHA for e16fa99

• [Model] Add Ultravox support for multiple audio chunks (#7963)

  petersalas

  authored
  2be8ec6

  Copy full SHA for 2be8ec6

• [Frontend] Multimodal support in offline chat (#8098)

  DarkLight1337

  authored
  855c262

  Copy full SHA for 855c262

• chore: Update check-wheel-size.py to read MAX_SIZE_MB from env (#8103)

  haitwang-cloud

  authored
  ccd7207

  Copy full SHA for ccd7207

• [Bugfix] remove post_layernorm in siglip (#8106)

  wnma3mz

  authored
  d331156

  Copy full SHA for d331156

• [MISC] Consolidate FP8 kv-cache tests (#8131)

  comaniac

  authored
  2ad2e56

  Copy full SHA for 2ad2e56

• [CI/Build][ROCm] Enabling LoRA tests on ROCm (#7369)

  Show description for d1dec64
  

  alexeykondrat

  and

  simon-mo

  authored
  d1dec64

  Copy full SHA for d1dec64

• [CI] Change test input in Gemma LoRA test (#8163)

  WoosukKwon

  authored
  561d6f8

  Copy full SHA for 561d6f8

• [Feature] OpenAI-Compatible Tools API + Streaming for Hermes & Mistral models (#5649)

  Show description for e02ce49
  

  K-Mistele

  and

  constellate

  authored
  e02ce49

  Copy full SHA for e02ce49

• [MISC] Replace input token throughput with total token throughput (#8164)

  Show description for 77d9e51
  

  comaniac

  and

  mgoin

  authored
  77d9e51

  Copy full SHA for 77d9e51

• [Neuron] Adding support for adding/ overriding neuron configuration a… (#8062)

  Show description for 008cf88
  

  hbikki

  and

  Harsha Bikki

  authored
  008cf88

  Copy full SHA for 008cf88

• Bump version to v0.6.0 (#8166)

  simon-mo

  authored
  32e7db2

  Copy full SHA for 32e7db2

• [Doc] [Misc] Create CODE_OF_CONDUCT.md (#8161)

  mmcelaney

  authored
  e01c2be

  Copy full SHA for e01c2be

Commits on Sep 5, 2024

• [bugfix] >1.43 constraint for openai (#8169)

  Show description for 1afc931
  

  SolitaryThinker

  and

  mgoin

  authored
  1afc931

  Copy full SHA for 1afc931

• [Misc] Clean up RoPE forward_native (#8076)

  WoosukKwon

  authored
  4624d98

  Copy full SHA for 4624d98

• [ci] Mark LoRA test as soft-fail (#8160)

  Show description for ba262c4
  khluu

  authored
  ba262c4

  Copy full SHA for ba262c4

• [Core/Bugfix] Add query dtype as per FlashInfer API requirements. (#8173)

  elfiegg

  authored
  e39ebf5

  Copy full SHA for e39ebf5

• [Doc] Indicate more information about supported modalities (#8181)

  DarkLight1337

  authored
  288a938

  Copy full SHA for 288a938

• Inclusion of InternVLChatModel In PP_SUPPORTED_MODELS(Pipeline Parallelism) (#7860)

  Manikandan-Thangaraj-ZS0321

  authored
  8685ba1

  Copy full SHA for 8685ba1

• [MODEL] Qwen Multimodal Support (Qwen-VL / Qwen-VL-Chat) (#8029)

  Show description for 9da25a8
  

  alex-jw-brooks

  and

  DarkLight1337

  authored
  9da25a8

  Copy full SHA for 9da25a8

• Move verify_marlin_supported to GPTQMarlinLinearMethod (#8165)

  mgoin

  authored
  2ee4528

  Copy full SHA for 2ee4528

• [Documentation][Spec Decode] Add documentation about lossless guarantees in Speculative Decoding in vLLM (#7962)

  sroy745

  authored
  2febcf2

  Copy full SHA for 2febcf2

Commits on Sep 6, 2024

• [Core] Support load and unload LoRA in api server (#6566)

  Show description for db3bf7c
  

  Jeffwan

  and

  jeejeelee

  authored
  db3bf7c

  Copy full SHA for db3bf7c

• [BugFix] Fix Granite model configuration (#8216)

  njhill

  authored
  baa5467

  Copy full SHA for baa5467

• [Frontend] Add --logprobs argument to benchmark_serving.py (#8191)

  afeldman-nm

  authored
  e5cab71

  Copy full SHA for e5cab71

• [Misc] Use ray[adag] dependency instead of cuda (#7938)

  ruisearch42

  authored
  de80783

  Copy full SHA for de80783

• [CI/Build] Increasing timeout for multiproc worker tests (#8203)

  alexeykondrat

  authored
  1447c97

  Copy full SHA for 1447c97

• [Kernel] [Triton] Memory optimization for awq_gemm and awq_dequantize, 2x throughput (#8248)

  rasmith

  authored
  9db52ea

  Copy full SHA for 9db52ea

• [Misc] Remove SqueezeLLM (#8220)

  dsikka

  authored
  23f3222

  Copy full SHA for 23f3222

• [Model] Allow loading from original Mistral format (#8168)

  Show description for 29f49cd
  

  patrickvonplaten

  and

  mgoin

  authored
  29f49cd

  Copy full SHA for 29f49cd

Commits on Sep 7, 2024

• [misc] [doc] [frontend] LLM torch profiler support (#7943)

  SolitaryThinker

  authored
  12dd715

  Copy full SHA for 12dd715

• [Bugfix] Fix Hermes tool call chat template bug (#8256)

  Show description for 41e95c5
  K-Mistele

  authored
  41e95c5

  Copy full SHA for 41e95c5

• [Model] Multi-input support for LLaVA (#8238)

  DarkLight1337

  authored
  2f707fc

  Copy full SHA for 2f707fc

• Enable Random Prefix Caching in Serving Profiling Tool (benchmark_serving.py) (#8241)

  wschin

  authored
  795b662

  Copy full SHA for 795b662

• [tpu][misc] fix typo (#8260)

  youkaichao

  authored
  ce2702a

  Copy full SHA for ce2702a

• [Bugfix] Fix broken OpenAI tensorizer test (#8258)

  DarkLight1337

  authored
  9f68e00

  Copy full SHA for 9f68e00

• [Model][VLM] Support multi-images inputs for InternVL2 models (#8201)

  Isotr0py

  authored
  e807125

  Copy full SHA for e807125

• [Model][VLM] Decouple weight loading logic for Paligemma (#8269)

  Isotr0py

  authored
  36bf815

  Copy full SHA for 36bf815

• ppc64le: Dockerfile fixed, and a script for buildkite (#8026)

  sumitd2

  authored
  b962ee1

  Copy full SHA for b962ee1

• [CI/Build] Use python 3.12 in cuda image (#8133)

  Show description for cfe712b
  joerunde

  authored
  cfe712b

  Copy full SHA for cfe712b

Commits on Sep 8, 2024

• [Bugfix] Fix async postprocessor in case of preemption (#8267)

  alexm-redhat

  authored
  4ef41b8

  Copy full SHA for 4ef41b8

Commits on Sep 9, 2024

• [Bugfix] Streamed tool calls now more strictly follow OpenAI's format; ensures Vercel AI SDK compatibility (#8272)

  K-Mistele

  authored
  08287ef

  Copy full SHA for 08287ef

• [Frontend] Add progress reporting to run_batch.py (#8060)

  Show description for 58fcc85
  

  alugowski

  and

  Adam Lugowski

  authored
  58fcc85

  Copy full SHA for 58fcc85

• [Bugfix] Correct adapter usage for cohere and jamba (#8292)

  vladislavkruglikov

  authored
  f9b4a2d

  Copy full SHA for f9b4a2d

• [Misc] GPTQ Activation Ordering (#8135)

  kylesayrs

  authored
  c7cb5c3

  Copy full SHA for c7cb5c3

Commits on Sep 10, 2024

• [Misc] Fused MoE Marlin support for GPTQ (#8217)

  dsikka

  authored
  6cd5e5b

  Copy full SHA for 6cd5e5b

• Add NVIDIA Meetup slides, announce AMD meetup, and add contact info (#8319)

  simon-mo

  authored
  a1d8742

  Copy full SHA for a1d8742

• [Bugfix] Fix missing post_layernorm in CLIP (#8155)

  DarkLight1337

  authored
  da1a844

  Copy full SHA for da1a844

• [CI/Build] enable ccache/scccache for HIP builds (#8327)

  dtrifiro

  authored
  6234385

  Copy full SHA for 6234385

• [Frontend] Clean up type annotations for mistral tokenizer (#8314)

  DarkLight1337

  authored
  8c054b7

  Copy full SHA for 8c054b7

• [CI/Build] Enabling kernels tests for AMD, ignoring some of then that fail (#8130)

  alexeykondrat

  authored
  f421f3c

  Copy full SHA for f421f3c

• Fix ppc64le buildkite job (#8309)

  sumitd2

  authored
  02751a7

  Copy full SHA for 02751a7

• [Spec Decode] Move ops.advance_step to flash attn advance_step (#8224)

  kevin314

  authored
  5faedf1

  Copy full SHA for 5faedf1

• [Misc] remove peft as dependency for prompt models (#8162)

  prashantgupta24

  authored
  04e7c4e

  Copy full SHA for 04e7c4e

• [MISC] Keep chunked prefill enabled by default with long context when prefix caching is enabled (#8342)

  comaniac

  authored
  b1f3e18

  Copy full SHA for b1f3e18

• [Bugfix] lookahead block table with cuda graph max capture (#8340)

  Show description for 22f3a4b
  alexm-redhat

  authored
  22f3a4b

  Copy full SHA for 22f3a4b

• [Core/Bugfix] pass VLLM_ATTENTION_BACKEND to ray workers (#8172)

  SolitaryThinker

  authored
  1d5e397

  Copy full SHA for 1d5e397

• [CI/Build][Kernel] Update CUTLASS to 3.5.1 tag (#8043)

  tlrmchlsmth

  authored
  94144e7

  Copy full SHA for 94144e7

Commits on Sep 11, 2024

• [Misc] Skip loading extra bias for Qwen2-MOE GPTQ models (#8329)

  jeejeelee

  authored
  e497b8a

  Copy full SHA for e497b8a

• [Bugfix] Fix InternVL2 vision embeddings process with pipeline parallel (#8299)

  Isotr0py

  authored
  1230263

  Copy full SHA for 1230263

• [Hardware][NV] Add support for ModelOpt static scaling checkpoints. (#6112)

  pavanimajety

  authored
  efcf946

  Copy full SHA for efcf946

• [model] Support for Llava-Next-Video model (#7559)

  Show description for 6a512a0
  

  3 peopleauthored
  6a512a0

  Copy full SHA for 6a512a0

• [Frontend] Create ErrorResponse instead of raising exceptions in run_batch (#8347)

  pooyadavoodi

  authored
  cea95df

  Copy full SHA for cea95df

• [Model][VLM] Add Qwen2-VL model support (#7905)

  Show description for 3b7fea7
  

  3 peopleauthored
  3b7fea7

  Copy full SHA for 3b7fea7

• [Hardware][Intel] Support compressed-tensor W8A8 for CPU backend (#7257)

  bigPYJ1151

  authored
  0b952af

  Copy full SHA for 0b952af

• [CI/Build] Excluding test_moe.py from AMD Kernels tests for investigation (#8373)

  alexeykondrat

  authored
  aea02f3

  Copy full SHA for aea02f3

• [Bugfix] Add missing attributes in mistral tokenizer (#8364)

  DarkLight1337

  authored
  7015417

  Copy full SHA for 7015417

• [Kernel][Misc] register ops to prevent graph breaks (#6917)

  Show description for 73202db
  

  bnellnm

  and

  SageMoore

  authored
  73202db

  Copy full SHA for 73202db

• [Misc] Move device options to a single place (#8322)

  akx

  authored
  8baa454

  Copy full SHA for 8baa454

• [Speculative Decoding] Test refactor (#8317)

  Show description for 775f00f
  

  LiuXiaoxuanPKU

  and

  youkaichao

  authored
  775f00f

  Copy full SHA for 775f00f

• Pixtral (#8377)

  Show description for d394787
  

  patrickvonplaten

  and

  ywang96

  authored
  d394787

  Copy full SHA for d394787

• Bump version to v0.6.1 (#8379)

  simon-mo

  authored
  3fd2b0d

  Copy full SHA for 3fd2b0d

Commits on Sep 12, 2024

• [MISC] Dump model runner inputs when crashing (#8305)

  comaniac

  authored
  a65cb16

  Copy full SHA for a65cb16

• [misc] remove engine_use_ray (#8126)

  youkaichao

  authored
  f842a7a

  Copy full SHA for f842a7a

• [TPU] Use Ray for default distributed backend (#8389)

  WoosukKwon

  authored
  b71c956

  Copy full SHA for b71c956

• Fix the AMD weight loading tests (#8390)

  mgoin

  authored
  b6c75e1

  Copy full SHA for b6c75e1

• [Bugfix]: Fix the logic for deciding if tool parsing is used (#8366)

  tomeras91

  authored
  5a60699

  Copy full SHA for 5a60699

• [Gemma2] add bitsandbytes support for Gemma2 (#8338)

  blueyo0

  authored
  1bf2dd9

  Copy full SHA for 1bf2dd9

• [Misc] Raise error when using encoder/decoder model with cpu backend (#8355)

  kevin314

  authored
  295c473

  Copy full SHA for 295c473

• [Misc] Use RoPE cache for MRoPE (#8396)

  WoosukKwon

  authored
  42ffba1

  Copy full SHA for 42ffba1

• [torch.compile] hide slicing under custom op for inductor (#8384)

  youkaichao

  authored
  7de49aa

  Copy full SHA for 7de49aa

• [Hotfix][VLM] Fixing max position embeddings for Pixtral (#8399)

  ywang96

  authored
  520ca38

  Copy full SHA for 520ca38

• [Bugfix] Fix InternVL2 inference with various num_patches (#8375)

  Show description for e56bf27
  

  Isotr0py

  and

  DarkLight1337

  authored
  e56bf27

  Copy full SHA for e56bf27

• [Model] Support multiple images for qwen-vl (#8247)

  Show description for c6202da
  

  alex-jw-brooks

  and

  DarkLight1337

  authored
  c6202da

  Copy full SHA for c6202da

• [BugFix] lazy init _copy_stream to avoid torch init wrong gpu instance (#8403)

  lnykww

  authored
  8a23e93

  Copy full SHA for 8a23e93

• [BugFix] Fix Duplicate Assignment in Hermes2ProToolParser (#8423)

  vegaluisjose

  authored
  1f0c75a

  Copy full SHA for 1f0c75a

• [Bugfix] Offline mode fix (#8376)

  Show description for f2e263b
  joerunde

  authored
  f2e263b

  Copy full SHA for f2e263b

• [multi-step] add flashinfer backend (#7928)

  SolitaryThinker

  authored
  a6c0f36

  Copy full SHA for a6c0f36

• [Core] Add engine option to return only deltas or final output (#7381)

  njhill

  authored
  551ce01

  Copy full SHA for 551ce01

• [Bugfix] multi-step + flashinfer: ensure cuda graph compatible (#8427)

  alexm-redhat

  authored
  0198772

  Copy full SHA for 0198772

• [Hotfix][Core][VLM] Disable chunked prefill by default and prefix caching for multimodal models (#8425)

  ywang96

  authored
  c163694

  Copy full SHA for c163694

• [CI/Build] Disable multi-node test for InternVL2 (#8428)

  ywang96

  authored
  b61bd98

  Copy full SHA for b61bd98

• [Hotfix][Pixtral] Fix multiple images bugs (#8415)

  patrickvonplaten

  authored
  d31174a

  Copy full SHA for d31174a

• [Bugfix] Fix weight loading issue by rename variable. (#8293)

  wenxcs

  authored
  a480939

  Copy full SHA for a480939

Commits on Sep 13, 2024

• [Misc] Update Pixtral example (#8431)

  ywang96

  authored
  360ddbd

  Copy full SHA for 360ddbd

• [BugFix] fix group_topk (#8430)

  dsikka

  authored
  8f44a92

  Copy full SHA for 8f44a92

• [Core] Factor out input preprocessing to a separate class (#7329)

  DarkLight1337

  authored
  5ec9c0f

  Copy full SHA for 5ec9c0f

• [Bugfix] Mapping physical device indices for e2e test utils (#8290)

  ShangmingCai

  authored
  40c3965

  Copy full SHA for 40c3965

• [Bugfix] Bump fastapi and pydantic version (#8435)

  DarkLight1337

  authored
  3f79bc3

  Copy full SHA for 3f79bc3

• [CI/Build] Update pixtral tests to use JSON (#8436)

  DarkLight1337

  authored
  8427550

  Copy full SHA for 8427550

• [Bugfix] Fix async log stats (#8417)

  alexm-redhat

  authored
  6821020

  Copy full SHA for 6821020

• [bugfix] torch profiler bug for single gpu with GPUExecutor (#8354)

  SolitaryThinker

  authored
  ba77527

  Copy full SHA for ba77527

• bump version to v0.6.1.post1 (#8440)

  simon-mo

  authored
  acda0b3

  Copy full SHA for acda0b3

• [CI/Build] Enable InternVL2 PP test only on single node (#8437)

  Isotr0py

  authored
  9b4a3b2

  Copy full SHA for 9b4a3b2

• [doc] recommend pip instead of conda (#8446)

  youkaichao

  authored
  cab69a1

  Copy full SHA for cab69a1

• [Misc] Skip loading extra bias for Qwen2-VL GPTQ-Int8 (#8442)

  jeejeelee

  authored
  06311e2

  Copy full SHA for 06311e2

• [misc][ci] fix quant test (#8449)

  youkaichao

  authored
  a246912

  Copy full SHA for a246912

• [Installation] Gate FastAPI version for Python 3.8 (#8456)

  DarkLight1337

  authored
  ecd7a1d

  Copy full SHA for ecd7a1d

• [plugin][torch.compile] allow to add custom compile backend (#8445)

  youkaichao

  authored
  0a4806f

  Copy full SHA for 0a4806f

• [CI/Build] Reorganize models tests (#7820)

  DarkLight1337

  authored
  a84e598

  Copy full SHA for a84e598

• [Doc] Add oneDNN installation to CPU backend documentation (#8467)

  Isotr0py

  authored
  f57092c

  Copy full SHA for f57092c

• [HotFix] Fix final output truncation with stop string + streaming (#8468)

  njhill

  authored
  18e9e1f

  Copy full SHA for 18e9e1f

• bump version to v0.6.1.post2 (#8473)

  simon-mo

  authored
  9ba0817

  Copy full SHA for 9ba0817

• [Hardware][intel GPU] bump up ipex version to 2.3 (#8365)

  Show description for 8517252
  

  jikunshang

  and

  yma11

  authored
  8517252

  Copy full SHA for 8517252

Commits on Sep 14, 2024

• [Kernel][Hardware][Amd]Custom paged attention kernel for rocm (#8310)

  charlifu

  authored
  1ef0d2e

  Copy full SHA for 1ef0d2e

• [Model] support minicpm3 (#8297)

  Show description for 8a0cf1d
  

  SUDA-HLT-ywfang

  and

  DarkLight1337

  authored
  8a0cf1d

  Copy full SHA for 8a0cf1d

• [torch.compile] fix functionalization (#8480)

  youkaichao

  authored
  a36e070

  Copy full SHA for a36e070

• [torch.compile] add a flag to disable custom op (#8488)

  youkaichao

  authored
  47790f3

  Copy full SHA for 47790f3

• [TPU] Implement multi-step scheduling (#8489)

  WoosukKwon

  authored
  50e9ec4

  Copy full SHA for 50e9ec4

Commits on Sep 15, 2024

• [Bugfix][Model] Fix Python 3.8 compatibility in Pixtral model by updating type annotations (#8490)

  chrisociepa

  authored
  3724d5f

  Copy full SHA for 3724d5f

• [Bugfix][Kernel] Add IQ1_M quantization implementation to GGUF kernel (#8357)

  Isotr0py

  authored
  fc990f9

  Copy full SHA for fc990f9

Commits on Sep 16, 2024

• [Kernel] Enable 8-bit weights in Fused Marlin MoE (#8032)

  Show description for a091e2d
  

  ElizaWszola

  and

  dsikka

  authored
  a091e2d

  Copy full SHA for a091e2d

• [Frontend] Expose revision arg in OpenAI server (#8501)

  lewtun

  authored
  837c196

  Copy full SHA for 837c196

• [BugFix] Fix clean shutdown issues (#8492)

  njhill

  authored
  acd5511

  Copy full SHA for acd5511

• [Bugfix][Kernel] Fix build for sm_60 in GGUF kernel (#8506)

  sasha0552

  authored
  781e3b9

  Copy full SHA for 781e3b9

• [Kernel] AQ AZP 3/4: Asymmetric quantization kernels (#7270)

  ProExpertProg

  authored
  5d73ae4

  Copy full SHA for 5d73ae4

• [doc] update doc on testing and debugging (#8514)

  youkaichao

  authored
  2759a43

  Copy full SHA for 2759a43

• [Bugfix] Bind api server port before starting engine (#8491)

  kevin314

  authored
  47f5e03

  Copy full SHA for 47f5e03

• [perf bench] set timeout to debug hanging (#8516)

  simon-mo

  authored
  5478c4b

  Copy full SHA for 5478c4b

• [misc] small qol fixes for release process (#8517)

  simon-mo

  authored
  5ce45eb

  Copy full SHA for 5ce45eb

Commits on Sep 17, 2024

• [Bugfix] Fix 3.12 builds on main (#8510)

  Show description for cca6164
  joerunde

  authored
  cca6164

  Copy full SHA for cca6164

• [refactor] remove triton based sampler (#8524)

  simon-mo

  authored
  546034b

  Copy full SHA for 546034b

• [Frontend] Improve Nullable kv Arg Parsing (#8525)

  Show description for 1c1bb38
  alex-jw-brooks

  authored
  1c1bb38

  Copy full SHA for 1c1bb38

• [Misc][Bugfix] Disable guided decoding for mistral tokenizer (#8521)

  ywang96

  authored
  ee2bcea

  Copy full SHA for ee2bcea

• [torch.compile] register allreduce operations as custom ops (#8526)

  youkaichao

  authored
  99aa4ed

  Copy full SHA for 99aa4ed

• [Misc] Limit to ray[adag] 2.35 to avoid backward incompatible change (#8509)

  Show description for cbdb252
  ruisearch42

  authored
  cbdb252

  Copy full SHA for cbdb252

• [Benchmark] Support sample from HF datasets and image input for benchmark_serving (#8495)

  Isotr0py

  authored
  1b6de83

  Copy full SHA for 1b6de83

• [Encoder decoder] Add cuda graph support during decoding for encoder-decoder models (#7631)

  sroy745

  authored
  1009e93

  Copy full SHA for 1009e93

• [Feature][kernel] tensor parallelism with bitsandbytes quantization (#8434)

  chenqianfzh

  authored
  9855b99

  Copy full SHA for 9855b99

• [Model] Add mistral function calling format to all models loaded with "mistral" format (#8515)

  Show description for a54ed80
  

  patrickvonplaten

  and

  DarkLight1337

  authored
  a54ed80

  Copy full SHA for a54ed80

• [Misc] Don't dump contents of kvcache tensors on errors (#8527)

  njhill

  authored
  56c3de0

  Copy full SHA for 56c3de0

• [Bugfix] Fix TP > 1 for new granite (#8544)

  Show description for 98f9713
  joerunde

  authored
  98f9713

  Copy full SHA for 98f9713

• [doc] improve installation doc (#8550)

  Show description for fa0c114
  

  youkaichao

  and

  Imss27

  authored
  fa0c114

  Copy full SHA for fa0c114

• [CI/Build] Excluding kernels/test_gguf.py from ROCm (#8520)

  alexeykondrat

  authored
  09deb47

  Copy full SHA for 09deb47

• [Kernel] Change interface to Mamba causal_conv1d_update for continuous batching (#8012)

  tlrmchlsmth

  authored
  8110e44

  Copy full SHA for 8110e44

Commits on Sep 18, 2024

• [CI/Build] fix Dockerfile.cpu on podman (#8540)

  dtrifiro

  authored
  95965d3

  Copy full SHA for 95965d3

• [Misc] Add argument to disable FastAPI docs (#8554)

  Jeffwan

  authored
  e351572

  Copy full SHA for e351572

• [CI/Build] Avoid CUDA initialization (#8534)

  DarkLight1337

  authored
  6ffa3f3

  Copy full SHA for 6ffa3f3

• [CI/Build] Update Ruff version (#8469)

  Show description for 9d104b5
  

  aarnphm

  and

  DarkLight1337

  authored
  9d104b5

  Copy full SHA for 9d104b5

• [Core][Bugfix][Perf] Introduce MQLLMEngine to avoid asyncio OH (#8157)

  Show description for 7c7714d
  

  4 peopleauthored
  7c7714d

  Copy full SHA for 7c7714d

• [Core] *Prompt* logprobs support in Multi-step (#8199)

  afeldman-nm

  authored
  a8c1d16

  Copy full SHA for a8c1d16

• [Core] zmq: bind only to 127.0.0.1 for local-only usage (#8543)

  Show description for d65798f
  russellb

  authored
  d65798f

  Copy full SHA for d65798f

• [Model] Support Solar Model (#8386)

  Show description for e18749f
  

  shing100

  and

  mgoin

  authored
  e18749f

  Copy full SHA for e18749f

• [AMD][ROCm]Quantization methods on ROCm; Fix _scaled_mm call (#8380)

  Show description for b3195bc
  

  3 peopleauthored
  b3195bc

  Copy full SHA for b3195bc

• [Kernel] Change interface to Mamba selective_state_update for continuous batching (#8039)

  tlrmchlsmth

  authored
  db9120c

  Copy full SHA for db9120c

• [BugFix] Nonzero exit code if MQLLMEngine startup fails (#8572)

  njhill

  authored
  d9cd78e

  Copy full SHA for d9cd78e

• [Bugfix] add dead_error property to engine client (#8574)

  Show description for 0d47bf3
  joerunde

  authored
  0d47bf3

  Copy full SHA for 0d47bf3

Commits on Sep 19, 2024

• [Kernel] Remove marlin moe templating on thread_m_blocks (#8573)

  Show description for 4c34ce8
  tlrmchlsmth

  authored
  4c34ce8

  Copy full SHA for 4c34ce8

• [Bugfix] [Encoder-Decoder] Bugfix for encoder specific metadata construction during decode of encoder-decoder models. (#8545)

  sroy745

  authored
  3118f63

  Copy full SHA for 3118f63

• Revert "[Misc][Bugfix] Disable guided decoding for mistral tokenizer" (#8593)

  ywang96

  authored
  02c9afa

  Copy full SHA for 02c9afa