Code

[WIP] Add HPU support to vLLM v1#487

Code

Draft

kzawora-intel wants to merge 36 commits intohabana_mainHabanaAI/vllm-fork:habana_mainfrom private/kzawora/vllm_v1_hpuHabanaAI/vllm-fork:private/kzawora/vllm_v1_hpuCopy head branch name to clipboard

+2,820-46

Commits

Commits on Nov 12, 2024

• vLLM v1 HPU prototype

  kzawora-intel

  committed
  2191184

• copy gpu model runner code, add hpugraphs support and profile run

  kzawora-intel

  committed
  fd77180

Commits on Nov 13, 2024

• i am very much struggling

  kzawora-intel

  committed
  4dadef5

• it's hopeless

  kzawora-intel

  committed
  9db1409

Commits on Nov 14, 2024

• [wip] bypass prefill chunking in v1 scheduler

  kzawora-intel

  committed
  3b3098c

• colonoscopy

  kzawora-intel

  committed
  c24adb5

• prefill runs, decode has deadlock, idk why

  kzawora-intel

  committed
  2da069e

• i'm done for today

  kzawora-intel

  committed
  932ce93

• do better job at prefill chunking detection

  kzawora-intel

  committed
  fc6a1c2

Commits on Nov 15, 2024

• mixed batch scheduling is still a problem

  kzawora-intel

  committed
  ff0ed54

Commits on Nov 18, 2024

• Merge remote-tracking branch 'origin/habana_main' into HEAD

  kzawora-intel

  committed
  50aa6b3

• general hpu code rewrite

  kzawora-intel

  committed
  debec16

• add debug stuff, it seems like prefill is functional

  kzawora-intel

  committed
  0c1d0b6

• slight code cleanup

  kzawora-intel

  committed
  35d3e38

• remove garbage changes

  kzawora-intel

  committed
  491f991

Commits on Nov 19, 2024

• gsm8k now produces 69% acc on llama3.1

  kzawora-intel

  committed
  e29b84a

• add config not warmed up warnings

  kzawora-intel

  committed
  27b4f32

• add bucketinggit add -u .!

  kzawora-intel

  committed
  087b5d2

• llama3.1 now gives 81% in gsm8k without contiguous pa

  kzawora-intel

  committed
  6fdb6a9

• disable contiguous pa by default

  kzawora-intel

  committed
  8714f9d

• async data copy

  kzawora-intel

  committed
  40ff0ac

• add split sampler optimization

  kzawora-intel

  committed
  28f2ac5

Commits on Nov 20, 2024

• add prompt batching

  kzawora-intel

  committed
  df7a1d4

• padded logits_indices and sampling + documentation

  kzawora-intel

  committed
  623ed10

• update docs

  kzawora-intel

  committed
  0371c31

• fix first-party random and greedy sampler for hpu

  kzawora-intel

  committed
  d7b2a06

• format.sh

  kzawora-intel

  committed
  c934e60

Commits on Nov 21, 2024

• add warmup w/ sampler (it doesn't work great tho)

  kzawora-intel

  committed
  e0f4c26

• add hpugraph check

  kzawora-intel

  committed
  58c8f5d

Commits on Nov 22, 2024

• fix async engine, fix sampler corner cases

  kzawora-intel

  committed
  0c8b075

Commits on Nov 25, 2024

• Add padding-aware scheduling

  kzawora-intel

  committed
  fecedb5

• Merge remote-tracking branch 'origin/habana_main' into HEAD

  kzawora-intel

  committed
  2ab1ac8

Commits on Nov 26, 2024

• bucketing refactor, enable contiguous pa, defrag blocks

  kzawora-intel

  committed
  0d41073

• FreeKVCacheBlockHeapQueue bugfixes

  kzawora-intel

  committed
  5645523

• [wip] add prefix caching support (it was actually really hard)

  kzawora-intel

  committed
  fd62723

Commits on Dec 4, 2024

• fix hpugraphs and long seq corner case

  kzawora-intel

  committed
  e80f2be