Code

Convert to Triton Punica kernels#658

Code

Merged

tgaddair merged 77 commits intomainpredibase/lorax:mainfrom optimizationspredibase/lorax:optimizationsCopy head branch name to clipboardNov 5, 2024

+4,198-954

Commits

Commits on Oct 23, 2024

• Collect timings

  tgaddair

  committed
  832d905

  Copy full SHA for 832d905

• Profiler

  tgaddair

  committed
  697bf4d

  Copy full SHA for 697bf4d

• Allow max batch prefill tokens < max input length

  tgaddair

  committed
  d155163

  Copy full SHA for d155163

• Fix fallback

  tgaddair

  committed
  ca3280c

  Copy full SHA for ca3280c

• Vectorize test

  tgaddair

  committed
  830ce3d

  Copy full SHA for 830ce3d

Commits on Oct 24, 2024

• Triton punica kernels

  tgaddair

  committed
  7f250fe

  Copy full SHA for 7f250fe

• Use triton punica

  tgaddair

  committed
  e4fb765

  Copy full SHA for e4fb765

• Fix format

  tgaddair

  committed
  634c8e2

  Copy full SHA for 634c8e2

• Plumb weights

  tgaddair

  committed
  7870729

  Copy full SHA for 7870729

• Fixed issues

  tgaddair

  committed
  0e057f0

  Copy full SHA for 0e057f0

• Fixed cuda graphs

  tgaddair

  committed
  c8ad4cb

  Copy full SHA for c8ad4cb

• Remove debug

  tgaddair

  committed
  a82eb64

  Copy full SHA for a82eb64

• Remove debug

  tgaddair

  committed
  f68d2c0

  Copy full SHA for f68d2c0

• Move init to warmup

  tgaddair

  committed
  2ffc1db

  Copy full SHA for 2ffc1db

• Fix preloaded and speculators

  tgaddair

  committed
  ea6c86d

  Copy full SHA for ea6c86d

• Docker test

  tgaddair

  committed
  0497a76

  Copy full SHA for 0497a76

• Profiling docs

  tgaddair

  committed
  9e2a29d

  Copy full SHA for 9e2a29d

Commits on Oct 25, 2024

• Revert timings

  tgaddair

  committed
  94e3742

  Copy full SHA for 94e3742

• Fixed merge

  tgaddair

  committed
  0abeccc

  Copy full SHA for 0abeccc

Commits on Oct 26, 2024

• Added LORAX_SPECULATION_MAX_BATCH_SIZE

  tgaddair

  committed
  6f5a976

  Copy full SHA for 6f5a976

Commits on Oct 27, 2024

• Try separate trees per adapter

  tgaddair

  committed
  f89ee87

  Copy full SHA for f89ee87

• Allow refcount==0

  tgaddair

  committed
  23a77d2

  Copy full SHA for 23a77d2

Commits on Oct 28, 2024

• Message

  tgaddair

  committed
  22ed54d

  Copy full SHA for 22ed54d

• Docker test

  tgaddair

  committed
  327bb91

  Copy full SHA for 327bb91

• Cleanup

  tgaddair

  committed
  fbb2b3f

  Copy full SHA for fbb2b3f

• Padding

  tgaddair

  committed
  f0693e9

  Copy full SHA for f0693e9

• Fixed turbo lora + compile

  tgaddair

  committed
  e62e0f8

  Copy full SHA for e62e0f8

• Fix

  tgaddair

  committed
  66d8676

  Copy full SHA for 66d8676

Commits on Oct 30, 2024

• Fix adapter root node id

  tgaddair

  committed
  55e5c41

  Copy full SHA for 55e5c41

• More tests

  tgaddair

  committed
  a6f3a17

  Copy full SHA for a6f3a17

• Docker test

  tgaddair

  committed
  352c92a

  Copy full SHA for 352c92a

• Bump flashinfer

  tgaddair

  committed
  1ea8d6e

  Copy full SHA for 1ea8d6e

Commits on Oct 31, 2024

• Added logprobs fix

  tgaddair

  committed
  c0640f2

  Copy full SHA for c0640f2

• Fix slots

  tgaddair

  committed
  54c36c9

  Copy full SHA for 54c36c9

• No debugging

  tgaddair

  committed
  88cd932

  Copy full SHA for 88cd932

• Docker test

  tgaddair

  committed
  3505b52

  Copy full SHA for 3505b52

• Fixed slot filtering

  tgaddair

  committed
  cf3d2d9

  Copy full SHA for cf3d2d9

• Triton kernels

  tgaddair

  committed
  d1ff7b4

  Copy full SHA for d1ff7b4

• Fix ragged

  tgaddair

  committed
  57c33d7

  Copy full SHA for 57c33d7

• More fixes

  tgaddair

  committed
  ece47f7

  Copy full SHA for ece47f7

• Merge

  tgaddair

  committed
  779bff3

  Copy full SHA for 779bff3

• Revert docker

  tgaddair

  committed
  cb99320

  Copy full SHA for cb99320

• Renamed sgmv -> punica

  tgaddair

  committed
  466ea37

  Copy full SHA for 466ea37

• Refactor PunicaWrapper

  tgaddair

  committed
  2f80c6a

  Copy full SHA for 2f80c6a

• More configuration

  tgaddair

  committed
  47bfd0c

  Copy full SHA for 47bfd0c

• More logs

  tgaddair

  committed
  2343d78

  Copy full SHA for 2343d78

• Fixes

  tgaddair

  committed
  f915abe

  Copy full SHA for f915abe

Commits on Nov 1, 2024

• Guard init

  tgaddair

  committed
  ad460c0

  Copy full SHA for ad460c0

• Guard model has lm_head

  tgaddair

  committed
  43c129b

  Copy full SHA for 43c129b

• Determine trace set from preloaded adapter set

  tgaddair

  committed
  1c70ec6

  Copy full SHA for 1c70ec6

• Plumb skip_lm_head

  tgaddair

  committed
  3ebcbea

  Copy full SHA for 3ebcbea

• Cleanup comments

  tgaddair

  committed
  922c5d6

  Copy full SHA for 922c5d6

• Fixed orient for rank

  tgaddair

  committed
  b2de54f

  Copy full SHA for b2de54f

• Format

  tgaddair

  committed
  35c7de2

  Copy full SHA for 35c7de2

• Fixed tests

  tgaddair

  committed
  295829f

  Copy full SHA for 295829f

• Fixed CausalLM and embedding model

  tgaddair

  committed
  ef86071

  Copy full SHA for ef86071

• Replace flume

  tgaddair

  committed
  0d78a0a

  Copy full SHA for 0d78a0a

• Remove unused dep

  tgaddair

  committed
  8cb79b2

  Copy full SHA for 8cb79b2

• Update axum

  tgaddair

  committed
  045a45a

  Copy full SHA for 045a45a

• Client debug mode, fixed /

  tgaddair

  committed
  20cf752

  Copy full SHA for 20cf752

• Docker test

  tgaddair

  committed
  2868acc

  Copy full SHA for 2868acc

• Fixed unused imports

  tgaddair

  committed
  2131dc1

  Copy full SHA for 2131dc1

• Revert docker

  tgaddair

  committed
  b727a94

  Copy full SHA for b727a94

• Add back tracing

  tgaddair

  committed
  cc17d47

  Copy full SHA for cc17d47

• Debug

  tgaddair

  committed
  68991ba

  Copy full SHA for 68991ba

• Docker test

  tgaddair

  committed
  5380426

  Copy full SHA for 5380426

• Debug registration

  tgaddair

  committed
  89abd51

  Copy full SHA for 89abd51

• Update tag

  tgaddair

  committed
  3c7b69b

  Copy full SHA for 3c7b69b

Commits on Nov 4, 2024

• Don't skip filter

  tgaddair

  committed
  d52f530

  Copy full SHA for d52f530

• Docker test

  tgaddair

  committed
  45c6c53

  Copy full SHA for 45c6c53

• Remove register

  tgaddair

  committed
  3ad4d66

  Copy full SHA for 3ad4d66

• Revert docker

  tgaddair

  committed
  b45c219

  Copy full SHA for b45c219

• Fixed tests

  tgaddair

  committed
  a4a2d5f

  Copy full SHA for a4a2d5f

• ruff

  tgaddair

  committed
  4a264bc

  Copy full SHA for 4a264bc

• Fix tests

  tgaddair

  committed
  e1067a0

  Copy full SHA for e1067a0

• Clear cache

  tgaddair

  committed
  848b4c7

  Copy full SHA for 848b4c7

Commits on Nov 5, 2024

• Check for key in lora weights

  tgaddair

  committed
  107be9a

  Copy full SHA for 107be9a