Code

Support FP8 KV Cache#652

Code

Merged

ajtejankar merged 27 commits intomainpredibase/lorax:mainfrom fp8-kv-flash-inferpredibase/lorax:fp8-kv-flash-inferCopy head branch name to clipboardOct 29, 2024

+243-39

Commits

Commits on Oct 8, 2024

• (feat) : support fp8 kv cache

  ajtejankar

  committed
  6f887aa

Commits on Oct 17, 2024

• fix a few things

  ajtejankar

  committed
  93fc0d1

• add support for fp8 kv cache using flash infer

  ajtejankar

  committed
  939b479

Commits on Oct 18, 2024

• add logging

  ajtejankar

  committed
  56589ed

• merge back previous code to provide fp8_kv as a quantization option

  ajtejankar

  committed
  9590a71

• remove unnecessary comments

  ajtejankar

  committed
  1517b16

• remove is_fp8_kv_supported function

  ajtejankar

  committed
  a5f1c25

• fix attention api and kv_dtype

  ajtejankar

  committed
  52021c0

• keep ruff happy

  ajtejankar

  committed
  7dd10f3

Commits on Oct 24, 2024

• use fp16 prefill with fp8 kv

  ajtejankar

  committed
  787b58e

Commits on Oct 25, 2024

• use fp16 prefill for fp8 kv cache (without prefix caching)

  ajtejankar

  committed
  8dae2d8

• Merge branch 'main' into fp8-kv-flash-infer

  ajtejankar

  committed
  8540461

• add window_left option

  ajtejankar

  committed
  d880464

• fix merge conflicts

  ajtejankar

  committed
  bb75730

• move paged_attention import location

  ajtejankar

  committed
  3031ba4

• add llama and qwen models

  ajtejankar

  committed
  1742cde

Commits on Oct 26, 2024

• refactor according to Travis' comments

  ajtejankar

  committed
  c732810

• refactor llama and qwen models

  ajtejankar

  committed
  90e85f5

• fix flash_attn call bug in qwen

  ajtejankar

  committed
  be6e9ff

• fix flash_attn call bug in llama

  ajtejankar

  committed
  f2f9a08

• refactor flash_causal_lm for better git diff

  ajtejankar

  committed
  7974ff9

• refactor to use fp8_kv in reshape_and_cache

  ajtejankar

  committed
  cd29d93

• add support for gemma models and fix ruff error in qwen

  ajtejankar

  committed
  488d428

Commits on Oct 28, 2024

• fix is_fp8_supported function

  ajtejankar

  committed
  813b64b

• fix fp8 error handling

  ajtejankar

  committed
  5dfb5e5

• keep ruff happy

  ajtejankar

  committed
  bca4daf

Commits on Oct 29, 2024

• rename fp8_kv quantization to fp8-kv

  ajtejankar

  committed
  422d9a4