Code

Features integration#5

Code

Closed

gshtras wants to merge 190 commits intomainROCm/vllm:mainfrom integrationROCm/vllm:integrationCopy head branch name to clipboard

+8,572-228

Commits

Commits on Jan 27, 2024

• [ROCm] Fixup arch checks for ROCM

  Show description for 9c2367c
  dllehr-amd

  committed
  9c2367c

  Copy full SHA for 9c2367c

• yapf cleanup

  dllehr-amd

  committed
  a9d752c

  Copy full SHA for a9d752c

Commits on Feb 5, 2024

• Add hip_fp8 datatype and conversions

  Show description for ad53f74
  gshtras

  committed
  ad53f74

  Copy full SHA for ad53f74

Commits on Feb 6, 2024

• Add 3rdparty quantizer utility and usage to quantize models (HF default)

  HaiShaw

  committed
  9b1577a

  Copy full SHA for 9b1577a

Commits on Feb 7, 2024

• Update 3rdparty quantizer utility and usage with ammo updates

  HaiShaw

  committed
  644b165

  Copy full SHA for 644b165

• Use e4m3 and e5m2 interchangeably

  gshtras

  committed
  0ed1d98

  Copy full SHA for 0ed1d98

• Using fp8 in any cache tests that could support it

  gshtras

  committed
  18b2516

  Copy full SHA for 18b2516

Commits on Feb 8, 2024

• Integrate e4m3 alongside e5m2 and adapt cache tests

  gshtras

  committed
  81a6859

  Copy full SHA for 81a6859

• Add gfx942 to the arch list

  gshtras

  committed
  83089d0

  Copy full SHA for 83089d0

• Less forgiving atol in fp8 tests

  gshtras

  committed
  926e2b8

  Copy full SHA for 926e2b8

Commits on Feb 9, 2024

• Merge pull request #1 from ROCm/greg/fp8_tests

  Show description for 475a2ef
  HaiShaw

  authored
  475a2ef

  Copy full SHA for 475a2ef

• enable fp8-e4m3 kv cache on rocm

  

  AdrianAbeyta

  authored and

  AdrianAbeyta

  committed
  1b8bc9f

  Copy full SHA for 1b8bc9f

• Rename remaining fp8_e5m2 to general fp8

  Show description for 777cc35
  gshtras

  committed
  777cc35

  Copy full SHA for 777cc35

• Fix or comparisons

  mawong-amd

  committed
  a2897d6

  Copy full SHA for a2897d6

• Add e4m3 to attention kernels

  gshtras

  committed
  a06ddac

  Copy full SHA for a06ddac

• Remove remaining mentions of e5m2 where it refers to general fp8

  gshtras

  committed
  f358dcd

  Copy full SHA for f358dcd

• Address naming conventions

  

  AdrianAbeyta

  authored and

  AdrianAbeyta

  committed
  4db0038

  Copy full SHA for 4db0038

• Merge branch 'fp8-e4m3-kvcache-rocm' of https://github.com/ROCm/vllm-fp8 into fp8-e4m3-kvcache-rocm

  

  AdrianAbeyta

  authored and

  AdrianAbeyta

  committed
  2e85bc7

  Copy full SHA for 2e85bc7

• More verbose help message for fp8 cache type

  gshtras

  committed
  17a91a0

  Copy full SHA for 17a91a0

• Updated fp8 help text in additional files sililar to arg_utils

  gshtras

  committed
  4fbc915

  Copy full SHA for 4fbc915

• Merge pull request #3 from ROCm/greg/tweaks

  Show description for 7f5623d
  HaiShaw

  authored
  7f5623d

  Copy full SHA for 7f5623d

• Merge branch 'fp8_kv' of https://github.com/ROCm/vllm-fp8 into fp8-e4m3-kvcache-rocm

  

  AdrianAbeyta

  authored and

  AdrianAbeyta

  committed
  0da44bc

  Copy full SHA for 0da44bc

• Fix merge conflict

  

  AdrianAbeyta

  authored and

  AdrianAbeyta

  committed
  2c525de

  Copy full SHA for 2c525de

• generalize fp8 convention

  

  AdrianAbeyta

  authored and

  AdrianAbeyta

  committed
  54d1d4d

  Copy full SHA for 54d1d4d

• Merge pull request #2 from ROCm/fp8-e4m3-kvcache-rocm

  Show description for 4a0d880
  HaiShaw

  authored
  4a0d880

  Copy full SHA for 4a0d880

Commits on Feb 10, 2024

• Update log info and args description w.r.t. FP8 KV cache.

  HaiShaw

  committed
  7a9db00

  Copy full SHA for 7a9db00

• Initial port of gradlib gemm tuner

  dllehr-amd

  committed
  20b5f10

  Copy full SHA for 20b5f10

Commits on Feb 11, 2024

• Enable torchrun vs Ray

  Show description for 6f28107
  dllehr-amd

  committed
  6f28107

  Copy full SHA for 6f28107

• Add custom matvec kernels and sampler matmul call tuned_gemm

  dllehr-amd

  committed
  184806e

  Copy full SHA for 184806e

Commits on Feb 13, 2024

• Initial skeleton; scaling factors in CacheEngine and PagedAttention

  mawong-amd

  committed
  0d59cfb

  Copy full SHA for 0d59cfb

Commits on Feb 14, 2024

• Add silu gemm fusion when batch and seq_len = 1

  dllehr-amd

  committed
  af9e9d1

  Copy full SHA for af9e9d1

• Add tunable flags to VLLM

  dllehr-amd

  committed
  5f8eac3

  Copy full SHA for 5f8eac3

• Allow benchmark_latency to take a list of input/output/batches for faster execution

  Show description for 22766b4
  dllehr-amd

  committed
  22766b4

  Copy full SHA for 22766b4

Commits on Feb 15, 2024

• Add dynamic tuning feature to vllm

  dllehr-amd

  committed
  87b4c1b

  Copy full SHA for 87b4c1b

• Add rpd tracer controls to benchmark_latency.py

  dllehr-amd

  committed
  694ae1d

  Copy full SHA for 694ae1d

• Initial conversion back to KV cache scales in model, using float scaling factor instead of Tensor, should be working out the box

  mawong-amd

  committed
  eaf08ff

  Copy full SHA for eaf08ff

Commits on Feb 20, 2024

• Completing KV cache scaling factors ingest (TP>1 todo), clean up code; Isolating math works

  HaiShaw

  committed
  0a80226

  Copy full SHA for 0a80226

• Fix typos, add a few more sanity checks to the KV cache scales loader, remove PT support from scales extraction utility

  mawong-amd

  committed
  7b26ec9

  Copy full SHA for 7b26ec9

• Add additional checks to the scaling factor loader and fail gracefully on errors

  mawong-amd

  committed
  936821e

  Copy full SHA for 936821e

• Remove lingering PT fallback in extraction utility

  mawong-amd

  committed
  74b2b3f

  Copy full SHA for 74b2b3f

• Add ROCm clarification to extract scales script

  mawong-amd

  committed
  2c7ce96

  Copy full SHA for 2c7ce96

• Merge pull request #4 from ROCm/fp8_ingest_stage1_model

  Show description for 5985e25
  AdrianAbeyta

  authored
  5985e25

  Copy full SHA for 5985e25

Commits on Feb 21, 2024

• Preliminary TP rank > 1 extraction and loading support

  mawong-amd

  committed
  763b283

  Copy full SHA for 763b283

• Ensure loaded dictionary has same TP size as currently running engine

  mawong-amd

  committed
  ef26716

  Copy full SHA for ef26716

Commits on Feb 22, 2024

• Fix Dockerfile errors

  dllehr-amd

  committed
  0e73aed

  Copy full SHA for 0e73aed

• Add llama2 run script

  dllehr-amd

  committed
  90df0c9

  Copy full SHA for 90df0c9

• Increase Partition and Num threads for attention blocks

  dllehr-amd

  committed
  ab67280

  Copy full SHA for ab67280

• Fix WORKDIR

  dllehr-amd

  committed
  1d53722

  Copy full SHA for 1d53722

• Add accuracy flag to benchmark_latency.py

  dllehr-amd

  committed
  5148aa5

  Copy full SHA for 5148aa5

Commits on Feb 23, 2024

• Add tp_size argument for user to specify TP size to expect in quantized model. Renamed --model argument to --quantized_model to be explicit.

  mawong-amd

  committed
  c7e2587

  Copy full SHA for c7e2587

• Add specific FP8 E4M3 and ROCm flavor text to the --quantized_model argument

  mawong-amd

  committed
  61f2046

  Copy full SHA for 61f2046

• Small tweak on expected TP size flavor text for clarity

  mawong-amd

  committed
  dc71088

  Copy full SHA for dc71088

• Add output filename argument, rename output_path to output_dir, and clean up some logic

  mawong-amd

  committed
  4cd76e9

  Copy full SHA for 4cd76e9

• Fix up remaining 'output_path's from the rename

  mawong-amd

  committed
  ad8b841

  Copy full SHA for ad8b841

• Add scaling factor correction for ROCm FP8

  mawong-amd

  committed
  fec2232

  Copy full SHA for fec2232

• Add example output for extract_scales

  

  root

  committed
  7fdcf10

  Copy full SHA for 7fdcf10

• Strip out download functionality in scale extraction utility

  mawong-amd

  committed
  6a6bbcd

  Copy full SHA for 6a6bbcd

• Merge pull request #5 from ROCm/fp8_ingest_stage1_model

  Show description for 9336cdb
  AdrianAbeyta

  authored
  9336cdb

  Copy full SHA for 9336cdb

• Correcting a stray type hint

  mawong-amd

  committed
  8e108d3

  Copy full SHA for 8e108d3

• Merge branch 'fp8_kv' into fp8_ingest_scales_correction

  mawong-amd

  authored
  31ebfa6

  Copy full SHA for 31ebfa6

• Correct a stray type hint

  mawong-amd

  committed
  4dd7d1e

  Copy full SHA for 4dd7d1e

• Create README.md and add usage example

  AdrianAbeyta

  authored
  9a03b96

  Copy full SHA for 9a03b96

• Added benchmark description

  AdrianAbeyta

  authored
  4064973

  Copy full SHA for 4064973

Commits on Feb 24, 2024

• Clean up readme

  AdrianAbeyta

  authored
  988ffc3

  Copy full SHA for 988ffc3

Commits on Feb 26, 2024

• Merge pull request #6 from ROCm/fp8_ingest_scales_correction

  Show description for 0650ae4
  AdrianAbeyta

  authored
  0650ae4

  Copy full SHA for 0650ae4

• Don't broadcast when using torchrun

  dllehr-amd

  committed
  534dcff

  Copy full SHA for 534dcff

• Change convention: Initialize scaling factors always if KV cache is FP8 and exceptions are handled, else terminate program execution

  mawong-amd

  committed
  ee6ba29

  Copy full SHA for ee6ba29

• Updated example descriptions

  AdrianAbeyta

  authored
  4007656

  Copy full SHA for 4007656

• Merge pull request #8 from ROCm/fp8_ingest_stage1_model

  Show description for 6f2b248
  AdrianAbeyta

  authored
  6f2b248

  Copy full SHA for 6f2b248

• Merge pull request #7 from ROCm/fp8_doc

  Show description for 0a45612
  AdrianAbeyta

  authored
  0a45612

  Copy full SHA for 0a45612

Commits on Feb 27, 2024

• Kernel and Device functions to enable FP8 KV cache scaling factors

  HaiShaw

  committed
  c8059c2

  Copy full SHA for c8059c2

• Make KV cache scaling factors default to 1.0 instead of None

  HaiShaw

  committed
  fc2cdaf

  Copy full SHA for fc2cdaf

• Update KV cache scales loader name to clarify that we are not using an iterator at present

  mawong-amd

  committed
  76c6058

  Copy full SHA for 76c6058

Commits on Feb 28, 2024

• Fix test cases from the introduction of KV cache scaling factors, using default kv_scale=1.0

  HaiShaw

  committed
  c825bb3

  Copy full SHA for c825bb3

Commits on Feb 29, 2024

• Cleanup comments according to reviews

  HaiShaw

  committed
  4f574cd

  Copy full SHA for 4f574cd

• Merge pull request #9 from ROCm/fp8_kv_cache

  Show description for 86f06ca
  HaiShaw

  authored
  86f06ca

  Copy full SHA for 86f06ca

Commits on Mar 1, 2024

• Add hip_fp8 datatype and conversions

  Show description for f325cb0
  

  gshtras

  authored and

  AdrianAbeyta

  committed
  f325cb0

  Copy full SHA for f325cb0

• Add 3rdparty quantizer utility and usage to quantize models (HF default)

  

  HaiShaw

  authored and

  AdrianAbeyta

  committed
  4bb8dac

  Copy full SHA for 4bb8dac

• Update 3rdparty quantizer utility and usage with ammo updates

  

  HaiShaw

  authored and

  AdrianAbeyta

  committed
  8b1279b

  Copy full SHA for 8b1279b

• Use e4m3 and e5m2 interchangeably

  

  gshtras

  authored and

  AdrianAbeyta

  committed
  9c4226e

  Copy full SHA for 9c4226e

• Using fp8 in any cache tests that could support it

  

  gshtras

  authored and

  AdrianAbeyta

  committed
  30bba1c

  Copy full SHA for 30bba1c

• Integrate e4m3 alongside e5m2 and adapt cache tests

  

  gshtras

  authored and

  AdrianAbeyta

  committed
  692f5ad

  Copy full SHA for 692f5ad

• Add gfx942 to the arch list

  

  gshtras

  authored and

  AdrianAbeyta

  committed
  c9321a0

  Copy full SHA for c9321a0

• Less forgiving atol in fp8 tests

  

  gshtras

  authored and

  AdrianAbeyta

  committed
  4e1f89a

  Copy full SHA for 4e1f89a

• enable fp8-e4m3 kv cache on rocm

  

  AdrianAbeyta

  authored and

  AdrianAbeyta

  committed
  7bc2574

  Copy full SHA for 7bc2574

• Address naming conventions

  

  AdrianAbeyta

  authored and

  AdrianAbeyta

  committed
  ebf7542

  Copy full SHA for ebf7542

• Fix or comparisons

  

  mawong-amd

  authored and

  AdrianAbeyta

  committed
  ad44055

  Copy full SHA for ad44055

• Rename remaining fp8_e5m2 to general fp8

  Show description for e5e0e7c
  

  gshtras

  authored and

  AdrianAbeyta

  committed
  e5e0e7c

  Copy full SHA for e5e0e7c

• Add e4m3 to attention kernels

  

  gshtras

  authored and

  AdrianAbeyta

  committed
  c86b2ec

  Copy full SHA for c86b2ec

• Remove remaining mentions of e5m2 where it refers to general fp8

  

  gshtras

  authored and

  AdrianAbeyta

  committed
  a432815

  Copy full SHA for a432815

• Updated fp8 help text in additional files sililar to arg_utils

  

  gshtras

  authored and

  AdrianAbeyta

  committed
  4b77126

  Copy full SHA for 4b77126

• generalize fp8 convention

  

  AdrianAbeyta

  authored and

  AdrianAbeyta

  committed
  bbf6d49

  Copy full SHA for bbf6d49

• Update log info and args description w.r.t. FP8 KV cache.

  

  HaiShaw

  authored and

  AdrianAbeyta

  committed
  4dfb26d

  Copy full SHA for 4dfb26d

• Initial skeleton; scaling factors in CacheEngine and PagedAttention

  

  mawong-amd

  authored and

  AdrianAbeyta

  committed
  0f492f5

  Copy full SHA for 0f492f5

• Initial conversion back to KV cache scales in model, using float scaling factor instead of Tensor, should be working out the box

  

  mawong-amd

  authored and

  AdrianAbeyta

  committed
  030c9eb

  Copy full SHA for 030c9eb

• Completing KV cache scaling factors ingest (TP>1 todo), clean up code; Isolating math works

  

  HaiShaw

  authored and

  AdrianAbeyta

  committed
  e00a86d

  Copy full SHA for e00a86d

• Fix typos, add a few more sanity checks to the KV cache scales loader, remove PT support from scales extraction utility

  

  mawong-amd

  authored and

  AdrianAbeyta

  committed
  d3e98f3

  Copy full SHA for d3e98f3

• Add additional checks to the scaling factor loader and fail gracefully on errors

  

  mawong-amd

  authored and

  AdrianAbeyta

  committed
  714e42c

  Copy full SHA for 714e42c

• Remove lingering PT fallback in extraction utility

  

  mawong-amd

  authored and

  AdrianAbeyta

  committed
  3ff51b1

  Copy full SHA for 3ff51b1

• Add ROCm clarification to extract scales script

  

  mawong-amd

  authored and

  AdrianAbeyta

  committed
  e97a31e

  Copy full SHA for e97a31e

• Add scaling factor correction for ROCm FP8

  

  mawong-amd

  authored and

  AdrianAbeyta

  committed
  0ba975d

  Copy full SHA for 0ba975d

• Correcting a stray type hint

  

  mawong-amd

  authored and

  AdrianAbeyta

  committed
  6991c59

  Copy full SHA for 6991c59

• Preliminary TP rank > 1 extraction and loading support

  

  mawong-amd

  authored and

  AdrianAbeyta

  committed
  2292776

  Copy full SHA for 2292776

• Ensure loaded dictionary has same TP size as currently running engine

  

  mawong-amd

  authored and

  AdrianAbeyta

  committed
  221699b

  Copy full SHA for 221699b

• Add tp_size argument for user to specify TP size to expect in quantized model. Renamed --model argument to --quantized_model to be explicit.

  

  mawong-amd

  authored and

  AdrianAbeyta

  committed
  96a7546

  Copy full SHA for 96a7546

• Add specific FP8 E4M3 and ROCm flavor text to the --quantized_model argument

  

  mawong-amd

  authored and

  AdrianAbeyta

  committed
  9d08a92

  Copy full SHA for 9d08a92

• Small tweak on expected TP size flavor text for clarity

  

  mawong-amd

  authored and

  AdrianAbeyta

  committed
  553209b

  Copy full SHA for 553209b

• Add output filename argument, rename output_path to output_dir, and clean up some logic

  

  mawong-amd

  authored and

  AdrianAbeyta

  committed
  c852549

  Copy full SHA for c852549

• Fix up remaining 'output_path's from the rename

  

  mawong-amd

  authored and

  AdrianAbeyta

  committed
  e23379e

  Copy full SHA for e23379e

• Strip out download functionality in scale extraction utility

  

  mawong-amd

  authored and

  AdrianAbeyta

  committed
  40171c9

  Copy full SHA for 40171c9

• Correct a stray type hint

  

  mawong-amd

  authored and

  AdrianAbeyta

  committed
  7666587

  Copy full SHA for 7666587

• Change convention: Initialize scaling factors always if KV cache is FP8 and exceptions are handled, else terminate program execution

  

  mawong-amd

  authored and

  AdrianAbeyta

  committed
  7c0bf6e

  Copy full SHA for 7c0bf6e

• Add example output for extract_scales

  

  root

  authored and

  AdrianAbeyta

  committed
  42e2aef

  Copy full SHA for 42e2aef

• Create README.md and add usage example

  

  AdrianAbeyta

  authored and

  AdrianAbeyta

  committed
  d3dbb1a

  Copy full SHA for d3dbb1a

• Added benchmark description

  

  AdrianAbeyta

  authored and

  AdrianAbeyta

  committed
  f39839b

  Copy full SHA for f39839b

• Clean up readme

  

  AdrianAbeyta

  authored and

  AdrianAbeyta

  committed
  2cfea65

  Copy full SHA for 2cfea65

• Updated example descriptions

  

  AdrianAbeyta

  authored and

  AdrianAbeyta

  committed
  257a7da

  Copy full SHA for 257a7da

• Update KV cache scales loader name to clarify that we are not using an iterator at present

  

  mawong-amd

  authored and

  AdrianAbeyta

  committed
  730562a

  Copy full SHA for 730562a

• Kernel and Device functions to enable FP8 KV cache scaling factors

  

  HaiShaw

  authored and

  AdrianAbeyta

  committed
  f20eceb

  Copy full SHA for f20eceb

• Make KV cache scaling factors default to 1.0 instead of None

  

  HaiShaw

  authored and

  AdrianAbeyta

  committed
  8834917

  Copy full SHA for 8834917

• Fix test cases from the introduction of KV cache scaling factors, using default kv_scale=1.0

  

  HaiShaw

  authored and

  AdrianAbeyta

  committed
  3187582

  Copy full SHA for 3187582

• Cleanup comments according to reviews

  

  HaiShaw

  authored and

  AdrianAbeyta

  committed
  49df502

  Copy full SHA for 49df502

Commits on Mar 4, 2024

• Remove load_dummy_kv_cache_scales as convention change in PR#9 renders it superfluous

  mawong-amd

  committed
  12f7650

  Copy full SHA for 12f7650

Commits on Mar 5, 2024

• Add back removal of gather cached kv kernel for use with FP8

  

  AdrianAbeyta

  authored and

  AdrianAbeyta

  committed
  4a8d06c

  Copy full SHA for 4a8d06c

• Clean up IFU

  

  AdrianAbeyta

  authored and

  AdrianAbeyta

  committed
  b87aec1

  Copy full SHA for b87aec1

• Clean up IFU

  

  AdrianAbeyta

  authored and

  AdrianAbeyta

  committed
  fa6fbce

  Copy full SHA for fa6fbce

Commits on Mar 6, 2024

• Schema change: preliminary changes to extract script, TODO: loading logic refactor

  mawong-amd

  committed
  65f70d7

  Copy full SHA for 65f70d7

• Fix runtime issues with upstream rebase

  

  AdrianAbeyta

  authored and

  AdrianAbeyta

  committed
  f5c0236

  Copy full SHA for f5c0236

• [Minor fix] The domain dns.google may cause a socket.gaierror exception (vllm-project#3176)

  Show description for d2a42f9
  

  2 peopleauthored and

  AdrianAbeyta

  committed
  d2a42f9

  Copy full SHA for d2a42f9

Commits on Mar 7, 2024

• Preliminary refactoring: KV cache scales JSON into general scales JSON schema

  mawong-amd

  committed
  4b3e4b0

  Copy full SHA for 4b3e4b0

• Merge branch 'fp8_kv' into fp8_ingest_stage1_model

  mawong-amd

  committed
  00f5113

  Copy full SHA for 00f5113

• Fixing stray syntax errors and typos, refactoring rank_keyword detection

  Show description for a7e6e81
  mawong-amd

  committed
  a7e6e81

  Copy full SHA for a7e6e81

• Address reviewer comments

  mawong-amd

  committed
  ef85f98

  Copy full SHA for ef85f98

• Address Greg's strong type checking :)

  mawong-amd

  committed
  d8b2843

  Copy full SHA for d8b2843

• Add an additional TODO

  mawong-amd

  committed
  52df603

  Copy full SHA for 52df603

• Merge remote-tracking branch 'upstream/main' into IFU-2024-03-01-fp8-kv

  

  AdrianAbeyta

  authored and

  AdrianAbeyta

  committed
  b484112

  Copy full SHA for b484112

• Merge pull request #16 from ROCm/fp8_ingest_stage1_model

  Show description for 7b72159
  AdrianAbeyta

  authored
  7b72159

  Copy full SHA for 7b72159

• Fix OOM bug in quantize script, remove extraneous model_export

  mawong-amd

  committed
  18c55d2

  Copy full SHA for 18c55d2

• Fix rocm build conditions

  

  AdrianAbeyta

  authored and

  AdrianAbeyta

  committed
  7d0fa2f

  Copy full SHA for 7d0fa2f

• Keep previous build flow for neuron

  

  AdrianAbeyta

  authored and

  AdrianAbeyta

  committed
  e7db6af

  Copy full SHA for e7db6af

• Merge remote-tracking branch 'origin/fp8_kv' into IFU-2024-03-01-fp8-kv

  

  AdrianAbeyta

  authored and

  AdrianAbeyta

  committed
  660dbb3

  Copy full SHA for 660dbb3

• Measure model memory usage (vllm-project#3120)

  

  mgoin

  authored and

  AdrianAbeyta

  committed
  ca1b39c

  Copy full SHA for ca1b39c

• Possible fix for conflict between Automated Prefix Caching (vllm-project#2762) and multi-LoRA support (vllm-project#1804) (vllm-project#3263)

  

  jacobthebanana

  authored and

  AdrianAbeyta

  committed
  fd6e57e

  Copy full SHA for fd6e57e

• Update fp8 examples

  

  AdrianAbeyta

  authored and

  AdrianAbeyta

  committed
  90c2cd4

  Copy full SHA for 90c2cd4

Commits on Mar 8, 2024

• Merge remote-tracking branch 'upstream/main' into IFU-2024-03-01-fp8-kv

  

  AdrianAbeyta

  authored and

  AdrianAbeyta

  committed
  2d520f2

  Copy full SHA for 2d520f2

• [FIX] Make flash_attn optional (vllm-project#3269)

  

  WoosukKwon

  authored and

  AdrianAbeyta

  committed
  9e6144a

  Copy full SHA for 9e6144a

• Fix setup.py up to where it should be before the excitement of the last 24h (upstream flashattn changes were broken)

  mawong-amd

  committed
  fd01e9a

  Copy full SHA for fd01e9a

• Fix missing enable FP8_E4M3 flag and cherry pick newest load convention

  mawong-amd

  committed
  6edfbf1

  Copy full SHA for 6edfbf1

• Merge branch 'IFU-2024-03-01-fp8-kv' of https://github.com/ROCm/vllm-fp8 into IFU-2024-03-01-fp8-kv

  

  AdrianAbeyta

  authored and

  AdrianAbeyta

  committed
  be92918

  Copy full SHA for be92918

• Add model flag as example option

  

  AdrianAbeyta

  authored and

  AdrianAbeyta

  committed
  dd469df

  Copy full SHA for dd469df

• Merge pull request #17 from ROCm/IFU-2024-03-01-fp8-kv

  Show description for b3d81e0
  AdrianAbeyta

  authored
  b3d81e0

  Copy full SHA for b3d81e0

Commits on Mar 11, 2024

• Merge branch 'main' into fp8_kv

  AdrianAbeyta

  authored
  5a2d747

  Copy full SHA for 5a2d747

• Merge remote-tracking branch 'upstream/main' into fp8_kv

  

  AdrianAbeyta

  authored and

  AdrianAbeyta

  committed
  267c847

  Copy full SHA for 267c847

Commits on Mar 13, 2024

• Fix ruff syntax errors

  

  AdrianAbeyta

  authored and

  AdrianAbeyta

  committed
  2f60ad7

  Copy full SHA for 2f60ad7

• Merge remote-tracking branch 'upstream/main' into fp8_kv

  

  AdrianAbeyta

  authored and

  AdrianAbeyta

  committed
  31d96dd

  Copy full SHA for 31d96dd

• Update model config for scales path

  

  AdrianAbeyta

  authored and

  AdrianAbeyta

  committed
  94c2e7c

  Copy full SHA for 94c2e7c

• Add .rst for fp8_e4m3_kvcache and rename fp8_kvcache to fp8_e5m2

  

  AdrianAbeyta

  authored and

  AdrianAbeyta

  committed
  a350641

  Copy full SHA for a350641

Commits on Mar 14, 2024

• Skip fp8 UT test on CUDA for e4m3

  

  AdrianAbeyta

  authored and

  AdrianAbeyta

  committed
  eb8e3d8

  Copy full SHA for eb8e3d8

• Fix device id formatting

  gshtras

  committed
  f9eba0c

  Copy full SHA for f9eba0c

Commits on Mar 15, 2024

• Fix scales_path location

  mawong-amd

  committed
  db8f29c

  Copy full SHA for db8f29c

• Fix yapf formatting

  mawong-amd

  committed
  49d1593

  Copy full SHA for 49d1593

• Adding new rocm triton flash attention kernel

  Show description for e569133
  

  jpvillam-amd

  and

  vgokhale

  committed
  e569133

  Copy full SHA for e569133

• Skipping certain cache tests when using fp8 cache with e5m2 type. These were only initially enabled in light of the support for e4m3

  gshtras

  committed
  b5ebb41

  Copy full SHA for b5ebb41

• Fix yapf ci error

  

  AdrianAbeyta

  authored and

  AdrianAbeyta

  committed
  45d5912

  Copy full SHA for 45d5912

Commits on Mar 18, 2024

• Merge remote-tracking branch 'origin/main' into v0.3.3_greg

  gshtras

  committed
  8cb05bc

  Copy full SHA for 8cb05bc

• Removed gradlib and its tuned gemm in favor of tunable ops

  gshtras

  committed
  be708d0

  Copy full SHA for be708d0

• Merge remote-tracking branch 'origin/fp8_kv' into v0.3.3_greg

  gshtras

  committed
  a2311fd

  Copy full SHA for a2311fd

• Merge branch 'jpvillam/v0.3.3_triton' into v0.3.3_greg

  gshtras

  committed
  8f266e8

  Copy full SHA for 8f266e8

• Initializing scaling factors for kv cache in flash attention backend

  gshtras

  committed
  1510843

  Copy full SHA for 1510843

Commits on Mar 19, 2024

• Removed obsolete parts. Made rocm attention defaults define guarded

  gshtras

  committed
  3d82eea

  Copy full SHA for 3d82eea

• Adding functionality with benchmarks/measure_ppl_MC_small.py. The script is supposed to be a quick correctness benchmark.

  Alexei-V-Ivanov-AMD

  committed
  86062a5

  Copy full SHA for 86062a5

• Merge pull request #6 from ROCm/integration_alexei

  Show description for 6ce8e8f
  gshtras

  authored
  6ce8e8f

  Copy full SHA for 6ce8e8f

• Fix triton build condition

  gshtras

  committed
  ce7078d

  Copy full SHA for ce7078d

• Merge branch 'integration' of github.com:ROCm/vllm into integration

  gshtras

  committed
  54d82e6

  Copy full SHA for 54d82e6

• Small fix on dockerfile

  jpvillam-amd

  committed
  0e63661

  Copy full SHA for 0e63661

• Update description of measure_ppl_MC_small.py

  Show description for c45547b
  Alexei-V-Ivanov-AMD

  authored
  c45547b

  Copy full SHA for c45547b

• Merge remote-tracking branch 'origin/main' into jpvillam/v0.3.3_triton

  jpvillam-amd

  committed
  c89c0e3

  Copy full SHA for c89c0e3

• Rebase updates and PR review changes

  Show description for d4cb905
  jpvillam-amd

  committed
  d4cb905

  Copy full SHA for d4cb905

Commits on Mar 20, 2024

• Merge remote-tracking branch 'origin/jpvillam/v0.3.3_triton' into integration

  gshtras

  committed
  9cb2bbd

  Copy full SHA for 9cb2bbd

Commits on Mar 21, 2024

• Introducing torchrun multi GPU support

  gshtras

  committed
  9d96fdb

  Copy full SHA for 9d96fdb

• add use case for custom kernel for matvec operation

  charlifu

  committed
  51ce9f5

  Copy full SHA for 51ce9f5

• Merge branch 'greg/torchrun' into integration

  gshtras

  committed
  ec9a8c0

  Copy full SHA for ec9a8c0

• Merge branch 'integration' of github.com:ROCm/vllm into integration

  gshtras

  committed
  bea2883

  Copy full SHA for bea2883

• Remove ignored file

  gshtras

  committed
  47c560e

  Copy full SHA for 47c560e

• limit the custom kernel under is_hip

  charlifu

  committed
  97e1978

  Copy full SHA for 97e1978

• Fix parameter

  gshtras

  committed
  ed96036

  Copy full SHA for ed96036

• Unused import

  gshtras

  committed
  42324b6

  Copy full SHA for 42324b6

• fix custom kernel

  charlifu

  committed
  9b1388c

  Copy full SHA for 9b1388c

• Refactor torchrun executor to reuse single gpu executor code

  gshtras

  committed
  6b186bb

  Copy full SHA for 6b186bb

Commits on Mar 22, 2024

• Fixed mixed up values

  gshtras

  committed
  6ff0272

  Copy full SHA for 6ff0272