Enable scaled FP8 (e4m3fn) KV cache on ROCm (AMD GPU)#3290

Merged

simon-mo merged 178 commits intovllm-project:mainfrom

Apr 3, 2024

Commits on Feb 5, 2024

Add hip_fp8 datatype and conversions
gshtras
committed

Commits on Feb 6, 2024

Add 3rdparty quantizer utility and usage to quantize models (HF default)
HaiShaw
committed

Commits on Feb 7, 2024

Commits on Feb 8, 2024

Commits on Feb 9, 2024

Merge pull request #1 from ROCm/greg/fp8_tests
HaiShaw
authored
enable fp8-e4m3 kv cache on rocm

AdrianAbeyta
authored and
AdrianAbeyta
committed
Rename remaining fp8_e5m2 to general fp8
gshtras
committed
Fix or comparisons
mawong-amd
committed
Add e4m3 to attention kernels
gshtras
committed
Remove remaining mentions of e5m2 where it refers to general fp8
gshtras
committed
Address naming conventions

AdrianAbeyta
authored and
AdrianAbeyta
committed
Merge branch 'fp8-e4m3-kvcache-rocm' of https://github.com/ROCm/vllm-fp8 into fp8-e4m3-kvcache-rocm

AdrianAbeyta
authored and
AdrianAbeyta
committed
More verbose help message for fp8 cache type
gshtras
committed
Updated fp8 help text in additional files sililar to arg_utils
gshtras
committed
Merge pull request #3 from ROCm/greg/tweaks
HaiShaw
authored
Merge branch 'fp8_kv' of https://github.com/ROCm/vllm-fp8 into fp8-e4m3-kvcache-rocm

AdrianAbeyta
authored and
AdrianAbeyta
committed
Fix merge conflict

AdrianAbeyta
authored and
AdrianAbeyta
committed
generalize fp8 convention

AdrianAbeyta
authored and
AdrianAbeyta
committed
Merge pull request #2 from ROCm/fp8-e4m3-kvcache-rocm
HaiShaw
authored

Commits on Feb 10, 2024

Update log info and args description w.r.t. FP8 KV cache.
HaiShaw
committed

Commits on Feb 13, 2024

Initial skeleton; scaling factors in CacheEngine and PagedAttention
mawong-amd
committed

Commits on Feb 15, 2024

Initial conversion back to KV cache scales in model, using float scaling factor instead of Tensor, should be working out the box
mawong-amd
committed

Commits on Feb 21, 2024

Commits on Feb 23, 2024

Commits on Feb 24, 2024

Clean up readme
AdrianAbeyta
authored

Commits on Feb 27, 2024

Commits on Feb 28, 2024

Fix test cases from the introduction of KV cache scaling factors, using default kv_scale=1.0
HaiShaw
committed

Commits on Feb 29, 2024

Commits on Mar 1, 2024

Add hip_fp8 datatype and conversions

gshtras
authored and
AdrianAbeyta
committed
Add 3rdparty quantizer utility and usage to quantize models (HF default)

HaiShaw
authored and
AdrianAbeyta
committed
Update 3rdparty quantizer utility and usage with ammo updates

HaiShaw
authored and
AdrianAbeyta
committed
Use e4m3 and e5m2 interchangeably

gshtras
authored and
AdrianAbeyta
committed
Using fp8 in any cache tests that could support it

gshtras
authored and
AdrianAbeyta
committed
Integrate e4m3 alongside e5m2 and adapt cache tests

gshtras
authored and
AdrianAbeyta
committed
Add gfx942 to the arch list

gshtras
authored and
AdrianAbeyta
committed
Less forgiving atol in fp8 tests

gshtras
authored and
AdrianAbeyta
committed
enable fp8-e4m3 kv cache on rocm

AdrianAbeyta
authored and
AdrianAbeyta
committed
Address naming conventions

AdrianAbeyta
authored and
AdrianAbeyta
committed
Fix or comparisons

mawong-amd
authored and
AdrianAbeyta
committed
Rename remaining fp8_e5m2 to general fp8

gshtras
authored and
AdrianAbeyta
committed
Add e4m3 to attention kernels

gshtras
authored and
AdrianAbeyta
committed
Remove remaining mentions of e5m2 where it refers to general fp8

gshtras
authored and
AdrianAbeyta
committed
Updated fp8 help text in additional files sililar to arg_utils

gshtras
authored and
AdrianAbeyta
committed
generalize fp8 convention

AdrianAbeyta
authored and
AdrianAbeyta
committed
Update log info and args description w.r.t. FP8 KV cache.

HaiShaw
authored and
AdrianAbeyta
committed
Initial skeleton; scaling factors in CacheEngine and PagedAttention

mawong-amd
authored and
AdrianAbeyta
committed
Initial conversion back to KV cache scales in model, using float scaling factor instead of Tensor, should be working out the box

mawong-amd
authored and
AdrianAbeyta
committed
Completing KV cache scaling factors ingest (TP>1 todo), clean up code; Isolating math works

HaiShaw
authored and
AdrianAbeyta
committed
Fix typos, add a few more sanity checks to the KV cache scales loader, remove PT support from scales extraction utility

mawong-amd
authored and
AdrianAbeyta
committed
Add additional checks to the scaling factor loader and fail gracefully on errors

mawong-amd
authored and
AdrianAbeyta
committed
Remove lingering PT fallback in extraction utility

mawong-amd
authored and
AdrianAbeyta
committed
Add ROCm clarification to extract scales script

mawong-amd
authored and
AdrianAbeyta
committed
Add scaling factor correction for ROCm FP8

mawong-amd
authored and
AdrianAbeyta
committed
Correcting a stray type hint

mawong-amd
authored and
AdrianAbeyta
committed
Preliminary TP rank > 1 extraction and loading support

mawong-amd
authored and
AdrianAbeyta
committed
Ensure loaded dictionary has same TP size as currently running engine

mawong-amd
authored and
AdrianAbeyta
committed
Add tp_size argument for user to specify TP size to expect in quantized model. Renamed --model argument to --quantized_model to be explicit.

mawong-amd
authored and
AdrianAbeyta
committed
Add specific FP8 E4M3 and ROCm flavor text to the --quantized_model argument

mawong-amd
authored and
AdrianAbeyta
committed
Small tweak on expected TP size flavor text for clarity

mawong-amd
authored and
AdrianAbeyta
committed
Add output filename argument, rename output_path to output_dir, and clean up some logic

mawong-amd
authored and
AdrianAbeyta
committed
Fix up remaining 'output_path's from the rename

mawong-amd
authored and
AdrianAbeyta
committed
Strip out download functionality in scale extraction utility

mawong-amd
authored and
AdrianAbeyta
committed
Correct a stray type hint

mawong-amd
authored and
AdrianAbeyta
committed
Change convention: Initialize scaling factors always if KV cache is FP8 and exceptions are handled, else terminate program execution

mawong-amd
authored and
AdrianAbeyta
committed
Add example output for extract_scales

root
authored and
AdrianAbeyta
committed
Create README.md and add usage example

AdrianAbeyta
authored and
AdrianAbeyta
committed
Added benchmark description

AdrianAbeyta
authored and
AdrianAbeyta
committed
Clean up readme

AdrianAbeyta
authored and
AdrianAbeyta
committed
Updated example descriptions

AdrianAbeyta
authored and
AdrianAbeyta
committed
Update KV cache scales loader name to clarify that we are not using an iterator at present

mawong-amd
authored and
AdrianAbeyta
committed
Kernel and Device functions to enable FP8 KV cache scaling factors

HaiShaw
authored and
AdrianAbeyta
committed
Make KV cache scaling factors default to 1.0 instead of None

HaiShaw
authored and
AdrianAbeyta
committed
Fix test cases from the introduction of KV cache scaling factors, using default kv_scale=1.0

HaiShaw
authored and
AdrianAbeyta
committed
Cleanup comments according to reviews

HaiShaw
authored and
AdrianAbeyta
committed

Commits on Mar 4, 2024

Remove load_dummy_kv_cache_scales as convention change in PR#9 renders it superfluous
mawong-amd
committed

Commits on Mar 5, 2024

Add back removal of gather cached kv kernel for use with FP8

AdrianAbeyta
authored and
AdrianAbeyta
committed
Clean up IFU

AdrianAbeyta
authored and
AdrianAbeyta
committed
Clean up IFU

AdrianAbeyta
authored and
AdrianAbeyta
committed

Commits on Mar 6, 2024

Schema change: preliminary changes to extract script, TODO: loading logic refactor
mawong-amd
committed
Fix runtime issues with upstream rebase

AdrianAbeyta
authored and
AdrianAbeyta
committed
[Minor fix] The domain dns.google may cause a socket.gaierror exception (#3176)

authored and
AdrianAbeyta
committed

Commits on Mar 7, 2024

Preliminary refactoring: KV cache scales JSON into general scales JSON schema
mawong-amd
committed
Merge branch 'fp8_kv' into fp8_ingest_stage1_model
mawong-amd
committed
Fixing stray syntax errors and typos, refactoring rank_keyword detection
mawong-amd
committed
Address reviewer comments
mawong-amd
committed
Address Greg's strong type checking :)
mawong-amd
committed
Add an additional TODO
mawong-amd
committed
Merge remote-tracking branch 'upstream/main' into IFU-2024-03-01-fp8-kv

AdrianAbeyta
authored and
AdrianAbeyta
committed
Merge pull request #16 from ROCm/fp8_ingest_stage1_model
AdrianAbeyta
authored
Fix OOM bug in quantize script, remove extraneous model_export
mawong-amd
committed
Fix rocm build conditions

AdrianAbeyta
authored and
AdrianAbeyta
committed
Keep previous build flow for neuron

AdrianAbeyta
authored and
AdrianAbeyta
committed
Merge remote-tracking branch 'origin/fp8_kv' into IFU-2024-03-01-fp8-kv

AdrianAbeyta
authored and
AdrianAbeyta
committed
Measure model memory usage (#3120)

mgoin
authored and
AdrianAbeyta
committed
Possible fix for conflict between Automated Prefix Caching (#2762) and multi-LoRA support (#1804) (#3263)

jacobthebanana
authored and
AdrianAbeyta
committed
Update fp8 examples

AdrianAbeyta
authored and
AdrianAbeyta
committed

Commits on Mar 8, 2024

Merge remote-tracking branch 'upstream/main' into IFU-2024-03-01-fp8-kv

AdrianAbeyta
authored and
AdrianAbeyta
committed
[FIX] Make `flash_attn` optional (#3269)

WoosukKwon
authored and
AdrianAbeyta
committed
Fix setup.py up to where it should be before the excitement of the last 24h (upstream flashattn changes were broken)
mawong-amd
committed
Fix missing enable FP8_E4M3 flag and cherry pick newest load convention
mawong-amd
committed
Merge branch 'IFU-2024-03-01-fp8-kv' of https://github.com/ROCm/vllm-fp8 into IFU-2024-03-01-fp8-kv

AdrianAbeyta
authored and
AdrianAbeyta
committed
Add model flag as example option

AdrianAbeyta
authored and
AdrianAbeyta
committed
Merge pull request #17 from ROCm/IFU-2024-03-01-fp8-kv
AdrianAbeyta
authored

Commits on Mar 11, 2024

Merge branch 'main' into fp8_kv
AdrianAbeyta
authored
Merge remote-tracking branch 'upstream/main' into fp8_kv

AdrianAbeyta
authored and
AdrianAbeyta
committed

Commits on Mar 13, 2024

Fix ruff syntax errors

AdrianAbeyta
authored and
AdrianAbeyta
committed
Merge remote-tracking branch 'upstream/main' into fp8_kv

AdrianAbeyta
authored and
AdrianAbeyta
committed
Update model config for scales path

AdrianAbeyta
authored and
AdrianAbeyta
committed
Add .rst for fp8_e4m3_kvcache and rename fp8_kvcache to fp8_e5m2

AdrianAbeyta
authored and
AdrianAbeyta
committed

Commits on Mar 14, 2024

Skip fp8 UT test on CUDA for e4m3

AdrianAbeyta
authored and
AdrianAbeyta
committed
Fix device id formatting
gshtras
committed

Commits on Mar 15, 2024

Fix scales_path location
mawong-amd
committed
Fix yapf formatting
mawong-amd
committed
Skipping certain cache tests when using fp8 cache with e5m2 type. These were only initially enabled in light of the support for e4m3
gshtras
committed
Fix yapf ci error

AdrianAbeyta
authored and
AdrianAbeyta
committed

Commits on Mar 19, 2024

Commits on Mar 28, 2024

Fix things left out from latest merge
mawong-amd
committed