Support W8A8 inference in vllm#1508

Closed

AniZpZ wants to merge 110 commits intovllm-project:mainfrom

Commits on Sep 20, 2023

Commits on Sep 21, 2023

Commits on Oct 30, 2023

add int8gemm
AniZpZ
committed
support int8 inference
AniZpZ
committed
Reduce alpha,beta unnecessary d2h

sleepcoo
authored and
AniZpZ
committed
fix weight load
AniZpZ
committed
fix weight load
AniZpZ
committed
fix ln layer init
AniZpZ
committed
rms norm fusion
AniZpZ
committed
fix w8a8 linear
AniZpZ
committed
use same scale across tensors
AniZpZ
committed
add ftgemm

zhangying169
authored and
AniZpZ
committed
fix cublas linear
AniZpZ
committed
clean cublass gemm code
AniZpZ
committed
code clean
AniZpZ
committed
fuse dequant silu and quant

zhangying169
authored and
AniZpZ
committed
fuse dequant and add residual

zhangying169
authored and
AniZpZ
committed
fuse dequant, add residual, rms_norm and quant

zhangying169
authored and
AniZpZ
committed
fuse dequant and pos_encoding

zhangying169
authored and
AniZpZ
committed
setup for fused kernels

zhangying169
authored and
AniZpZ
committed
fix bugs

zhangying169
authored and
AniZpZ
committed
add tests for fusion kernels

zhangying169
authored and
AniZpZ
committed
fix uncontiguous tensor case

zhangying169
authored and
AniZpZ
committed
add quant, dequant kernel

zhangying169
authored and
AniZpZ
committed
optimize layernorm kernel

zhangying169
authored and
AniZpZ
committed
add python class DequantAddResidualI8RMSNormQuant, DequantPagedAttentionWithRoPEQuant, DequantSiluAndMulQuant

zhangying169
authored and
AniZpZ
committed
add tests

zhangying169
authored and
AniZpZ
committed
add w8a8linear without quant and dequant

zhangying169
authored and
AniZpZ
committed
adjust code for fusion

zhangying169
authored and
AniZpZ
committed
rm obsolete file
AniZpZ
committed
fix llama
AniZpZ
committed
remove cutlass dependency
AniZpZ
committed
add sq quantized linear
AniZpZ
committed
rm unit test for w8a8 linear
AniZpZ
committed
adjust i8 llama weight load
AniZpZ
committed
add fusion.py
AniZpZ
committed
code clean
AniZpZ
committed
support kv cache quantization

Lin Pengyun
authored and
AniZpZ
committed
fix python code

Lin Pengyun
authored and
AniZpZ
committed
merge and reformat

Lin Pengyun
authored and
AniZpZ
committed
support generating kv quant parameters and evaluting kv quant models

Lin Pengyun
authored and
AniZpZ
committed
modify test functions

Lin Pengyun
authored and
AniZpZ
committed
fix test code

Lin Pengyun
authored and
AniZpZ
committed
fix test attention
AniZpZ
committed
modify attention kernel test using pytest

Lin Pengyun
authored and
AniZpZ
committed
fix quant parameter passing

Lin Pengyun
authored and
AniZpZ
committed
code clean
AniZpZ
committed
code clean
AniZpZ
committed
simplify code

zhangying169
authored and
AniZpZ
committed
simplify code

zhangying169
authored and
AniZpZ
committed

Commits on Nov 2, 2023

Commits on Nov 3, 2023

Commits on Nov 15, 2023

fix merge
AniZpZ
committed

Commits on Nov 20, 2023

fix reshape_and_cache_quantized
AniZpZ
committed

Commits on Nov 21, 2023

fix load weight bug
zhangying169
committed
update w8a8 kernels
zhangying169
committed
fix w8a8
AniZpZ
committed

Commits on Nov 22, 2023

update kernel tests
zhangying169
committed
fix merge problems
zhangying169
committed
fix bugs
zhangying169
committed
fix pos_emb bugs
zhangying169
committed
fix merge problems
zhangying169
committed
fix merge problems
zhangying169
committed
fix w8a8
zhangying169
committed
code format
AniZpZ
committed
fix rotary_embedding bug
zhangying169
committed
tmp fix
AniZpZ
committed
tmp fix2
AniZpZ
committed

Commits on Nov 23, 2023

fix linear
AniZpZ
committed
fix int8 linear bugs
zhangying169
committed
update kv-quant kernels
zhangying169
committed
add kv-quant kernel tests
zhangying169
committed
support kv-quant
zhangying169
committed
Merge branch 'kv_quant' into vllmq
zhangying169
committed

Commits on Nov 24, 2023

fix merge problems
zhangying169
committed
fix work bugs
zhangying169
committed

Commits on Dec 1, 2023

fix kernel bugs
zhangying169
committed
support per-token quant tensor parallel
zhangying169
committed
fix tp load weight
zhangying169
committed
fix work bugs

zhangying169
authored and
AniZpZ
committed
support per-token quant tensor parallel

zhangying169
authored and
AniZpZ
committed
fix tp load weight

zhangying169
authored and
AniZpZ
committed

Commits on Dec 5, 2023

fix kv-quant args
zhangying169
committed
fix I8CUGEMM duplicate instance issue
zhangying169
committed
fix I8CUGEMM duplicate instance issue
zhangying169
committed

Commits on Dec 7, 2023

Merge tag 'v0.2.3' into vllmq_0.2.3
zhangying169
committed

Commits on Dec 13, 2023

optimize kernels and dequant scale
zhangying169
committed
fix dtype bug
zhangying169
committed

Commits on Jan 8, 2024

optimize kernels and dequant scale
zhangying169
committed
fix kv-quant bugs
zhangying169
committed
fix attention params
zhangying169
committed
merge utils.cuh to quant_utils.cuh
zhangying169
committed
fix bugs
zhangying169
committed

Commits on Jan 9, 2024

Merge tag 'v0.2.7' into vllmq_0.2.7
zhangying169
committed

Commits on Jan 16, 2024

refactor smoothquant code
zhangying169
committed
Merge remote-tracking branch 'origin/vllmq_merge' into w8a8_v0.2.7
zhangying169
committed
format code
zhangying169
committed
add .buildkite
zhangying169
committed

Commits on Jan 31, 2024

fix quant config
zhangying169
committed
fix tp bug
zhangying169
committed