GPTQ performance in llmcompressor

[nirey10]

Describe the bug
The result of running gptq on a model with llmcompressor doesnt provide the expected results, atleast not like auto_gptq. i think there might be a bug in the implementation.

Expected behavior
A much higher accuracy on several metrics after running gptq

Environment
Include all relevant environment information:

1. Ubuntu 22.04.3 LTS
2. Python 3.10.12
3. LLM Compressor 0.4.0:
4. torch 2.4.0, transformers 4.48.2
5. Cuda 12.2

To Reproduce
Use model "meta-llama/Llama-3.1-8B-Instruct"
Run quantization only with the 2of4_w4a16_group-128_recipe.yaml example to quantize to 4 bit
Run quantization with auto_gptq to 4 bit witht the same parameters.
evaluate with lm_eval

llmcomrpessor results:
arc-c: 0.35
winogrande: 0.6
wikitext: 9.2

auto_gptq results:
arc-c: 0.49
winogrande: 0.72
wikitext: 9.12

base_results:
arc-c: 0.51
winogrande: 0.73
wikitext: 8.6420

As we can see there is a big difference while both running GPTQ method.

[nirey10]

The results differ from the gptq paper itself while using the same models and same configuration.

[kylesayrs]

Hi @nirey10,

The 2of4_w4a16_group-128_recipe.yaml is not equivalent to auto_gptq with 4 bit quantization. The 2of4 in the recipe name refers to applying structured sparsity, which yields higher performance at the cost of reduced accuracy

[kylesayrs]

If you want to perform replication, please use this example and open up a new ticket if you observe large differences.

Thanks!

[nirey10]

Hey, sorry i wasnt clear enough. i removed from the recipe the sparsification anf fine-tuning part, i did quantization only. i just mentioned the example so you know where i started from.

[kylesayrs]

@nirey10 I see. Our last regression evaluation was done with this PR last month, where we achieved 0.7545 on winogrande with Llama-3-8B-Instruct.

auto_gptq and LLM Compressor have very small implementation differences which can lead to significant accuracy differences of ~3% on winogrande. The numbers you've reported are much larger than that, so I'll run another regression test today and let you know the results.

Thanks for bringing it up!

[kylesayrs]

Hi @nirey10!

This evaluation was produced by removing the sparsity steps from 2of4_w4a16_group-128_recipe.yaml

vllm (pretrained=/home/kyle/llm-compressor/Llama-3.1-8B-Instruct-W4A16-G128-FROM_RECIPE,dtype=bfloat16,max_model_len=4096,gpu_memory_utilization=0.8,add_bos_token=True,trust_remote_code=True), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: 1
|  Tasks   |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|----------|------:|------|-----:|------|---|-----:|---|-----:|
|winogrande|      1|none  |     5|acc   |↑  |0.7814|±  |0.0116|

This evaluation was produced by running with examples/quantization_w4a16/llama_example.py with llama-3.1

vllm (pretrained=/home/kyle/llm-compressor/Llama-3.1-8B-Instruct-W4A16-G128-STANDARD,dtype=bfloat16,max_model_len=4096,gpu_memory_utilization=0.8,add_bos_token=True,trust_remote_code=True), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: 1
|  Tasks   |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|----------|------:|------|-----:|------|---|-----:|---|-----:|
|winogrande|      1|none  |     5|acc   |↑  |0.7814|±  |0.0116|

Without the fewshot examples, the evaluation matches auto_gptq

vllm (pretrained=/home/kyle/llm-compressor/Llama-3.1-8B-Instruct-W4A16-G128-STANDARD,dtype=bfloat16,max_model_len=4096,gpu_memory_utilization=0.8,add_bos_token=True,trust_remote_code=True), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 1
|  Tasks   |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|----------|------:|------|-----:|------|---|-----:|---|-----:|
|winogrande|      1|none  |     0|acc   |↑  |0.7269|±  |0.0125|

As you can see, they are identical and higher than the initially reported values. Let me know if any of the lm_eval configurations may have played a role.

[nirey10]

Hey, thank you for the detailed answer, i think i found the problem. i should have added 'add_bos_token=True' parameter. In my last issue we found out that the dtype=bfloat16 was missing and you added it to the example :) maybe it is a good thing to add this too for other users.
Thanks again!

[kylesayrs]

@nirey10 Glad to help! The eval parameters can definitely be finnicky, but we provide example lm_eval commands and results on all of our released models on huggingface

https://huggingface.co/neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8