[CORE] [QUANT] Support for GPTQModel's dynamic quantization per module override/control

[Qubitium]

GPTQModel v0.9.10-dev0 main branch has merged dynamic, per layer/module support of different gptq bits, sym, desc_act using a regex style definition. This is a work in process and we are awaiting feedback before release. We are targeting both vllm and sglang compat with the quant so would like to work with vllm to see if what is the best way forward.

Previously a gptq model has a single config that applies to all layers and all modules within nested layers. This change allows pin-point targeting of different gptq quantization config for specific layers and/or specific modules within specific layers for better optimization.

Sample model: https://huggingface.co/ModelCloud/TinyLlama-1.1B-Chat-v1.0-dynamic-GPTQ-2024-8-3

full quant config for sample:

{
  "bits": 4,
  "dynamic": {
    ".*\\.(?:1[0-5])\\..*": {
      "bits": 8
    },
    ".*\\.(?:1[6-9]|20|21)\\..*": {
      "bits": 8,
      "group_size": 64
    }
  },
  "group_size": 128,
  "desc_act": true,
  "static_groups": false,
  "sym": true,
  "lm_head": false,
  "damp_percent": 0.005,
  "damp_auto_increment": 0.0015,
  "true_sequential": true,
  "model_name_or_path": "./test_dynamic_model",
  "model_file_base_name": "model",
  "quant_method": "gptq",
  "checkpoint_format": "gptq",
  "meta": {
    "quantizer": "gptqmodel:0.9.10-dev0"
  }
}

Dynamic config explained:

# sample tinyllama 1.1B model has 22 layers
# default is 4bit, group_size 128
# layer index start at 0

# last 1/2 of the layers 10-21 has 8bit vs 4bit for 0-9
# last 1/4 of the layers 16-21 has 8bit and group_size 64
dynamic = {
  # `.*\.` matches the layers_node prefix
  r".*\.(?:1[0-5])\..*": {"bits": 8,}, # match layer 10-15
  r".*\.(?:1[6-9]|20|21)\..*": {"bits": 8, "group_size": 64,}, # match layer 16-21
}

Same code to quantize using dynamic control: https://github.com/ModelCloud/GPTQModel/blob/main/tests/test_dynamic.py

Design choices:

1. Need a def table to notify quantizer (GPTQModel) and infer engine (vllm) which layers has dynamic (override) quant config.
2. Possible to generate a static all inclusive per layer/module def/table in json but content would not be human friendly as each nested layer with each nested module would need an entry. If a model has 44 layers and each layer has 6-8 modules, we are looking a t a 44x8 lines of json minimum.
3. GPTQModel decided on a design where a simple regex: str key mapped to dict[str, int or bool] for both quantization and model inference/loading. Multiple regex/dynamic pairs can be defined and for matching, the rules are looped and first one that match, is applied.
4. Upload loading, and looping over each layer/module, we check for dynamic (override) match and if matches, override the static quant config files for that layer/module.

Compat Notes:

dynamic config require that the model inference does not remerge the layers with different dyanmic/quant param values. MergedColumnParallel in Llama model in vllm for example merges mlp.gate and mlp.up. Dynamic override works but in this case, because they are fused/merged, these two layers must have exact same quant config values. Can't have one with 4bit and the other with 8bits.

TODO:

1. unit test
2. finalize design of loading so GPTQModel and vllm can agree, how to best pass/share dynamic layer/module quant override config via quantize_config json

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which consists a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of default ones by unblocking the steps in your fast-check build on Buildkite UI.

Once the PR is approved and ready to go, please make sure to run full CI as it is required to merge (or just use auto-merge).

To run full CI, you can do one of these:

• Comment /ready on the PR
• Add ready label to the PR
• Enable auto-merge.

🚀

[mgoin]

Hi @Qubitium thanks for sharing your interesting work!

We have the notion of variable quantization already in vLLM through our compressed-tensors integration. With this we can blend integer and float quantization of weights and/or activations within a single model config in a similar explicit target or regex manner. I recommend digging into the non-uniform support we already have for compressed-tensors and fp8 methods.
It would be interesting to see if your library could export into compressed-tensors format so it would work out-of-the-box in vLLM and Transformers!

Regarding merged layers, I think the performance and complexity cost of needing to support possibly unmerging layers like QKV or GateUp is too high. I want to recommend keeping the quantization level of merged layers the same so we (and several other inference engines) don't run into this issue.

If you are still open to editing your format, I also think dynamic isn't a clear term here since there is already the notion of static or dynamic quantization, which means something else. Also, the quantization isn't changing in any dynamic way. I would recommend using a name like non-uniform quantization, since we are not performing uniform quantization anymore but have settled on a non-uniform scheme.

[Qubitium]

@mgoin Wow, I totally missed this PR. After cursory check of the https://github.com/vllm-project/vllm/pull/6515/files PR, our pr is entirely redundant. The core concept is similar including re matching. The only little advantage of this pr, and very little at this point, is minimal code-change to bootstrap gptq flexible layer/module quant.

I will need to digest the vllm pr/unit tests to test with gptqmodel export. If gptq model can integrate with compressed_config protocol, then there is zero reason for this pr.

Regarding merged layers, I think the performance and complexity cost of needing to support possibly unmerging layers like >QKV or GateUp is too high. I want to recommend keeping the quantization level of merged layers the same so we (and >several other inference engines) don't run into this issue.

Yes, this our finding as well. Merged layers should retain the same scheme.

If you are still open to editing your format, I also think dynamic isn't a clear term here since there is already the notion of >static or dynamic quantization, which means something else. Also, the quantization isn't changing in any dynamic way. I >would recommend using a name like non-uniform quantization, since we are not performing uniform quantization ?>anymore but have settled on a non-uniform scheme.

I want the config to to be compatible to vllm/sglang, and since sglang for the most part re-uses/import vllm model weight/model layers. Do not want another protocol parser so if vllm compressed_config protocol works like I think it does, then this is good base moving forward for gptqmodel as well.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Qubitium.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[Qubitium]

@mgoin Fixed. Turned out we had to add # noqa: to both lines if a logical line was split into two lines.

[Qubitium]

@mgoin Not ready. Found bug during our internal testing. For non-Marlin kernel code path,dynamic is not correctly applied.

[Qubitium]

@mgoin Ready for re-view. Sorry this is taking too long. We want to get it right.

Changes since your last review:

• Dynamic override get/logic moved to gptq_utils.py
• Fixed gptq cuda kernel not applying dynamic correctly: previously only Marlin kernel was fully tested
• Cleaned up related ci tests
• Lint/pre-commit CI passing

[mgoin]

Thanks for fixing the issues, I think this should be good to go if the CI is green with the GPTQ changes.

[mgoin]

Thank you! The failing tests are unrelated so we will merge shortly

[liweiqing1997]

Can this method support MOE models such as DeepSeekv3?

[Qubitium]

@liweiqing1997 There is no limitation to this feature. If you use GPTQModel for quantization, you can switch any module to any bit rate/group_size. It is not limited to specific model.

[liweiqing1997]

@Qubitium

The issue described involves a KeyError when attempting to deploy a quantized DeepSeek-V2-Lite-Chat model using the vllm framework. The error occurs during the weight-loading process, where the key names in the model's named_parameters() dictionary (params_dict) do not match the key names in the quantized weight file. Specifically:

Key Mismatch:

The original key in self.named_parameters() is 'model.layers.21.mlp.experts.w2_qweight'.
The processed key in the quantized weight file is 'model.layers.21.mlp.experts.w2_weight'.

Error Cause:

When loading the weights, the code attempts to access param = params_dict[name], but the name from the weight file does not exist in params_dict, resulting in a KeyError.

How can this issue be resolved?

Error stack trace:

File "/mnt/lwq/lwq/quant/vllm/vllm-main-debd6bb/vllm/executor/mp_distributed_executor.py", line 185, in _run_workers
    driver_worker_output = run_method(self.driver_worker, sent_method,
  File "/mnt/lwq/lwq/quant/vllm/vllm-main-debd6bb/vllm/utils.py", line 2238, in run_method
    return func(*args, **kwargs)
  File "/mnt/lwq/lwq/quant/vllm/vllm-main-debd6bb/vllm/worker/worker.py", line 183, in load_model
    self.model_runner.load_model()
  File "/mnt/lwq/lwq/quant/vllm/vllm-main-debd6bb/vllm/worker/model_runner.py", line 1113, in load_model
    self.model = get_model(vllm_config=self.vllm_config)
  File "/mnt/lwq/lwq/quant/vllm/vllm-main-debd6bb/vllm/model_executor/model_loader/__init__.py", line 14, in get_model
    return loader.load_model(vllm_config=vllm_config)
  File "/mnt/lwq/lwq/quant/vllm/vllm-main-debd6bb/vllm/model_executor/model_loader/loader.py", line 426, in load_model
    loaded_weights = model.load_weights(
  File "/mnt/lwq/lwq/quant/vllm/vllm-main-debd6bb/vllm/model_executor/models/deepseek_v2.py", line 790, in load_weights
    param = params_dict[name]
KeyError: 'model.layers.10.mlp.experts.w2_weight

My dynamic quantization settings are set to the default configuration from the gptqmodel homepage:

python

dynamic = {
# .*\. matches the layers_node prefix
# layer index starts at 0

# positive match: layer 19, gate module 
r"+:.*\.18\..*gate.*": {"bits": 4, "group_size": 32},  

# positive match: layer 20, gate module (prefix defaults to positive if missing)
r".*\.19\..*gate.*": {"bits": 8, "group_size": 64},  

# negative match: skip layer 21, gate module
r"-:.*\.20\..*gate.*": {}, 

# negative match: skip all down modules for all layers
r"-:.*down.*": {},  

}

The config after quantization is:

{
  "_name_or_path": "/mnt/models/deepseek-ai/DeepSeek-V2-Lite-Chat/",
  "architectures": [
    "DeepseekV2ForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "auto_map": {
    "AutoConfig": "configuration_deepseek.DeepseekV2Config",
    "AutoModel": "modeling_deepseek.DeepseekV2Model",
    "AutoModelForCausalLM": "modeling_deepseek.DeepseekV2ForCausalLM"
  },
  "aux_loss_alpha": 0.001,
  "bos_token_id": 100000,
  "eos_token_id": 100001,
  "ep_size": 1,
  "first_k_dense_replace": 1,
  "hidden_act": "silu",
  "hidden_size": 2048,
  "initializer_range": 0.02,
  "intermediate_size": 10944,
  "kv_lora_rank": 512,
  "max_position_embeddings": 163840,
  "model_type": "deepseek_v2",
  "moe_intermediate_size": 1408,
  "moe_layer_freq": 1,
  "n_group": 1,
  "n_routed_experts": 64,
  "n_shared_experts": 2,
  "norm_topk_prob": false,
  "num_attention_heads": 16,
  "num_experts_per_tok": 6,
  "num_hidden_layers": 27,
  "num_key_value_heads": 16,
  "pretraining_tp": 1,
  "q_lora_rank": null,
  "qk_nope_head_dim": 128,
  "qk_rope_head_dim": 64,
  "quantization_config": {
    "bits": 8,
    "checkpoint_format": "gptq",
    "desc_act": false,
    "dynamic": {
      "+:.*\\.18\\..*gate.*": {
        "bits": 4,
        "group_size": 32
      },
      "-:.*\\.20\\..*gate.*": {},
      "-:.*down.*": {},
      ".*\\.19\\..*gate.*": {
        "bits": 8,
        "group_size": 64
      }
    },
    "group_size": 64,
    "lm_head": false,
    "meta": {
      "damp_auto_increment": 0.0025,
      "damp_percent": 0.01,
      "mse": 0.0,
      "quantizer": [
        "gptqmodel:2.0.0-dev"
      ],
      "static_groups": false,
      "true_sequential": true,
      "uri": "https://github.com/modelcloud/gptqmodel"
    },
    "pack_dtype": "int32",
    "quant_method": "gptq",
    "sym": true
  },
  "rms_norm_eps": 1e-06,
  "rope_scaling": {
    "beta_fast": 32,
    "beta_slow": 1,
    "factor": 40,
    "mscale": 0.707,
    "mscale_all_dim": 0.707,
    "original_max_position_embeddings": 4096,
    "type": "yarn"
  },
  "rope_theta": 10000,
  "routed_scaling_factor": 1.0,
  "scoring_func": "softmax",
  "seq_aux": true,
  "tie_word_embeddings": false,
  "topk_group": 1,
  "topk_method": "greedy",
  "torch_dtype": "bfloat16",
  "transformers_version": "4.48.3",
  "use_cache": true,
  "v_head_dim": 128,
  "vocab_size": 102400
}

[Qubitium]

@liweiqing1997 Please open an issue at GPTQmodel.

1. please use latest gptqmodel buid from main
2. first test loading using GPTQModel.load api

I need to isolate issue to gptqmodel first before checking is vllm compat issue

[liweiqing1997]

@Qubitium

GPTQModel.load is ok