Error when benchmarking relaxml/Llama-2-13b-QTIP-4Bit

[HsChen-sys]

I used this command python interactive_gen.py --hf_path relaxml/Llama-2-13b-QTIP-4Bit --max_new_tokens 512

  File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl     [0/1996]
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/qtip/model/llama.py", line 1437, in forward
    outputs = self.model(
              ^^^^^^^^^^^
  File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/qtip/model/llama.py", line 1242, in forward
    layer_outputs = decoder_layer(
                    ^^^^^^^^^^^^^^
  File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/qtip/model/llama.py", line 965, in forward 
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
                                                          ^^^^^^^^^^^^^^^
  File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/qtip/model/llama.py", line 831, in forward 
    query_states = self.q_proj(hidden_states)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/qtip/lib/linear/quantized_linear.py", line 86, in forward
    return self.no_ckpt_forward(input)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/qtip/lib/linear/quantized_linear.py", line 138, in no_ckpt_forward
    result = self.codebook_class(input,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/qtip/lib/codebook/bitshift.py", line 444, in forward
    wrapper = getattr(
              ^^^^^^^^
  File "/venv/main/lib/python3.12/site-packages/torch/_ops.py", line 1267, in __getattr__
    raise AttributeError(
AttributeError: '_OpNamespace' 'quip_lib' object has no attribute 'decompress_matvec_qtip_5120_1_5120_4'

It seems there is some kernel missing.

[tsengalb99]

That appears to be true. If you need to urgently run 13B, add the missing kernels to https://github.com/Cornell-RelaxML/qtip/blob/main/qtip-kernels/src/qtip_torch.cu https://github.com/Cornell-RelaxML/qtip/blob/main/qtip-kernels/src/wrapper.cpp https://github.com/Cornell-RelaxML/qtip/blob/main/lib/codebook/__init__.py and make sure to recompile the kernels. Otherwise, I will probably eventually get around to adding the missing kernels. From: Haisheng Chen ***@***.***> Reply-To: Cornell-RelaxML/qtip ***@***.***> Date: Tuesday, September 2, 2025 at 7:51 PM To: Cornell-RelaxML/qtip ***@***.***> Cc: Subscribed ***@***.***> Subject: [Cornell-RelaxML/qtip] Error when benchmarking relaxml/Llama-2-7b-QTIP-4Bit (Issue #32) [Image removed by sender.]HsChen-sys created an issue (Cornell-RelaxML/qtip#32)<#32> I used this command python interactive_gen.py --hf_path relaxml/Llama-2-13b-QTIP-4Bit --max_new_tokens 512 File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl [0/1996] return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/workspace/qtip/model/llama.py", line 1437, in forward outputs = self.model( ^^^^^^^^^^^ File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/workspace/qtip/model/llama.py", line 1242, in forward layer_outputs = decoder_layer( ^^^^^^^^^^^^^^ File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/workspace/qtip/model/llama.py", line 965, in forward hidden_states, self_attn_weights, present_key_value = self.self_attn( ^^^^^^^^^^^^^^^ File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/workspace/qtip/model/llama.py", line 831, in forward query_states = self.q_proj(hidden_states) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/workspace/qtip/lib/linear/quantized_linear.py", line 86, in forward return self.no_ckpt_forward(input) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/workspace/qtip/lib/linear/quantized_linear.py", line 138, in no_ckpt_forward result = self.codebook_class(input, ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/workspace/qtip/lib/codebook/bitshift.py", line 444, in forward wrapper = getattr( ^^^^^^^^ File "/venv/main/lib/python3.12/site-packages/torch/_ops.py", line 1267, in __getattr__ raise AttributeError( AttributeError: '_OpNamespace' 'quip_lib' object has no attribute 'decompress_matvec_qtip_5120_1_5120_4' It seems there is some kernel missing. — Reply to this email directly, view it on GitHub<#32>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AH6WZSEB6YTMTYU3BZO5SYT3QYUOTAVCNFSM6AAAAACFO2BVXKVHI2DSMVQWIX3LMV43ASLTON2WKOZTGM3TOOBRHA2DMNA>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>