[Tests] Fix GPTQ Tests #1692
[Tests] Fix GPTQ Tests #1692
Conversation
|
👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review. Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @dsikka, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request addresses failing GPTQ tests by updating the test setup to use a more stable and appropriate model, TinyLlama-1.1B-Chat-v1.0, instead of the problematic llama2.c-stories110M. To manage runtime with the larger model, the quantization process is now specifically targeted at a single layer, and corresponding test assertions have been updated to reflect this change, ensuring the tests pass reliably.
Highlights
- Test Model Update: Swapped the test model from
Xenova/llama2.c-stories110MtoTinyLlama/TinyLlama-1.1B-Chat-v1.0to resolve issues with the previous model'slm_headdevice type and improve test stability. - Targeted Quantization: Modified the GPTQ recipe configurations to quantize only a specific layer (
re:.*model.layers.2.self_attn.q_proj$) instead of allLinearlayers. This change was made to contain runtime given the larger size of the new test model. - Assertion Alignment: Updated test assertions to correctly validate the quantization of the newly targeted layer (
model.layers[2].self_attn.q_proj) and to accommodate potential variations in theignorelist forlm_head.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
There was a problem hiding this comment.
Code Review
This pull request fixes failing GPTQ tests by switching from a problematic model to TinyLlama. To manage test runtime, the quantization is now targeted at a single specific layer. The changes correctly update the test recipes and assertions to reflect this new targeted approach. My review includes a suggestion to refactor the repeated hardcoded layer name and index into constants to improve the test's maintainability.
There was a problem hiding this comment.
It'd be really nice if we could maintain our own version of this model, or use the smaller version. This test is less complete now
https://huggingface.co/nm-testing/llama2.c-stories15M
I can make a copy later today
Not really. The point of this test is to check if gptq was applied correctly. I would argue this is a better test as it checks for gptq being applied but using a case which is usually more error-prone. We have plenty of cases where we check for quantization in the entire model. We can maintain a separate model definition but that is separate from the usefulness of this test. |
SUMMARY: - The current tests are failing because when loading the tinystories model, the lm_head is ending up with device type "meta" - This model is generally problematic so we swap to use TinyLlama - With the size of the model being large, we target just one layer for quantization to contain runtime, while updating the asserts to be reflective of the just one layer being quantized TESTING: - All tests pass with these changes Signed-off-by: Derek Kozikowski <[email protected]>
SUMMARY:
TESTING: