Initial README #3
Conversation
README.md
Outdated
|
|
||
| oneshot( | ||
| model="TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T", # sample model |
There was a problem hiding this comment.
I would update this to use TinyLlama/TinyLlama-1.1B-Chat-v1.0
There was a problem hiding this comment.
I think we should instead pass a dataset that the user created here, like this example
The "open_platypus" is a bit too opaque
There was a problem hiding this comment.
agreed - wanted to keep the code minimal in the readme, taking a look at what we can do
| # sets parameters for the GPTQ algorithms - target Linear layer weights at 4 bits | ||
| gptq = GPTQModifier(scheme="W4A16", targets="Linear") | ||
|
|
There was a problem hiding this comment.
Can we initialize the model outside of the one-shot via AutoModelForCausalLM or does this require SparseAutoModelForCausalLM?
There was a problem hiding this comment.
after the HFQuantizer is upstreamed the base auto model will work
| ``` | ||
|
|
||
| ### Inference with vLLM |
There was a problem hiding this comment.
Could we also add example code for inference with transformers? Just to make it clear that both are supported with the caveat that transformers runs in fake quant mode
There was a problem hiding this comment.
great idea, let's add this after we can get the HFQuantizer integration landed since that will affect that pathway
Define BaseModels for Quantization
No description provided.