Quantized Gorilla

[CharlieJCJ]

Resolved #77 , demo displaying local inference with textwebui.

K-quantized gorilla models can be found on Huggingface: Llama-based, MPT-Based, Falcon-Based, gorilla-openfunctions-v0-gguf, gorilla-openfunctions-v1-gguf

A tutorial walkthrough on how to quantize model using llama.cpp with different quantization methods documented in .

Running local inference with Gorilla on a clean interface is simple. Demoed using text-generation-webui, add your desired models, and run inference.

More details in /inference README

Co-authored-by: Pranav Ramesh [email protected]

[ShishirPatil]

Thanks for the PR @CharlieJCJ and @pranramesh!

Did you get a chance to test all the three models? I remember for Falcon and MPT they needed some minor tweaks. So, good to test they work - functionally, and the outputs make logical sense.

Also, while you are at it, do you mind quantizing the openfunctions-v0 and openfunctions-v1as well? You don't have to have the inference scripts ready now.

Rest of it looks good, I'll go ahead and merge.

[CharlieJCJ]

Yep, I'll take care of quantizing openfunctions-v0 and openfunctions-v1, and add their links to README as well in a day. I've tested MPT and Falcon ones, and their outputs are consistent with the original models.

[CharlieJCJ]

openfunctions-v0 and openfunctions-v1 quantized versions generated and uploaded.

K-quantized gorilla-openfunctions-v0 and gorilla-openfunctions-v1 models can be found on Huggingface: gorilla-openfunctions-v0-gguf, gorilla-openfunctions-v1-gguf

/inference README updated.

[ShishirPatil]

LGTM.

One potential follow-up is to benchmark the performance of these models with respect to the full precision ones.