AI Model Optimization Toolkit for the ONNX Runtime

Given a model and targeted hardware, Olive (abbreviation of Onnx LIVE) composes the best suitable optimization techniques to output the most efficient ONNX model(s) for inferencing on the cloud or edge, while taking a set of constraints such as accuracy and latency into consideration.

• Reduce frustration of manual trial-and-error model optimization experimentation. Define your target and precision and let Olive automatically produce the best model for you.
• 40+ built-in model optimization components covering industry-leading techniques across model compression, optimization, finetuning, and compilation.
• Easy-to-use CLI for common model optimization tasks.
• Workflows to orchestrate model transformations and optimizations steps.
• Support for compiling LoRA adapters for MultiLoRA serving.
• Seamless integration with Hugging Face and Azure AI.
• Built-in caching mechanism to improve productivity.

Here are some recent videos, blog articles and labs that highlight Olive:

• [ Feb 2025 ] New Notebook available - Finetune and Optimize DeepSeek R1 with Olive 🐋
• [ Nov 2024 ] Democratizing AI Model optimization with the new Olive CLI
• [ Nov 2024 ] Unlocking NLP Potential: Fine-Tuning with Microsoft Olive (Ignite Pre-Day Lab PRE016)
• [ Nov 2024 ] Olive supports generating models for MultiLoRA serving on the ONNX Runtime
• [ Oct 2024 ] Windows Dev Chat: Optimizing models from Hugging Face for the ONNX Runtime (video)
• [ May 2024 ] AI Toolkit - VS Code Extension that uses Olive to fine tune models

For a full list of news and blogs, read the news archive.

The following notebooks are available that demonstrate key optimization workflows with Olive and include the application code to inference the optimized models on the ONNX Runtime.

If you prefer using the command line directly instead of Jupyter notebooks, we've outlined the quickstart commands here.

We recommend installing Olive in a virtual environment or a conda environment.

pip install olive-ai[auto-opt]
pip install transformers onnxruntime-genai

In this quickstart you'll be optimizing Qwen/Qwen2.5-0.5B-Instruct, which has many model files in the Hugging Face repo for different precisions that are not required by Olive.

Run the automatic optimization:

olive optimize \
    --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \
    --precision int4 \
    --output_path models/qwen

olive optimize `
   --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct `
   --output_path models/qwen `
   --precision int4

The automatic optimizer will:

1. Acquire the model from the the Hugging Face model repo.
2. Quantize the model to int4 using GPTQ.
3. Capture the ONNX Graph and store the weights in an ONNX data file.
4. Optimize the ONNX Graph.

Olive can automatically optimize popular model architectures like Llama, Phi, Qwen, Gemma, etc out-of-the-box - see detailed list here. Also, you can optimize other model architectures by providing details on the input/outputs of the model (io_config).

The ONNX Runtime (ORT) is a fast and light-weight cross-platform inference engine with bindings for popular programming language such as Python, C/C++, C#, Java, JavaScript, etc. ORT enables you to infuse AI models into your applications so that inference is handled on-device.

The sample chat app to run is found as model-chat.py in the onnxruntime-genai Github repository.

• Documentation
• Examples

• We welcome contributions! Please read the contribution guidelines for more details on how to contribute to the Olive project.
• For feature requests or bug reports, file a GitHub Issue.
• For general discussion or questions, use GitHub Discussions.

Copyright (c) Microsoft Corporation. All rights reserved.

Licensed under the MIT License.