The Endpoint of OpenAI model is not allow to change, how should I add 3rd party LLMs?

[freemank1224]

I notice that I can modify some files in the src folder, but this still can't work when I use data_formulator. So can you provide the users the oppertunity to use 3rd party endpoints ?

[Chenglong-MS]

In order to be able to modify source code and reflect it in data formulator, you should follow the development build (instead of using pip install data_formulator).

You can check out the development build here https://github.com/microsoft/data-formulator/blob/main/DEVELOPMENT.md

You'll need to complete both backend build and frontend build, and visit http://localhost:3000/ to see the live version.

Some tips: if you made updates to backend, you'll need to restart the backend server with .\local_server.sh to reflect changes to the custom data formulator you are developing.

[pmccabe157]

Where in the app can i change the endpoint e.g. to use an OLLAMA model?

[ramixpe]

i would love to see the Ollama as well as back end please

[Chenglong-MS]

Here is the place to update endpoint to use other models. https://github.com/microsoft/data-formulator/blob/main/py-src/data_formulator/agents/client_utils.py

There seems to be enough interests? We can update data formulator to support Ollama as part of the new version release.

[82ndAirborneDiv]

I would love to see a way to use other LLM. And possibly change the API URL. And also to use it locally with Ollama.

[Chenglong-MS]

Yeah, that sounds great. I also find Ollama a good addition given that open source models have much better performance nowadays.

Community -- for closed source models, if you all have suggestions on priorities, feel free to make suggestions. We could potentially start with claude models or hugging face keys.

[alikalik9]

Azure OpenAI would be great (and Microsoft related 😊)

[Chittaranjans]

To enable seamless integration with additional LLMs. we propose updating the LLM client initialization in client_utils.py to accept custom endpoints and model parameters using LiteLLM’s unified interface. Additionally, there introduce a user-friendly UI settings panel where users can input the Endpoint URL (e.g., http://localhost:11434 for Ollama), API Key and Model Name . This enhancement empowers users to switch between OpenAI, local models, or third-party providers.

export LLM_ENDPOINT="http://localhost:11434"(ollama)
defult:
export LLM_MODEL="llama2"
export LLM_API_KEY=""

Example LiteLLM configuration:

from litellm import completion 
import os

def get_llm_client():
    custom_endpoint = os.getenv("LLM_ENDPOINT", "https://api.openai.com/v1")  
    api_key = os.getenv("LLM_API_KEY", "")
    model = os.getenv("LLM_MODEL", "gpt-4")  # Default 

    if "ollama" in custom_endpoint.lower():
        model = f"ollama/{model}"  

    return completion(
        model=model,
        api_base=custom_endpoint,
        api_key=api_key,
        custom_llm_provider="ollama" if "ollama" in custom_endpoint else None
    )

[Chenglong-MS]

yes, this sounds like a very good approach. We'll test it out and do a PR soon.

[Chittaranjans]

OKay

[Chenglong-MS]

A work in progress supporting non-openai models is here: https://github.com/microsoft/data-formulator/tree/dev

I have tested with Ollama/OpenAI/AzureOpenAI, would do a bit more testing before doing a PR.

Checkout client support here: https://github.com/microsoft/data-formulator/blob/dev/py-src/data_formulator/agents/client_utils.py
Frontend update can be tracked at: https://github.com/microsoft/data-formulator/blob/dev/src/views/ModelSelectionDialog.tsx