How can i use the functions attribute provided by OpenAI with open source models

[necromorph98]

While using langchain integration with fastchat, i tried out the function calling API of OpenAI with Vicuna 7B v1.3 but i am getting an AttributeError. How can i get structured output for unstructured input?

[surak]

Hey @necromorph98 , can you give us more information? I would like to see how you are trying things and what exactly are the issues you are experiencing.

[rchen19]

Hey @necromorph98 , can you give us more information? I would like to see how you are trying things and what exactly are the issues you are experiencing.

I believe the OP is referring to the function calling API, see their official article here. It basically allows you to force the model to generate a structured response (i.e., JSON).

I have been looking into the same as the OP, I would very much like to be able to force open source models served using fastchat to return JSON.

However according to OpenAI's API Doc, it seems the model needs to be trained on a set that uses the same syntax of prompts:

Under the hood, functions are injected into the system message in a syntax the model has been trained on.

[yaofeng]

From the code in openai_api_protocol.py:

class ChatCompletionRequest(BaseModel):
    model: str
    messages: Union[str, List[Dict[str, str]]]
    temperature: Optional[float] = 0.7
    top_p: Optional[float] = 1.0
    top_k: Optional[int] = -1
    n: Optional[int] = 1
    max_tokens: Optional[int] = None
    stop: Optional[Union[str, List[str]]] = None
    stream: Optional[bool] = False
    presence_penalty: Optional[float] = 0.0
    frequency_penalty: Optional[float] = 0.0
    user: Optional[str] = None

There's no field "tools", so it dosen't support function call yet.

I hope it can support function call soon.

[rchen19]

From the code in openai_api_protocol.py:

class ChatCompletionRequest(BaseModel):
    model: str
    messages: Union[str, List[Dict[str, str]]]
    temperature: Optional[float] = 0.7
    top_p: Optional[float] = 1.0
    top_k: Optional[int] = -1
    n: Optional[int] = 1
    max_tokens: Optional[int] = None
    stop: Optional[Union[str, List[str]]] = None
    stream: Optional[bool] = False
    presence_penalty: Optional[float] = 0.0
    frequency_penalty: Optional[float] = 0.0
    user: Optional[str] = None

There's no field "tools", so it dosen't support function call yet.

I hope it can support function call soon.

I second this. "Function call" or "tool use" will be very useful, especially with Llama-3.1 tuned for this.

[surak]

There was an attempt here: #2085, perhaps it's of help?