You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/features/tool_calling.md
+35-31Lines changed: 35 additions & 31 deletions
Original file line number
Diff line number
Diff line change
@@ -1,10 +1,10 @@
1
1
# Tool Calling
2
2
3
-
vLLM currently supports named function calling, as well as the `auto`, `required` (as of `vllm>=0.8.3`) and `none` options for the `tool_choice` field in the chat completion API.
3
+
vLLM currently supports named function calling, as well as the `auto`, `required` (as of `vllm>=0.8.3`), and `none` options for the `tool_choice` field in the chat completion API.
4
4
5
5
## Quickstart
6
6
7
-
Start the server with tool calling enabled. This example uses Meta's Llama 3.1 8B model, so we need to use the llama3 tool calling chat template from the vLLM examples directory:
7
+
Start the server with tool calling enabled. This example uses Meta's Llama 3.1 8B model, so we need to use the `llama3_json` tool calling chat template from the vLLM examples directory:
Next, make a request to the model that should result in it using the available tools:
16
+
Next, make a request that triggers the model to use the available tools:
17
17
18
18
??? code
19
19
@@ -73,7 +73,7 @@ This example demonstrates:
73
73
74
74
You can also specify a particular function using named function calling by setting `tool_choice={"type": "function", "function": {"name": "get_weather"}}`. Note that this will use the guided decoding backend - so the first time this is used, there will be several seconds of latency (or more) as the FSM is compiled for the first time before it is cached for subsequent requests.
75
75
76
-
Remember that it's the callers responsibility to:
76
+
Remember that it's the caller's responsibility to:
77
77
78
78
1. Define appropriate tools in the request
79
79
2. Include relevant context in the chat messages
@@ -84,7 +84,7 @@ For more advanced usage, including parallel tool calls and different model-speci
84
84
## Named Function Calling
85
85
86
86
vLLM supports named function calling in the chat completion API by default. It does so using Outlines through guided decoding, so this is
87
-
enabled by default, and will work with any supported model. You are guaranteed a validly-parsable function call - not a
87
+
enabled by default and will work with any supported model. You are guaranteed a validly-parsable function call - not a
88
88
high-quality one.
89
89
90
90
vLLM will use guided decoding to ensure the response matches the tool parameter object defined by the JSON schema in the `tools` parameter.
@@ -95,7 +95,7 @@ specify the `name` of one of the tools in the `tool_choice` parameter of the cha
95
95
96
96
## Required Function Calling
97
97
98
-
vLLM supports the `tool_choice='required'` option in the chat completion API. Similar to the named function calling, it also uses guided decoding, so this is enabled by default and will work with any supported model. The required guided decoding features (JSON schema with `anyOf`) are currently only supported in the V0 engine with the guided decoding backend `outlines`. However, support for alternative decoding backends are on the [roadmap](../usage/v1_guide.md#features) for the V1 engine.
98
+
vLLM supports the `tool_choice='required'` option in the chat completion API. Similar to the named function calling, it also uses guided decoding, so this is enabled by default and will work with any supported model. The guided decoding features for `tool_choice='required'` (such as JSON schema with `anyOf`) are currently only supported in the V0 engine with the guided decoding backend `outlines`. However, support for alternative decoding backends are on the [roadmap](../usage/v1_guide.md#features) for the V1 engine.
99
99
100
100
When tool_choice='required' is set, the model is guaranteed to generate one or more tool calls based on the specified tool list in the `tools` parameter. The number of tool calls depends on the user's query. The output format strictly follows the schema defined in the `tools` parameter.
101
101
@@ -109,16 +109,16 @@ However, when `tool_choice='none'` is specified, vLLM includes tool definitions
109
109
110
110
To enable this feature, you should set the following flags:
111
111
112
-
*`--enable-auto-tool-choice` -- **mandatory** Auto tool choice. tells vLLM that you want to enable the model to generate its own tool calls when it
112
+
*`--enable-auto-tool-choice` -- **mandatory** Auto tool choice. It tells vLLM that you want to enable the model to generate its own tool calls when it
113
113
deems appropriate.
114
114
*`--tool-call-parser` -- select the tool parser to use (listed below). Additional tool parsers
115
-
will continue to be added in the future, and also can register your own tool parsers in the `--tool-parser-plugin`.
115
+
will continue to be added in the future. You can also register your own tool parsers in the `--tool-parser-plugin`.
116
116
*`--tool-parser-plugin` -- **optional** tool parser plugin used to register user defined tool parsers into vllm, the registered tool parser name can be specified in `--tool-call-parser`.
117
-
*`--chat-template` -- **optional** for auto tool choice. the path to the chat template which handles `tool`-role messages and `assistant`-role messages
117
+
*`--chat-template` -- **optional** for auto tool choice. It's the path to the chat template which handles `tool`-role messages and `assistant`-role messages
118
118
that contain previously generated tool calls. Hermes, Mistral and Llama models have tool-compatible chat templates in their
119
119
`tokenizer_config.json` files, but you can specify a custom template. This argument can be set to `tool_use` if your model has a tool use-specific chat
120
120
template configured in the `tokenizer_config.json`. In this case, it will be used per the `transformers` specification. More on this [here](https://huggingface.co/docs/transformers/en/chat_templating#why-do-some-models-have-multiple-templates)
121
-
from HuggingFace; and you can find an example of this in a `tokenizer_config.json`[here](https://huggingface.co/NousResearch/Hermes-2-Pro-Llama-3-8B/blob/main/tokenizer_config.json)
121
+
from HuggingFace; and you can find an example of this in a `tokenizer_config.json`[here](https://huggingface.co/NousResearch/Hermes-2-Pro-Llama-3-8B/blob/main/tokenizer_config.json).
122
122
123
123
If your favorite tool-calling model is not supported, please feel free to contribute a parser & tool use chat template!
124
124
@@ -130,7 +130,7 @@ All Nous Research Hermes-series models newer than Hermes 2 Pro should be support
130
130
*`NousResearch/Hermes-2-Theta-*`
131
131
*`NousResearch/Hermes-3-*`
132
132
133
-
_Note that the Hermes 2 **Theta** models are known to have degraded tool call quality & capabilities due to the merge
133
+
_Note that the Hermes 2 **Theta** models are known to have degraded tool call quality and capabilities due to the merge
134
134
step in their creation_.
135
135
136
136
Flags: `--tool-call-parser hermes`
@@ -146,13 +146,13 @@ Known issues:
146
146
147
147
1. Mistral 7B struggles to generate parallel tool calls correctly.
148
148
2. Mistral's `tokenizer_config.json` chat template requires tool call IDs that are exactly 9 digits, which is
149
-
much shorter than what vLLM generates. Since an exception is thrown when this condition
150
-
is not met, the following additional chat templates are provided:
149
+
much shorter than what vLLM generates. Since an exception is thrown when this condition
150
+
is not met, the following additional chat templates are provided:
151
151
152
-
*<gh-file:examples/tool_chat_template_mistral.jinja> - this is the "official" Mistral chat template, but tweaked so that
153
-
it works with vLLM's tool call IDs (provided `tool_call_id` fields are truncated to the last 9 digits)
154
-
*<gh-file:examples/tool_chat_template_mistral_parallel.jinja> - this is a "better" version that adds a tool-use system prompt
155
-
when tools are provided, that results in much better reliability when working with parallel tool calling.
152
+
*<gh-file:examples/tool_chat_template_mistral.jinja> - this is the "official" Mistral chat template, but tweaked so that
153
+
it works with vLLM's tool call IDs (provided `tool_call_id` fields are truncated to the last 9 digits)
154
+
*<gh-file:examples/tool_chat_template_mistral_parallel.jinja> - this is a "better" version that adds a tool-use system prompt
155
+
when tools are provided, that results in much better reliability when working with parallel tool calling.
@@ -166,17 +166,17 @@ All Llama 3.1, 3.2 and 4 models should be supported.
166
166
*`meta-llama/Llama-3.2-*`
167
167
*`meta-llama/Llama-4-*`
168
168
169
-
The tool calling that is supported is the [JSONbased tool calling](https://llama.meta.com/docs/model-cards-and-prompt-formats/llama3_1/#json-based-tool-calling). For [pythonic tool calling](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/text_prompt_format.md#zero-shot-function-calling) introduced by the Llama-3.2 models, see the `pythonic` tool parser below. As for llama 4 models, it is recommended to use the `llama4_pythonic` tool parser.
169
+
The tool calling that is supported is the [JSON-based tool calling](https://llama.meta.com/docs/model-cards-and-prompt-formats/llama3_1/#json-based-tool-calling). For [pythonic tool calling](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/text_prompt_format.md#zero-shot-function-calling) introduced by the Llama-3.2 models, see the `pythonic` tool parser below. As for Llama 4 models, it is recommended to use the `llama4_pythonic` tool parser.
170
170
171
171
Other tool calling formats like the built in python tool calling or custom tool calling are not supported.
172
172
173
173
Known issues:
174
174
175
-
1. Parallel tool calls are not supported for llama 3, but it is supported in llama 4 models.
176
-
2. The model can generate parameters with a wrong format, such as generating
175
+
1. Parallel tool calls are not supported for Llama 3, but it is supported in Llama 4 models.
176
+
2. The model can generate parameters in an incorrect format, such as generating
177
177
an array serialized as string instead of an array.
178
178
179
-
VLLM provides two JSONbased chat templates for Llama 3.1 and 3.2:
179
+
VLLM provides two JSON-based chat templates for Llama 3.1 and 3.2:
180
180
181
181
*<gh-file:examples/tool_chat_template_llama3.1_json.jinja> - this is the "official" chat template for the Llama 3.1
182
182
models, but tweaked so that it works better with vLLM.
VLLM also provides a pythonic and JSON based chat template for Llama 4, but pythonic tool calling is recommended:
188
+
VLLM also provides a pythonic and JSON-based chat template for Llama 4, but pythonic tool calling is recommended:
189
+
189
190
*<gh-file:examples/tool_chat_template_llama4_pythonic.jinja> - this is based on the [official chat template](https://www.llama.com/docs/model-cards-and-prompt-formats/llama4/) for the Llama 4 models.
190
191
191
192
For Llama 4 model, use `--tool-call-parser llama4_pythonic --chat-template examples/tool_chat_template_llama4_pythonic.jinja`.
<gh-file:examples/tool_chat_template_granite.jinja>: this is a modified chat template from the original on Huggingface. Parallel function calls are supported.
202
+
<gh-file:examples/tool_chat_template_granite.jinja>: this is a modified chat template from the original on Hugging Face. Parallel function calls are supported.
202
203
203
204
*`ibm-granite/granite-3.1-8b-instruct`
204
205
205
-
Recommended flags: `--tool-call-parser granite`
206
+
Recommended flags: `--tool-call-parser granite`
206
207
207
-
The chat template from Huggingface can be used directly. Parallel function calls are supported.
208
+
The chat template from Huggingface can be used directly. Parallel function calls are supported.
<gh-file:examples/tool_chat_template_granite_20b_fc.jinja>: this is a modified chat template from the original on Huggingface, which is not vLLMcompatible. It blends function description elements from the Hermes template and follows the same system prompt as "Response Generation" mode from [the paper](https://arxiv.org/abs/2407.00121). Parallel function calls are supported.
214
+
<gh-file:examples/tool_chat_template_granite_20b_fc.jinja>: this is a modified chat template from the original on Hugging Face, which is not vLLM-compatible. It blends function description elements from the Hermes template and follows the same system prompt as "Response Generation" mode from [the paper](https://arxiv.org/abs/2407.00121). Parallel function calls are supported.
214
215
215
216
### InternLM Models (`internlm`)
216
217
@@ -246,10 +247,12 @@ The xLAM tool parser is designed to support models that generate tool calls in v
246
247
Parallel function calls are supported, and the parser can effectively separate text content from tool calls.
Llama's smaller models frequently fail to emit tool calls in the correct format. Your mileage may vary.
332
+
Llama's smaller models frequently fail to emit tool calls in the correct format. Results may vary depending on the model.
329
333
330
-
## How to write a tool parser plugin
334
+
## How to Write a ToolParser Plugin
331
335
332
336
A tool parser plugin is a Python file containing one or more ToolParser implementations. You can write a ToolParser similar to the `Hermes2ProToolParser` in <gh-file:vllm/entrypoints/openai/tool_parsers/hermes_tool_parser.py>.
0 commit comments