Skip to content

Commit 669feed

Browse files
committed
[Docs] Fix bullets and grammars in tool_calling.md
Signed-off-by: windsonsea <[email protected]>
1 parent 2f5c14d commit 669feed

File tree

1 file changed

+35
-31
lines changed

1 file changed

+35
-31
lines changed

docs/features/tool_calling.md

Lines changed: 35 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
11
# Tool Calling
22

3-
vLLM currently supports named function calling, as well as the `auto`, `required` (as of `vllm>=0.8.3`) and `none` options for the `tool_choice` field in the chat completion API.
3+
vLLM currently supports named function calling, as well as the `auto`, `required` (as of `vllm>=0.8.3`), and `none` options for the `tool_choice` field in the chat completion API.
44

55
## Quickstart
66

7-
Start the server with tool calling enabled. This example uses Meta's Llama 3.1 8B model, so we need to use the llama3 tool calling chat template from the vLLM examples directory:
7+
Start the server with tool calling enabled. This example uses Meta's Llama 3.1 8B model, so we need to use the `llama3_json` tool calling chat template from the vLLM examples directory:
88

99
```bash
1010
vllm serve meta-llama/Llama-3.1-8B-Instruct \
@@ -13,7 +13,7 @@ vllm serve meta-llama/Llama-3.1-8B-Instruct \
1313
--chat-template examples/tool_chat_template_llama3.1_json.jinja
1414
```
1515

16-
Next, make a request to the model that should result in it using the available tools:
16+
Next, make a request that triggers the model to use the available tools:
1717

1818
??? code
1919

@@ -73,7 +73,7 @@ This example demonstrates:
7373

7474
You can also specify a particular function using named function calling by setting `tool_choice={"type": "function", "function": {"name": "get_weather"}}`. Note that this will use the guided decoding backend - so the first time this is used, there will be several seconds of latency (or more) as the FSM is compiled for the first time before it is cached for subsequent requests.
7575

76-
Remember that it's the callers responsibility to:
76+
Remember that it's the caller's responsibility to:
7777

7878
1. Define appropriate tools in the request
7979
2. Include relevant context in the chat messages
@@ -84,7 +84,7 @@ For more advanced usage, including parallel tool calls and different model-speci
8484
## Named Function Calling
8585

8686
vLLM supports named function calling in the chat completion API by default. It does so using Outlines through guided decoding, so this is
87-
enabled by default, and will work with any supported model. You are guaranteed a validly-parsable function call - not a
87+
enabled by default and will work with any supported model. You are guaranteed a validly-parsable function call - not a
8888
high-quality one.
8989

9090
vLLM will use guided decoding to ensure the response matches the tool parameter object defined by the JSON schema in the `tools` parameter.
@@ -95,7 +95,7 @@ specify the `name` of one of the tools in the `tool_choice` parameter of the cha
9595

9696
## Required Function Calling
9797

98-
vLLM supports the `tool_choice='required'` option in the chat completion API. Similar to the named function calling, it also uses guided decoding, so this is enabled by default and will work with any supported model. The required guided decoding features (JSON schema with `anyOf`) are currently only supported in the V0 engine with the guided decoding backend `outlines`. However, support for alternative decoding backends are on the [roadmap](../usage/v1_guide.md#features) for the V1 engine.
98+
vLLM supports the `tool_choice='required'` option in the chat completion API. Similar to the named function calling, it also uses guided decoding, so this is enabled by default and will work with any supported model. The guided decoding features for `tool_choice='required'` (such as JSON schema with `anyOf`) are currently only supported in the V0 engine with the guided decoding backend `outlines`. However, support for alternative decoding backends are on the [roadmap](../usage/v1_guide.md#features) for the V1 engine.
9999

100100
When tool_choice='required' is set, the model is guaranteed to generate one or more tool calls based on the specified tool list in the `tools` parameter. The number of tool calls depends on the user's query. The output format strictly follows the schema defined in the `tools` parameter.
101101

@@ -109,16 +109,16 @@ However, when `tool_choice='none'` is specified, vLLM includes tool definitions
109109

110110
To enable this feature, you should set the following flags:
111111

112-
* `--enable-auto-tool-choice` -- **mandatory** Auto tool choice. tells vLLM that you want to enable the model to generate its own tool calls when it
112+
* `--enable-auto-tool-choice` -- **mandatory** Auto tool choice. It tells vLLM that you want to enable the model to generate its own tool calls when it
113113
deems appropriate.
114114
* `--tool-call-parser` -- select the tool parser to use (listed below). Additional tool parsers
115-
will continue to be added in the future, and also can register your own tool parsers in the `--tool-parser-plugin`.
115+
will continue to be added in the future. You can also register your own tool parsers in the `--tool-parser-plugin`.
116116
* `--tool-parser-plugin` -- **optional** tool parser plugin used to register user defined tool parsers into vllm, the registered tool parser name can be specified in `--tool-call-parser`.
117-
* `--chat-template` -- **optional** for auto tool choice. the path to the chat template which handles `tool`-role messages and `assistant`-role messages
117+
* `--chat-template` -- **optional** for auto tool choice. It's the path to the chat template which handles `tool`-role messages and `assistant`-role messages
118118
that contain previously generated tool calls. Hermes, Mistral and Llama models have tool-compatible chat templates in their
119119
`tokenizer_config.json` files, but you can specify a custom template. This argument can be set to `tool_use` if your model has a tool use-specific chat
120120
template configured in the `tokenizer_config.json`. In this case, it will be used per the `transformers` specification. More on this [here](https://huggingface.co/docs/transformers/en/chat_templating#why-do-some-models-have-multiple-templates)
121-
from HuggingFace; and you can find an example of this in a `tokenizer_config.json` [here](https://huggingface.co/NousResearch/Hermes-2-Pro-Llama-3-8B/blob/main/tokenizer_config.json)
121+
from HuggingFace; and you can find an example of this in a `tokenizer_config.json` [here](https://huggingface.co/NousResearch/Hermes-2-Pro-Llama-3-8B/blob/main/tokenizer_config.json).
122122

123123
If your favorite tool-calling model is not supported, please feel free to contribute a parser & tool use chat template!
124124

@@ -130,7 +130,7 @@ All Nous Research Hermes-series models newer than Hermes 2 Pro should be support
130130
* `NousResearch/Hermes-2-Theta-*`
131131
* `NousResearch/Hermes-3-*`
132132

133-
_Note that the Hermes 2 **Theta** models are known to have degraded tool call quality & capabilities due to the merge
133+
_Note that the Hermes 2 **Theta** models are known to have degraded tool call quality and capabilities due to the merge
134134
step in their creation_.
135135

136136
Flags: `--tool-call-parser hermes`
@@ -146,13 +146,13 @@ Known issues:
146146

147147
1. Mistral 7B struggles to generate parallel tool calls correctly.
148148
2. Mistral's `tokenizer_config.json` chat template requires tool call IDs that are exactly 9 digits, which is
149-
much shorter than what vLLM generates. Since an exception is thrown when this condition
150-
is not met, the following additional chat templates are provided:
149+
much shorter than what vLLM generates. Since an exception is thrown when this condition
150+
is not met, the following additional chat templates are provided:
151151

152-
* <gh-file:examples/tool_chat_template_mistral.jinja> - this is the "official" Mistral chat template, but tweaked so that
153-
it works with vLLM's tool call IDs (provided `tool_call_id` fields are truncated to the last 9 digits)
154-
* <gh-file:examples/tool_chat_template_mistral_parallel.jinja> - this is a "better" version that adds a tool-use system prompt
155-
when tools are provided, that results in much better reliability when working with parallel tool calling.
152+
* <gh-file:examples/tool_chat_template_mistral.jinja> - this is the "official" Mistral chat template, but tweaked so that
153+
it works with vLLM's tool call IDs (provided `tool_call_id` fields are truncated to the last 9 digits)
154+
* <gh-file:examples/tool_chat_template_mistral_parallel.jinja> - this is a "better" version that adds a tool-use system prompt
155+
when tools are provided, that results in much better reliability when working with parallel tool calling.
156156

157157
Recommended flags: `--tool-call-parser mistral --chat-template examples/tool_chat_template_mistral_parallel.jinja`
158158

@@ -166,17 +166,17 @@ All Llama 3.1, 3.2 and 4 models should be supported.
166166
* `meta-llama/Llama-3.2-*`
167167
* `meta-llama/Llama-4-*`
168168

169-
The tool calling that is supported is the [JSON based tool calling](https://llama.meta.com/docs/model-cards-and-prompt-formats/llama3_1/#json-based-tool-calling). For [pythonic tool calling](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/text_prompt_format.md#zero-shot-function-calling) introduced by the Llama-3.2 models, see the `pythonic` tool parser below. As for llama 4 models, it is recommended to use the `llama4_pythonic` tool parser.
169+
The tool calling that is supported is the [JSON-based tool calling](https://llama.meta.com/docs/model-cards-and-prompt-formats/llama3_1/#json-based-tool-calling). For [pythonic tool calling](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/text_prompt_format.md#zero-shot-function-calling) introduced by the Llama-3.2 models, see the `pythonic` tool parser below. As for Llama 4 models, it is recommended to use the `llama4_pythonic` tool parser.
170170

171171
Other tool calling formats like the built in python tool calling or custom tool calling are not supported.
172172

173173
Known issues:
174174

175-
1. Parallel tool calls are not supported for llama 3, but it is supported in llama 4 models.
176-
2. The model can generate parameters with a wrong format, such as generating
175+
1. Parallel tool calls are not supported for Llama 3, but it is supported in Llama 4 models.
176+
2. The model can generate parameters in an incorrect format, such as generating
177177
an array serialized as string instead of an array.
178178

179-
VLLM provides two JSON based chat templates for Llama 3.1 and 3.2:
179+
VLLM provides two JSON-based chat templates for Llama 3.1 and 3.2:
180180

181181
* <gh-file:examples/tool_chat_template_llama3.1_json.jinja> - this is the "official" chat template for the Llama 3.1
182182
models, but tweaked so that it works better with vLLM.
@@ -185,7 +185,8 @@ images.
185185

186186
Recommended flags: `--tool-call-parser llama3_json --chat-template {see_above}`
187187

188-
VLLM also provides a pythonic and JSON based chat template for Llama 4, but pythonic tool calling is recommended:
188+
VLLM also provides a pythonic and JSON-based chat template for Llama 4, but pythonic tool calling is recommended:
189+
189190
* <gh-file:examples/tool_chat_template_llama4_pythonic.jinja> - this is based on the [official chat template](https://www.llama.com/docs/model-cards-and-prompt-formats/llama4/) for the Llama 4 models.
190191

191192
For Llama 4 model, use `--tool-call-parser llama4_pythonic --chat-template examples/tool_chat_template_llama4_pythonic.jinja`.
@@ -196,21 +197,21 @@ Supported models:
196197

197198
* `ibm-granite/granite-3.0-8b-instruct`
198199

199-
Recommended flags: `--tool-call-parser granite --chat-template examples/tool_chat_template_granite.jinja`
200+
Recommended flags: `--tool-call-parser granite --chat-template examples/tool_chat_template_granite.jinja`
200201

201-
<gh-file:examples/tool_chat_template_granite.jinja>: this is a modified chat template from the original on Huggingface. Parallel function calls are supported.
202+
<gh-file:examples/tool_chat_template_granite.jinja>: this is a modified chat template from the original on Hugging Face. Parallel function calls are supported.
202203

203204
* `ibm-granite/granite-3.1-8b-instruct`
204205

205-
Recommended flags: `--tool-call-parser granite`
206+
Recommended flags: `--tool-call-parser granite`
206207

207-
The chat template from Huggingface can be used directly. Parallel function calls are supported.
208+
The chat template from Huggingface can be used directly. Parallel function calls are supported.
208209

209210
* `ibm-granite/granite-20b-functioncalling`
210211

211-
Recommended flags: `--tool-call-parser granite-20b-fc --chat-template examples/tool_chat_template_granite_20b_fc.jinja`
212+
Recommended flags: `--tool-call-parser granite-20b-fc --chat-template examples/tool_chat_template_granite_20b_fc.jinja`
212213

213-
<gh-file:examples/tool_chat_template_granite_20b_fc.jinja>: this is a modified chat template from the original on Huggingface, which is not vLLM compatible. It blends function description elements from the Hermes template and follows the same system prompt as "Response Generation" mode from [the paper](https://arxiv.org/abs/2407.00121). Parallel function calls are supported.
214+
<gh-file:examples/tool_chat_template_granite_20b_fc.jinja>: this is a modified chat template from the original on Hugging Face, which is not vLLM-compatible. It blends function description elements from the Hermes template and follows the same system prompt as "Response Generation" mode from [the paper](https://arxiv.org/abs/2407.00121). Parallel function calls are supported.
214215

215216
### InternLM Models (`internlm`)
216217

@@ -246,10 +247,12 @@ The xLAM tool parser is designed to support models that generate tool calls in v
246247
Parallel function calls are supported, and the parser can effectively separate text content from tool calls.
247248

248249
Supported models:
250+
249251
* Salesforce Llama-xLAM models: `Salesforce/Llama-xLAM-2-8B-fc-r`, `Salesforce/Llama-xLAM-2-70B-fc-r`
250252
* Qwen-xLAM models: `Salesforce/xLAM-1B-fc-r`, `Salesforce/xLAM-3B-fc-r`, `Salesforce/Qwen-xLAM-32B-fc-r`
251253

252254
Flags:
255+
253256
* For Llama-based xLAM models: `--tool-call-parser xlam --chat-template examples/tool_chat_template_xlam_llama.jinja`
254257
* For Qwen-based xLAM models: `--tool-call-parser xlam --chat-template examples/tool_chat_template_xlam_qwen.jinja`
255258

@@ -292,9 +295,10 @@ Flags: `--tool-call-parser kimi_k2`
292295

293296
Supported models:
294297

295-
* `tencent/Hunyuan-A13B-Instruct` (chat template already included huggingface model file.)
298+
* `tencent/Hunyuan-A13B-Instruct` (The chat template is already included in the Hugging Face model files.)
296299

297300
Flags:
301+
298302
* For non-reasoning: `--tool-call-parser hunyuan_a13b`
299303
* For reasoning: `--tool-call-parser hunyuan_a13b --reasoning-parser hunyuan_a13b --enable_reasoning`
300304

@@ -325,9 +329,9 @@ Example supported models:
325329
Flags: `--tool-call-parser pythonic --chat-template {see_above}`
326330

327331
!!! warning
328-
Llama's smaller models frequently fail to emit tool calls in the correct format. Your mileage may vary.
332+
Llama's smaller models frequently fail to emit tool calls in the correct format. Results may vary depending on the model.
329333

330-
## How to write a tool parser plugin
334+
## How to Write a ToolParser Plugin
331335

332336
A tool parser plugin is a Python file containing one or more ToolParser implementations. You can write a ToolParser similar to the `Hermes2ProToolParser` in <gh-file:vllm/entrypoints/openai/tool_parsers/hermes_tool_parser.py>.
333337

0 commit comments

Comments
 (0)