Add think chunk #21333

juliendenize · 2025-07-21T21:15:54Z

github-actions · 2025-07-21T21:16:01Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

gemini-code-assist

Code Review

This pull request introduces support for a 'think' content chunk, which seems to be used for representing reasoning steps from models like Mistral. The changes involve updating chat utilities to recognize and parse this new content type, and modifying the DeepSeekR1ReasoningParser to handle special tokens from Mistral's tokenizer.

I have raised a high-severity concern regarding the modification of DeepSeekR1ReasoningParser. Adding Mistral-specific logic to a class named and registered for DeepSeek models is a design flaw that will impact maintainability and clarity. I've recommended creating a separate parser for Mistral to keep the components clean and decoupled.

vllm/reasoning/deepseek_r1_reasoning_parser.py

vllm/transformers_utils/tokenizers/mistral.py

vllm/reasoning/deepseek_r1_reasoning_parser.py

vllm/transformers_utils/tokenizers/mistral.py

vllm/reasoning/deepseek_r1_reasoning_parser.py

DarkLight1337 · 2025-07-23T10:42:52Z

tests/reasoning/utils.py

@@ -56,29 +57,44 @@ def run_reasoning_extraction(

 def run_reasoning_extraction_nonstreaming(
    reasoning_parser: ReasoningParser,
-    model_output: list[str],
+    model_output: Union[list[str], list[int]],


I don't really like how we have combined paths when the inputs are actually not valid for all tokenizers. Perhaps @aarnphm has a better idea of how to refactor this. Since @aarnphm is OOO lately, I'm fine with this for now

Added utils just for mistral so that it removes the Union, @aarnphm might have another idea but at least this avoids changing previous code.

tests/reasoning/utils.py

mergify · 2025-07-23T20:56:02Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @juliendenize.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: Julien Denize <[email protected]>

Signed-off-by: Julien Denize <[email protected]> Signed-off-by: 董巍 <[email protected]>

Signed-off-by: Julien Denize <[email protected]> Signed-off-by: avigny <[email protected]>

Signed-off-by: Julien Denize <[email protected]> Signed-off-by: shuw <[email protected]>

Signed-off-by: Julien Denize <[email protected]> Signed-off-by: x22x22 <[email protected]>

Signed-off-by: Julien Denize <[email protected]>

Signed-off-by: Julien Denize <[email protected]> Signed-off-by: Jinzhen Lin <[email protected]>

juliendenize requested a review from aarnphm as a code owner July 21, 2025 21:15

mergify bot added deepseek Related to DeepSeek models frontend labels Jul 21, 2025

gemini-code-assist bot reviewed Jul 21, 2025

View reviewed changes

vllm/reasoning/deepseek_r1_reasoning_parser.py Outdated Show resolved Hide resolved

juliendenize requested review from DarkLight1337, robertgshaw2-redhat and simon-mo as code owners July 22, 2025 10:21

juliendenize force-pushed the add_think_chunk branch from b24960a to d284a34 Compare July 22, 2025 11:30

juliendenize commented Jul 22, 2025

View reviewed changes

vllm/transformers_utils/tokenizers/mistral.py Outdated Show resolved Hide resolved

juliendenize force-pushed the add_think_chunk branch 2 times, most recently from 1ad0d70 to df19b71 Compare July 23, 2025 09:30

patrickvonplaten reviewed Jul 23, 2025

View reviewed changes

vllm/reasoning/deepseek_r1_reasoning_parser.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed Jul 23, 2025

View reviewed changes

vllm/reasoning/deepseek_r1_reasoning_parser.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed Jul 23, 2025

View reviewed changes

vllm/transformers_utils/tokenizers/mistral.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed Jul 23, 2025

View reviewed changes

vllm/transformers_utils/tokenizers/mistral.py Outdated Show resolved Hide resolved

DarkLight1337 reviewed Jul 23, 2025

View reviewed changes

mgoin added the ready ONLY add when PR is ready to merge/full CI is needed label Jul 23, 2025

juliendenize commented Jul 23, 2025

View reviewed changes

tests/reasoning/utils.py Outdated Show resolved Hide resolved

juliendenize force-pushed the add_think_chunk branch from 7150b0c to 035817d Compare July 23, 2025 11:19

mergify bot added the ci/build label Jul 23, 2025

mergify bot added the needs-rebase label Jul 23, 2025

juliendenize and others added 7 commits July 23, 2025 23:01

Add ThinkChunk

8d2a6d4

Signed-off-by: Julien Denize <[email protected]>

clean

f28d316

Signed-off-by: Julien Denize <[email protected]>

Add Mistral reasoning parser

ce2da5f

Signed-off-by: Julien Denize <[email protected]>

clean

5e07129

Signed-off-by: Julien Denize <[email protected]>

clean

d2511d0

Signed-off-by: Julien Denize <[email protected]>

typing

dab21e3

Signed-off-by: Julien Denize <[email protected]>

fix

95ec295

Signed-off-by: Julien Denize <[email protected]>

juliendenize added 6 commits July 23, 2025 23:01

refactor

70cc547

Signed-off-by: Julien Denize <[email protected]>

nit

8a23934

Signed-off-by: Julien Denize <[email protected]>

Fix doc

6ecbb76

Signed-off-by: Julien Denize <[email protected]>

fix

6afdcfc

Signed-off-by: Julien Denize <[email protected]>

refactor

4750762

Signed-off-by: Julien Denize <[email protected]>

update requirements

e15343c

Signed-off-by: Julien Denize <[email protected]>

juliendenize force-pushed the add_think_chunk branch from 3d2af6c to e15343c Compare July 23, 2025 21:03

mergify bot removed the needs-rebase label Jul 23, 2025

simon-mo added this to the v0.10.0 milestone Jul 23, 2025

DarkLight1337 enabled auto-merge (squash) July 24, 2025 03:23

simon-mo disabled auto-merge July 24, 2025 04:51

simon-mo merged commit 6d8d0a2 into vllm-project:main Jul 24, 2025
97 of 98 checks passed

DW934 pushed a commit to DW934/vllm that referenced this pull request Jul 28, 2025

Add think chunk (vllm-project#21333)

9b0d624

Signed-off-by: Julien Denize <[email protected]> Signed-off-by: 董巍 <[email protected]>

avigny pushed a commit to avigny/vllm that referenced this pull request Jul 31, 2025

Add think chunk (vllm-project#21333)

5cefc7f

Signed-off-by: Julien Denize <[email protected]> Signed-off-by: avigny <[email protected]>

wenscarl pushed a commit to wenscarl/vllm that referenced this pull request Aug 4, 2025

Add think chunk (vllm-project#21333)

3fd5153

Signed-off-by: Julien Denize <[email protected]> Signed-off-by: shuw <[email protected]>

x22x22 pushed a commit to x22x22/vllm that referenced this pull request Aug 5, 2025

Add think chunk (vllm-project#21333)

972af6b

Signed-off-by: Julien Denize <[email protected]> Signed-off-by: x22x22 <[email protected]>

Pradyun92 pushed a commit to Pradyun92/vllm that referenced this pull request Aug 6, 2025

Add think chunk (vllm-project#21333)

63ec849

Signed-off-by: Julien Denize <[email protected]>

npanpaliya pushed a commit to odh-on-pz/vllm-upstream that referenced this pull request Aug 6, 2025

Add think chunk (vllm-project#21333)

9ba6f0f

Signed-off-by: Julien Denize <[email protected]>

jinzhen-lin pushed a commit to jinzhen-lin/vllm that referenced this pull request Aug 9, 2025

Add think chunk (vllm-project#21333)

452850c

Signed-off-by: Julien Denize <[email protected]> Signed-off-by: Jinzhen Lin <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Add think chunk #21333

Add think chunk #21333

Uh oh!

juliendenize commented Jul 21, 2025

Uh oh!

github-actions bot commented Jul 21, 2025

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DarkLight1337 Jul 23, 2025 •

edited

Loading

Uh oh!

juliendenize Jul 23, 2025

Uh oh!

Uh oh!

mergify bot commented Jul 23, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Add think chunk #21333

Add think chunk #21333

Uh oh!

Conversation

juliendenize commented Jul 21, 2025

Uh oh!

github-actions bot commented Jul 21, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DarkLight1337 Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

juliendenize Jul 23, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mergify bot commented Jul 23, 2025

Uh oh!

Uh oh!

Uh oh!

DarkLight1337 Jul 23, 2025 •

edited

Loading