Skip to content

feat: Support the Structural Tag in guided decoding #4066

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 10 commits into from
May 12, 2025
Merged

feat: Support the Structural Tag in guided decoding #4066

merged 10 commits into from
May 12, 2025

Conversation

Ubospica
Copy link
Contributor

@Ubospica Ubospica commented May 5, 2025 •
edited
Loading

This PR supports the structural tag in guided decoding.

It also adds support for the previously unsupported response_format parameter in the OpenAI API protocol, including both JSON format and structural tag format.

Signed-off-by: Ubospica [email protected]

@Ubospica
finish
3a24871 
Signed-off-by: Ubospica <[email protected]>
@syuoni syuoni requested review from LinPoly, kaiyux and syuoni May 6, 2025 07:09
@syuoni syuoni added Community want to contribute PRs initiated from Community Community Engagement help/insights needed from community labels May 6, 2025
LinPoly
LinPoly approved these changes May 6, 2025
View reviewed changes
Copy link
Collaborator

@LinPoly LinPoly left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, since this PR also enables guided decoding in trtllm-serve, can you pls add test for both json and structural_tag in chat test?

LinPoly
LinPoly reviewed May 6, 2025
View reviewed changes
@LinPoly LinPoly changed the title [feat] Support the Structural Tag in guided decoding feat: Support the Structural Tag in guided decoding May 6, 2025
syuoni
syuoni reviewed May 6, 2025
View reviewed changes
Ubospica and others added 6 commits May 8, 2025 03:52
@Ubospica
update
d56370a 
Signed-off-by: Ubospica <[email protected]>
@Ubospica
update
6f803aa 
Signed-off-by: Ubospica <[email protected]>
@syuoni
fix
40c5f78 
Signed-off-by: Enwei Zhu <[email protected]>
@syuoni
exc overlap scheduler
f05e9f1 
Signed-off-by: Enwei Zhu <[email protected]>
@syuoni
add test
0f3672c 
Signed-off-by: Enwei Zhu <[email protected]>
@syuoni
Merge branch 'main' into main-dev/2025-05-01-upgrade-xgrammar
310fda9 
Signed-off-by: Enwei Zhu <[email protected]>
@syuoni
Copy link
Collaborator

syuoni commented May 9, 2025

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #4620 [ run ] triggered by Bot

Ubospica
Ubospica commented May 9, 2025
View reviewed changes
Copy link
Contributor Author

@Ubospica Ubospica left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tests look great. Thanks @syuoni!

@syuoni
fix api ref
ac14ba2 
Signed-off-by: Enwei Zhu <[email protected]>
@syuoni syuoni requested a review from a team as a code owner May 9, 2025 04:14
@syuoni
Copy link
Collaborator

syuoni commented May 9, 2025

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #4639 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #4620 [ run ] completed with state ABORTED
/LLM/main/L0_MergeRequest_PR pipeline #3331 completed with status: 'FAILURE'

@syuoni syuoni requested a review from juney-nvidia May 9, 2025 05:42
juney-nvidia
juney-nvidia approved these changes May 9, 2025
View reviewed changes
Copy link
Collaborator

@juney-nvidia juney-nvidia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
Thanks for the contribution!

@syuoni
Copy link
Collaborator

syuoni commented May 9, 2025

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #4696 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #4639 [ run ] completed with state ABORTED

@tensorrt-cicd
Copy link
Collaborator

PR_Github #4696 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #3386 completed with status: 'FAILURE'

@syuoni
Copy link
Collaborator

syuoni commented May 10, 2025

/bot run --disable-fail-fast

@syuoni
Copy link
Collaborator

syuoni commented May 10, 2025

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #4747 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #4747 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #3426 completed with status: 'FAILURE'

@syuoni
merge main
c1e91d2 
Signed-off-by: Enwei Zhu <[email protected]>
@syuoni syuoni force-pushed the main-dev/2025-05-01-upgrade-xgrammar branch from 1ad65ce to c1e91d2 Compare May 12, 2025 00:39
@syuoni
Copy link
Collaborator

syuoni commented May 12, 2025

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #4794 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #4794 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #3468 completed with status: 'SUCCESS'

@syuoni
Copy link
Collaborator

syuoni commented May 12, 2025

Thanks @Ubospica for the contribution!

@syuoni syuoni merged commit c90ebad into NVIDIA:main May 12, 2025
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@LinPoly LinPoly LinPoly approved these changes

@Funatiq Funatiq Funatiq approved these changes

@syuoni syuoni syuoni approved these changes

@dongxuy04 dongxuy04 dongxuy04 approved these changes

@juney-nvidia juney-nvidia juney-nvidia approved these changes

@kaiyux kaiyux Awaiting requested review from kaiyux

@dcampora dcampora Awaiting requested review from dcampora

@HuiGao-NV HuiGao-NV Awaiting requested review from HuiGao-NV

Assignees

No one assigned

Labels

Community Engagement help/insights needed from community Community want to contribute PRs initiated from Community

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@Ubospica @syuoni @tensorrt-cicd @Funatiq @dongxuy04 @LinPoly @juney-nvidia