question about prompts_text processing

[AGI-player]

Reproduction

in grpo_trainer.py
prompts_text = self.processing_class.batch_decode(
prompt_ids, skip_special_tokens=False, clean_up_tokenization_spaces=False
)

the sample for Qwen models like:
<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
hello, who are you?<|im_end|>
<|im_start|>assistant

due to the padding, after self.processing_class.batch_decode it maybe as follows:
<|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
hello, who are you?<|im_end|>
<|im_start|>assistant

i think it should exclude "<|endoftext|>...<|endoftext|>", otherwise it maybe get very strange response when sending such sample to vllm_server

System Info

transformers 4.51.3
trl 0.20.0.dev0

Checklist

• I have checked that my issue isn't already filed (see open issues)
• I have included my system information
• Any code provided is minimal, complete, and reproducible (more on MREs)
• Any code provided is properly formatted in code blocks, (no screenshot, more on code blocks)
• Any traceback provided is complete

[pramodith]

Thanks for raising this! I'm looking into this now.

[pramodith]

@AGI-player just to follow up, I don't think this should matter since vllm is going to tokenize the input prompt sent to it and those would correctly be mapped to pad_token_id (in this context I think the model uses the same token_id for pad and end_of_text) or do you see any other issue with having the <|endoftext|> token?

[AGI-player]

@AGI-player just to follow up, I don't think this should matter since vllm is going to tokenize the input prompt sent to it and those would correctly be mapped to pad_token_id (in this context I think the model uses the same token_id for pad and end_of_text) or do you see any other issue with having the <|endoftext|> token?

@pramodith many bad samples can be found in the training log, 2 cases as follow:

case1 for english

input:
”“”<|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|>
......(it's too long, i didn't copy it)......
<|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|im_start|>system
You are a function calling AI model. You are provided with function signatures within XML tags. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions.

Tools

You may call one or more functions to assist with the user query.

You are provided with function signatures within XML tags:

{"type": "function", "function": {"name": "create_user", "description": "Create a new user", "parameters": {"type": "object", "properties": {"name": {"type": "string", "description": "The name of the user"}, "email": {"type": "string", "description": "The email address of the user"}, "password": {"type": "string", "description": "The password for the user"}}, "required": ["name", "email", "password"]}}}
{"type": "function", "function": {"name": "create_new_contact", "description": "Create a new contact in the address book", "parameters": {"type": "object", "properties": {"name": {"type": "string", "description": "The name of the contact"}, "phone": {"type": "string", "description": "The phone number of the contact"}, "email": {"type": "string", "description": "The email address of the contact"}}, "required": ["name", "phone"]}}}

For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
<tool_call>
{"name": , "arguments": }
</tool_call><|im_end|>
<|im_start|>user
I want to create a new account.<|im_end|>
<|im_start|>assistant
Sure, I can help with that. Could you please provide me with your name, email address, and a password?<|im_end|>
<|im_start|>user
My name is John Doe, email is johndoe@example.com and password is mysecretpassword.<|im_end|>
<|im_start|>assistant
“”“

response from vllm:
”“”Okay, I to assist with the user query.

The tools
<tool_call>
<tool_call>
<tool_call>
<tool_call>
<tool_call>“”“

case2 for chinese

input:
”“”<|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|>
......(it's too long, i didn't copy it)......
<|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
请以JSON格式给出5位知名探险家的名字并使用幽默诙谐的语气呈现。<|im_end|>
<|im_start|>assistant
“”“

response from vllm:
”“”各康复教师的简历
{}'. def two(long long int num)?
use_task)“”“

the two cases get very strange response. And these probelms can be solved if i don‘t use self.processing_class.batch_decode. It's ok for me, cause all my training data length(encoded with chat template) are less than max_prompt_length.

[pramodith]

hmmm ok that's strange, I'll take a deeper look.

[pramodith]

So, found out what the problem is. When tokenizing text with pad_tokens already present in them the tokenizer doesn't set the attention_mask for those tokens to 0, which is why you see different outputs for what we'd consider to be effectively the same prompt.

tokenizer = AutoProcessor.from_pretrained("Qwen/Qwen3-0.6B")
tokenizer.pad_token
>>> <|endoftext|>

text_with_pad = "<|endoftext|><|endoftext|><|im_start|>system\nYou are a music teacher<|im_end|>\n<|im_start|>user\nDo re mi fa so la<|im_end|>\n<|im_start|>assistant\n'"
text_without_pad = "<|im_start|>system\nYou are a music teacher<|im_end|>\n<|im_start|>user\nDo re mi fa so la<|im_end|>\n<|im_start|>assistant\n'"

 inputs = tokenizer([text_with_pad, text_without_pad], padding=True, padding_side="left")

assert inputs["input_ids"][0] == inputs["input_ids"][1], "Input ids don't match!"
assert inputs["attention_mask"][0] == inputs["attention_mask"][1], "Attention masks don't match!"
>>> AssertionError: Attention masks don't match!

print(f"Attention masks of original text with pad/eos tokens\n {inputs['attention_mask'][0]}")
print(f"Attention masks of original text without pad/eos tokens\n {inputs['attention_mask'][1]}")
>>> Attention masks of original text with pad/eos tokens
 [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
Attention masks of original text without pad/eos tokens
 [0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]

I'll have a PR to fix this by eod Monday.

[AGI-player]

So, found out what the problem is. When tokenizing text with pad_tokens already present in them the tokenizer doesn't set the attention_mask for those tokens to 0, which is why you see different outputs for what we'd consider to be effectively the same prompt.

tokenizer = AutoProcessor.from_pretrained("Qwen/Qwen3-0.6B")
tokenizer.pad_token
>>> <|endoftext|>

text_with_pad = "<|endoftext|><|endoftext|><|im_start|>system\nYou are a music teacher<|im_end|>\n<|im_start|>user\nDo re mi fa so la<|im_end|>\n<|im_start|>assistant\n'"
text_without_pad = "<|im_start|>system\nYou are a music teacher<|im_end|>\n<|im_start|>user\nDo re mi fa so la<|im_end|>\n<|im_start|>assistant\n'"

 inputs = tokenizer([text_with_pad, text_without_pad], padding=True, padding_side="left")

assert inputs["input_ids"][0] == inputs["input_ids"][1], "Input ids don't match!"
assert inputs["attention_mask"][0] == inputs["attention_mask"][1], "Attention masks don't match!"
>>> AssertionError: Attention masks don't match!

print(f"Attention masks of original text with pad/eos tokens\n {inputs['attention_mask'][0]}")
print(f"Attention masks of original text without pad/eos tokens\n {inputs['attention_mask'][1]}")
>>> Attention masks of original text with pad/eos tokens
 [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
Attention masks of original text without pad/eos tokens
 [0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]

I'll have a PR to fix this by eod Monday.

nice~