📏 Completion length logging fix + remainder logging fix

[shirinyamani]

What does this PR do?

Fixes #3476

I'm testing using this tiny script on two gpus:

for server:

CUDA_VISIBLE_DEVICES=0 trl vllm-serve --model Qwen/Qwen3-0.6B

trianing script:

from datasets import load_dataset
from trl import GRPOTrainer, GRPOConfig

dataset = load_dataset("trl-lib/tldr", split="train[:1%]")

# Dummy reward function: count the number of unique characters in the completions
def reward_num_unique_chars(completions, **kwargs):
    return [len(set(c)) for c in completions]

training_args = GRPOConfig(
    output_dir="qwen3-mask-completions",
    bf16=True,
    gradient_checkpointing=True,
    logging_steps=10,
    use_vllm=True,
    mask_truncated_completions=True,
)

trainer = GRPOTrainer(
    model="Qwen/Qwen3-0.6B",
    args=training_args,
    reward_funcs=reward_num_unique_chars,
    train_dataset=dataset,
)

trainer.train()

for training:

CUDA_VISIBLE_DEVICES=0 accelerate launch trl/trainer/test.py

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a GitHub issue? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[qgallouedec]

It seems like the PR isn't working, see the CI

[qgallouedec]

I'm taking over this PR so that we can have this fix in the next release.

[qgallouedec]

TLDR: don't use gather_for_metrics (use gather instead) in GRPO

I just found a subtle bug in GRPO (thankfully with no major consequences). Since we’re rewriting the sampler and ensure there’s no remainder (see here), but accelerate doesn’t detect that and assumes a remainder still exists in the dataset. As a result, using gather_for_metrics is unsafe, as it tries to truncate data based on a nonexistent remainder, which can lead to a mismatched shape error, see this CI
It's pretty amazing that it hasn't caused any bugs before.