[GRPO] Reference model initialization bug fix

[LeonEricsson]

What does this PR do?

The existing issue

The current check for if FSDP is enabled is incorrect

trl/trl/trainer/grpo_trainer.py

Line 412 in 09b669f

elif is_deepspeed_zero3_enabled() or args.fsdp_config is not None:

as the fsdp_config is always post initialized to a dictionary. You can easily test this with

from trl import GRPOConfig, GRPOTrainer, TrlParser


model_name_or_path = "Qwen/Qwen2.5-0.5B-Instruct"
parser = TrlParser(GRPOConfig)
(training_args,) = parser.parse_args_into_dataclasses()

trainer = GRPOTrainer(model=model_name_or_path, reward_funcs=[], args=training_args)

and printing the ref_model

Solution

I propose to move the ref_model initialization till after Trainer has been initialized, leaving the check to Accelerate, and then using self.is_fsdp_enabled. But I am open to alternatives, we could use args.fsdp, or the ACCELERATE_USE_FSDP environment variable.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a GitHub issue? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[LeonEricsson]

@kashif is there something I missed in #3260?

[kashif]

yes that looks good!