Raise error and suggestion when using custom optimizer with Fairscale or Deepspeed

[allanj]

What does this PR do?

1st Issues: OOM when saving the optimizer, #14542
This issue happens in consolidating the optimizer, we add an argument save_optimizer_state to give an option on whether we want to save it.

2nd issue: Using a custom optimizer has a problem with Fairscale and Deepspeed, #15784

We simply raise an error and warn the user to proceed with different solution.

Fixes # (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

trainer: @sgugger

[sgugger]

I'm not really in favor of adding an argument to not save the optimizer as there is no point checkpointing if the optimizer is not saved. For the fairscale problem of OOM, there is an option that was detailed in #14542 to use force_broadcast_object=True with the newest version of fairscale.

[allanj]

Oh, I just saw that solution. Maybe the second part where we raise the error is enough for this PR? Do you think it is necessary

if (self.sharded_ddp is not None or args.deepspeed) and (self.optimizer is not None or self.lr_scheduler is not None):
  raise RuntimeError(
      "Passing `optimizers` is not allowed if Fairscale or Deepspeed is enabled."
      "You should subclass `Trainer` and override the `create_optimizer_and_scheduler` method."
  )

[sgugger]

This change I agree with :-) If you want to remove the others, we can merge the PR.
Make sure to run make style after so that the quality check passes.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[allanj]

Yeah, I will remove the others and make a check

[sgugger]

There is still one error of quality script, so you'll need to run make style on your branch :-)
Could you also rename the PR so that the title reflects the changes you actually did?

[sgugger]

Thanks! You have to mark the PR as ready for review, as GitHub won't let me merge a draft PR :-)

[allanj]

Thanks. It is ready for review now.