Error when resuming a training

[Pharaun85]

Search before asking

• I have searched the YOLOv5 issues and found no similar bug report.

YOLOv5 Component

Training

Bug

Due to GPU time restrictions, I train until it cuts off and then relaunch the training with the --resume option.
Until yesterday it worked perfectly, but today I get this error whatever the checkpoint with which I relaunch the training. I have tried with some old ones that I know worked, and the error is the same. Has anything changed in the optimizer structure?

Environment

Using torch 1.10.0+cu111 (Tesla K80)

Google Colab

Minimal Reproducible Example

!python train.py --img 1280 --batch 16 --epochs 50 --data /content/drive/MyDrive/OIv6/dataset.yaml --project /content/drive/MyDrive/OIv6/runs/train --weights yolov5s6.pt --hyp hyp.VOC.yaml --optimizer AdamW --device 0

!python train.py --resume /content/drive/MyDrive/OIv6/runs/train/exp/weights/last.pt --device 0

Traceback (most recent call last):
File "train.py", line 667, in
main(opt)
File "train.py", line 562, in main
train(opt.hyp, opt, device, callbacks)
File "train.py", line 191, in train
optimizer.load_state_dict(ckpt['optimizer'])
File "/usr/local/lib/python3.7/dist-packages/torch/optim/optimizer.py", line 146, in load_state_dict
raise ValueError("loaded state dict contains a parameter group "
ValueError: loaded state dict contains a parameter group that doesn't match the size of optimizer's group

Additional

No response

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[github-actions]

👋 Hello @Pharaun85, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset images, training logs, screenshots, and a public link to online W&B logging if available.

For business inquiries or professional support requests please visit https://ultralytics.com or email support@ultralytics.com.

Requirements

Python>=3.7.0 with all requirements.txt installed including PyTorch>=1.7. To get started:

git clone https://github.com/ultralytics/yolov5  # clone
cd yolov5
pip install -r requirements.txt  # install

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Google Colab and Kaggle notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training (train.py), validation (val.py), inference (detect.py) and export (export.py) on macOS, Windows, and Ubuntu every 24 hours and on every commit.

[glenn-jocher]

@Pharaun85 👋 Hello! Thanks for asking about resuming training. YOLOv5 🚀 Learning Rate (LR) schedulers follow predefined LR curves for the fixed number of --epochs defined at training start (default=300), and are designed to fall to a minimum LR on the final epoch for best training results. For this reason you can not modify the number of epochs once training has started.

If your training was interrupted for any reason you may continue where you left off using the --resume argument. If your training fully completed, you can start a new training from any model using the --weights argument. Examples:

Resume Single-GPU

You may not change settings when resuming, and no additional arguments other than --resume should be passed, with an optional path to the checkpoint you'd like to resume from. If no checkpoint is passed the most recently updated last.pt in your yolov5/ directory is automatically found and used:

python train.py --resume  # automatically find latest checkpoint (searches yolov5/ directory)
python train.py --resume path/to/last.pt  # specify resume checkpoint

Resume Multi-GPU

Multi-GPU DDP trainings must be resumed with the same GPUs and DDP command, i.e. assuming 8 GPUs:

python -m torch.distributed.run --nproc_per_node 8 train.py --resume  # resume latest checkpoint
python -m torch.distributed.run --nproc_per_node 8 train.py --resume path/to/last.pt  # specify resume checkpoint

Start from Pretrained

If you would like to start training from a fully trained model, use the --weights argument, not the --resume argument:

python train.py --weights path/to/best.pt  # start from pretrained model

Good luck 🍀 and let us know if you have any other questions!

[Pharaun85]

Thanks for your response @glenn-jocher . For resuming the training im using the code you posted:

python train.py --resume  # automatically find latest checkpoint (searches yolov5/ directory)
python train.py --resume path/to/last.pt  # specify resume checkpoint

The thing is that yesterday worked, and today raises this error:

Traceback (most recent call last):
  File "train.py", line 667, in <module>
    main(opt)
  File "train.py", line 562, in main
    train(opt.hyp, opt, device, callbacks)
  File "train.py", line 191, in train
    optimizer.load_state_dict(ckpt['optimizer'])
  File "/usr/local/lib/python3.7/dist-packages/torch/optim/optimizer.py", line 146, in load_state_dict
    raise ValueError("loaded state dict contains a parameter group "
ValueError: loaded state dict contains a parameter group that doesn't match the size of optimizer's group

Is there any changes in the resume options to load the optimizer?

[glenn-jocher]

@Pharaun85 compatibility for resuming runs is not guaranteed across commits. If you are resuming a run you should always try to do it with the same commit that you started training with. In your case it looks like this PR is the root cause: #7376

[Pharaun85]

Thank you very much for your reply!
The truth is that yes, that seems to have been the problem.
I'll have to get a real graphics card XD