Unable to resume training when using DDP

[bsugerman]

I'm running training on 2 GPUs without any problems as follows:

 python -m torch.distributed.launch --nproc_per_node 2 --master_port 1111  train.py --cfg yolov5s.yaml --weights '' --epochs 3 --batch-size 12 --workers 64 --device 0,1 --data data/coco128.yaml

However, if I have to kill the job (so someone else can use the processors for a bit), I can not restart the training. I've tried

python -m torch.distributed.launch --nproc_per_node 2 --master_port 1111  train.py --cfg yolov5s.yaml --weights '' --epochs 3 --batch-size 12 --workers 64 --device 0,1 --data data/coco128.yaml --resume 

and a number of variants by leaving out different arguments related to multi-processors. The training gets to:

Transferred 370/370 items from ./runs/exp1/weights/last.pt
Using DDP

sits for a few seconds, then issues the second

Using DDP

and then just hangs. On the GPU, 3 processors are started. Two on processor 0, each using 2250Mb, and one on processor 1 using 965Mb.

Any ideas?

[bsugerman]

I should note that resumption works fine when just using 1 gpu.

[glenn-jocher]

@bsugerman I have not tested --resume on multi-gpu, but a problem above I see is that --resume is incapable of modifying original arguments. The only supported use cases are
python train.py --resume or
python train.py --resume path/to/last.pt

You might want to try this with your DDP command.

[ljmiao]

I also meet the problem when I try to resume training with ddp, can you fix the bug? @glenn-jocher

[bsugerman]

I have tried these, and it doesn't work. I have isolated the problem to the line in train.py that converts the model to DDP:

    # DDP mode
    if cuda and rank != -1:
         model = DDP(model, device_ids=[opt.local_rank], output_device=opt.local_rank)

This is where the program seems to hang, for me at least.

@bsugerman I have not tested --resume on multi-gpu, but a problem above I see is that --resume is incapable of modifying original arguments. The only supported use cases are
python train.py --resume or
python train.py --resume path/to/last.pt

You might want to try this with your DDP command.

[tyomj]

A simple workaround, for now, is to comment the lines with opt replacement in train.py:
https://github.com/ultralytics/yolov5/blob/master/train.py#L422-L423
because every time you read this config you get local_rank = 0.
Then run training as follows:
python -m torch.distributed.launch --nproc_per_node 8 train.py --%your opt params% --resume

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[chongkuiqi]

A simple workaround, for now, is to comment the lines with opt replacement in train.py:
https://github.com/ultralytics/yolov5/blob/master/train.py#L422-L423
because every time you read this config you get local_rank = 0.
Then run training as follows:
python -m torch.distributed.launch --nproc_per_node 8 train.py --%your opt params% --resume

Thanks! But sorry i'm a little confused about your method to solve the problem, which line should i comment ? And how to configure "--%your opt params%" ? Thank you!

[tyomj]

@chongkuiqi
Sorry, I should have provided the link to a specific version since the line positions are unstable.
Here are the lines to comment:

yolov5/train.py

Lines 475 to 476 in 685d601

	with open(Path(ckpt).parent.parent / 'opt.yaml') as f:
	opt = argparse.Namespace(**yaml.load(f, Loader=yaml.FullLoader)) # replace

[tyomj]

@ how to configure "--%your opt params%"
Since you will no longer override these params with the cached opt.yaml, just make sure you are using the same params manually.

[chongkuiqi]

@ how to configure "--%your opt params%"
Since you will no longer override these params with the cached opt.yaml, just make sure you are using the same params manually.

Thank you ! It works ! And now i understand what caused the problem. Thanks for your quick reply !

[glenn-jocher]

@chongkuiqi @tyomj hi guys, I'm glad to hear there is a workaround. Is this something that might be codeable into a PR to help future people?

Also might there be a workaround to include opt.yaml? opt.yaml carries all the previous argparser arguments that need to be applied the same way on --resume.