Problems with DDP + hydra

[ashleve]

There have been numerous issues about using DDP with hydra:
#231 #289 #229 #226 #194 #352

Current state of things is well described here:
facebookresearch/hydra#2070

tl;dr:
You should be good when using current lightning-hydra-template with ddp_spawn:

# run ddp_spawn on 4 GPUs
python train.py trainer.strategy=ddp_spawn trainer.accelerator=gpu trainer.devices=4

# simulate ddp_spawn on CPU on 4 processes (for testing)
python train.py trainer.strategy=ddp_spawn trainer.accelerator=cpu trainer.devices=4

This works correctly with normal runs as well as multiruns as far as I'm aware.

(ddp_spawn works a bit slower than normal ddp and should be run with datamodule.num_workers=0 only)

Normal ddp computes correctly but generates multiple output directories.

I have not tested what happens when using SLURM.

For now, I don't see anything that can be done on the template part to fix this. This might change with future hydra releases.

Update (April 2023):
Nornal DDP seems to be working correctly with current lightning release (2.0.2). There are no longer multiple output directories.

[bwdeng20]

Thanks for the summary👍🏻. Looking forward to future fixs.

[turian]

@ashleve Can you explain why ddp_spawn is better than ddp? (I skimmed the issues but couldn't grok it.) I understand from the lightning docs that spawn is slower than ddp

[ashleve]

@turian ddp_spawn is not better but it's the only ddp mode that works correctly with hydra right now.

As I mentioned, normal DDP generates multiple unwanted files. This is due to the fact that ddp launches a new process for each GPU, which doesn't go well with the way hydra creates different output dir each time a program is launched.

The problem doesn't exist with ddp_spawn which uses a different strategy of launching new GPU processes.

[turian]

@ashleve Just curious because I am using hydra + DDP in a current project. How would I be able to detect if this issue is occurring for me? What evidence should I look for? Thank you for the tip

[ashleve]

@turian There will be more output directories as explained in facebookresearch/hydra#2070

Just to make this clear, normal ddp actually computes correctly in hydra single run mode, but you will have multiple output directories with .hydra files