[QUESTION]Pipeoffload does not operate normally

[SuperCB]

I tried to run pipeoffload according to what's explained in the readme，but it blocked at

[huanggx-sea]

Hi, thanks for reporting this issue!

To further investigate the problem, could you help provide the following details?

• The command to run the code. If you change the default settings in examples/pretrain_offload.sh, please help to post the modified script.
• The full output like the picture you posted, including full Megatron settings.
• Your environment: number of GPUs and their spec. The output of pip freeze and nvidia-smi.

I would appreciate it if you could use py-spy to print the stack of each rank so that it can help us quickly locate where they are stuck:

py-spy dump --pid <megatron python process id>