Cannot run PPO policy for summarization RLHF example

[imraviagrawal]

🐛 Describe the bug

accelerate launch --config_file ./configs/default_accelerate_config.yaml trlx_gptj_text_summarization.py

Exception: Error: Trying to access a method that has not been registered

Using all of the default parameters as provided in the example.

Which trlX version are you using?

0.3.0

Additional system and package information

pytorch=1.10.2+cu113, transformer=4.23.1

[PhungVanDuy]

@imraviagrawal can you try to run in the root folder like: accelerate launch --config_file ./configs/default_accelerate_config.yaml examples/summarize_rlhf/trlx_gptj_text_summarization.py?

[imraviagrawal]

This worked, now I have another issue, using all of the default parameters

[PhungVanDuy]

This worked, now I have another issue, using all of the default parameters

@imraviagrawal can you upgrade trlx with the newest version, I tried to re-run but cannot get this issue.

[agassi4013]

This worked, now I have another issue, using all of the default parameters

@imraviagrawal can you upgrade trlx with the newest version, I tried to re-run but cannot get this issue.

I run this example built yesterday, got same issue

pytorch=1.10.0+cu113, transformer=4.26.0

trained sft and reward model based on gpt-neo-1.3B

[PhungVanDuy]

This worked, now I have another issue, using all of the default parameters

@imraviagrawal can you upgrade trlx with the newest version, I tried to re-run but cannot get this issue.

I run this example built yesterday, got same issue

pytorch=1.10.0+cu113, transformer=4.26.0

trained sft and reward model based on gpt-neo-1.3B

The problem above showed the different between haft and double tensor, can you review your deepspeed config with fp16?

[agassi4013]

this is my ds config:

{
"train_micro_batch_size_per_gpu": "auto",
"fp16": {
"enabled": true,
"min_loss_scale": 0.5,
"fp16_scale_tolerance": 0.25,
"opt_level": "O2"
},
"zero_optimization": {
"stage": "auto",
"offload_optimizer": {
"device": "auto"
},
"offload_param": {
"device": "auto"
},
"allgather_partitions": true,
"allgather_bucket_size": 5e8,
"contiguous_gradients": true
},
"gradient_clipping": "auto",
"train_batch_size": "auto",
"gradient_accumulation_steps": "auto"
}

[PhungVanDuy]

@agassi4013 can you please review your build trlx version and deepspeed configs (default.yml and ds_config.json)? I just try to re-run GPT-NeoX-1.3B, but still not getting that error. My run here: https://wandb.ai/pvduy/trlx/runs/3r0nsa7t

[agassi4013]

@agassi4013 can you please review your build trlx version and deepspeed configs (default.yml and ds_config.json)? I just try to re-run GPT-NeoX-1.3B, but still not getting that error. My run here: https://wandb.ai/pvduy/trlx/runs/3r0nsa7t

This is the default yml:

compute_environment: LOCAL_MACHINE
deepspeed_config:
deepspeed_config_file: /app/trlx/examples/summarize_rlhf/configs/ds_config_trlx_acc.json
zero3_init_flag: false
distributed_type: DEEPSPEED
downcast_bf16: 'no'
dynamo_backend: 'NO'
fsdp_config: {}
machine_rank: 0
main_training_function: main
megatron_lm_config: {}
num_machines: 1
num_processes: 1
rdzv_backend: static
same_network: true
use_cpu: false

ds config is above, deepspeed 0.8.0, trlx 0.3.0 downloaded yesterday, I'm trying to modify fp16 setting and see if it works

[PhungVanDuy]

@agassi4013 can you please review your build trlx version and deepspeed configs (default.yml and ds_config.json)? I just try to re-run GPT-NeoX-1.3B, but still not getting that error. My run here: https://wandb.ai/pvduy/trlx/runs/3r0nsa7t

This is the default yml:

compute_environment: LOCAL_MACHINE deepspeed_config: deepspeed_config_file: /app/trlx/examples/summarize_rlhf/configs/ds_config_trlx_acc.json zero3_init_flag: false distributed_type: DEEPSPEED downcast_bf16: 'no' dynamo_backend: 'NO' fsdp_config: {} machine_rank: 0 main_training_function: main megatron_lm_config: {} num_machines: 1 num_processes: 1 rdzv_backend: static same_network: true use_cpu: false

ds config is above, deepspeed 0.8.0, trlx 0.3.0 downloaded yesterday, I'm trying to modify fp16 setting and see if it works

Did you try to pull lastest code from master and re-install?

[agassi4013]

@agassi4013 can you please review your build trlx version and deepspeed configs (default.yml and ds_config.json)? I just try to re-run GPT-NeoX-1.3B, but still not getting that error. My run here: https://wandb.ai/pvduy/trlx/runs/3r0nsa7t

This is the default yml:
compute_environment: LOCAL_MACHINE deepspeed_config: deepspeed_config_file: /app/trlx/examples/summarize_rlhf/configs/ds_config_trlx_acc.json zero3_init_flag: false distributed_type: DEEPSPEED downcast_bf16: 'no' dynamo_backend: 'NO' fsdp_config: {} machine_rank: 0 main_training_function: main megatron_lm_config: {} num_machines: 1 num_processes: 1 rdzv_backend: static same_network: true use_cpu: false
ds config is above, deepspeed 0.8.0, trlx 0.3.0 downloaded yesterday, I'm trying to modify fp16 setting and see if it works

Did you try to pull lastest code from master and re-install?

I'll try it later

[agassi4013]

I rebuild the environment again, still have same issue, maybe it is caused by pytorch version?

[PhungVanDuy]

I rebuild the environment again, still have same issue, maybe it is caused by pytorch version?

Can you try to run with another problem like sentiment, random walk (use deepspeed)?

[agassi4013]

I rebuild the environment again, still have same issue, maybe it is caused by pytorch version?

Can you try to run with another problem like sentiment, random walk (use deepspeed)?

I tried deepspeed ppo_randomwalks

there's an error:

[agassi4013]

I rebuild the environment again, still have same issue, maybe it is caused by pytorch version?

updated to torch 1.12.1, bug fixed