GRPO + LoRA Fine-tuning Qwen3-32B Failed on 8x A2 910B3 with VLLM-Ascend 0.9.1

[1103036128]

System Info

----------Python Info----------
Version : 3.11.13
Compiler : GCC 11.4.0
Build : ('main', 'Jul 26 2025 07:27:32')
Arch : ('64bit', '')
------------Pip Info-----------
Version : 25.1.1
Directory : /usr/local/python3.11.13/lib/python3.11/site-packages/pip
vllm : 0.9.1+empty
sglang : not found.
ray : 2.46.0
torch : 2.5.1
----------verl Info-----------
Version : 0.5.0.dev
Directory : /bigdata/sde/Verl/verl/verl
Commit Hash : 0d4541f
----------Platform Info----------
Platform : Linux-5.10.0-60.18.0.50.oe2203.aarch64-aarch64-with-glibc2.35
system : Linux
node : localhost.localdomain
release : 5.10.0-60.18.0.50.oe2203.aarch64
version : #1 SMP Wed Mar 30 02:43:08 UTC 2022
----------Environment----------
CUDA is not available.
----------System Info----------
Failed to execute nvidia-smi command.
CPU Memory : 2010.33 GB
GPU Count : 0
root@localhost:/bigdata/sde/Verl/verl# npu-smi info
+------------------------------------------------------------------------------------------------+
| npu-smi 24.1.rc3 Version: 24.1.rc3 |
+---------------------------+---------------+----------------------------------------------------+
| NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page)|
| Chip | Bus-Id | AICore(%) Memory-Usage(MB) HBM-Usage(MB) |
+===========================+===============+====================================================+
| 0 910B3 | OK | 112.5 37 0 / 0 |
| 0 | 0000:C1:00.0 | 6 0 / 0 33551/ 65536 |
+===========================+===============+====================================================+
| 1 910B3 | OK | 103.6 36 0 / 0 |
| 0 | 0000:C2:00.0 | 6 0 / 0 42641/ 65536 |
+===========================+===============+====================================================+
| 2 910B3 | OK | 103.0 37 0 / 0 |
| 0 | 0000:81:00.0 | 6 0 / 0 42642/ 65536 |
+===========================+===============+====================================================+
| 3 910B3 | OK | 104.0 37 0 / 0 |
| 0 | 0000:82:00.0 | 6 0 / 0 42642/ 65536 |
+===========================+===============+====================================================+
| 4 910B3 | OK | 103.7 40 0 / 0 |
| 0 | 0000:01:00.0 | 6 0 / 0 42641/ 65536 |
+===========================+===============+====================================================+
| 5 910B3 | OK | 107.2 43 0 / 0 |
| 0 | 0000:02:00.0 | 6 0 / 0 42639/ 65536 |
+===========================+===============+====================================================+
| 6 910B3 | OK | 110.2 40 0 / 0 |
| 0 | 0000:41:00.0 | 6 0 / 0 42641/ 65536 |
+===========================+===============+====================================================+
| 7 910B3 | OK | 100.0 41 0 / 0 |
| 0 | 0000:42:00.0 | 6 0 / 0 42641/ 65536 |
+===========================+===============+====================================================+
+---------------------------+---------------+----------------------------------------------------+
| NPU Chip | Process id | Process name | Process memory(MB) |
+===========================+===============+====================================================+
| 0 0 | 108846 | rayWorkerDict | 30213 |
+===========================+===============+====================================================+
| 1 0 | 108847 | rayWorkerDict | 39299 |
+===========================+===============+====================================================+
| 2 0 | 108848 | rayWorkerDict | 39299 |
+===========================+===============+====================================================+
| 3 0 | 108849 | rayWorkerDict | 39299 |
+===========================+===============+====================================================+
| 4 0 | 108850 | rayWorkerDict | 39299 |
+===========================+===============+====================================================+
| 5 0 | 108851 | rayWorkerDict | 39299 |
+===========================+===============+====================================================+
| 6 0 | 108852 | rayWorkerDict | 39299 |
+===========================+===============+====================================================+
| 7 0 | 108853 | rayWorkerDict | 39299 |
+===========================+===============+====================================================+

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Environment:
Hardware: 1x Atlas 800 (8x A2 910B3, 64GB VRAM each)
VLLM-Ascend: 0.9.1
VerL: main branch
Model: Qwen3-32B
Training Method: GRPO + LoRA
CANN: 8.2.0.0.201 (RC1)
Driver: 24.1.rc3
Name: torch-npu Version: 2.5.1.post1
Name: torch Version: 2.5.1

Log:

+ export WANDB_MODE=disabled
+ export HYDRA_FULL_ERROR=1
+ project_name=GRPO-Qwen3
+ exp_name=GRPO-Qwen3-32B-npu-lora
+ gen_tp=8
+ RAY_DATA_HOME=/root/verl
+ MODEL_PATH=/bigdata/sdb/Qwen3-32B
+ CKPTS_DIR=/bigdata/sde/Verl/verl/ckpts/GRPO-Qwen3/GRPO-Qwen3-32B-npu-lora
+ TRAIN_FILE=/bigdata/sde/Verl/dataset/gsm8k/train.parquet
+ TEST_FILE=/bigdata/sde/Verl/dataset/gsm8k/test.parquet
+ python3 -m verl.trainer.main_ppo algorithm.adv_estimator=grpo data.train_files=/bigdata/sde/Verl/dataset/gsm8k/train.parquet data.val_files=/bigdata/sde/Verl/dataset/gsm8k/test.parquet data.train_batch_size=8 data.max_prompt_length=2048 data.max_response_length=512 data.filter_overlong_prompts=False data.truncation=error data.shuffle=False actor_rollout_ref.model.path=/bigdata/sdb/Qwen3-32B actor_rollout_ref.model.use_shm=True actor_rollout_ref.model.lora_rank=8 actor_rollout_ref.model.lora_alpha=16 actor_rollout_ref.model.target_modules=all-linear actor_rollout_ref.actor.optim.lr=1e-6 actor_rollout_ref.model.use_remove_padding=True actor_rollout_ref.actor.ppo_mini_batch_size=8 actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=2 actor_rollout_ref.actor.use_kl_loss=True actor_rollout_ref.actor.kl_loss_coef=0.001 actor_rollout_ref.actor.kl_loss_type=low_var_kl actor_rollout_ref.actor.entropy_coeff=0 actor_rollout_ref.rollout.max_num_seqs=8 actor_rollout_ref.rollout.max_model_len=2560 actor_rollout_ref.rollout.max_num_batched_tokens=2560 actor_rollout_ref.model.enable_gradient_checkpointing=True actor_rollout_ref.actor.fsdp_config.param_offload=False actor_rollout_ref.actor.fsdp_config.optimizer_offload=False actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=2 actor_rollout_ref.rollout.tensor_model_parallel_size=8 actor_rollout_ref.rollout.name=vllm actor_rollout_ref.rollout.gpu_memory_utilization=0.2 actor_rollout_ref.rollout.n=2 actor_rollout_ref.rollout.load_format=safetensors actor_rollout_ref.rollout.layered_summon=True actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=2 actor_rollout_ref.ref.fsdp_config.param_offload=True actor_rollout_ref.ref.fsdp_config.optimizer_offload=True algorithm.use_kl_in_reward=False trainer.critic_warmup=0 'trainer.logger=["console","wandb"]' trainer.project_name=GRPO-Qwen3 trainer.experiment_name=GRPO-Qwen3-32B-npu-lora trainer.n_gpus_per_node=8 trainer.nnodes=1 trainer.default_local_dir=/bigdata/sde/Verl/verl/ckpts/GRPO-Qwen3/GRPO-Qwen3-32B-npu-lora trainer.device=npu trainer.resume_mode=auto actor_rollout_ref.actor.fsdp_config.forward_prefetch=True actor_rollout_ref.ref.fsdp_config.forward_prefetch=True ++actor_rollout_ref.actor.entropy_from_logits_with_chunking=True ++actor_rollout_ref.ref.entropy_from_logits_with_chunking=True trainer.val_before_train=True trainer.save_freq=20 trainer.test_freq=5 trainer.total_epochs=2
ray init kwargs: {'num_cpus': None, 'runtime_env': {'env_vars': {'TOKENIZERS_PARALLELISM': 'true', 'NCCL_DEBUG': 'WARN', 'VLLM_LOGGING_LEVEL': 'WARN', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': 'true', 'CUDA_DEVICE_MAX_CONNECTIONS': '1', 'NCCL_CUMEM_ENABLE': '0'}, 'working_dir': None}}
2025-09-22 10:49:13,042	INFO worker.py:1879 -- Started a local Ray instance. View the dashboard at http://127.0.0.1:8265 
(TaskRunner pid=91866) TaskRunner hostname: localhost.localdomain, PID: 91866
(TaskRunner pid=91866) {'actor_rollout_ref': {'actor': {'_target_': 'verl.workers.config.FSDPActorConfig',
(.............
(TaskRunner pid=91866) WARNING 09-22 10:49:26 [env_override.py:17] NCCL_CUMEM_ENABLE is set to 0, skipping override. This may increase memory overhead with cudagraph+allreduce: https://github.com/NVIDIA/nccl/issues/1234
(TaskRunner pid=91866) WARNING 09-22 10:49:26 [importing.py:29] Triton is not installed. Using dummy decorators. Install it via `pip install triton` to enable kernel compilation.
(TaskRunner pid=91866) WARNING 09-22 10:49:29 [_custom_ops.py:22] Failed to import from vllm._C with ModuleNotFoundError("No module named 'vllm._C'")
(TaskRunner pid=91866) /bigdata/sde/Verl/verl/verl/trainer/main_ppo.py:264: UserWarning: Disabled critic as algorithm.adv_estimator != gae. If it is not intended, please set critic.enable=True
(TaskRunner pid=91866)   use_critic=need_critic(config),
(TaskRunner pid=91866) [validate_config] All configuration checks passed successfully!
(TaskRunner pid=91866) [WARNING]: The memory model path /dev/shm/verl-cache/ab35dd8c922b74d470b9a5d969f8a17e/Qwen3-32B already exists. If it is not you want, please clear it and restart the task.
(TaskRunner pid=91866) /bigdata/sde/Verl/verl/verl/utils/profiler/config.py:49: UserWarning: Torch profiler tool config is not fully supported now.
(TaskRunner pid=91866)   warnings.warn("Torch profiler tool config is not fully supported now.", stacklevel=1)
(TaskRunner pid=91866) Using dataset class: RLHFDataset
(TaskRunner pid=91866) dataset len: 7473
(TaskRunner pid=91866) Using dataset class: RLHFDataset
(TaskRunner pid=91866) dataset len: 1319
(TaskRunner pid=91866) Size of train dataloader: 934, Size of val dataloader: 1
(TaskRunner pid=91866) Total training steps: 1868
(TaskRunner pid=91866) colocated worker base class <class 'verl.single_controller.base.worker.Worker'>
(TaskRunner pid=91866) /bigdata/sde/Verl/verl/verl/trainer/ppo/ray_trainer.py:324: UserWarning: Disabled critic as algorithm.adv_estimator != gae. If it is not intended, please set critic.enable=True
(TaskRunner pid=91866)   self.use_critic = need_critic(self.config)
(pid=108846) WARNING 09-22 10:50:31 [env_override.py:17] NCCL_CUMEM_ENABLE is set to 0, skipping override. This may increase memory overhead with cudagraph+allreduce: https://github.com/NVIDIA/nccl/issues/1234
(pid=108846) WARNING 09-22 10:50:31 [importing.py:29] Triton is not installed. Using dummy decorators. Install it via `pip install triton` to enable kernel compilation.
(pid=108850) WARNING 09-22 10:50:35 [_custom_ops.py:22] Failed to import from vllm._C with ModuleNotFoundError("No module named 'vllm._C'")
(WorkerDict pid=108846) [W922 10:50:38.015328500 compiler_depend.ts:989] Warning: The watchdog timeout 600000ms(which is set by init_process_group) is less than or equal to HCCL execution timeout 1836000ms! The plog may not be recorded. (function ProcessGroupHCCL)
(WorkerDict pid=108847) [WARNING]: The memory model path /dev/shm/verl-cache/ab35dd8c922b74d470b9a5d969f8a17e/Qwen3-32B already exists. If it is not you want, please clear it and restart the task.
(pid=108853) WARNING 09-22 10:50:32 [env_override.py:17] NCCL_CUMEM_ENABLE is set to 0, skipping override. This may increase memory overhead with cudagraph+allreduce: https://github.com/NVIDIA/nccl/issues/1234 [repeated 7x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)
(pid=108853) WARNING 09-22 10:50:32 [importing.py:29] Triton is not installed. Using dummy decorators. Install it via `pip install triton` to enable kernel compilation. [repeated 7x across cluster]
Loading checkpoint shards:   0%|          | 0/17 [00:00<?, ?it/s]
Loading checkpoint shards:   6%|?         | 1/17 [00:03<01:00,  3.75s/it]
(WorkerDict pid=108846) Model config after override: Qwen3Config {
(WorkerDict pid=108846)   "architectures": [
(WorkerDict pid=108846)     "Qwen3ForCausalLM"
(WorkerDict pid=108846)   ],
(WorkerDict pid=108846)   "attention_bias": false,
(WorkerDict pid=108846)   "attention_dropout": 0.0,
(WorkerDict pid=108846)   "eos_token_id": 151645,
(WorkerDict pid=108846)   "head_dim": 128,
(WorkerDict pid=108846)   "hidden_act": "silu",
(WorkerDict pid=108846)   "hidden_size": 5120,
(WorkerDict pid=108846)   "initializer_range": 0.02,
(WorkerDict pid=108846)   "intermediate_size": 25600,
(WorkerDict pid=108846)   "max_position_embeddings": 40960,
(WorkerDict pid=108846)   "max_window_layers": 64,
(WorkerDict pid=108846)   "model_type": "qwen3",
(WorkerDict pid=108846)   "num_attention_heads": 64,
(WorkerDict pid=108846)   "num_hidden_layers": 64,
(WorkerDict pid=108846)   "num_key_value_heads": 8,
(WorkerDict pid=108846)   "pad_token_id": 151643,
(WorkerDict pid=108846)   "rms_norm_eps": 1e-06,
(WorkerDict pid=108846)   "rope_scaling": null,
(WorkerDict pid=108846)   "rope_theta": 1000000,
(WorkerDict pid=108846)   "sliding_window": null,
(WorkerDict pid=108846)   "tie_word_embeddings": false,
(WorkerDict pid=108846)   "torch_dtype": "bfloat16",
(WorkerDict pid=108846)   "transformers_version": "4.52.4",
(WorkerDict pid=108846)   "use_cache": true,
(WorkerDict pid=108846)   "use_sliding_window": false,
(WorkerDict pid=108846)   "vocab_size": 151936
(WorkerDict pid=108846) }
(WorkerDict pid=108846) 
(pid=108853) WARNING 09-22 10:50:36 [_custom_ops.py:22] Failed to import from vllm._C with ModuleNotFoundError("No module named 'vllm._C'") [repeated 7x across cluster]
(WorkerDict pid=108853) [WARNING]: The memory model path /dev/shm/verl-cache/ab35dd8c922b74d470b9a5d969f8a17e/Qwen3-32B already exists. If it is not you want, please clear it and restart the task. [repeated 7x across cluster]
Loading checkpoint shards:   0%|          | 0/17 [00:00<?, ?it/s] [repeated 7x across cluster]
Loading checkpoint shards:   6%|?         | 1/17 [00:03<00:57,  3.59s/it] [repeated 14x across cluster]
Loading checkpoint shards:  24%|???       | 4/17 [00:15<00:50,  3.89s/it] [repeated 12x across cluster]
Loading checkpoint shards:  29%|???       | 5/17 [00:20<00:48,  4.05s/it] [repeated 12x across cluster]
Loading checkpoint shards:  41%|????      | 7/17 [00:26<00:36,  3.63s/it] [repeated 12x across cluster]
Loading checkpoint shards:  53%|??????    | 9/17 [00:31<00:27,  3.45s/it] [repeated 12x across cluster]
Loading checkpoint shards:  59%|??????    | 10/17 [00:37<00:25,  3.70s/it] [repeated 12x across cluster]
Loading checkpoint shards:  71%|???????   | 12/17 [00:42<00:17,  3.52s/it] [repeated 12x across cluster]
Loading checkpoint shards:  76%|????????  | 13/17 [00:47<00:14,  3.58s/it] [repeated 12x across cluster]
Loading checkpoint shards:  88%|????????? | 15/17 [00:53<00:06,  3.45s/it] [repeated 13x across cluster]
Loading checkpoint shards:  94%|??????????| 16/17 [00:56<00:03,  3.39s/it]
Loading checkpoint shards:  88%|????????? | 15/17 [00:52<00:06,  3.37s/it] [repeated 8x across cluster]
(WorkerDict pid=108847) Monkey patch state_dict in AutoModelForCausalLMWithValueHead. 
(WorkerDict pid=108847) Monkey patch _flash_attention_forward in transformers.integrations.flash_attention
(WorkerDict pid=108847) Skipping monkey patch for Qwen3ForCausalLM as use_fused_kernels is False or fused_kernels_backend is torch
Loading checkpoint shards: 100%|??????????| 17/17 [00:59<00:00,  3.49s/it]
(WorkerDict pid=108847) Applying LoRA to actor module
Loading checkpoint shards:  94%|??????????| 16/17 [01:00<00:03,  3.58s/it] [repeated 6x across cluster]
Loading checkpoint shards: 100%|??????????| 17/17 [00:58<00:00,  3.42s/it] [repeated 7x across cluster]
(WorkerDict pid=108846) Monkey patch state_dict in AutoModelForCausalLMWithValueHead.  [repeated 7x across cluster]
(WorkerDict pid=108846) Monkey patch _flash_attention_forward in transformers.integrations.flash_attention [repeated 7x across cluster]
(WorkerDict pid=108846) Skipping monkey patch for Qwen3ForCausalLM as use_fused_kernels is False or fused_kernels_backend is torch [repeated 7x across cluster]
(WorkerDict pid=108853) Applying LoRA to actor module [repeated 6x across cluster]
(WorkerDict pid=108846) PeftModelForCausalLM contains 32.83B parameters
(WorkerDict pid=108846) wrap_policy: functools.partial(<function _or_policy at 0xffd01af7af20>, policies=[functools.partial(<function lambda_auto_wrap_policy at 0xffd01af7b6a0>, lambda_fn=<function get_fsdp_wrap_policy.<locals>.lambda_policy_fn at 0xffcf50133600>), functools.partial(<function transformer_auto_wrap_policy at 0xffd01af7b060>, transformer_layer_cls={<class 'transformers.models.qwen3.modeling_qwen3.Qwen3DecoderLayer'>})])
(WorkerDict pid=108847) /bigdata/sde/Verl/verl/verl/utils/profiler/config.py:49: UserWarning: Torch profiler tool config is not fully supported now.
(WorkerDict pid=108847)   warnings.warn("Torch profiler tool config is not fully supported now.", stacklevel=1)
Loading checkpoint shards:  94%|??????????| 16/17 [00:55<00:03,  3.33s/it]
(WorkerDict pid=108846) Total steps: 1868, num_warmup_steps: 0
(WorkerDict pid=108846) Actor use_remove_padding=True
(WorkerDict pid=108846) Actor use_fused_kernels=False
(WorkerDict pid=108846) Applying LoRA to actor module
(WorkerDict pid=108848) [WARNING]: The memory model path /dev/shm/verl-cache/ab35dd8c922b74d470b9a5d969f8a17e/Qwen3-32B already exists. If it is not you want, please clear it and restart the task.
(WorkerDict pid=108848) [WARNING]: The memory model path /dev/shm/verl-cache/ab35dd8c922b74d470b9a5d969f8a17e/Qwen3-32B already exists. If it is not you want, please clear it and restart the task.
(WorkerDict pid=108847) WARNING 09-22 10:52:38 [registry.py:401] Model architecture DeepSeekMTPModel is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_mtp:CustomDeepSeekMTP.
(WorkerDict pid=108847) WARNING 09-22 10:52:38 [registry.py:401] Model architecture Qwen2ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2:CustomQwen2ForCausalLM.
(WorkerDict pid=108847) WARNING 09-22 10:52:38 [registry.py:401] Model architecture Qwen2VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2_vl:AscendQwen2VLForConditionalGeneration.
(WorkerDict pid=108847) WARNING 09-22 10:52:38 [registry.py:401] Model architecture Qwen2_5_VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2_5_vl:AscendQwen2_5_VLForConditionalGeneration.
(WorkerDict pid=108847) WARNING 09-22 10:52:38 [registry.py:401] Model architecture DeepseekV2ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_v2:CustomDeepseekV2ForCausalLM.
(WorkerDict pid=108847) WARNING 09-22 10:52:38 [registry.py:401] Model architecture DeepseekV3ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_v2:CustomDeepseekV3ForCausalLM.
(WorkerDict pid=108847) WARNING 09-22 10:52:38 [registry.py:401] Model architecture Qwen3MoeForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen3_moe:CustomQwen3MoeForCausalLM.
(WorkerDict pid=108847) WARNING 09-22 10:52:38 [registry.py:401] Model architecture Qwen3ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen3:CustomQwen3ForCausalLM.
(WorkerDict pid=108848) WARNING 09-22 10:52:54 [utils.py:2737] Methods add_prompt_adapter,cache_config,compilation_config,current_platform,list_prompt_adapters,load_config,pin_prompt_adapter,remove_prompt_adapter not implemented in <vllm_ascend.worker.worker.NPUWorker object at 0xffcf24471910>
(WorkerDict pid=108846) [WARNING]: The memory model path /dev/shm/verl-cache/ab35dd8c922b74d470b9a5d969f8a17e/Qwen3-32B already exists. If it is not you want, please clear it and restart the task. [repeated 22x across cluster]
(WorkerDict pid=108849) WARNING 09-22 10:52:39 [registry.py:401] Model architecture DeepSeekMTPModel is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_mtp:CustomDeepSeekMTP. [repeated 7x across cluster]
(WorkerDict pid=108849) WARNING 09-22 10:52:39 [registry.py:401] Model architecture Qwen3ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen3:CustomQwen3ForCausalLM. [repeated 49x across cluster]
Loading safetensors checkpoint shards:   0% Completed | 0/17 [00:00<?, ?it/s]
(WorkerDict pid=108846) /bigdata/sde/Verl/verl/verl/utils/profiler/config.py:49: UserWarning: Torch profiler tool config is not fully supported now. [repeated 7x across cluster]
(WorkerDict pid=108846)   warnings.warn("Torch profiler tool config is not fully supported now.", stacklevel=1) [repeated 7x across cluster]
Loading safetensors checkpoint shards:   6% Completed | 1/17 [00:00<00:02,  6.10it/s]
Loading safetensors checkpoint shards:  12% Completed | 2/17 [00:00<00:04,  3.60it/s]
Loading safetensors checkpoint shards:  18% Completed | 3/17 [00:00<00:04,  3.19it/s]
Loading safetensors checkpoint shards:  24% Completed | 4/17 [00:01<00:03,  3.67it/s]
Loading safetensors checkpoint shards:  29% Completed | 5/17 [00:01<00:02,  4.41it/s]
Loading safetensors checkpoint shards:  35% Completed | 6/17 [00:01<00:02,  4.41it/s]
Loading safetensors checkpoint shards:  41% Completed | 7/17 [00:01<00:02,  4.45it/s]
Loading safetensors checkpoint shards:  47% Completed | 8/17 [00:01<00:02,  4.43it/s]
Loading safetensors checkpoint shards:  53% Completed | 9/17 [00:02<00:01,  4.58it/s]
Loading safetensors checkpoint shards:  59% Completed | 10/17 [00:02<00:01,  4.68it/s]
Loading safetensors checkpoint shards:  65% Completed | 11/17 [00:02<00:01,  4.76it/s]
Loading safetensors checkpoint shards:  71% Completed | 12/17 [00:02<00:01,  4.78it/s]
Loading safetensors checkpoint shards:  76% Completed | 13/17 [00:02<00:00,  4.82it/s]
Loading safetensors checkpoint shards:  82% Completed | 14/17 [00:03<00:00,  4.89it/s]
Loading safetensors checkpoint shards:  88% Completed | 15/17 [00:03<00:00,  4.99it/s]
Loading safetensors checkpoint shards:  94% Completed | 16/17 [00:03<00:00,  4.84it/s]
Loading safetensors checkpoint shards: 100% Completed | 17/17 [00:03<00:00,  4.51it/s]
Loading safetensors checkpoint shards: 100% Completed | 17/17 [00:03<00:00,  4.48it/s]
(WorkerDict pid=108846) 
(WorkerDict pid=108853) /usr/local/python3.11.13/lib/python3.11/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
(WorkerDict pid=108853)   warnings.warn(
(WorkerDict pid=108853) kwargs: {'n': 1, 'logprobs': 0, 'max_tokens': 512, 'repetition_penalty': 1.0, 'detokenize': False, 'temperature': 1.0, 'top_k': -1, 'top_p': 1, 'ignore_eos': False}
(WorkerDict pid=108850) WARNING 09-22 10:52:55 [utils.py:2737] Methods add_prompt_adapter,cache_config,compilation_config,current_platform,list_prompt_adapters,load_config,pin_prompt_adapter,remove_prompt_adapter not implemented in <vllm_ascend.worker.worker.NPUWorker object at 0xffcf0079d110> [repeated 7x across cluster]
(TaskRunner pid=91866) Checkpoint tracker file does not exist: /bigdata/sde/Verl/verl/ckpts/GRPO-Qwen3/GRPO-Qwen3-32B-npu-lora/latest_checkpointed_iteration.txt
(TaskRunner pid=91866) Training from scratch
(TaskRunner pid=91866) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': False, 'validate': True, 'global_steps': 0}
(WorkerDict pid=108852) kwargs: {'n': 1, 'logprobs': 0, 'max_tokens': 512, 'repetition_penalty': 1.0, 'detokenize': False, 'temperature': 1.0, 'top_k': -1, 'top_p': 1, 'ignore_eos': False} [repeated 7x across cluster]
(WorkerDict pid=108851) WARNING 09-22 10:53:31 [tokenizer.py:295] No tokenizer found in /simon-stub-path, using base model tokenizer instead. (Exception: Repo id must use alphanumeric chars or '-', '_', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is 96: '/simon-stub-path'.)

The log hasn't been refreshed for an hour.
Please help me check what the issue is. Thank you.

Expected behavior

The log hasn't been refreshed for an hour.
Please help me check what the issue is. Thank you.

[1103036128]

WorkerDict pid=108853) kwargs: {'n': 1, 'logprobs': 0, 'max_tokens': 512, 'repetition_penalty': 1.0, 'detokenize': False, 'temperature': 1.0, 'top_k': -1, 'top_p': 1, 'ignore_eos': False}
(WorkerDict pid=108850) WARNING 09-22 10:52:55 [utils.py:2737] Methods add_prompt_adapter,cache_config,compilation_config,current_platform,list_prompt_adapters,load_config,pin_prompt_adapter,remove_prompt_adapter not implemented in <vllm_ascend.worker.worker.NPUWorker object at 0xffcf0079d110> [repeated 7x across cluster]
(TaskRunner pid=91866) Checkpoint tracker file does not exist: /bigdata/sde/Verl/verl/ckpts/GRPO-Qwen3/GRPO-Qwen3-32B-npu-lora/latest_checkpointed_iteration.txt
(TaskRunner pid=91866) Training from scratch
(TaskRunner pid=91866) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': False, 'validate': True, 'global_steps': 0}
(WorkerDict pid=108852) kwargs: {'n': 1, 'logprobs': 0, 'max_tokens': 512, 'repetition_penalty': 1.0, 'detokenize': False, 'temperature': 1.0, 'top_k': -1, 'top_p': 1, 'ignore_eos': False} [repeated 7x across cluster]
(WorkerDict pid=108851) WARNING 09-22 10:53:31 [tokenizer.py:295] No tokenizer found in /simon-stub-path, using base model tokenizer instead. (Exception: Repo id must use alphanumeric chars or '-', '', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is 96: '/simon-stub-path'.)
(TaskRunner pid=91866) validation generation end
(WorkerDict pid=108849) WARNING 09-22 10:53:31 [tokenizer.py:295] No tokenizer found in /simon-stub-path, using base model tokenizer instead. (Exception: Repo id must use alphanumeric chars or '-', '', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is 96: '/simon-stub-path'.) [repeated 7x across cluster]
(TaskRunner pid=91866) [prompt] user
(TaskRunner pid=91866) Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market? Let's think step by step and output the final answer after "####". /no_think
(TaskRunner pid=91866) assistant
(TaskRunner pid=91866)
(TaskRunner pid=91866) [response]
(TaskRunner pid=91866)
(TaskRunner pid=91866)
(TaskRunner pid=91866)
(TaskRunner pid=91866) Let's break this down step by step:
(TaskRunner pid=91866)
(TaskRunner pid=91866) 1. Total eggs laid per day:
(TaskRunner pid=91866) Janet’s ducks lay 16 eggs per day.
(TaskRunner pid=91866)
(TaskRunner pid=91866) 2. Eggs used for breakfast:
(TaskRunner pid=91866) She eats 3 eggs for breakfast.
(TaskRunner pid=91866)
(TaskRunner pid=91866) 3. Eggs used for baking:
(TaskRunner pid=91866) She uses 4 eggs for baking muffins.
(TaskRunner pid=91866)
(TaskRunner pid=91866) 4. Total eggs used per day:
(TaskRunner pid=91866) $ 3 + 4 = 7 $ eggs used.
(TaskRunner pid=91866)
(TaskRunner pid=91866) 5. Eggs remaining to sell:
(TaskRunner pid=91866) $ 16 - 7 = 9 $ eggs left to sell.
(TaskRunner pid=91866)
(TaskRunner pid=91866) 6. Price per egg:
(TaskRunner pid=91866) She sells each egg for $2.
(TaskRunner pid=91866)
(TaskRunner pid=91866) 7. Total daily earnings from the farmers' market:
(TaskRunner pid=91866) $ 9 \times 2 = 18 $ dollars.
(TaskRunner pid=91866)
(TaskRunner pid=91866) ---
(TaskRunner pid=91866)
(TaskRunner pid=91866) Final Answer:
(TaskRunner pid=91866) #### 18
(TaskRunner pid=91866) [ground_truth] 18
(TaskRunner pid=91866) [score] 1.0
(TaskRunner pid=91866) len reward_extra_infos_dict['reward']: 1319
(TaskRunner pid=91866) ("Initial validation metrics: {'val-core/openai/gsm8k/reward/mean@1': "
(TaskRunner pid=91866) '0.5420773313115997}')
(TaskRunner pid=91866) step:0 - val-core/openai/gsm8k/reward/mean@1:0.5420773313115997
Training Progress: 0%| | 0/1868 [00:00<?, ?it/s]
(WorkerDict pid=108852) /usr/local/python3.11.13/lib/python3.11/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html . [repeated 7x across cluster]
(WorkerDict pid=108852) warnings.warn( [repeated 7x across cluster]
(WorkerDict pid=108847) WARNING 09-22 15:32:07 [tokenizer.py:295] No tokenizer found in /simon-stub-path, using base model tokenizer instead. (Exception: Repo id must use alphanumeric chars or '-', '', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is 96: '/simon-stub-path'.)
(WorkerDict pid=108848) WARNING 09-22 15:32:07 [tokenizer.py:295] No tokenizer found in /simon-stub-path, using base model tokenizer instead. (Exception: Repo id must use alphanumeric chars or '-', '', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is 96: '/simon-stub-path'.)
Error executing job with overrides: ['algorithm.adv_estimator=grpo', 'data.train_files=/bigdata/sde/Verl/dataset/gsm8k/train.parquet', 'data.val_files=/bigdata/sde/Verl/dataset/gsm8k/test.parquet', 'data.train_batch_size=8', 'data.max_prompt_length=2048', 'data.max_response_length=512', 'data.filter_overlong_prompts=False', 'data.truncation=error', 'data.shuffle=False', 'actor_rollout_ref.model.path=/bigdata/sdb/Qwen3-32B', 'actor_rollout_ref.model.use_shm=True', 'actor_rollout_ref.model.lora_rank=8', 'actor_rollout_ref.model.lora_alpha=16', 'actor_rollout_ref.model.target_modules=all-linear', 'actor_rollout_ref.actor.optim.lr=1e-6', 'actor_rollout_ref.model.use_remove_padding=True', 'actor_rollout_ref.actor.ppo_mini_batch_size=8', 'actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=2', 'actor_rollout_ref.actor.use_kl_loss=True', 'actor_rollout_ref.actor.kl_loss_coef=0.001', 'actor_rollout_ref.actor.kl_loss_type=low_var_kl', 'actor_rollout_ref.actor.entropy_coeff=0', 'actor_rollout_ref.rollout.max_num_seqs=8', 'actor_rollout_ref.rollout.max_model_len=2560', 'actor_rollout_ref.rollout.max_num_batched_tokens=2560', 'actor_rollout_ref.model.enable_gradient_checkpointing=True', 'actor_rollout_ref.actor.fsdp_config.param_offload=False', 'actor_rollout_ref.actor.fsdp_config.optimizer_offload=False', 'actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=2', 'actor_rollout_ref.rollout.tensor_model_parallel_size=8', 'actor_rollout_ref.rollout.name=vllm', 'actor_rollout_ref.rollout.gpu_memory_utilization=0.2', 'actor_rollout_ref.rollout.n=2', 'actor_rollout_ref.rollout.load_format=safetensors', 'actor_rollout_ref.rollout.layered_summon=True', 'actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=2', 'actor_rollout_ref.ref.fsdp_config.param_offload=True', 'actor_rollout_ref.ref.fsdp_config.optimizer_offload=True', 'algorithm.use_kl_in_reward=False', 'trainer.critic_warmup=0', 'trainer.logger=["console","wandb"]', 'trainer.project_name=GRPO-Qwen3', 'trainer.experiment_name=GRPO-Qwen3-32B-npu-lora', 'trainer.n_gpus_per_node=8', 'trainer.nnodes=1', 'trainer.default_local_dir=/bigdata/sde/Verl/verl/ckpts/GRPO-Qwen3/GRPO-Qwen3-32B-npu-lora', 'trainer.device=npu', 'trainer.resume_mode=auto', 'actor_rollout_ref.actor.fsdp_config.forward_prefetch=True', 'actor_rollout_ref.ref.fsdp_config.forward_prefetch=True', '++actor_rollout_ref.actor.entropy_from_logits_with_chunking=True', '++actor_rollout_ref.ref.entropy_from_logits_with_chunking=True', 'trainer.val_before_train=True', 'trainer.save_freq=20', 'trainer.test_freq=5', 'trainer.total_epochs=2']
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/bigdata/sde/Verl/verl/verl/trainer/main_ppo.py", line 412, in
main()
File "/usr/local/python3.11.13/lib/python3.11/site-packages/hydra/main.py", line 94, in decorated_main
_run_hydra(
File "/usr/local/python3.11.13/lib/python3.11/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
_run_app(
File "/usr/local/python3.11.13/lib/python3.11/site-packages/hydra/_internal/utils.py", line 457, in _run_app
run_and_report(
File "/usr/local/python3.11.13/lib/python3.11/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
raise ex
File "/usr/local/python3.11.13/lib/python3.11/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
return func()
^^^^^^
File "/usr/local/python3.11.13/lib/python3.11/site-packages/hydra/_internal/utils.py", line 458, in
lambda: hydra.run(
^^^^^^^^^^
File "/usr/local/python3.11.13/lib/python3.11/site-packages/hydra/_internal/hydra.py", line 132, in run
_ = ret.return_value
^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.13/lib/python3.11/site-packages/hydra/core/utils.py", line 260, in return_value
raise self._return_value
File "/usr/local/python3.11.13/lib/python3.11/site-packages/hydra/core/utils.py", line 186, in run_job
ret.return_value = task_function(task_cfg)
^^^^^^^^^^^^^^^^^^^^^^^
File "/bigdata/sde/Verl/verl/verl/trainer/main_ppo.py", line 42, in main
run_ppo(config)
File "/bigdata/sde/Verl/verl/verl/trainer/main_ppo.py", line 85, in run_ppo
ray.get(runner.run.remote(config))
File "/usr/local/python3.11.13/lib/python3.11/site-packages/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.13/lib/python3.11/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.13/lib/python3.11/site-packages/ray/_private/worker.py", line 2822, in get
values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.13/lib/python3.11/site-packages/ray/_private/worker.py", line 930, in get_objects
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(RaySystemError): ray::TaskRunner.run() (pid=91866, ip=141.71.22.115, actor_id=1fb38b693ddd22a646b9ec9001000000, repr=<main_ppo.TaskRunner object at 0xffff9a0c4550>)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/bigdata/sde/Verl/verl/verl/trainer/main_ppo.py", line 316, in run
trainer.fit()
File "/bigdata/sde/Verl/verl/verl/trainer/ppo/ray_trainer.py", line 1029, in fit
old_log_prob = self.actor_rollout_wg.compute_log_prob(batch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/bigdata/sde/Verl/verl/verl/single_controller/ray/base.py", line 48, in call
output = ray.get(output)
^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ray.exceptions.RaySystemError: System error: Failed to unpickle serialized exception
traceback: Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/site-packages/ray/exceptions.py", line 51, in from_ray_exception
return pickle.loads(ray_exception.serialized_exception)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: BackendCompilerFailed.init() missing 1 required positional argument: 'inner_exception'

The above exception was the direct cause of the following exception:

ray::TaskRunner.run() (pid=91866, ip=141.71.22.115, actor_id=1fb38b693ddd22a646b9ec9001000000, repr=<main_ppo.TaskRunner object at 0xffff9a0c4550>)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.13/lib/python3.11/site-packages/ray/exceptions.py", line 45, in from_bytes
return RayError.from_ray_exception(ray_exception)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.13/lib/python3.11/site-packages/ray/exceptions.py", line 54, in from_ray_exception
raise RuntimeError(msg) from e
RuntimeError: Failed to unpickle serialized exception
[ERROR] 2025-09-22-15:36:02 (PID:76532, Device:-1, RankID:-1) ERR99999 UNKNOWN applicaiton exception
(TaskRunner pid=91866) Failed to unpickle serialized exception
(TaskRunner pid=91866) Traceback (most recent call last):
(TaskRunner pid=91866) File "/usr/local/python3.11.13/lib/python3.11/site-packages/ray/exceptions.py", line 51, in from_ray_exception
(TaskRunner pid=91866) return pickle.loads(ray_exception.serialized_exception)
(TaskRunner pid=91866) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=91866) TypeError: BackendCompilerFailed.init() missing 1 required positional argument: 'inner_exception'
(TaskRunner pid=91866)
(TaskRunner pid=91866) The above exception was the direct cause of the following exception:
(TaskRunner pid=91866)
(TaskRunner pid=91866) Traceback (most recent call last):
(TaskRunner pid=91866) File "/usr/local/python3.11.13/lib/python3.11/site-packages/ray/_private/serialization.py", line 460, in deserialize_objects
(TaskRunner pid=91866) obj = self._deserialize_object(data, metadata, object_ref)
(TaskRunner pid=91866) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=91866) File "/usr/local/python3.11.13/lib/python3.11/site-packages/ray/_private/serialization.py", line 342, in deserialize_object
(TaskRunner pid=91866) return RayError.from_bytes(obj)
(TaskRunner pid=91866) ^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=91866) File "/usr/local/python3.11.13/lib/python3.11/site-packages/ray/exceptions.py", line 45, in from_bytes
(TaskRunner pid=91866) return RayError.from_ray_exception(ray_exception)
(TaskRunner pid=91866) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=91866) File "/usr/local/python3.11.13/lib/python3.11/site-packages/ray/exceptions.py", line 54, in from_ray_exception
(TaskRunner pid=91866) raise RuntimeError(msg) from e
(TaskRunner pid=91866) RuntimeError: Failed to unpickle serialized exception
(WorkerDict pid=108852) WARNING 09-22 15:32:07 [tokenizer.py:295] No tokenizer found in /simon-stub-path, using base model tokenizer instead. (Exception: Repo id must use alphanumeric chars or '-', '', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is 96: '/simon-stub-path'.) [repeated 6x across cluster]

[1103036128]

An error occurred 5 hours later.

[1103036128]

Ray logs ‘/tmp/ray/session_latest/logs’ find error:

[1103036128]

cat worker-8924704be9668d85154e08c3a696f9c1d03fa6c99cdcaec9cc5d1d2a-01000000-108852.err
:job_id:01000000
:actor_name:WorkerDict
Loading checkpoint shards: 100%|██████████| 17/17 [01:02<00:00, 3.67s/it]
/bigdata/sde/Verl/verl/verl/utils/profiler/config.py:49: UserWarning: Torch profiler tool config is not fully supported now.
warnings.warn("Torch profiler tool config is not fully supported now.", stacklevel=1)
/usr/local/python3.11.13/lib/python3.11/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
warnings.warn(
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
self.run()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/utils/multiprocess_util.py", line 91, in run
key, func, args, kwargs = self.task_q.get(timeout=TIMEOUT)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 2, in get
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/managers.py", line 822, in _callmethod
kind, result = conn.recv()
^^^^^^^^^^^
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/connection.py", line 430, in _recv_bytes
buf = self._recv(4)
^^^^^^^^^^^^^
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/connection.py", line 399, in _recv
raise EOFError
EOFError
/usr/local/python3.11.13/lib/python3.11/multiprocessing/resource_tracker.py:123: UserWarning: resource_tracker: process died unexpectedly, relaunching. Some resources might leak.
warnings.warn('resource_tracker: process died unexpectedly, '
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/resource_tracker.py", line 239, in main
cache[rtype].remove(name)
KeyError: '/mp-zlt_5z30'
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/resource_tracker.py", line 239, in main
cache[rtype].remove(name)
KeyError: '/mp-8ejaz1re'
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/resource_tracker.py", line 239, in main
cache[rtype].remove(name)
KeyError: '/mp-ubshk2ql'
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/resource_tracker.py", line 239, in main
cache[rtype].remove(name)
KeyError: '/mp-tnqk4s9f'
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/resource_tracker.py", line 239, in main
cache[rtype].remove(name)
KeyError: '/mp-5mel5wqm'
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/resource_tracker.py", line 239, in main
cache[rtype].remove(name)
KeyError: '/mp-1oyvgarh'
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/resource_tracker.py", line 239, in main
cache[rtype].remove(name)
KeyError: '/mp-ccb_igve'
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/resource_tracker.py", line 239, in main
cache[rtype].remove(name)
KeyError: '/mp-jetw9snm'
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/resource_tracker.py", line 239, in main
cache[rtype].remove(name)
KeyError: '/mp-3bgmmn7s'
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/resource_tracker.py", line 239, in main
cache[rtype].remove(name)
KeyError: '/mp-w0my1f01'
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/resource_tracker.py", line 239, in main
cache[rtype].remove(name)
KeyError: '/mp-3c3xumvd'
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/resource_tracker.py", line 239, in main
cache[rtype].remove(name)
KeyError: '/mp-fp_t89b1'
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/resource_tracker.py", line 239, in main
cache[rtype].remove(name)
KeyError: '/mp-j0d0wt_r'
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/resource_tracker.py", line 239, in main
cache[rtype].remove(name)
KeyError: '/mp-s88oxgcu'
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/resource_tracker.py", line 239, in main
cache[rtype].remove(name)
KeyError: '/mp-k6djopgp'
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/resource_tracker.py", line 239, in main
cache[rtype].remove(name)
KeyError: '/mp-484r4ie1'
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/resource_tracker.py", line 239, in main
cache[rtype].remove(name)
KeyError: '/mp-y8ikpre8'
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/resource_tracker.py", line 239, in main
cache[rtype].remove(name)
KeyError: '/mp-61zske35'
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/resource_tracker.py", line 239, in main
cache[rtype].remove(name)
KeyError: '/mp-6i4g4ayg'
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/resource_tracker.py", line 239, in main
cache[rtype].remove(name)
KeyError: '/mp-i8_0ghg7'
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/resource_tracker.py", line 239, in main
cache[rtype].remove(name)
KeyError: '/mp-_xt7mwba'
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/resource_tracker.py", line 239, in main
cache[rtype].remove(name)
KeyError: '/mp-pth7qi8s'
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/resource_tracker.py", line 239, in main
cache[rtype].remove(name)
KeyError: '/mp-bh39zpfp'
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/resource_tracker.py", line 239, in main
cache[rtype].remove(name)
KeyError: '/mp-tkgjnvwk'
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/resource_tracker.py", line 239, in main
cache[rtype].remove(name)
KeyError: '/mp-xuibr16x'
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/resource_tracker.py", line 239, in main
cache[rtype].remove(name)
KeyError: '/mp-hh7qyp6z'
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/resource_tracker.py", line 239, in main
cache[rtype].remove(name)
KeyError: '/mp-pmo7th1e'
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/resource_tracker.py", line 239, in main
cache[rtype].remove(name)
KeyError: '/mp-ocwo8t_2'
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/resource_tracker.py", line 239, in main
cache[rtype].remove(name)
KeyError: '/mp-s9nsxlwv'
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/resource_tracker.py", line 239, in main
cache[rtype].remove(name)
KeyError: '/mp-rsl1ul8h'
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/managers.py", line 814, in _callmethod
conn = self._tls.connection
^^^^^^^^^^^^^^^^^^^^
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/util.py", line 303, in _run_finalizers
finalizer()
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/util.py", line 227, in call
res = self._callback(*self._args, **self._kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 54, in wrapper
return func(cls, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 208, in finalize
cls.cann_kb_mgr.finalize()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/cann_kb_manager/knowledge_bank_manager.py", line 42, in finalize
self.__static_mgr.close()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/cann_kb_manager/base_manager.py", line 339, in close
for name in copy.deepcopy(self.write_kb.keys()):
^^^^^^^^^^^^^^^^^^^^
File "", line 2, in keys
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/managers.py", line 818, in _callmethod
self._connect()
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/managers.py", line 805, in _connect
conn = self._Client(self._token.address, authkey=self._authkey)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/connection.py", line 519, in Client
c = SocketClient(address)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/connection.py", line 647, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
Process ForkServerProcess-10:
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/utils/common.py", line 110, in daemon_process
os.kill(mgr_pid, signal.SIGKILL)
ProcessLookupError: [Errno 3] No such proces

[1103036128]

The script can be executed after refreshing. Please refer to:

export WANDB_MODE=disabled
set -x

project_name='GRPO-Qwen32B'
exp_name='GRPO-Qwen3-32B-npu-lora'
gen_tp=8
RAY_DATA_HOME=${RAY_DATA_HOME:-"${HOME}/verl"}
MODEL_PATH=/bigdata/sdb/Qwen3-32B
CKPTS_DIR=/bigdata/sde/Verl/verl/ckpts/${project_name}/${exp_name}
TRAIN_FILE=/bigdata/sde/Verl/dataset/gsm8k/train.parquet
TEST_FILE=/bigdata/sde/Verl/dataset/gsm8k/test.parquet

python3 -m verl.trainer.main_ppo \
    algorithm.adv_estimator=grpo \
    data.train_files="${TRAIN_FILE}" \
    data.val_files="${TEST_FILE}" \
    data.train_batch_size=64 \
    data.max_prompt_length=512 \
    data.max_response_length=1024 \
    data.filter_overlong_prompts=False \
    data.truncation='error' \
    actor_rollout_ref.model.path=${MODEL_PATH} \
    actor_rollout_ref.actor.optim.lr=1e-6 \
    actor_rollout_ref.model.use_remove_padding=True \
    actor_rollout_ref.model.use_shm=True \
    actor_rollout_ref.rollout.layered_summon=True \
    actor_rollout_ref.actor.ppo_mini_batch_size=16 \
    actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=8 \
    actor_rollout_ref.actor.use_kl_loss=True \
    actor_rollout_ref.actor.kl_loss_coef=0.001 \
    actor_rollout_ref.actor.kl_loss_type=low_var_kl \
    actor_rollout_ref.model.lora_rank=8 \
    actor_rollout_ref.model.lora_alpha=16 \
    actor_rollout_ref.rollout.load_format="safetensors" \
    actor_rollout_ref.model.target_modules="all-linear" \
    actor_rollout_ref.actor.entropy_coeff=0 \
    actor_rollout_ref.actor.use_torch_compile=False \
    actor_rollout_ref.ref.use_torch_compile=False \
    actor_rollout_ref.model.enable_gradient_checkpointing=True \
    actor_rollout_ref.actor.fsdp_config.param_offload=False \
    actor_rollout_ref.actor.fsdp_config.optimizer_offload=False \
    actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=8 \
    actor_rollout_ref.rollout.tensor_model_parallel_size=${gen_tp} \
    actor_rollout_ref.rollout.name=vllm \
    actor_rollout_ref.rollout.gpu_memory_utilization=0.3 \
    actor_rollout_ref.rollout.n=4 \
    actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=8 \
    actor_rollout_ref.ref.fsdp_config.param_offload=True \
    algorithm.use_kl_in_reward=False \
    trainer.critic_warmup=0 \
    trainer.logger='["console","wandb"]' \
    trainer.project_name="${project_name}" \
    trainer.experiment_name="${exp_name}" \
    trainer.n_gpus_per_node=8 \
    trainer.nnodes=1 \
    trainer.default_local_dir=${CKPTS_DIR} \
    trainer.device=npu \
    trainer.resume_mode=auto \
    actor_rollout_ref.actor.fsdp_config.forward_prefetch=True \
    actor_rollout_ref.ref.fsdp_config.forward_prefetch=True \
    ++actor_rollout_ref.actor.entropy_from_logits_with_chunking=True \
    ++actor_rollout_ref.ref.entropy_from_logits_with_chunking=True \
    trainer.val_before_train=True \
    trainer.save_freq=20 \
    trainer.test_freq=5 \
    trainer.total_epochs=5