[SGLang Async Rollout] CUDA error: an illegal memory access was encountered

[jybsuper]

I encountered the following CUDA error while using async SGLang for multiturn GRPO RL.

[2025-05-20 18:43:20 TP3] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/jobuser/.local/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 2269, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/home/jobuser/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/home/jobuser/.local/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 658, in event_loop_overlap
    batch = self.get_next_batch_to_run()
  File "/home/jobuser/.local/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 1304, in get_next_batch_to_run
    self.running_batch = self.update_running_batch(self.running_batch)
  File "/home/jobuser/.local/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 1502, in update_running_batch
    batch.prepare_for_decode()
  File "/home/jobuser/.local/lib/python3.10/site-packages/sglang/srt/managers/schedule_batch.py", line 1461, in prepare_for_decode
    locs = self.seq_lens.clone()
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

I constantly see this error in the validation step before training if I use QwQ-32b or Qwen3-32b but everything worked well when training with Qwen2.5-32b-instruct. To verify if this is actually an OOM issue which is mentioned in SGLang doc, I reduced the validation dataset to only 1 data point but still hit the same error when tp size is 4 or 8. Here is the an overview of the GPU memory usage when the error happened, which does not seem like OOM to me(only ~65% GPU Mem utilization before failure):

I am using up-to-date Verl and have the following dependencies installed:

torch==2.6.0+cu124
torch_memory_saver==0.0.6
transformers==4.51.1
sgl-kernel==0.1.2.post1
sglang==0.4.6.post4
ray==2.46.0
numpy==1.26.4
nvidia-cublas-cu12==12.4.5.8
nvidia-cuda-cupti-cu12==12.4.127
nvidia-cuda-nvrtc-cu12==12.4.127
nvidia-cuda-runtime-cu12==12.4.127
nvidia-cudnn-cu12==9.1.0.70
nvidia-cufft-cu12==11.2.1.3
nvidia-curand-cu12==10.3.5.147
nvidia-cusolver-cu12==11.6.1.9
nvidia-cusparse-cu12==12.3.1.170
nvidia-cusparselt-cu12==0.6.2
nvidia-ml-py==12.570.86
nvidia-nccl-cu12==2.21.5
nvidia-nvjitlink-cu12==12.4.127
nvidia-nvtx-cu12==12.4.127
flash_attn==2.7.4.post1+cu12torch2.6cxx11abiFALSE

Here is my full training config:

{
  "actor_rollout_ref": {
    "actor": {
      "checkpoint": {
        "contents": ["model", "optimizer", "extra"]
      },
      "clip_ratio": 0.2,
      "clip_ratio_c": 3.0,
      "clip_ratio_high": 0.2,
      "clip_ratio_low": 0.2,
      "entropy_coeff": 0,
      "fsdp_config": {
        "fsdp_size": -1,
        "offload_policy": false,
        "optimizer_offload": true,
        "param_offload": true,
        "reshard_after_forward": true,
        "wrap_policy": {"min_num_params": 0}
      },
      "grad_clip": 1.0,
      "kl_loss_coef": 0.001,
      "kl_loss_type": "low_var_kl",
      "loss_agg_mode": "token-mean",
      "optim": {
        "lr": 1e-6,
        "lr_warmup_steps": -1,
        "lr_warmup_steps_ratio": 0.0,
        "min_lr_ratio": 0.0,
        "num_cycles": 0.5,
        "total_training_steps": -1,
        "warmup_style": "constant",
        "weight_decay": 0.01
      },
      "ppo_epochs": 1,
      "ppo_max_token_len_per_gpu": 16384,
      "ppo_micro_batch_size": null,
      "ppo_micro_batch_size_per_gpu": 1,
      "ppo_mini_batch_size": 8,
      "shuffle": false,
      "strategy": "fsdp",
      "ulysses_sequence_parallel_size": 1,
      "use_dynamic_bsz": false,
      "use_kl_loss": true,
      "use_torch_compile": true
    },
    "hybrid_engine": true,
    "model": {
      "enable_gradient_checkpointing": true,
      "external_lib": null,
      "override_config": {},
      "path": "/shared/public/elr-models/Qwen/QwQ-32B/976055f8c83f394f35dbd3ab09a285a984907bd0",
      "trust_remote_code": false,
      "use_fused_kernels": false,
      "use_liger": false,
      "use_remove_padding": true
    },
    "ref": {
      "fsdp_config": {
        "param_offload": true,
        "reshard_after_forward": true,
        "wrap_policy": {"min_num_params": 0}
      },
      "log_prob_max_token_len_per_gpu": 16384,
      "log_prob_micro_batch_size": null,
      "log_prob_micro_batch_size_per_gpu": 32,
      "log_prob_use_dynamic_bsz": false,
      "strategy": "fsdp",
      "ulysses_sequence_parallel_size": 1,
      "use_torch_compile": true
    },
    "rollout": {
      "chat_scheduler": null,
      "disable_log_stats": true,
      "do_sample": true,
      "dtype": "bfloat16",
      "enable_chunked_prefill": true,
      "enforce_eager": true,
      "engine_kwargs": {"swap_space": null},
      "free_cache_engine": true,
      "gpu_memory_utilization": 0.6,
      "ignore_eos": false,
      "load_format": "dummy_dtensor",
      "log_prob_max_token_len_per_gpu": 16384,
      "log_prob_micro_batch_size": null,
      "log_prob_micro_batch_size_per_gpu": 32,
      "log_prob_use_dynamic_bsz": false,
      "max_model_len": null,
      "max_num_batched_tokens": 8192,
      "max_num_seqs": 1024,
      "mode": "sync",
      "multi_turn": {
        "enable": true,
        "format": "qwen",
        "max_turns": 5,
        "tool_config_path": null,
        "tool_list": "tools.tool_list"
      },
      "n": 8,
      "name": "sglang_async",
      "prompt_length": 4096,
      "response_length": 1024,
      "temperature": 1.0,
      "tensor_model_parallel_size": 8,
      "top_k": -1,
      "top_p": 1,
      "use_fire_sampling": false,
      "val_kwargs": {
        "do_sample": false,
        "n": 1,
        "temperature": 0,
        "top_k": -1,
        "top_p": 1.0
      }
    }
  },
  "algorithm": {
    "adv_estimator": "grpo",
    "gamma": 1.0,
    "kl_ctrl": {
      "horizon": 10000,
      "kl_coef": 0.001,
      "target_kl": 0.1,
      "type": "fixed"
    },
    "kl_penalty": "kl",
    "lam": 1.0,
    "norm_adv_by_std_in_grpo": true,
    "use_kl_in_reward": false
  },
  "critic": {
    "checkpoint": {"contents": ["model", "optimizer", "extra"]},
    "cliprange_value": 0.5,
    "forward_max_token_len_per_gpu": 32768,
    "forward_micro_batch_size": null,
    "forward_micro_batch_size_per_gpu": null,
    "grad_clip": 1.0,
    "model": {
      "enable_gradient_checkpointing": true,
      "external_lib": null,
      "fsdp_config": {
        "fsdp_size": -1,
        "offload_policy": false,
        "optimizer_offload": false,
        "param_offload": false,
        "reshard_after_forward": true,
        "wrap_policy": {"min_num_params": 0}
      },
      "override_config": {},
      "path": "~/models/deepseek-llm-7b-chat",
      "tokenizer_path": "/shared/public/elr-models/Qwen/QwQ-32B/976055f8c83f394f35dbd3ab09a285a984907bd0",
      "trust_remote_code": false,
      "use_remove_padding": false
    },
    "optim": {
      "lr": 1e-5,
      "lr_warmup_steps_ratio": 0.0,
      "min_lr_ratio": null,
      "total_training_steps": -1,
      "warmup_style": "constant",
      "weight_decay": 0.01
    },
    "ppo_epochs": 1,
    "ppo_max_token_len_per_gpu": 32768,
    "ppo_micro_batch_size": null,
    "ppo_micro_batch_size_per_gpu": null,
    "ppo_mini_batch_size": 8,
    "rollout_n": 8,
    "shuffle": false,
    "strategy": "fsdp",
    "ulysses_sequence_parallel_size": 1,
    "use_dynamic_bsz": false
  },
  "custom_reward_function": {
    "name": "compute_score",
    "path": "...",
    "reward_kwargs": {}
  },
  "data": {
    "custom_cls": {"name": null, "path": null},
    "filter_overlong_prompts": true,
    "filter_overlong_prompts_workers": 1,
    "image_key": "images",
    "max_prompt_length": 4096,
    "max_response_length": 1024,
    "prompt_key": "prompt",
    "return_full_prompt": false,
    "return_raw_chat": true,
    "return_raw_input_ids": false,
    "reward_fn_key": "data_source",
    "shuffle": true,
    "tokenizer": null,
    "train_batch_size": 8,
    "train_files": "...",
    "truncation": "error",
    "val_batch_size": 1,
    "val_files": "...",
    "video_key": "videos"
  },
  "ray_init": {
    "num_cpus": null
  },
  "reward_manager": "naive",
  "sandbox_fusion": {
    "max_concurrent": 64,
    "url": null
  },
  "strategy": "fsdp",
  "ulysses_sequence_parallel_size": 1,
  "use_dynamic_bsz": false,
  "trainer": {
    "balance_batch": true,
    "critic_warmup": 0,
    "default_hdfs_dir": null,
    "default_local_dir": "...",
    "del_local_ckpt_after_load": false,
    "experiment_name": "qwq-32b",
    "log_val_generations": 0,
    "logger": ["mlflow"],
    "max_actor_ckpt_to_keep": null,
    "max_critic_ckpt_to_keep": null,
    "n_gpus_per_node": 8,
    "nnodes": 1,
    "project_name": "yajiang-project/multiturn",
    "ray_wait_register_center_timeout": 300,
    "resume_from_path": null,
    "resume_mode": "disable",
    "rollout_data_dir": null,
    "save_freq": 10,
    "test_freq": 10,
    "total_epochs": 5,
    "total_training_steps": null,
    "val_before_train": true,
    "validation_data_dir": null
  },
  "reward_model": {
    "enable": false,
    "forward_max_token_len_per_gpu": 32768,
    "launch_reward_fn_async": false,
    "max_length": null,
    "micro_batch_size": null,
    "micro_batch_size_per_gpu": null,
    "model": {
      "external_lib": null,
      "fsdp_config": {
        "fsdp_size": -1,
        "param_offload": false,
        "reshard_after_forward": true,
        "wrap_policy": {"min_num_params": 0}
      },
      "input_tokenizer": "/shared/public/elr-models/Qwen/QwQ-32B/976055f8c83f394f35dbd3ab09a285a984907bd0",
      "path": "~/models/FsfairX-LLaMA3-RM-v0.1",
      "trust_remote_code": false,
      "use_remove_padding": false
    }
  }
}

[jybsuper]

Below is the full error log:

(TaskRunner pid=1714137) Using dataset class: RLHFDataset
(TaskRunner pid=1714137) dataset len: 45
(TaskRunner pid=1714137) filter dataset len: 45
(TaskRunner pid=1714137) Using dataset class: RLHFDataset
(TaskRunner pid=1714137) dataset len: 1
(TaskRunner pid=1714137) filter dataset len: 1
(TaskRunner pid=1714137) WARNING: val_batch_size is deprecated. Validation datasets are sent to inference engines as a whole batch, which will schedule the memory themselves.
(TaskRunner pid=1714137) [validate_config] All configuration checks passed successfully!
(TaskRunner pid=1714137) Size of train dataloader: 5, Size of val dataloader: 1
(TaskRunner pid=1714137) Total training steps: 25
(TaskRunner pid=1714137) colocated worker base class <class 'verl.single_controller.base.worker.Worker'>
(WorkerDict pid=1732013) Model config after override: Qwen2Config {
(WorkerDict pid=1732013)   "architectures": [
(WorkerDict pid=1732013)     "Qwen2ForCausalLM"
(WorkerDict pid=1732013)   ],
(WorkerDict pid=1732013)   "attention_dropout": 0.0,
(WorkerDict pid=1732013)   "eos_token_id": 151645,
(WorkerDict pid=1732013)   "hidden_act": "silu",
(WorkerDict pid=1732013)   "hidden_size": 5120,
(WorkerDict pid=1732013)   "initializer_range": 0.02,
(WorkerDict pid=1732013)   "intermediate_size": 27648,
(WorkerDict pid=1732013)   "max_position_embeddings": 40960,
(WorkerDict pid=1732013)   "max_window_layers": 64,
(WorkerDict pid=1732013)   "model_type": "qwen2",
(WorkerDict pid=1732013)   "num_attention_heads": 40,
(WorkerDict pid=1732013)   "num_hidden_layers": 64,
(WorkerDict pid=1732013)   "num_key_value_heads": 8,
(WorkerDict pid=1732013)   "pad_token_id": 151643,
(WorkerDict pid=1732013)   "rms_norm_eps": 1e-05,
(WorkerDict pid=1732013)   "rope_scaling": null,
(WorkerDict pid=1732013)   "rope_theta": 1000000.0,
(WorkerDict pid=1732013)   "sliding_window": 32768,
(WorkerDict pid=1732013)   "tie_word_embeddings": false,
(WorkerDict pid=1732013)   "torch_dtype": "bfloat16",
(WorkerDict pid=1732013)   "transformers_version": "4.51.1",
(WorkerDict pid=1732013)   "use_cache": true,
(WorkerDict pid=1732013)   "use_sliding_window": false,
(WorkerDict pid=1732013)   "vocab_size": 152064
(WorkerDict pid=1732013) }
(WorkerDict pid=1732013) 
(WorkerDict pid=1733167) Monkey patch _flash_attention_forward in transformers.integrations.flash_attention
(WorkerDict pid=1732013) Qwen2ForCausalLM contains 32.76B parameters
(WorkerDict pid=1732013) wrap_policy: functools.partial(<function _or_policy at 0x71c356939000>, policies=[functools.partial(<function transformer_auto_wrap_policy at 0x71c356938ee0>, transformer_layer_cls={<class 'transformers.models.qwen2.modeling_qwen2.Qwen2DecoderLayer'>})])
(WorkerDict pid=1732013) NCCL version 2.21.5+cuda12.4
(WorkerDict pid=1732013) Actor use_remove_padding=True
(WorkerDict pid=1732013) Actor use_fused_kernels=False
(WorkerDict pid=1732013) Monkey patch _flash_attention_forward in transformers.integrations.flash_attention [repeated 7x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)
(WorkerDict pid=1733170) wrap_policy: functools.partial(<function _or_policy at 0x77e34f3e1000>, policies=[functools.partial(<function transformer_auto_wrap_policy at 0x77e34f3e0ee0>, transformer_layer_cls={<class 'transformers.models.qwen2.modeling_qwen2.Qwen2DecoderLayer'>})]) [repeated 7x across cluster]
(WorkerDict pid=1732013) Model config after override: Qwen2Config {
(WorkerDict pid=1732013)   "architectures": [
(WorkerDict pid=1732013)     "Qwen2ForCausalLM"
(WorkerDict pid=1732013)   ],
(WorkerDict pid=1732013)   "attention_dropout": 0.0,
(WorkerDict pid=1732013)   "eos_token_id": 151645,
(WorkerDict pid=1732013)   "hidden_act": "silu",
(WorkerDict pid=1732013)   "hidden_size": 5120,
(WorkerDict pid=1732013)   "initializer_range": 0.02,
(WorkerDict pid=1732013)   "intermediate_size": 27648,
(WorkerDict pid=1732013)   "max_position_embeddings": 40960,
(WorkerDict pid=1732013)   "max_window_layers": 64,
(WorkerDict pid=1732013)   "model_type": "qwen2",
(WorkerDict pid=1732013)   "num_attention_heads": 40,
(WorkerDict pid=1732013)   "num_hidden_layers": 64,
(WorkerDict pid=1732013)   "num_key_value_heads": 8,
(WorkerDict pid=1732013)   "pad_token_id": 151643,
(WorkerDict pid=1732013)   "rms_norm_eps": 1e-05,
(WorkerDict pid=1732013)   "rope_scaling": null,
(WorkerDict pid=1732013)   "rope_theta": 1000000.0,
(WorkerDict pid=1732013)   "sliding_window": 32768,
(WorkerDict pid=1732013)   "tie_word_embeddings": false,
(WorkerDict pid=1732013)   "torch_dtype": "bfloat16",
(WorkerDict pid=1732013)   "transformers_version": "4.51.1",
(WorkerDict pid=1732013)   "use_cache": true,
(WorkerDict pid=1732013)   "use_sliding_window": false,
(WorkerDict pid=1732013)   "vocab_size": 152064
(WorkerDict pid=1732013) Flash Attention 2.0 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen2ForCausalLM is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `torch_dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", torch_dtype=torch.float16)`
(WorkerDict pid=1733172) You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. [repeated 7x across cluster]
Loading checkpoint shards:   0%|          | 0/14 [00:00<?, ?it/s] [repeated 7x across cluster]
Loading checkpoint shards:  57%|█████▋    | 8/14 [00:00<00:00, 72.33it/s]
Loading checkpoint shards: 100%|██████████| 14/14 [00:00<00:00, 75.03it/s]
Loading checkpoint shards: 100%|██████████| 14/14 [00:00<00:00, 89.86it/s] [repeated 5x across cluster]
Loading checkpoint shards:   7%|▋         | 1/14 [00:04<00:56,  4.37s/it]
(WorkerDict pid=1733170) Flash Attention 2.0 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen2ForCausalLM is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `torch_dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", torch_dtype=torch.float16)` [repeated 7x across cluster]
Loading checkpoint shards:   0%|          | 0/14 [00:00<?, ?it/s] [repeated 8x across cluster]
Loading checkpoint shards:  14%|█▍        | 2/14 [00:09<00:59,  4.98s/it] [repeated 11x across cluster]
Loading checkpoint shards:  21%|██▏       | 3/14 [00:15<00:56,  5.18s/it] [repeated 9x across cluster]
Loading checkpoint shards:  29%|██▊       | 4/14 [00:20<00:51,  5.19s/it] [repeated 8x across cluster]
Loading checkpoint shards:  36%|███▌      | 5/14 [00:26<00:47,  5.23s/it] [repeated 8x across cluster]
Loading checkpoint shards:  43%|████▎     | 6/14 [00:31<00:42,  5.28s/it] [repeated 8x across cluster]
Loading checkpoint shards:  57%|█████▋    | 8/14 [00:36<00:27,  4.60s/it] [repeated 10x across cluster]
Loading checkpoint shards:  57%|█████▋    | 8/14 [00:42<00:31,  5.31s/it] [repeated 9x across cluster]
Loading checkpoint shards:  64%|██████▍   | 9/14 [00:47<00:26,  5.30s/it] [repeated 8x across cluster]
Loading checkpoint shards:  71%|███████▏  | 10/14 [00:52<00:21,  5.31s/it] [repeated 8x across cluster]
Loading checkpoint shards:  93%|█████████▎| 13/14 [00:57<00:04,  4.29s/it]
Loading checkpoint shards:  79%|███████▊  | 11/14 [00:58<00:15,  5.33s/it] [repeated 8x across cluster]
Loading checkpoint shards: 100%|██████████| 14/14 [00:59<00:00,  4.23s/it]
Loading checkpoint shards:  93%|█████████▎| 13/14 [00:59<00:04,  4.57s/it] [repeated 2x across cluster]
Loading checkpoint shards:  86%|████████▌ | 12/14 [01:03<00:10,  5.33s/it] [repeated 5x across cluster]
Loading checkpoint shards: 100%|██████████| 14/14 [01:01<00:00,  4.40s/it] [repeated 2x across cluster]
Loading checkpoint shards:  93%|█████████▎| 13/14 [01:08<00:05,  5.31s/it] [repeated 2x across cluster]
Loading checkpoint shards:  86%|████████▌ | 12/14 [01:04<00:10,  5.43s/it] [repeated 3x across cluster]
Loading checkpoint shards: 100%|██████████| 14/14 [01:06<00:00,  4.78s/it]
Loading checkpoint shards: 100%|██████████| 14/14 [01:10<00:00,  5.05s/it]
(WorkerDict pid=1733167) /home/jobuser/.local/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
(WorkerDict pid=1733167)   warnings.warn(
Loading checkpoint shards:  93%|█████████▎| 13/14 [01:09<00:05,  5.39s/it] [repeated 3x across cluster]
Loading checkpoint shards: 100%|██████████| 14/14 [01:12<00:00,  5.15s/it] [repeated 3x across cluster]
Capturing batches (avail_mem=52.88 GB):   0%|          | 0/35 [00:00<?, ?it/s]
(WorkerDict pid=1733170) /home/jobuser/.local/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html . [repeated 6x across cluster]
(WorkerDict pid=1733170)   warnings.warn( [repeated 6x across cluster]
Capturing batches (avail_mem=52.41 GB):   3%|▎         | 1/35 [00:00<00:27,  1.24it/s]
Capturing batches (avail_mem=52.22 GB):   6%|▌         | 2/35 [00:01<00:21,  1.53it/s]
Capturing batches (avail_mem=52.03 GB):   9%|▊         | 3/35 [00:02<00:21,  1.48it/s]
Capturing batches (avail_mem=51.85 GB):  11%|█▏        | 4/35 [00:02<00:19,  1.62it/s]
Capturing batches (avail_mem=51.68 GB):  14%|█▍        | 5/35 [00:03<00:17,  1.75it/s]
Capturing batches (avail_mem=51.51 GB):  17%|█▋        | 6/35 [00:03<00:16,  1.79it/s]
Capturing batches (avail_mem=51.35 GB):  20%|██        | 7/35 [00:04<00:15,  1.84it/s]
Capturing batches (avail_mem=51.19 GB):  23%|██▎       | 8/35 [00:04<00:14,  1.85it/s]
Capturing batches (avail_mem=51.04 GB):  26%|██▌       | 9/35 [00:05<00:13,  1.90it/s]
Capturing batches (avail_mem=50.89 GB):  29%|██▊       | 10/35 [00:05<00:13,  1.91it/s]
Capturing batches (avail_mem=50.75 GB):  31%|███▏      | 11/35 [00:06<00:12,  1.93it/s]
Capturing batches (avail_mem=50.62 GB):  34%|███▍      | 12/35 [00:06<00:12,  1.91it/s]
Capturing batches (avail_mem=50.49 GB):  37%|███▋      | 13/35 [00:07<00:11,  1.91it/s]
Capturing batches (avail_mem=50.37 GB):  40%|████      | 14/35 [00:07<00:10,  1.91it/s]
Capturing batches (avail_mem=50.26 GB):  43%|████▎     | 15/35 [00:08<00:10,  1.90it/s]
Capturing batches (avail_mem=50.15 GB):  46%|████▌     | 16/35 [00:08<00:09,  1.92it/s]
Capturing batches (avail_mem=50.11 GB):  49%|████▊     | 17/35 [00:09<00:09,  1.94it/s]
Capturing batches (avail_mem=50.09 GB):  51%|█████▏    | 18/35 [00:09<00:08,  1.95it/s]
Capturing batches (avail_mem=49.99 GB):  54%|█████▍    | 19/35 [00:10<00:08,  1.95it/s]
Capturing batches (avail_mem=49.90 GB):  57%|█████▋    | 20/35 [00:10<00:07,  1.94it/s]
Capturing batches (avail_mem=49.82 GB):  60%|██████    | 21/35 [00:11<00:07,  1.95it/s]
Capturing batches (avail_mem=49.74 GB):  63%|██████▎   | 22/35 [00:11<00:06,  1.95it/s]
Capturing batches (avail_mem=49.67 GB):  66%|██████▌   | 23/35 [00:12<00:06,  1.95it/s]
Capturing batches (avail_mem=49.61 GB):  69%|██████▊   | 24/35 [00:12<00:05,  1.95it/s]
Capturing batches (avail_mem=49.59 GB):  71%|███████▏  | 25/35 [00:13<00:05,  1.95it/s]
Capturing batches (avail_mem=49.52 GB):  74%|███████▍  | 26/35 [00:13<00:04,  1.96it/s]
Capturing batches (avail_mem=49.49 GB):  77%|███████▋  | 27/35 [00:14<00:04,  1.95it/s]
Capturing batches (avail_mem=49.43 GB):  80%|████████  | 28/35 [00:14<00:03,  1.93it/s]
Capturing batches (avail_mem=49.42 GB):  83%|████████▎ | 29/35 [00:15<00:03,  1.95it/s]
Capturing batches (avail_mem=49.41 GB):  86%|████████▌ | 30/35 [00:15<00:02,  1.97it/s]
Capturing batches (avail_mem=49.39 GB):  89%|████████▊ | 31/35 [00:16<00:02,  1.97it/s]
Capturing batches (avail_mem=49.38 GB):  91%|█████████▏| 32/35 [00:16<00:01,  1.97it/s]
Capturing batches (avail_mem=49.37 GB):  94%|█████████▍| 33/35 [00:17<00:01,  1.98it/s]
Capturing batches (avail_mem=49.36 GB):  97%|█████████▋| 34/35 [00:17<00:00,  1.97it/s]
Capturing batches (avail_mem=49.36 GB): 100%|██████████| 35/35 [00:18<00:00,  1.89it/s]
(WorkerDict pid=1732013) /home/jobuser/.local/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
(WorkerDict pid=1732013)   warnings.warn(
(WorkerDict pid=1732013) [2025-05-21 01:27:52 TP2] Scheduler hit an exception: Traceback (most recent call last):
(WorkerDict pid=1732013)   File "/home/jobuser/.local/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 2269, in run_scheduler_process
(WorkerDict pid=1732013)     scheduler.event_loop_overlap()
(WorkerDict pid=1732013)   File "/home/jobuser/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(WorkerDict pid=1732013)     return func(*args, **kwargs)
(WorkerDict pid=1732013)   File "/home/jobuser/.local/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 658, in event_loop_overlap
(WorkerDict pid=1732013)     batch = self.get_next_batch_to_run()
(WorkerDict pid=1732013)   File "/home/jobuser/.local/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 1304, in get_next_batch_to_run
(WorkerDict pid=1732013)     self.running_batch = self.update_running_batch(self.running_batch)
(WorkerDict pid=1732013)   File "/home/jobuser/.local/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 1502, in update_running_batch
(WorkerDict pid=1732013)     batch.prepare_for_decode()
(WorkerDict pid=1732013)   File "/home/jobuser/.local/lib/python3.10/site-packages/sglang/srt/managers/schedule_batch.py", line 1461, in prepare_for_decode
(WorkerDict pid=1732013)     locs = self.seq_lens.clone()
(WorkerDict pid=1732013) RuntimeError: CUDA error: an illegal memory access was encountered
(WorkerDict pid=1732013) CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(WorkerDict pid=1732013) For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(WorkerDict pid=1732013) Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(WorkerDict pid=1732013) 
(WorkerDict pid=1732013) 
(WorkerDict pid=1732013) [2025-05-21 01:27:52] Received sigquit from a child process. It usually means the child failed.
(TaskRunner pid=1714137) Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::WorkerDict.actor_rollout_generate_sequences() (pid=1733173, ip=100.96.253.216, actor_id=537b3c0b0720bff85b98982501000000, repr=<verl.single_controller.ray.base.WorkerDict object at 0x7757c3aafa00>)
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/single_controller/ray/base.py", line 625, in func
(TaskRunner pid=1714137)     return getattr(self.worker_dict[key], name)(*args, **kwargs)
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/single_controller/base/decorator.py", line 534, in inner
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/workers/fsdp_workers.py", line 630, in generate_sequences
(TaskRunner pid=1714137)     output = self.rollout.generate_sequences_with_tools(prompts=prompts)
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/utils/debug/performance.py", line 78, in f
(TaskRunner pid=1714137)     return self.log(decorated_function, *args, **kwargs)
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/utils/debug/performance.py", line 88, in log
(TaskRunner pid=1714137)     output = func(*args, **kwargs)
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/workers/rollout/sglang_rollout/async_sglang_rollout.py", line 619, in generate_sequences_with_tools
(TaskRunner pid=1714137)     [sorted_output_req_list] = broadcast_pyobj(
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/workers/rollout/sglang_rollout/utils.py", line 55, in broadcast_pyobj
(TaskRunner pid=1714137)     dist.broadcast(tensor_size, src=src, group=dist_group)
(TaskRunner pid=1714137)   File "/home/jobuser/.local/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
(TaskRunner pid=1714137)   File "/home/jobuser/.local/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2726, in broadcast
(TaskRunner pid=1714137)     work = group.broadcast([tensor], opts)
(TaskRunner pid=1714137) torch.distributed.DistBackendError: [7] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes
(TaskRunner pid=1714137) Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:671 (most recent call first):
(TaskRunner pid=1714137) frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x778adc76c1b6 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libc10.so)
(TaskRunner pid=1714137) frame #1: <unknown function> + 0x6312a88 (0x775bc1256a88 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x367 (0x775bc1253817 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x775bc1253b9a in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x775bc125493a in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x775bc12015d1 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x775bc12015d1 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x775bc12015d1 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x775bc12015d1 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x775b86f91a9f in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
(TaskRunner pid=1714137) frame #10: c10d::ProcessGroupNCCL::initNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x1176 (0x775b86f9f666 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
(TaskRunner pid=1714137) frame #11: c10d::ProcessGroupNCCL::broadcast(std::vector<at::Tensor, std::allocator<at::Tensor> >&, c10d::BroadcastOptions const&) + 0x1265 (0x775b86faa175 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
(TaskRunner pid=1714137) frame #12: <unknown function> + 0x62b13b6 (0x775bc11f53b6 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #13: <unknown function> + 0x62bcd26 (0x775bc1200d26 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #14: <unknown function> + 0x58bc27b (0x775bc080027b in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #15: <unknown function> + 0x58b9af9 (0x775bc07fdaf9 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #16: <unknown function> + 0x1aca9f8 (0x775bbca0e9f8 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #17: <unknown function> + 0x62c339a (0x775bc120739a in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #18: <unknown function> + 0x62d3b30 (0x775bc1217b30 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #19: <unknown function> + 0xe9fac5 (0x778ac792eac5 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
(TaskRunner pid=1714137) frame #20: <unknown function> + 0x518d37 (0x778ac6fa7d37 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
(TaskRunner pid=1714137) <omitting python frames>
(TaskRunner pid=1714137) frame #55: <unknown function> + 0x769d4f (0x778b78617d4f in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #56: <unknown function> + 0x7daaa8 (0x778b78688aa8 in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #57: <unknown function> + 0x769d4f (0x778b78617d4f in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #58: <unknown function> + 0x86fc56 (0x778b7871dc56 in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #59: std::_Function_handler<ray::Status (ray::rpc::Address const&, ray::rpc::TaskType, std::string, ray::core::RayFunction const&, std::unordered_map<std::string, double, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, double> > > const&, std::vector<std::shared_ptr<ray::RayObject>, std::allocator<std::shared_ptr<ray::RayObject> > > const&, std::vector<ray::rpc::ObjectReference, std::allocator<ray::rpc::ObjectReference> > const&, std::string const&, std::string const&, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*, std::shared_ptr<ray::LocalMemoryBuffer>&, bool*, std::string*, std::vector<ray::ConcurrencyGroup, std::allocator<ray::ConcurrencyGroup> > const&, std::string, bool, bool, bool, long), ray::Status (*)(ray::rpc::Address const&, ray::rpc::TaskType, std::string, ray::core::RayFunction const&, std::unordered_map<std::string, double, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, double> > > const&, std::vector<std::shared_ptr<ray::RayObject>, std::allocator<std::shared_ptr<ray::RayObject> > > const&, std::vector<ray::rpc::ObjectReference, std::allocator<ray::rpc::ObjectReference> > const&, std::string, std::string, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*, std::shared_ptr<ray::LocalMemoryBuffer>&, bool*, std::string*, std::vector<ray::ConcurrencyGroup, std::allocator<ray::ConcurrencyGroup> > const&, std::string, bool, bool, bool, long)>::_M_invoke(std::_Any_data const&, ray::rpc::Address const&, ray::rpc::TaskType&&, std::string&&, ray::core::RayFunction const&, std::unordered_map<std::string, double, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, double> > > const&, std::vector<std::shared_ptr<ray::RayObject>, std::allocator<std::shared_ptr<ray::RayObject> > > const&, std::vector<ray::rpc::ObjectReference, std::allocator<ray::rpc::ObjectReference> > const&, std::string const&, std::string const&, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*&&, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*&&, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*&&, std::shared_ptr<ray::LocalMemoryBuffer>&, bool*&&, std::string*&&, std::vector<ray::ConcurrencyGroup, std::allocator<ray::ConcurrencyGroup> > const&, std::string&&, bool&&, bool&&, bool&&, long&&) + 0x169 (0x778b7861ed39 in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #60: ray::core::CoreWorker::ExecuteTask(ray::TaskSpecification const&, std::optional<std::unordered_map<std::string, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > > > > > >, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*, google::protobuf::RepeatedPtrField<ray::rpc::ObjectReferenceCount>*, bool*, std::string*) + 0xdf5 (0x778b78830245 in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #61: std::_Function_handler<ray::Status (ray::TaskSpecification const&, std::optional<std::unordered_map<std::string, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > > > > > >, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*, google::protobuf::RepeatedPtrField<ray::rpc::ObjectReferenceCount>*, bool*, std::string*), std::_Bind<ray::Status (ray::core::CoreWorker::*(ray::core::CoreWorker*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Placeholder<3>, std::_Placeholder<4>, std::_Placeholder<5>, std::_Placeholder<6>, std::_Placeholder<7>, std::_Placeholder<8>))(ray::TaskSpecification const&, std::optional<std::unordered_map<std::string, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > > > > > >, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*, google::protobuf::RepeatedPtrField<ray::rpc::ObjectReferenceCount>*, bool*, std::string*)> >::_M_invoke(std::_Any_data const&, ray::TaskSpecification const&, std::optional<std::unordered_map<std::string, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > > > > > >&&, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*&&, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*&&, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*&&, google::protobuf::RepeatedPtrField<ray::rpc::ObjectReferenceCount>*&&, bool*&&, std::string*&&) + 0x75 (0x778b787ba845 in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #62: <unknown function> + 0x9f30ce (0x778b788a10ce in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) . This may indicate a possible application crash on rank 0 or a network set up issue.
(TaskRunner pid=1714137) Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::WorkerDict.actor_rollout_generate_sequences() (pid=1733172, ip=100.96.253.216, actor_id=b418c34c9373f957e605c79001000000, repr=<verl.single_controller.ray.base.WorkerDict object at 0x738b18890640>)
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/single_controller/ray/base.py", line 625, in func
(TaskRunner pid=1714137)     return getattr(self.worker_dict[key], name)(*args, **kwargs)
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/single_controller/base/decorator.py", line 534, in inner
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/workers/fsdp_workers.py", line 630, in generate_sequences
(TaskRunner pid=1714137)     output = self.rollout.generate_sequences_with_tools(prompts=prompts)
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/utils/debug/performance.py", line 78, in f
(TaskRunner pid=1714137)     return self.log(decorated_function, *args, **kwargs)
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/utils/debug/performance.py", line 88, in log
(TaskRunner pid=1714137)     output = func(*args, **kwargs)
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/workers/rollout/sglang_rollout/async_sglang_rollout.py", line 619, in generate_sequences_with_tools
(TaskRunner pid=1714137)     [sorted_output_req_list] = broadcast_pyobj(
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/workers/rollout/sglang_rollout/utils.py", line 55, in broadcast_pyobj
(TaskRunner pid=1714137)     dist.broadcast(tensor_size, src=src, group=dist_group)
(TaskRunner pid=1714137)   File "/home/jobuser/.local/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
(TaskRunner pid=1714137)   File "/home/jobuser/.local/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2726, in broadcast
(TaskRunner pid=1714137)     work = group.broadcast([tensor], opts)
(TaskRunner pid=1714137) torch.distributed.DistBackendError: [6] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes
(TaskRunner pid=1714137) Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:671 (most recent call first):
(TaskRunner pid=1714137) frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x73be1396c1b6 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libc10.so)
(TaskRunner pid=1714137) frame #1: <unknown function> + 0x6312a88 (0x738f1d256a88 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x367 (0x738f1d253817 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x738f1d253b9a in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x738f1d25493a in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x738f1d2015d1 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x738f1d2015d1 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x738f1d2015d1 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x738f1d2015d1 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x738ee2f91a9f in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
(TaskRunner pid=1714137) frame #10: c10d::ProcessGroupNCCL::initNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x1176 (0x738ee2f9f666 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
(TaskRunner pid=1714137) frame #11: c10d::ProcessGroupNCCL::broadcast(std::vector<at::Tensor, std::allocator<at::Tensor> >&, c10d::BroadcastOptions const&) + 0x1265 (0x738ee2faa175 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
(TaskRunner pid=1714137) frame #12: <unknown function> + 0x62b13b6 (0x738f1d1f53b6 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #13: <unknown function> + 0x62bcd26 (0x738f1d200d26 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #14: <unknown function> + 0x58bc27b (0x738f1c80027b in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #15: <unknown function> + 0x58b9af9 (0x738f1c7fdaf9 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #16: <unknown function> + 0x1aca9f8 (0x738f18a0e9f8 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #17: <unknown function> + 0x62c339a (0x738f1d20739a in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #18: <unknown function> + 0x62d3b30 (0x738f1d217b30 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #19: <unknown function> + 0xe9fac5 (0x73be12f2eac5 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
(TaskRunner pid=1714137) frame #20: <unknown function> + 0x518d37 (0x73be125a7d37 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
(TaskRunner pid=1714137) <omitting python frames>
(TaskRunner pid=1714137) frame #55: <unknown function> + 0x769d4f (0x73bec3400d4f in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #56: <unknown function> + 0x7daaa8 (0x73bec3471aa8 in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #57: <unknown function> + 0x769d4f (0x73bec3400d4f in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #58: <unknown function> + 0x86fc56 (0x73bec3506c56 in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #59: std::_Function_handler<ray::Status (ray::rpc::Address const&, ray::rpc::TaskType, std::string, ray::core::RayFunction const&, std::unordered_map<std::string, double, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, double> > > const&, std::vector<std::shared_ptr<ray::RayObject>, std::allocator<std::shared_ptr<ray::RayObject> > > const&, std::vector<ray::rpc::ObjectReference, std::allocator<ray::rpc::ObjectReference> > const&, std::string const&, std::string const&, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*, std::shared_ptr<ray::LocalMemoryBuffer>&, bool*, std::string*, std::vector<ray::ConcurrencyGroup, std::allocator<ray::ConcurrencyGroup> > const&, std::string, bool, bool, bool, long), ray::Status (*)(ray::rpc::Address const&, ray::rpc::TaskType, std::string, ray::core::RayFunction const&, std::unordered_map<std::string, double, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, double> > > const&, std::vector<std::shared_ptr<ray::RayObject>, std::allocator<std::shared_ptr<ray::RayObject> > > const&, std::vector<ray::rpc::ObjectReference, std::allocator<ray::rpc::ObjectReference> > const&, std::string, std::string, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*, std::shared_ptr<ray::LocalMemoryBuffer>&, bool*, std::string*, std::vector<ray::ConcurrencyGroup, std::allocator<ray::ConcurrencyGroup> > const&, std::string, bool, bool, bool, long)>::_M_invoke(std::_Any_data const&, ray::rpc::Address const&, ray::rpc::TaskType&&, std::string&&, ray::core::RayFunction const&, std::unordered_map<std::string, double, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, double> > > const&, std::vector<std::shared_ptr<ray::RayObject>, std::allocator<std::shared_ptr<ray::RayObject> > > const&, std::vector<ray::rpc::ObjectReference, std::allocator<ray::rpc::ObjectReference> > const&, std::string const&, std::string const&, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*&&, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*&&, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*&&, std::shared_ptr<ray::LocalMemoryBuffer>&, bool*&&, std::string*&&, std::vector<ray::ConcurrencyGroup, std::allocator<ray::ConcurrencyGroup> > const&, std::string&&, bool&&, bool&&, bool&&, long&&) + 0x169 (0x73bec3407d39 in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #60: ray::core::CoreWorker::ExecuteTask(ray::TaskSpecification const&, std::optional<std::unordered_map<std::string, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > > > > > >, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*, google::protobuf::RepeatedPtrField<ray::rpc::ObjectReferenceCount>*, bool*, std::string*) + 0xdf5 (0x73bec3619245 in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #61: std::_Function_handler<ray::Status (ray::TaskSpecification const&, std::optional<std::unordered_map<std::string, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > > > > > >, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*, google::protobuf::RepeatedPtrField<ray::rpc::ObjectReferenceCount>*, bool*, std::string*), std::_Bind<ray::Status (ray::core::CoreWorker::*(ray::core::CoreWorker*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Placeholder<3>, std::_Placeholder<4>, std::_Placeholder<5>, std::_Placeholder<6>, std::_Placeholder<7>, std::_Placeholder<8>))(ray::TaskSpecification const&, std::optional<std::unordered_map<std::string, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > > > > > >, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*, google::protobuf::RepeatedPtrField<ray::rpc::ObjectReferenceCount>*, bool*, std::string*)> >::_M_invoke(std::_Any_data const&, ray::TaskSpecification const&, std::optional<std::unordered_map<std::string, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > > > > > >&&, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*&&, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*&&, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*&&, google::protobuf::RepeatedPtrField<ray::rpc::ObjectReferenceCount>*&&, bool*&&, std::string*&&) + 0x75 (0x73bec35a3845 in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #62: <unknown function> + 0x9f30ce (0x73bec368a0ce in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) . This may indicate a possible application crash on rank 0 or a network set up issue.
(TaskRunner pid=1714137) Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::WorkerDict.actor_rollout_generate_sequences() (pid=1733171, ip=100.96.253.216, actor_id=d37e3c945e454da0bd149ec401000000, repr=<verl.single_controller.ray.base.WorkerDict object at 0x7cf46cb78640>)
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/single_controller/ray/base.py", line 625, in func
(TaskRunner pid=1714137)     return getattr(self.worker_dict[key], name)(*args, **kwargs)
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/single_controller/base/decorator.py", line 534, in inner
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/workers/fsdp_workers.py", line 630, in generate_sequences
(TaskRunner pid=1714137)     output = self.rollout.generate_sequences_with_tools(prompts=prompts)
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/utils/debug/performance.py", line 78, in f
(TaskRunner pid=1714137)     return self.log(decorated_function, *args, **kwargs)
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/utils/debug/performance.py", line 88, in log
(TaskRunner pid=1714137)     output = func(*args, **kwargs)
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/workers/rollout/sglang_rollout/async_sglang_rollout.py", line 619, in generate_sequences_with_tools
(TaskRunner pid=1714137)     [sorted_output_req_list] = broadcast_pyobj(
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/workers/rollout/sglang_rollout/utils.py", line 55, in broadcast_pyobj
(TaskRunner pid=1714137)     dist.broadcast(tensor_size, src=src, group=dist_group)
(TaskRunner pid=1714137)   File "/home/jobuser/.local/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
(TaskRunner pid=1714137)   File "/home/jobuser/.local/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2726, in broadcast
(TaskRunner pid=1714137)     work = group.broadcast([tensor], opts)
(TaskRunner pid=1714137) torch.distributed.DistBackendError: [5] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes
(TaskRunner pid=1714137) Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:671 (most recent call first):
(TaskRunner pid=1714137) frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7d277bb6c1b6 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libc10.so)
(TaskRunner pid=1714137) frame #1: <unknown function> + 0x6312a88 (0x7cf85d256a88 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x367 (0x7cf85d253817 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7cf85d253b9a in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7cf85d25493a in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7cf85d2015d1 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7cf85d2015d1 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7cf85d2015d1 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7cf85d2015d1 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7cf822f91a9f in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
(TaskRunner pid=1714137) frame #10: c10d::ProcessGroupNCCL::initNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x1176 (0x7cf822f9f666 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
(TaskRunner pid=1714137) frame #11: c10d::ProcessGroupNCCL::broadcast(std::vector<at::Tensor, std::allocator<at::Tensor> >&, c10d::BroadcastOptions const&) + 0x1265 (0x7cf822faa175 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
(TaskRunner pid=1714137) frame #12: <unknown function> + 0x62b13b6 (0x7cf85d1f53b6 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #13: <unknown function> + 0x62bcd26 (0x7cf85d200d26 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #14: <unknown function> + 0x58bc27b (0x7cf85c80027b in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #15: <unknown function> + 0x58b9af9 (0x7cf85c7fdaf9 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #16: <unknown function> + 0x1aca9f8 (0x7cf858a0e9f8 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #17: <unknown function> + 0x62c339a (0x7cf85d20739a in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #18: <unknown function> + 0x62d3b30 (0x7cf85d217b30 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #19: <unknown function> + 0xe9fac5 (0x7d276b92eac5 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
(TaskRunner pid=1714137) frame #20: <unknown function> + 0x518d37 (0x7d276afa7d37 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
(TaskRunner pid=1714137) <omitting python frames>
(TaskRunner pid=1714137) frame #55: <unknown function> + 0x769d4f (0x7d28176a5d4f in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #56: <unknown function> + 0x7daaa8 (0x7d2817716aa8 in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #57: <unknown function> + 0x769d4f (0x7d28176a5d4f in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #58: <unknown function> + 0x86fc56 (0x7d28177abc56 in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #59: std::_Function_handler<ray::Status (ray::rpc::Address const&, ray::rpc::TaskType, std::string, ray::core::RayFunction const&, std::unordered_map<std::string, double, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, double> > > const&, std::vector<std::shared_ptr<ray::RayObject>, std::allocator<std::shared_ptr<ray::RayObject> > > const&, std::vector<ray::rpc::ObjectReference, std::allocator<ray::rpc::ObjectReference> > const&, std::string const&, std::string const&, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*, std::shared_ptr<ray::LocalMemoryBuffer>&, bool*, std::string*, std::vector<ray::ConcurrencyGroup, std::allocator<ray::ConcurrencyGroup> > const&, std::string, bool, bool, bool, long), ray::Status (*)(ray::rpc::Address const&, ray::rpc::TaskType, std::string, ray::core::RayFunction const&, std::unordered_map<std::string, double, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, double> > > const&, std::vector<std::shared_ptr<ray::RayObject>, std::allocator<std::shared_ptr<ray::RayObject> > > const&, std::vector<ray::rpc::ObjectReference, std::allocator<ray::rpc::ObjectReference> > const&, std::string, std::string, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*, std::shared_ptr<ray::LocalMemoryBuffer>&, bool*, std::string*, std::vector<ray::ConcurrencyGroup, std::allocator<ray::ConcurrencyGroup> > const&, std::string, bool, bool, bool, long)>::_M_invoke(std::_Any_data const&, ray::rpc::Address const&, ray::rpc::TaskType&&, std::string&&, ray::core::RayFunction const&, std::unordered_map<std::string, double, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, double> > > const&, std::vector<std::shared_ptr<ray::RayObject>, std::allocator<std::shared_ptr<ray::RayObject> > > const&, std::vector<ray::rpc::ObjectReference, std::allocator<ray::rpc::ObjectReference> > const&, std::string const&, std::string const&, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*&&, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*&&, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*&&, std::shared_ptr<ray::LocalMemoryBuffer>&, bool*&&, std::string*&&, std::vector<ray::ConcurrencyGroup, std::allocator<ray::ConcurrencyGroup> > const&, std::string&&, bool&&, bool&&, bool&&, long&&) + 0x169 (0x7d28176acd39 in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #60: ray::core::CoreWorker::ExecuteTask(ray::TaskSpecification const&, std::optional<std::unordered_map<std::string, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > > > > > >, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*, google::protobuf::RepeatedPtrField<ray::rpc::ObjectReferenceCount>*, bool*, std::string*) + 0xdf5 (0x7d28178be245 in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #61: std::_Function_handler<ray::Status (ray::TaskSpecification const&, std::optional<std::unordered_map<std::string, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > > > > > >, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*, google::protobuf::RepeatedPtrField<ray::rpc::ObjectReferenceCount>*, bool*, std::string*), std::_Bind<ray::Status (ray::core::CoreWorker::*(ray::core::CoreWorker*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Placeholder<3>, std::_Placeholder<4>, std::_Placeholder<5>, std::_Placeholder<6>, std::_Placeholder<7>, std::_Placeholder<8>))(ray::TaskSpecification const&, std::optional<std::unordered_map<std::string, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > > > > > >, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*, google::protobuf::RepeatedPtrField<ray::rpc::ObjectReferenceCount>*, bool*, std::string*)> >::_M_invoke(std::_Any_data const&, ray::TaskSpecification const&, std::optional<std::unordered_map<std::string, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > > > > > >&&, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*&&, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*&&, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*&&, google::protobuf::RepeatedPtrField<ray::rpc::ObjectReferenceCount>*&&, bool*&&, std::string*&&) + 0x75 (0x7d2817848845 in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #62: <unknown function> + 0x9f30ce (0x7d281792f0ce in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) . This may indicate a possible application crash on rank 0 or a network set up issue.
(TaskRunner pid=1714137) Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::WorkerDict.actor_rollout_generate_sequences() (pid=1733169, ip=100.96.253.216, actor_id=4cb9864143683f98469fe90101000000, repr=<verl.single_controller.ray.base.WorkerDict object at 0x74b2dd004610>)
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/single_controller/ray/base.py", line 625, in func
(TaskRunner pid=1714137)     return getattr(self.worker_dict[key], name)(*args, **kwargs)
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/single_controller/base/decorator.py", line 534, in inner
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/workers/fsdp_workers.py", line 630, in generate_sequences
(TaskRunner pid=1714137)     output = self.rollout.generate_sequences_with_tools(prompts=prompts)
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/utils/debug/performance.py", line 78, in f
(TaskRunner pid=1714137)     return self.log(decorated_function, *args, **kwargs)
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/utils/debug/performance.py", line 88, in log
(TaskRunner pid=1714137)     output = func(*args, **kwargs)
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/workers/rollout/sglang_rollout/async_sglang_rollout.py", line 619, in generate_sequences_with_tools
(TaskRunner pid=1714137)     [sorted_output_req_list] = broadcast_pyobj(
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/workers/rollout/sglang_rollout/utils.py", line 55, in broadcast_pyobj
(TaskRunner pid=1714137)     dist.broadcast(tensor_size, src=src, group=dist_group)
(TaskRunner pid=1714137)   File "/home/jobuser/.local/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
(TaskRunner pid=1714137)   File "/home/jobuser/.local/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2726, in broadcast
(TaskRunner pid=1714137)     work = group.broadcast([tensor], opts)
(TaskRunner pid=1714137) torch.distributed.DistBackendError: [3] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes
(TaskRunner pid=1714137) Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:671 (most recent call first):
(TaskRunner pid=1714137) frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x74b720f6c1b6 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libc10.so)
(TaskRunner pid=1714137) frame #1: <unknown function> + 0x6312a88 (0x74b7006e5a88 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x367 (0x74b7006e2817 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x74b7006e2b9a in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x74b7006e393a in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x74b7006905d1 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x74b7006905d1 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x74b7006905d1 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x74b7006905d1 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x74b6c6420a9f in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
(TaskRunner pid=1714137) frame #10: c10d::ProcessGroupNCCL::initNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x1176 (0x74b6c642e666 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
(TaskRunner pid=1714137) frame #11: c10d::ProcessGroupNCCL::broadcast(std::vector<at::Tensor, std::allocator<at::Tensor> >&, c10d::BroadcastOptions const&) + 0x1265 (0x74b6c6439175 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
(TaskRunner pid=1714137) frame #12: <unknown function> + 0x62b13b6 (0x74b7006843b6 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #13: <unknown function> + 0x62bcd26 (0x74b70068fd26 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #14: <unknown function> + 0x58bc27b (0x74b6ffc8f27b in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #15: <unknown function> + 0x58b9af9 (0x74b6ffc8caf9 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #16: <unknown function> + 0x1aca9f8 (0x74b6fbe9d9f8 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #17: <unknown function> + 0x62c339a (0x74b70069639a in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #18: <unknown function> + 0x62d3b30 (0x74b7006a6b30 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #19: <unknown function> + 0xe9fac5 (0x74b71032eac5 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
(TaskRunner pid=1714137) frame #20: <unknown function> + 0x518d37 (0x74b70f9a7d37 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
(TaskRunner pid=1714137) <omitting python frames>
(TaskRunner pid=1714137) frame #55: <unknown function> + 0x769d4f (0x74e691c51d4f in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #56: <unknown function> + 0x7daaa8 (0x74e691cc2aa8 in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #57: <unknown function> + 0x769d4f (0x74e691c51d4f in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #58: <unknown function> + 0x86fc56 (0x74e691d57c56 in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #59: std::_Function_handler<ray::Status (ray::rpc::Address const&, ray::rpc::TaskType, std::string, ray::core::RayFunction const&, std::unordered_map<std::string, double, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, double> > > const&, std::vector<std::shared_ptr<ray::RayObject>, std::allocator<std::shared_ptr<ray::RayObject> > > const&, std::vector<ray::rpc::ObjectReference, std::allocator<ray::rpc::ObjectReference> > const&, std::string const&, std::string const&, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*, std::shared_ptr<ray::LocalMemoryBuffer>&, bool*, std::string*, std::vector<ray::ConcurrencyGroup, std::allocator<ray::ConcurrencyGroup> > const&, std::string, bool, bool, bool, long), ray::Status (*)(ray::rpc::Address const&, ray::rpc::TaskType, std::string, ray::core::RayFunction const&, std::unordered_map<std::string, double, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, double> > > const&, std::vector<std::shared_ptr<ray::RayObject>, std::allocator<std::shared_ptr<ray::RayObject> > > const&, std::vector<ray::rpc::ObjectReference, std::allocator<ray::rpc::ObjectReference> > const&, std::string, std::string, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*, std::shared_ptr<ray::LocalMemoryBuffer>&, bool*, std::string*, std::vector<ray::ConcurrencyGroup, std::allocator<ray::ConcurrencyGroup> > const&, std::string, bool, bool, bool, long)>::_M_invoke(std::_Any_data const&, ray::rpc::Address const&, ray::rpc::TaskType&&, std::string&&, ray::core::RayFunction const&, std::unordered_map<std::string, double, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, double> > > const&, std::vector<std::shared_ptr<ray::RayObject>, std::allocator<std::shared_ptr<ray::RayObject> > > const&, std::vector<ray::rpc::ObjectReference, std::allocator<ray::rpc::ObjectReference> > const&, std::string const&, std::string const&, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*&&, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*&&, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*&&, std::shared_ptr<ray::LocalMemoryBuffer>&, bool*&&, std::string*&&, std::vector<ray::ConcurrencyGroup, std::allocator<ray::ConcurrencyGroup> > const&, std::string&&, bool&&, bool&&, bool&&, long&&) + 0x169 (0x74e691c58d39 in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #60: ray::core::CoreWorker::ExecuteTask(ray::TaskSpecification const&, std::optional<std::unordered_map<std::string, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > > > > > >, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*, google::protobuf::RepeatedPtrField<ray::rpc::ObjectReferenceCount>*, bool*, std::string*) + 0xdf5 (0x74e691e6a245 in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #61: std::_Function_handler<ray::Status (ray::TaskSpecification const&, std::optional<std::unordered_map<std::string, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > > > > > >, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*, google::protobuf::RepeatedPtrField<ray::rpc::ObjectReferenceCount>*, bool*, std::string*), std::_Bind<ray::Status (ray::core::CoreWorker::*(ray::core::CoreWorker*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Placeholder<3>, std::_Placeholder<4>, std::_Placeholder<5>, std::_Placeholder<6>, std::_Placeholder<7>, std::_Placeholder<8>))(ray::TaskSpecification const&, std::optional<std::unordered_map<std::string, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > > > > > >, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*, google::protobuf::RepeatedPtrField<ray::rpc::ObjectReferenceCount>*, bool*, std::string*)> >::_M_invoke(std::_Any_data const&, ray::TaskSpecification const&, std::optional<std::unordered_map<std::string, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > > > > > >&&, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*&&, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*&&, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*&&, google::protobuf::RepeatedPtrField<ray::rpc::ObjectReferenceCount>*&&, bool*&&, std::string*&&) + 0x75 (0x74e691df4845 in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #62: <unknown function> + 0x9f30ce (0x74e691edb0ce in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) . This may indicate a possible application crash on rank 0 or a network set up issue.
(TaskRunner pid=1714137) Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::WorkerDict.actor_rollout_generate_sequences() (pid=1733168, ip=100.96.253.216, actor_id=82c7bbaa109872e66aff35e701000000, repr=<verl.single_controller.ray.base.WorkerDict object at 0x7fc525913a00>)
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/single_controller/ray/base.py", line 625, in func
(TaskRunner pid=1714137)     return getattr(self.worker_dict[key], name)(*args, **kwargs)
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/single_controller/base/decorator.py", line 534, in inner
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/workers/fsdp_workers.py", line 630, in generate_sequences
(TaskRunner pid=1714137)     output = self.rollout.generate_sequences_with_tools(prompts=prompts)
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/utils/debug/performance.py", line 78, in f
(TaskRunner pid=1714137)     return self.log(decorated_function, *args, **kwargs)
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/utils/debug/performance.py", line 88, in log
(TaskRunner pid=1714137)     output = func(*args, **kwargs)
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/workers/rollout/sglang_rollout/async_sglang_rollout.py", line 619, in generate_sequences_with_tools
(TaskRunner pid=1714137)     [sorted_output_req_list] = broadcast_pyobj(
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/workers/rollout/sglang_rollout/utils.py", line 55, in broadcast_pyobj
(TaskRunner pid=1714137)     dist.broadcast(tensor_size, src=src, group=dist_group)
(TaskRunner pid=1714137)   File "/home/jobuser/.local/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
(TaskRunner pid=1714137)   File "/home/jobuser/.local/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2726, in broadcast
(TaskRunner pid=1714137)     work = group.broadcast([tensor], opts)
(TaskRunner pid=1714137) torch.distributed.DistBackendError: [2] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes
(TaskRunner pid=1714137) Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:671 (most recent call first):
(TaskRunner pid=1714137) frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7ff81d1fb1b6 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libc10.so)
(TaskRunner pid=1714137) frame #1: <unknown function> + 0x6312a88 (0x7fc939256a88 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x367 (0x7fc939253817 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fc939253b9a in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fc93925493a in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc9392015d1 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc9392015d1 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc9392015d1 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc9392015d1 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fc8fef91a9f in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
(TaskRunner pid=1714137) frame #10: c10d::ProcessGroupNCCL::initNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x1176 (0x7fc8fef9f666 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
(TaskRunner pid=1714137) frame #11: c10d::ProcessGroupNCCL::broadcast(std::vector<at::Tensor, std::allocator<at::Tensor> >&, c10d::BroadcastOptions const&) + 0x1265 (0x7fc8fefaa175 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
(TaskRunner pid=1714137) frame #12: <unknown function> + 0x62b13b6 (0x7fc9391f53b6 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #13: <unknown function> + 0x62bcd26 (0x7fc939200d26 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #14: <unknown function> + 0x58bc27b (0x7fc93880027b in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #15: <unknown function> + 0x58b9af9 (0x7fc9387fdaf9 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #16: <unknown function> + 0x1aca9f8 (0x7fc934a0e9f8 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #17: <unknown function> + 0x62c339a (0x7fc93920739a in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #18: <unknown function> + 0x62d3b30 (0x7fc939217b30 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #19: <unknown function> + 0xe9fac5 (0x7ff81e12eac5 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
(TaskRunner pid=1714137) frame #20: <unknown function> + 0x518d37 (0x7ff81d7a7d37 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
(TaskRunner pid=1714137) <omitting python frames>
(TaskRunner pid=1714137) frame #55: <unknown function> + 0x769d4f (0x7ff8d6573d4f in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #56: <unknown function> + 0x7daaa8 (0x7ff8d65e4aa8 in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #57: <unknown function> + 0x769d4f (0x7ff8d6573d4f in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #58: <unknown function> + 0x86fc56 (0x7ff8d6679c56 in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #59: std::_Function_handler<ray::Status (ray::rpc::Address const&, ray::rpc::TaskType, std::string, ray::core::RayFunction const&, std::unordered_map<std::string, double, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, double> > > const&, std::vector<std::shared_ptr<ray::RayObject>, std::allocator<std::shared_ptr<ray::RayObject> > > const&, std::vector<ray::rpc::ObjectReference, std::allocator<ray::rpc::ObjectReference> > const&, std::string const&, std::string const&, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*, std::shared_ptr<ray::LocalMemoryBuffer>&, bool*, std::string*, std::vector<ray::ConcurrencyGroup, std::allocator<ray::ConcurrencyGroup> > const&, std::string, bool, bool, bool, long), ray::Status (*)(ray::rpc::Address const&, ray::rpc::TaskType, std::string, ray::core::RayFunction const&, std::unordered_map<std::string, double, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, double> > > const&, std::vector<std::shared_ptr<ray::RayObject>, std::allocator<std::shared_ptr<ray::RayObject> > > const&, std::vector<ray::rpc::ObjectReference, std::allocator<ray::rpc::ObjectReference> > const&, std::string, std::string, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*, std::shared_ptr<ray::LocalMemoryBuffer>&, bool*, std::string*, std::vector<ray::ConcurrencyGroup, std::allocator<ray::ConcurrencyGroup> > const&, std::string, bool, bool, bool, long)>::_M_invoke(std::_Any_data const&, ray::rpc::Address const&, ray::rpc::TaskType&&, std::string&&, ray::core::RayFunction const&, std::unordered_map<std::string, double, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, double> > > const&, std::vector<std::shared_ptr<ray::RayObject>, std::allocator<std::shared_ptr<ray::RayObject> > > const&, std::vector<ray::rpc::ObjectReference, std::allocator<ray::rpc::ObjectReference> > const&, std::string const&, std::string const&, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*&&, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*&&, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*&&, std::shared_ptr<ray::LocalMemoryBuffer>&, bool*&&, std::string*&&, std::vector<ray::ConcurrencyGroup, std::allocator<ray::ConcurrencyGroup> > const&, std::string&&, bool&&, bool&&, bool&&, long&&) + 0x169 (0x7ff8d657ad39 in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #60: ray::core::CoreWorker::ExecuteTask(ray::TaskSpecification const&, std::optional<std::unordered_map<std::string, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > > > > > >, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*, google::protobuf::RepeatedPtrField<ray::rpc::ObjectReferenceCount>*, bool*, std::string*) + 0xdf5 (0x7ff8d678c245 in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #61: std::_Function_handler<ray::Status (ray::TaskSpecification const&, std::optional<std::unordered_map<std::string, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > > > > > >, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*, google::protobuf::RepeatedPtrField<ray::rpc::ObjectReferenceCount>*, bool*, std::string*), std::_Bind<ray::Status (ray::core::CoreWorker::*(ray::core::CoreWorker*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Placeholder<3>, std::_Placeholder<4>, std::_Placeholder<5>, std::_Placeholder<6>, std::_Placeholder<7>, std::_Placeholder<8>))(ray::TaskSpecification const&, std::optional<std::unordered_map<std::string, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > > > > > >, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*, google::protobuf::RepeatedPtrField<ray::rpc::ObjectReferenceCount>*, bool*, std::string*)> >::_M_invoke(std::_Any_data const&, ray::TaskSpecification const&, std::optional<std::unordered_map<std::string, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > > > > > >&&, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*&&, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*&&, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*&&, google::protobuf::RepeatedPtrField<ray::rpc::ObjectReferenceCount>*&&, bool*&&, std::string*&&) + 0x75 (0x7ff8d6716845 in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #62: <unknown function> + 0x9f30ce (0x7ff8d67fd0ce in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) . This may indicate a possible application crash on rank 0 or a network set up issue.
(TaskRunner pid=1714137) Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::WorkerDict.actor_rollout_generate_sequences() (pid=1733167, ip=100.96.253.216, actor_id=82e75560ee1db7816b11a0f201000000, repr=<verl.single_controller.ray.base.WorkerDict object at 0x75b64a9dfa00>)
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/single_controller/ray/base.py", line 625, in func
(TaskRunner pid=1714137)     return getattr(self.worker_dict[key], name)(*args, **kwargs)
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/single_controller/base/decorator.py", line 534, in inner
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/workers/fsdp_workers.py", line 630, in generate_sequences
(TaskRunner pid=1714137)     output = self.rollout.generate_sequences_with_tools(prompts=prompts)
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/utils/debug/performance.py", line 78, in f
(TaskRunner pid=1714137)     return self.log(decorated_function, *args, **kwargs)
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/utils/debug/performance.py", line 88, in log
(TaskRunner pid=1714137)     output = func(*args, **kwargs)
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/workers/rollout/sglang_rollout/async_sglang_rollout.py", line 619, in generate_sequences_with_tools
(TaskRunner pid=1714137)     [sorted_output_req_list] = broadcast_pyobj(
(TaskRunner pid=1714137)   File "/home/jobuser/resources/verl/workers/rollout/sglang_rollout/utils.py", line 55, in broadcast_pyobj
(TaskRunner pid=1714137)     dist.broadcast(tensor_size, src=src, group=dist_group)
(TaskRunner pid=1714137)   File "/home/jobuser/.local/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
(TaskRunner pid=1714137)   File "/home/jobuser/.local/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2726, in broadcast
(TaskRunner pid=1714137)     work = group.broadcast([tensor], opts)
(TaskRunner pid=1714137) torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes
(TaskRunner pid=1714137) Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:671 (most recent call first):
(TaskRunner pid=1714137) frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x75b69036c1b6 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libc10.so)
(TaskRunner pid=1714137) frame #1: <unknown function> + 0x6312a88 (0x758765256a88 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x367 (0x758765253817 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x758765253b9a in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x75876525493a in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7587652015d1 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7587652015d1 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7587652015d1 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7587652015d1 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x75872af91a9f in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
(TaskRunner pid=1714137) frame #10: c10d::ProcessGroupNCCL::initNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x1176 (0x75872af9f666 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
(TaskRunner pid=1714137) frame #11: c10d::ProcessGroupNCCL::broadcast(std::vector<at::Tensor, std::allocator<at::Tensor> >&, c10d::BroadcastOptions const&) + 0x1265 (0x75872afaa175 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
(TaskRunner pid=1714137) frame #12: <unknown function> + 0x62b13b6 (0x7587651f53b6 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #13: <unknown function> + 0x62bcd26 (0x758765200d26 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #14: <unknown function> + 0x58bc27b (0x75876480027b in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #15: <unknown function> + 0x58b9af9 (0x7587647fdaf9 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #16: <unknown function> + 0x1aca9f8 (0x758760a0e9f8 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #17: <unknown function> + 0x62c339a (0x75876520739a in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #18: <unknown function> + 0x62d3b30 (0x758765217b30 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
(TaskRunner pid=1714137) frame #19: <unknown function> + 0xe9fac5 (0x75b67392eac5 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
(TaskRunner pid=1714137) frame #20: <unknown function> + 0x518d37 (0x75b672fa7d37 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
(TaskRunner pid=1714137) <omitting python frames>
(TaskRunner pid=1714137) frame #55: <unknown function> + 0x769d4f (0x75b720a41d4f in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #56: <unknown function> + 0x7daaa8 (0x75b720ab2aa8 in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #57: <unknown function> + 0x769d4f (0x75b720a41d4f in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #58: <unknown function> + 0x86fc56 (0x75b720b47c56 in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #59: std::_Function_handler<ray::Status (ray::rpc::Address const&, ray::rpc::TaskType, std::string, ray::core::RayFunction const&, std::unordered_map<std::string, double, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, double> > > const&, std::vector<std::shared_ptr<ray::RayObject>, std::allocator<std::shared_ptr<ray::RayObject> > > const&, std::vector<ray::rpc::ObjectReference, std::allocator<ray::rpc::ObjectReference> > const&, std::string const&, std::string const&, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*, std::shared_ptr<ray::LocalMemoryBuffer>&, bool*, std::string*, std::vector<ray::ConcurrencyGroup, std::allocator<ray::ConcurrencyGroup> > const&, std::string, bool, bool, bool, long), ray::Status (*)(ray::rpc::Address const&, ray::rpc::TaskType, std::string, ray::core::RayFunction const&, std::unordered_map<std::string, double, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, double> > > const&, std::vector<std::shared_ptr<ray::RayObject>, std::allocator<std::shared_ptr<ray::RayObject> > > const&, std::vector<ray::rpc::ObjectReference, std::allocator<ray::rpc::ObjectReference> > const&, std::string, std::string, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*, std::shared_ptr<ray::LocalMemoryBuffer>&, bool*, std::string*, std::vector<ray::ConcurrencyGroup, std::allocator<ray::ConcurrencyGroup> > const&, std::string, bool, bool, bool, long)>::_M_invoke(std::_Any_data const&, ray::rpc::Address const&, ray::rpc::TaskType&&, std::string&&, ray::core::RayFunction const&, std::unordered_map<std::string, double, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, double> > > const&, std::vector<std::shared_ptr<ray::RayObject>, std::allocator<std::shared_ptr<ray::RayObject> > > const&, std::vector<ray::rpc::ObjectReference, std::allocator<ray::rpc::ObjectReference> > const&, std::string const&, std::string const&, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*&&, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*&&, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*&&, std::shared_ptr<ray::LocalMemoryBuffer>&, bool*&&, std::string*&&, std::vector<ray::ConcurrencyGroup, std::allocator<ray::ConcurrencyGroup> > const&, std::string&&, bool&&, bool&&, bool&&, long&&) + 0x169 (0x75b720a48d39 in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #60: ray::core::CoreWorker::ExecuteTask(ray::TaskSpecification const&, std::optional<std::unordered_map<std::string, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > > > > > >, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*, google::protobuf::RepeatedPtrField<ray::rpc::ObjectReferenceCount>*, bool*, std::string*) + 0xdf5 (0x75b720c5a245 in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #61: std::_Function_handler<ray::Status (ray::TaskSpecification const&, std::optional<std::unordered_map<std::string, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > > > > > >, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*, google::protobuf::RepeatedPtrField<ray::rpc::ObjectReferenceCount>*, bool*, std::string*), std::_Bind<ray::Status (ray::core::CoreWorker::*(ray::core::CoreWorker*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Placeholder<3>, std::_Placeholder<4>, std::_Placeholder<5>, std::_Placeholder<6>, std::_Placeholder<7>, std::_Placeholder<8>))(ray::TaskSpecification const&, std::optional<std::unordered_map<std::string, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > > > > > >, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*, google::protobuf::RepeatedPtrField<ray::rpc::ObjectReferenceCount>*, bool*, std::string*)> >::_M_invoke(std::_Any_data const&, ray::TaskSpecification const&, std::optional<std::unordered_map<std::string, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > > > > > >&&, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*&&, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*&&, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*&&, google::protobuf::RepeatedPtrField<ray::rpc::ObjectReferenceCount>*&&, bool*&&, std::string*&&) + 0x75 (0x75b720be4845 in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) frame #62: <unknown function> + 0x9f30ce (0x75b720ccb0ce in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
(TaskRunner pid=1714137) . This may indicate a possible application crash on rank 0 or a network set up issue.
(WorkerDict pid=1732013) }
(WorkerDict pid=1732013) 
(WorkerDict pid=1733170) Actor use_remove_padding=True [repeated 7x across cluster]
(WorkerDict pid=1733170) Actor use_fused_kernels=False [repeated 7x across cluster]
(WorkerDict pid=1732013) Monkey patch _flash_attention_forward in transformers.integrations.flash_attention
(WorkerDict pid=1733168) Monkey patch _flash_attention_forward in transformers.integrations.flash_attention
(WorkerDict pid=1733172) Monkey patch _flash_attention_forward in transformers.integrations.flash_attention [repeated 2x across cluster]
(WorkerDict pid=1732013) Qwen2ForCausalLM contains 32.76B parameters
(WorkerDict pid=1732013) wrap_policy: functools.partial(<function _or_policy at 0x71c356939000>, policies=[functools.partial(<function transformer_auto_wrap_policy at 0x71c356938ee0>, transformer_layer_cls={<class 'transformers.models.qwen2.modeling_qwen2.Qwen2DecoderLayer'>})])
(WorkerDict pid=1733171) Monkey patch _flash_attention_forward in transformers.integrations.flash_attention [repeated 3x across cluster]
(WorkerDict pid=1733167) Total steps: 25, num_warmup_steps: 0
(WorkerDict pid=1733170) wrap_policy: functools.partial(<function _or_policy at 0x77e34f3e1000>, policies=[functools.partial(<function transformer_auto_wrap_policy at 0x77e34f3e0ee0>, transformer_layer_cls={<class 'transformers.models.qwen2.modeling_qwen2.Qwen2DecoderLayer'>})]) [repeated 7x across cluster]
(WorkerDict pid=1733169) Monkey patch _flash_attention_forward in transformers.integrations.flash_attention
(WorkerDict pid=1733172) Actor use_remove_padding=True
(WorkerDict pid=1733172) Actor use_fused_kernels=False
(WorkerDict pid=1732013) Total steps: 25, num_warmup_steps: 0 [repeated 7x across cluster]
(WorkerDict pid=1732013) Before building sglang_async rollout, memory allocated (GB): 0.00, memory reserved (GB): 0.00, device memory used/total (GB): 3.35/139.72
(WorkerDict pid=1733167) kwargs: {'n': 8, 'max_new_tokens': 1024, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'repetition_penalty': 1.0, 'temperature': 1.0, 'top_k': -1, 'top_p': 1, 'ignore_eos': False}
(WorkerDict pid=1732013) Actor use_remove_padding=True [repeated 7x across cluster]
(WorkerDict pid=1732013) Actor use_fused_kernels=False [repeated 7x across cluster]
(WorkerDict pid=1732013) NCCL version 2.21.5+cuda12.4
(WorkerDict pid=1733170) kwargs: {'n': 8, 'max_new_tokens': 1024, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'repetition_penalty': 1.0, 'temperature': 1.0, 'top_k': -1, 'top_p': 1, 'ignore_eos': False} [repeated 6x across cluster]
(WorkerDict pid=1732013) After building sglang_async rollout, memory allocated (GB): 0.00, memory reserved (GB): 0.00, device memory used/total (GB): 8.74/139.72
(WorkerDict pid=1732013) After building sharding manager, memory allocated (GB): 0.00, memory reserved (GB): 0.00, device memory used/total (GB): 8.74/139.72
(WorkerDict pid=1732013) kwargs: {'n': 8, 'max_new_tokens': 1024, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'repetition_penalty': 1.0, 'temperature': 1.0, 'top_k': -1, 'top_p': 1, 'ignore_eos': False}
(TaskRunner pid=1714137) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': False, 'validate': True}
Traceback (most recent call last):
  File "/home/jobuser/resources/verl/trainer/main_ppo.py", line 64, in main
    run_ppo(config)
  File "/home/jobuser/resources/verl/trainer/main_ppo.py", line 76, in run_ppo
    ray.get(runner.run.remote(config))
  File "/home/jobuser/.local/lib/python3.10/site-packages/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
    return fn(*args, **kwargs)
  File "/home/jobuser/.local/lib/python3.10/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
    return func(*args, **kwargs)
  File "/home/jobuser/.local/lib/python3.10/site-packages/ray/_private/worker.py", line 2822, in get
    values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
  File "/home/jobuser/.local/lib/python3.10/site-packages/ray/_private/worker.py", line 930, in get_objects
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(DistBackendError): ray::TaskRunner.run() (pid=1714137, ip=100.96.253.216, actor_id=b62deb437e3f5623886f56ac01000000, repr=<main_ppo.TaskRunner object at 0x700127cff010>)
  File "/home/jobuser/resources/verl/trainer/main_ppo.py", line 183, in run
    trainer.fit()
  File "/home/jobuser/resources/verl/trainer/ppo/ray_trainer.py", line 872, in fit
    val_metrics = self._validate()
  File "/home/jobuser/resources/verl/trainer/ppo/ray_trainer.py", line 607, in _validate
    test_output_gen_batch_padded = self.actor_rollout_wg.generate_sequences(test_gen_batch_padded)
  File "/home/jobuser/resources/verl/single_controller/ray/base.py", line 49, in func
    output = ray.get(output)
ray.exceptions.RayTaskError(DistBackendError): ray::WorkerDict.actor_rollout_generate_sequences() (pid=1733170, ip=100.96.253.216, actor_id=df3193cf46083e93d60809e201000000, repr=<verl.single_controller.ray.base.WorkerDict object at 0x77b180b2bfa0>)
  File "/home/jobuser/resources/verl/single_controller/ray/base.py", line 625, in func
    return getattr(self.worker_dict[key], name)(*args, **kwargs)
  File "/home/jobuser/resources/verl/single_controller/base/decorator.py", line 534, in inner
    return func(*args, **kwargs)
  File "/home/jobuser/resources/verl/workers/fsdp_workers.py", line 630, in generate_sequences
    output = self.rollout.generate_sequences_with_tools(prompts=prompts)
  File "/home/jobuser/resources/verl/utils/debug/performance.py", line 78, in f
    return self.log(decorated_function, *args, **kwargs)
  File "/home/jobuser/resources/verl/utils/debug/performance.py", line 88, in log
    output = func(*args, **kwargs)
  File "/home/jobuser/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/home/jobuser/resources/verl/workers/rollout/sglang_rollout/async_sglang_rollout.py", line 619, in generate_sequences_with_tools
    [sorted_output_req_list] = broadcast_pyobj(
  File "/home/jobuser/resources/verl/workers/rollout/sglang_rollout/utils.py", line 55, in broadcast_pyobj
    dist.broadcast(tensor_size, src=src, group=dist_group)
  File "/home/jobuser/.local/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
    return func(*args, **kwargs)
  File "/home/jobuser/.local/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2726, in broadcast
    work = group.broadcast([tensor], opts)
torch.distributed.DistBackendError: [4] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes
Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:671 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x77e36d3fb1b6 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x6312a88 (0x77b441256a88 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x367 (0x77b441253817 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x77b441253b9a in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x77b44125493a in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x77b4412015d1 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x77b4412015d1 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x77b4412015d1 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x77b4412015d1 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x77b406f91a9f in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #10: c10d::ProcessGroupNCCL::initNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x1176 (0x77b406f9f666 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #11: c10d::ProcessGroupNCCL::broadcast(std::vector<at::Tensor, std::allocator<at::Tensor> >&, c10d::BroadcastOptions const&) + 0x1265 (0x77b406faa175 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #12: <unknown function> + 0x62b13b6 (0x77b4411f53b6 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #13: <unknown function> + 0x62bcd26 (0x77b441200d26 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #14: <unknown function> + 0x58bc27b (0x77b44080027b in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #15: <unknown function> + 0x58b9af9 (0x77b4407fdaf9 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #16: <unknown function> + 0x1aca9f8 (0x77b43ca0e9f8 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #17: <unknown function> + 0x62c339a (0x77b44120739a in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #18: <unknown function> + 0x62d3b30 (0x77b441217b30 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #19: <unknown function> + 0xe9fac5 (0x77e36e32eac5 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #20: <unknown function> + 0x518d37 (0x77e36d9a7d37 in /home/jobuser/.local/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
<omitting python frames>
frame #55: <unknown function> + 0x769d4f (0x77e402733d4f in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
frame #56: <unknown function> + 0x7daaa8 (0x77e4027a4aa8 in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
frame #57: <unknown function> + 0x769d4f (0x77e402733d4f in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
frame #58: <unknown function> + 0x86fc56 (0x77e402839c56 in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
frame #59: std::_Function_handler<ray::Status (ray::rpc::Address const&, ray::rpc::TaskType, std::string, ray::core::RayFunction const&, std::unordered_map<std::string, double, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, double> > > const&, std::vector<std::shared_ptr<ray::RayObject>, std::allocator<std::shared_ptr<ray::RayObject> > > const&, std::vector<ray::rpc::ObjectReference, std::allocator<ray::rpc::ObjectReference> > const&, std::string const&, std::string const&, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*, std::shared_ptr<ray::LocalMemoryBuffer>&, bool*, std::string*, std::vector<ray::ConcurrencyGroup, std::allocator<ray::ConcurrencyGroup> > const&, std::string, bool, bool, bool, long), ray::Status (*)(ray::rpc::Address const&, ray::rpc::TaskType, std::string, ray::core::RayFunction const&, std::unordered_map<std::string, double, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, double> > > const&, std::vector<std::shared_ptr<ray::RayObject>, std::allocator<std::shared_ptr<ray::RayObject> > > const&, std::vector<ray::rpc::ObjectReference, std::allocator<ray::rpc::ObjectReference> > const&, std::string, std::string, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*, std::shared_ptr<ray::LocalMemoryBuffer>&, bool*, std::string*, std::vector<ray::ConcurrencyGroup, std::allocator<ray::ConcurrencyGroup> > const&, std::string, bool, bool, bool, long)>::_M_invoke(std::_Any_data const&, ray::rpc::Address const&, ray::rpc::TaskType&&, std::string&&, ray::core::RayFunction const&, std::unordered_map<std::string, double, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, double> > > const&, std::vector<std::shared_ptr<ray::RayObject>, std::allocator<std::shared_ptr<ray::RayObject> > > const&, std::vector<ray::rpc::ObjectReference, std::allocator<ray::rpc::ObjectReference> > const&, std::string const&, std::string const&, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*&&, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*&&, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*&&, std::shared_ptr<ray::LocalMemoryBuffer>&, bool*&&, std::string*&&, std::vector<ray::ConcurrencyGroup, std::allocator<ray::ConcurrencyGroup> > const&, std::string&&, bool&&, bool&&, bool&&, long&&) + 0x169 (0x77e40273ad39 in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
frame #60: ray::core::CoreWorker::ExecuteTask(ray::TaskSpecification const&, std::optional<std::unordered_map<std::string, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > > > > > >, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*, google::protobuf::RepeatedPtrField<ray::rpc::ObjectReferenceCount>*, bool*, std::string*) + 0xdf5 (0x77e40294c245 in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
frame #61: std::_Function_handler<ray::Status (ray::TaskSpecification const&, std::optional<std::unordered_map<std::string, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > > > > > >, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*, google::protobuf::RepeatedPtrField<ray::rpc::ObjectReferenceCount>*, bool*, std::string*), std::_Bind<ray::Status (ray::core::CoreWorker::*(ray::core::CoreWorker*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Placeholder<3>, std::_Placeholder<4>, std::_Placeholder<5>, std::_Placeholder<6>, std::_Placeholder<7>, std::_Placeholder<8>))(ray::TaskSpecification const&, std::optional<std::unordered_map<std::string, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > > > > > >, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*, google::protobuf::RepeatedPtrField<ray::rpc::ObjectReferenceCount>*, bool*, std::string*)> >::_M_invoke(std::_Any_data const&, ray::TaskSpecification const&, std::optional<std::unordered_map<std::string, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<std::pair<long, double>, std::allocator<std::pair<long, double> > > > > > >&&, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*&&, std::vector<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> >, std::allocator<std::pair<ray::ObjectID, std::shared_ptr<ray::RayObject> > > >*&&, std::vector<std::pair<ray::ObjectID, bool>, std::allocator<std::pair<ray::ObjectID, bool> > >*&&, google::protobuf::RepeatedPtrField<ray::rpc::ObjectReferenceCount>*&&, bool*&&, std::string*&&) + 0x75 (0x77e4028d6845 in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
frame #62: <unknown function> + 0x9f30ce (0x77e4029bd0ce in /home/jobuser/.local/lib/python3.10/site-packages/ray/_raylet.so)
. This may indicate a possible application crash on rank 0 or a network set up issue.

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

[jybsuper]

@hebiao064 @MrAta Could you take a look? Thanks!

[ysy-phoenix]

I also encountered what appears to be a similar error on the AMD MI300X while using async SGLang for multiturn GRPO RL.
Here is the error log:

[36m(TaskRunner pid=2036068)[0m Dumped generations to /home/aiscuser/data/gsm8k_tool/validation/0.jsonl
[36m(TaskRunner pid=2036068)[0m ("Initial validation metrics: {'val-core/openai/gsm8k/reward/mean@1': "
[36m(TaskRunner pid=2036068)[0m  '0.6808188021228203}')
[36m(TaskRunner pid=2036068)[0m step:0 - val-core/openai/gsm8k/reward/mean@1:0.681
[36m(WorkerDict pid=2039146)[0m RCCL version : 2.21.5-HEAD:648a58d[32m [repeated 4x across cluster][0m
[36m(WorkerDict pid=2039146)[0m HIP version  : 6.3.42131-fa1d09cbd[32m [repeated 4x across cluster][0m
[36m(WorkerDict pid=2039146)[0m ROCm version : 6.3.0.0-39-ce2be3b[32m [repeated 4x across cluster][0m
[36m(WorkerDict pid=2039146)[0m Hostname     : node-0[32m [repeated 4x across cluster][0m
[36m(WorkerDict pid=2039146)[0m Librccl path : /usr/local/lib/librccl.so.1[32m [repeated 4x across cluster][0m
[36m(TaskRunner pid=2036068)[0m list(reward_extra_infos_dict.keys())=[]
[36m(TaskRunner pid=2036068)[0m _StringKeys(dict_keys(['input_ids', 'loss_mask', 'responses', 'attention_mask', 'position_ids', 'prompts', 'response_mask', 'old_log_probs', 'token_level_scores', 'token_level_rewards', 'advantages', 'returns']))
[36m(TaskRunner pid=2036068)[0m Dumped generations to /home/aiscuser/data/gsm8k_tool/rollout/1.jsonl
[36m(TaskRunner pid=2036068)[0m step:1 - global_seqlen/min:329662.000 - global_seqlen/max:416832.000 - global_seqlen/minmax_diff:87170.000 - global_seqlen/balanced_min:373316.000 - global_seqlen/balanced_max:374944.000 - global_seqlen/mean:373520.125 - actor/entropy_loss:0.266 - actor/pg_loss:0.077 - actor/pg_clipfrac:0.000 - actor/ppo_kl:0.000 - actor/pg_clipfrac_lower:0.000 - actor/grad_norm:0.374 - perf/mfu/actor:0.137 - perf/max_memory_allocated_gb:43.985 - perf/max_memory_reserved_gb:93.621 - perf/cpu_memory_used_gb:882.024 - actor/lr:0.000 - training/global_step:1.000 - training/epoch:0.000 - critic/score/mean:0.712 - critic/score/max:1.000 - critic/score/min:0.000 - critic/rewards/mean:0.712 - critic/rewards/max:1.000 - critic/rewards/min:0.000 - critic/advantages/mean:0.006 - critic/advantages/max:3.750 - critic/advantages/min:-3.750 - critic/returns/mean:0.006 - critic/returns/max:3.750 - critic/returns/min:-3.750 - response_length/mean:364.805 - response_length/max:4096.000 - response_length/min:60.000 - response_length/clip_ratio:0.007 - prompt_length/mean:364.727 - prompt_length/max:449.000 - prompt_length/min:326.000 - prompt_length/clip_ratio:0.000 - timing_s/gen:38.104 - timing_s/reward:1.137 - timing_s/old_log_prob:17.170 - timing_s/adv:0.082 - timing_s/update_actor:43.592 - timing_s/dump_rollout_generations:2.431 - timing_s/step:102.846 - timing_per_token_ms/gen:0.026 - timing_per_token_ms/update_actor:0.015 - timing_per_token_ms/adv:0.000 - perf/total_num_tokens:2988161.000 - perf/time_per_step:102.846 - perf/throughput:3631.850[36m(TaskRunner pid=2036068)[0m 
Training Progress:   0%|          | 1/435 [01:43<12:28:45, 103.51s/it]
[36m(TaskRunner pid=2036068)[0m 
Training Progress:   0%|          | 2/435 [03:21<12:02:15, 100.08s/it]
[36m(TaskRunner pid=2036068)[0m 
Training Progress:   1%|          | 3/435 [05:06<12:17:21, 102.41s/it]
[36m(TaskRunner pid=2036068)[0m 
Training Progress:   1%|          | 4/435 [06:53<12:28:56, 104.26s/it]
[36m(WorkerDict pid=2039142)[0m Memory access fault by GPU node-4 (Agent handle: 0xe92da90) on address 0x7eb3435d7000. Reason: Unknown.
[36m(WorkerDict pid=2039142)[0m Fatal Python error: Aborted
[36m(WorkerDict pid=2039142)[0m 
[36m(WorkerDict pid=2039142)[0m Thread 0x00007eb3469fe640 (most recent call first):
[36m(WorkerDict pid=2039142)[0m   File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 1798 in watchdog_thread
[36m(WorkerDict pid=2039142)[0m   File "/usr/lib/python3.12/threading.py", line 1012 in run
[36m(WorkerDict pid=2039142)[0m   File "/usr/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
[36m(WorkerDict pid=2039142)[0m   File "/usr/lib/python3.12/threading.py", line 1032 in _bootstrap
[36m(WorkerDict pid=2039142)[0m 
[36m(WorkerDict pid=2039142)[0m Thread 0x00007eb3471ff640 (most recent call first):
[36m(WorkerDict pid=2039142)[0m   File "/usr/lib/python3.12/threading.py", line 355 in wait
[36m(WorkerDict pid=2039142)[0m   File "/usr/lib/python3.12/queue.py", line 171 in get
[36m(WorkerDict pid=2039142)[0m   File "/sgl-workspace/sglang/python/sglang/srt/managers/tp_worker_overlap_thread.py", line 130 in forward_thread_func_
[36m(WorkerDict pid=2039142)[0m   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116 in decorate_context
[36m(WorkerDict pid=2039142)[0m   File "/sgl-workspace/sglang/python/sglang/srt/managers/tp_worker_overlap_thread.py", line 118 in forward_thread_func
[36m(WorkerDict pid=2039142)[0m   File "/usr/lib/python3.12/threading.py", line 1012 in run
[36m(WorkerDict pid=2039142)[0m   File "/usr/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
[36m(WorkerDict pid=2039142)[0m   File "/usr/lib/python3.12/threading.py", line 1032 in _bootstrap
[36m(WorkerDict pid=2039142)[0m 
[36m(WorkerDict pid=2039142)[0m Thread 0x00007eb38a3ff640 (most recent call first):
[36m(WorkerDict pid=2039142)[0m   File "/usr/lib/python3.12/threading.py", line 359 in wait
[36m(WorkerDict pid=2039142)[0m   File "/usr/lib/python3.12/threading.py", line 655 in wait
[36m(WorkerDict pid=2039142)[0m   File "/usr/local/lib/python3.12/dist-packages/tqdm/_monitor.py", line 60 in run
[36m(WorkerDict pid=2039142)[0m   File "/usr/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
[36m(WorkerDict pid=2039142)[0m   File "/usr/lib/python3.12/threading.py", line 1032 in _bootstrap
[36m(WorkerDict pid=2039142)[0m 
[36m(WorkerDict pid=2039142)[0m Thread 0x00007f0a0d4a4640 (most recent call first):
[36m(WorkerDict pid=2039142)[0m   File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_worker/subproc_pool.py", line 55 in _recv_msg
[36m(WorkerDict pid=2039142)[0m   File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_worker/subproc_pool.py", line 191 in _read_thread
[36m(WorkerDict pid=2039142)[0m   File "/usr/lib/python3.12/threading.py", line 1012 in run
[36m(WorkerDict pid=2039142)[0m   File "/usr/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
[36m(WorkerDict pid=2039142)[0m   File "/usr/lib/python3.12/threading.py", line 1032 in _bootstrap
[36m(WorkerDict pid=2039142)[0m 
[36m(WorkerDict pid=2039142)[0m Thread 0x00007f0b58d26180 (most recent call first):
[36m(WorkerDict pid=2039142)[0m   File "/usr/local/lib/python3.12/dist-packages/torch/cuda/streams.py", line 224 in synchronize
[36m(WorkerDict pid=2039142)[0m   File "/sgl-workspace/sglang/python/sglang/srt/managers/tp_worker_overlap_thread.py", line 193 in resolve_last_batch_result
[36m(WorkerDict pid=2039142)[0m   File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler_output_processor_mixin.py", line 200 in process_batch_result_decode
[36m(WorkerDict pid=2039142)[0m   File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 1592 in process_batch_result
[36m(WorkerDict pid=2039142)[0m   File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 683 in event_loop_overlap
[36m(WorkerDict pid=2039142)[0m   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116 in decorate_context
[36m(WorkerDict pid=2039142)[0m   File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2269 in run_scheduler_process
[36m(WorkerDict pid=2039142)[0m   File "/usr/lib/python3.12/multiprocessing/process.py", line 108 in run
[36m(WorkerDict pid=2039142)[0m   File "/usr/lib/python3.12/multiprocessing/process.py", line 314 in _bootstrap
[36m(WorkerDict pid=2039142)[0m   File "/usr/lib/python3.12/multiprocessing/spawn.py", line 135 in _main
[36m(WorkerDict pid=2039142)[0m   File "/usr/lib/python3.12/multiprocessing/spawn.py", line 122 in spawn_main
[36m(WorkerDict pid=2039142)[0m   File "<string>", line 1 in <module>
[36m(WorkerDict pid=2039142)[0m 
[36m(WorkerDict pid=2039142)[0m Extension modules: msgpack._cmsgpack, google._upb._message, psutil._psutil_linux, psutil._psutil_posix, setproctitle, yaml._yaml, charset_normalizer.md, requests.packages.charset_normalizer.md, requests.packages.chardet.md, uvloop.loop, ray._raylet, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, multidict._multidict, yarl._quoting_c, propcache._helpers_c, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket.mask, aiohttp._websocket.reader_c, frozenlist._frozenlist, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, zmq.backend.cython._zmq, markupsafe._speedups, PIL._imaging, PIL._imagingft, hiredis.hiredis, sentencepiece._sentencepiece, msgspec._core, regex._regex, cython.cimports.libc.math, _cffi_backend, hip_utils, Cython.Utils, Cython.Plex.Actions, Cython.Plex.Transitions, Cython.Plex.Machines, Cython.Plex.DFA, Cython.Plex.Scanners, Cython.Compiler.Scanning, Cython.StringIOTree, Cython.Compiler.Code, pyarrow.lib, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pyarrow._compute, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, __triton_launcher (total: 106)

I can confirm that no OOM (Out of Memory) error occurred. The runtime environment is based on Dockerfile.rocm.

[MrAta]

I am looking into this.

[kemuxiaozi000]

Same error.

I am looking into this.

[liulurong]

I also have a "illegal memory access" problem when using VerlEngine(backend="server") i.e. HttpServerEngineAdapter
sglang 0.4.6.post4

export GLOO_SOCKET_IFNAME=bond1
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
MODEL=Qwen/Qwen2.5-0.5B-Instruct
python3 -m verl.trainer.main_ppo
algorithm.adv_estimator=grpo
data.train_files="$train_files"
data.val_files="$test_files"
data.train_batch_size=128
data.max_prompt_length=512
data.max_response_length=1024
data.filter_overlong_prompts=True
data.truncation='error'
actor_rollout_ref.model.path=${MODEL}
actor_rollout_ref.actor.optim.lr=1e-6
actor_rollout_ref.model.use_remove_padding=True
actor_rollout_ref.actor.ppo_mini_batch_size=64
actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=40
actor_rollout_ref.actor.use_kl_loss=True
actor_rollout_ref.actor.kl_loss_coef=0.001
actor_rollout_ref.actor.kl_loss_type=low_var_kl
actor_rollout_ref.actor.entropy_coeff=0
actor_rollout_ref.model.enable_gradient_checkpointing=True
actor_rollout_ref.actor.fsdp_config.param_offload=False
actor_rollout_ref.actor.fsdp_config.optimizer_offload=False
actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=40
actor_rollout_ref.rollout.tensor_model_parallel_size=1
actor_rollout_ref.rollout.enforce_eager=True
actor_rollout_ref.rollout.free_cache_engine=True
actor_rollout_ref.rollout.name=sglang
actor_rollout_ref.rollout.gpu_memory_utilization=0.6
actor_rollout_ref.rollout.n=5
actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=40
actor_rollout_ref.ref.fsdp_config.param_offload=True
algorithm.use_kl_in_reward=False
trainer.critic_warmup=0
trainer.logger=['console','wandb']
trainer.project_name='verl_grpo_example_gsm8k'
trainer.experiment_name='qwen2.5_0.5b_function_rm'
trainer.n_gpus_per_node=8
trainer.nnodes=1
trainer.save_freq=-1
trainer.test_freq=5
trainer.total_epochs=15 | tee test_qwen2.5_0.5b_fsdp.log

[liulurong]

full error log

(TaskRunner pid=845875) {'actor_rollout_ref': {'actor': {'checkpoint': {'contents': ['model',
(TaskRunner pid=845875)                                                              'optimizer',
(TaskRunner pid=845875)                                                              'extra']},
(TaskRunner pid=845875)                                  'clip_ratio': 0.2,
(TaskRunner pid=845875)                                  'clip_ratio_c': 3.0,
(TaskRunner pid=845875)                                  'clip_ratio_high': 0.2,
(TaskRunner pid=845875)                                  'clip_ratio_low': 0.2,
(TaskRunner pid=845875)                                  'entropy_coeff': 0,
(TaskRunner pid=845875)                                  'fsdp_config': {'fsdp_size': -1,
(TaskRunner pid=845875)                                                  'offload_policy': False,
(TaskRunner pid=845875)                                                  'optimizer_offload': False,
(TaskRunner pid=845875)                                                  'param_offload': False,
(TaskRunner pid=845875)                                                  'reshard_after_forward': True,
(TaskRunner pid=845875)                                                  'wrap_policy': {'min_num_params': 0}},
(TaskRunner pid=845875)                                  'grad_clip': 1.0,
(TaskRunner pid=845875)                                  'kl_loss_coef': 0.001,
(TaskRunner pid=845875)                                  'kl_loss_type': 'low_var_kl',
(TaskRunner pid=845875)                                  'loss_agg_mode': 'token-mean',
(TaskRunner pid=845875)                                  'optim': {'lr': 1e-06,
(TaskRunner pid=845875)                                            'lr_warmup_steps': -1,
(TaskRunner pid=845875)                                            'lr_warmup_steps_ratio': 0.0,
(TaskRunner pid=845875)                                            'min_lr_ratio': None,
(TaskRunner pid=845875)                                            'total_training_steps': -1,
(TaskRunner pid=845875)                                            'warmup_style': 'constant',
(TaskRunner pid=845875)                                            'weight_decay': 0.01},
(TaskRunner pid=845875)                                  'ppo_epochs': 1,
(TaskRunner pid=845875)                                  'ppo_max_token_len_per_gpu': 16384,
(TaskRunner pid=845875)                                  'ppo_micro_batch_size': None,
(TaskRunner pid=845875)                                  'ppo_micro_batch_size_per_gpu': 40,
(TaskRunner pid=845875)                                  'ppo_mini_batch_size': 64,
(TaskRunner pid=845875)                                  'shuffle': False,
(TaskRunner pid=845875)                                  'strategy': 'fsdp',
(TaskRunner pid=845875)                                  'ulysses_sequence_parallel_size': 1,
(TaskRunner pid=845875)                                  'use_dynamic_bsz': False,
(TaskRunner pid=845875)                                  'use_kl_loss': True,
(TaskRunner pid=845875)                                  'use_torch_compile': True},
(TaskRunner pid=845875)                        'hybrid_engine': True,
(TaskRunner pid=845875)                        'model': {'enable_gradient_checkpointing': True,
(TaskRunner pid=845875)                                  'external_lib': None,
(TaskRunner pid=845875)                                  'override_config': {},
(TaskRunner pid=845875)                                  'path': '/w/models/Qwen/Qwen2.5-0.5B-Instruct',
(TaskRunner pid=845875)                                  'trust_remote_code': False,
(TaskRunner pid=845875)                                  'use_liger': False,
(TaskRunner pid=845875)                                  'use_remove_padding': True},
(TaskRunner pid=845875)                        'ref': {'fsdp_config': {'param_offload': True,
(TaskRunner pid=845875)                                                'reshard_after_forward': True,
(TaskRunner pid=845875)                                                'wrap_policy': {'min_num_params': 0}},
(TaskRunner pid=845875)                                'log_prob_max_token_len_per_gpu': 16384,
(TaskRunner pid=845875)                                'log_prob_micro_batch_size': None,
(TaskRunner pid=845875)                                'log_prob_micro_batch_size_per_gpu': 40,
(TaskRunner pid=845875)                                'log_prob_use_dynamic_bsz': False,
(TaskRunner pid=845875)                                'strategy': 'fsdp',
(TaskRunner pid=845875)                                'ulysses_sequence_parallel_size': 1,
(TaskRunner pid=845875)                                'use_torch_compile': True},
(TaskRunner pid=845875)                        'rollout': {'chat_scheduler': None,
(TaskRunner pid=845875)                                    'disable_log_stats': True,
(TaskRunner pid=845875)                                    'do_sample': True,
(TaskRunner pid=845875)                                    'dtype': 'bfloat16',
(TaskRunner pid=845875)                                    'enable_chunked_prefill': True,
(TaskRunner pid=845875)                                    'enforce_eager': True,
(TaskRunner pid=845875)                                    'engine_kwargs': {'swap_space': None},
(TaskRunner pid=845875)                                    'free_cache_engine': True,
(TaskRunner pid=845875)                                    'gpu_memory_utilization': 0.6,
(TaskRunner pid=845875)                                    'ignore_eos': False,
(TaskRunner pid=845875)                                    'load_format': 'dummy_dtensor',
(TaskRunner pid=845875)                                    'log_prob_max_token_len_per_gpu': 16384,
(TaskRunner pid=845875)                                    'log_prob_micro_batch_size': None,
(TaskRunner pid=845875)                                    'log_prob_micro_batch_size_per_gpu': 40,
(TaskRunner pid=845875)                                    'log_prob_use_dynamic_bsz': False,
(TaskRunner pid=845875)                                    'max_model_len': None,
(TaskRunner pid=845875)                                    'max_num_batched_tokens': 8192,
(TaskRunner pid=845875)                                    'max_num_seqs': 1024,
(TaskRunner pid=845875)                                    'mode': 'sync',
(TaskRunner pid=845875)                                    'multi_turn': {'enable': False,
(TaskRunner pid=845875)                                                   'format': 'chatml',
(TaskRunner pid=845875)                                                   'max_turns': None,
(TaskRunner pid=845875)                                                   'tool_config_path': None},
(TaskRunner pid=845875)                                    'n': 5,
(TaskRunner pid=845875)                                    'name': 'sglang',
(TaskRunner pid=845875)                                    'prompt_length': 512,
(TaskRunner pid=845875)                                    'response_length': 1024,
(TaskRunner pid=845875)                                    'temperature': 1.0,
(TaskRunner pid=845875)                                    'tensor_model_parallel_size': 1,
(TaskRunner pid=845875)                                    'top_k': -1,
(TaskRunner pid=845875)                                    'top_p': 1,
(TaskRunner pid=845875)                                    'use_fire_sampling': False,
(TaskRunner pid=845875)                                    'val_kwargs': {'do_sample': False,
(TaskRunner pid=845875)                                                   'n': 1,
(TaskRunner pid=845875)                                                   'temperature': 0,
(TaskRunner pid=845875)                                                   'top_k': -1,
(TaskRunner pid=845875)                                                   'top_p': 1.0}}},
(TaskRunner pid=845875)  'algorithm': {'adv_estimator': 'grpo',
(TaskRunner pid=845875)                'gamma': 1.0,
(TaskRunner pid=845875)                'kl_ctrl': {'horizon': 10000,
(TaskRunner pid=845875)                            'kl_coef': 0.001,
(TaskRunner pid=845875)                            'target_kl': 0.1,
(TaskRunner pid=845875)                            'type': 'fixed'},
(TaskRunner pid=845875)                'kl_penalty': 'kl',
(TaskRunner pid=845875)                'lam': 1.0,
(TaskRunner pid=845875)                'norm_adv_by_std_in_grpo': True,
(TaskRunner pid=845875)                'use_kl_in_reward': False},
(TaskRunner pid=845875)  'critic': {'checkpoint': {'contents': ['model', 'optimizer', 'extra']},
(TaskRunner pid=845875)             'cliprange_value': 0.5,
(TaskRunner pid=845875)             'forward_max_token_len_per_gpu': 32768,
(TaskRunner pid=845875)             'forward_micro_batch_size': None,
(TaskRunner pid=845875)             'forward_micro_batch_size_per_gpu': None,
(TaskRunner pid=845875)             'grad_clip': 1.0,
(TaskRunner pid=845875)             'model': {'enable_gradient_checkpointing': True,
(TaskRunner pid=845875)                       'external_lib': None,
(TaskRunner pid=845875)                       'fsdp_config': {'fsdp_size': -1,
(TaskRunner pid=845875)                                       'offload_policy': False,
(TaskRunner pid=845875)                                       'optimizer_offload': False,
(TaskRunner pid=845875)                                       'param_offload': False,
(TaskRunner pid=845875)                                       'reshard_after_forward': True,
(TaskRunner pid=845875)                                       'wrap_policy': {'min_num_params': 0}},
(TaskRunner pid=845875)                       'override_config': {},
(TaskRunner pid=845875)                       'path': '~/models/deepseek-llm-7b-chat',
(TaskRunner pid=845875)                       'tokenizer_path': '/w/models/Qwen/Qwen2.5-0.5B-Instruct',
(TaskRunner pid=845875)                       'trust_remote_code': False,
(TaskRunner pid=845875)                       'use_remove_padding': False},
(TaskRunner pid=845875)             'optim': {'lr': 1e-05,
(TaskRunner pid=845875)                       'lr_warmup_steps_ratio': 0.0,
(TaskRunner pid=845875)                       'min_lr_ratio': None,
(TaskRunner pid=845875)                       'total_training_steps': -1,
(TaskRunner pid=845875)                       'warmup_style': 'constant',
(TaskRunner pid=845875)                       'weight_decay': 0.01},
(TaskRunner pid=845875)             'ppo_epochs': 1,
(TaskRunner pid=845875)             'ppo_max_token_len_per_gpu': 32768,
(TaskRunner pid=845875)             'ppo_micro_batch_size': None,
(TaskRunner pid=845875)             'ppo_micro_batch_size_per_gpu': None,
(TaskRunner pid=845875)             'ppo_mini_batch_size': 64,
(TaskRunner pid=845875)             'rollout_n': 5,
(TaskRunner pid=845875)             'shuffle': False,
(TaskRunner pid=845875)             'strategy': 'fsdp',
(TaskRunner pid=845875)             'ulysses_sequence_parallel_size': 1,
(TaskRunner pid=845875)             'use_dynamic_bsz': False},
(TaskRunner pid=845875)  'custom_reward_function': {'name': 'compute_score', 'path': None},
(TaskRunner pid=845875)  'data': {'custom_cls': {'name': None, 'path': None},
(TaskRunner pid=845875)           'filter_overlong_prompts': True,
(TaskRunner pid=845875)           'filter_overlong_prompts_workers': 1,
(TaskRunner pid=845875)           'image_key': 'images',
(TaskRunner pid=845875)           'max_prompt_length': 512,
(TaskRunner pid=845875)           'max_response_length': 1024,
(TaskRunner pid=845875)           'prompt_key': 'prompt',
(TaskRunner pid=845875)           'return_raw_chat': False,
(TaskRunner pid=845875)           'return_raw_input_ids': False,
(TaskRunner pid=845875)           'reward_fn_key': 'data_source',
(TaskRunner pid=845875)           'shuffle': True,
(TaskRunner pid=845875)           'tokenizer': None,
(TaskRunner pid=845875)           'train_batch_size': 128,
(TaskRunner pid=845875)           'train_files': ['/w/data/gsm8k/train.parquet',
(TaskRunner pid=845875)                           '/w/data/math/train.parquet'],
(TaskRunner pid=845875)           'truncation': 'error',
(TaskRunner pid=845875)           'val_batch_size': None,
(TaskRunner pid=845875)           'val_files': ['/w/data/gsm8k/test.parquet',
(TaskRunner pid=845875)                         '/w/data/math/test.parquet'],
(TaskRunner pid=845875)           'video_key': 'videos'},
(TaskRunner pid=845875)  'ray_init': {'num_cpus': None},
(TaskRunner pid=845875)  'reward_model': {'enable': False,
(TaskRunner pid=845875)                   'forward_max_token_len_per_gpu': 32768,
(TaskRunner pid=845875)                   'launch_reward_fn_async': False,
(TaskRunner pid=845875)                   'max_length': None,
(TaskRunner pid=845875)                   'micro_batch_size': None,
(TaskRunner pid=845875)                   'micro_batch_size_per_gpu': None,
(TaskRunner pid=845875)                   'model': {'external_lib': None,
(TaskRunner pid=845875)                             'fsdp_config': {'fsdp_size': -1,
(TaskRunner pid=845875)                                             'param_offload': False,
(TaskRunner pid=845875)                                             'reshard_after_forward': True,
(TaskRunner pid=845875)                                             'wrap_policy': {'min_num_params': 0}},
(TaskRunner pid=845875)                             'input_tokenizer': '/w/models/Qwen/Qwen2.5-0.5B-Instruct',
(TaskRunner pid=845875)                             'path': '~/models/FsfairX-LLaMA3-RM-v0.1',
(TaskRunner pid=845875)                             'trust_remote_code': False,
(TaskRunner pid=845875)                             'use_remove_padding': False},
(TaskRunner pid=845875)                   'reward_manager': 'naive',
(TaskRunner pid=845875)                   'strategy': 'fsdp',
(TaskRunner pid=845875)                   'ulysses_sequence_parallel_size': 1,
(TaskRunner pid=845875)                   'use_dynamic_bsz': False},
(TaskRunner pid=845875)  'trainer': {'balance_batch': True,
(TaskRunner pid=845875)              'critic_warmup': 0,
(TaskRunner pid=845875)              'default_hdfs_dir': None,
(TaskRunner pid=845875)              'default_local_dir': 'checkpoints/verl_grpo_example_gsm8k/qwen2_7b_function_rm',
(TaskRunner pid=845875)              'del_local_ckpt_after_load': False,
(TaskRunner pid=845875)              'experiment_name': 'qwen2_7b_function_rm',
(TaskRunner pid=845875)              'log_val_generations': 0,
Filtering prompts longer than 512 tokens:   0%|          | 0/14973 [00:00<?, ? examples/s]) 
Filtering prompts longer than 512 tokens:   7%|▋         | 1000/14973 [00:00<00:04, 3256.30 examples/s]
Filtering prompts longer than 512 tokens:  13%|█▎        | 2000/14973 [00:00<00:03, 3487.31 examples/s]
Filtering prompts longer than 512 tokens:  20%|██        | 3000/14973 [00:00<00:03, 3588.70 examples/s]
Filtering prompts longer than 512 tokens:  27%|██▋       | 4000/14973 [00:01<00:03, 3631.58 examples/s]
Filtering prompts longer than 512 tokens:  33%|███▎      | 5000/14973 [00:01<00:02, 3737.45 examples/s]
Filtering prompts longer than 512 tokens:  40%|████      | 6000/14973 [00:01<00:02, 3853.66 examples/s]
Filtering prompts longer than 512 tokens:  47%|████▋     | 7000/14973 [00:01<00:02, 3866.50 examples/s]
Filtering prompts longer than 512 tokens:  53%|█████▎    | 8000/14973 [00:02<00:01, 3819.58 examples/s]
Filtering prompts longer than 512 tokens:  60%|██████    | 9000/14973 [00:02<00:01, 3836.03 examples/s]
Filtering prompts longer than 512 tokens:  67%|██████▋   | 10000/14973 [00:02<00:01, 3797.57 examples/s]
Filtering prompts longer than 512 tokens:  73%|███████▎  | 11000/14973 [00:03<00:01, 3327.30 examples/s]
Filtering prompts longer than 512 tokens:  80%|████████  | 12000/14973 [00:03<00:00, 3411.38 examples/s]
Filtering prompts longer than 512 tokens:  87%|████████▋ | 13000/14973 [00:03<00:00, 3326.70 examples/s]
Filtering prompts longer than 512 tokens:  94%|█████████▎| 14000/14973 [00:03<00:00, 3361.59 examples/s]
Filtering prompts longer than 512 tokens: 100%|██████████| 14973/14973 [00:04<00:00, 3546.45 examples/s]
Filtering prompts longer than 512 tokens:   0%|          | 0/6319 [00:00<?, ? examples/s]
Filtering prompts longer than 512 tokens:  16%|█▌        | 1000/6319 [00:00<00:01, 3872.11 examples/s]
Filtering prompts longer than 512 tokens:  32%|███▏      | 2000/6319 [00:00<00:01, 3979.11 examples/s]
Filtering prompts longer than 512 tokens:  47%|████▋     | 3000/6319 [00:00<00:00, 3865.44 examples/s]
Filtering prompts longer than 512 tokens:  63%|██████▎   | 4000/6319 [00:01<00:00, 3460.83 examples/s]
Filtering prompts longer than 512 tokens:  79%|███████▉  | 5000/6319 [00:01<00:00, 3577.15 examples/s]
Filtering prompts longer than 512 tokens:  95%|█████████▍| 6000/6319 [00:01<00:00, 3541.37 examples/s]
Filtering prompts longer than 512 tokens: 100%|██████████| 6319/6319 [00:01<00:00, 3606.27 examples/s]
(TaskRunner pid=845875) DeprecationWarning: `ray.state.available_resources_per_node` is a private attribute and access will be removed in a future Ray version.
(TaskRunner pid=845875) WARNING:2025-05-23 18:01:19,425:Waiting for register center actor IwGFgu_register_center to be ready. Elapsed time: 0 seconds out of 300 seconds.
(WorkerDict pid=859188) You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
(WorkerDict pid=857940) Flash Attention 2.0 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen2ForCausalLM is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `torch_dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", torch_dtype=torch.float16)`
(WorkerDict pid=857940) You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. [repeated 7x across cluster]

(TaskRunner pid=845875)              'max_actor_ckpt_to_keep': None,
(TaskRunner pid=845875)              'max_critic_ckpt_to_keep': None,
(TaskRunner pid=845875)              'n_gpus_per_node': 8,
(TaskRunner pid=845875)              'nnodes': 1,
(TaskRunner pid=845875)              'project_name': 'verl_grpo_example_gsm8k',
(TaskRunner pid=845875)              'ray_wait_register_center_timeout': 300,
(TaskRunner pid=845875)              'resume_from_path': None,
(TaskRunner pid=845875)              'resume_mode': 'auto',
(TaskRunner pid=845875)              'rollout_data_dir': None,
(TaskRunner pid=845875)              'save_freq': -1,
(TaskRunner pid=845875)              'test_freq': 5,
(TaskRunner pid=845875)              'total_epochs': 15,
(TaskRunner pid=845875)              'total_training_steps': None,
(TaskRunner pid=845875)              'val_before_train': True,
(TaskRunner pid=845875)              'validation_data_dir': None}}
(TaskRunner pid=845875) [2025-05-23 18:01:08,339] [WARNING] [real_accelerator.py:174:get_accelerator] Setting accelerator to CPU. If you have GPU or other accelerator, we were unable to detect it.
(TaskRunner pid=845875) [2025-05-23 18:01:08,340] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cpu (auto detect)
(TaskRunner pid=845875) Using dataset class: RLHFDataset
(TaskRunner pid=845875) dataset len: 14973
(TaskRunner pid=845875) filter dataset len: 14898
(TaskRunner pid=845875) Using dataset class: RLHFDataset
(TaskRunner pid=845875) dataset len: 6319
(TaskRunner pid=845875) filter dataset len: 6291
(TaskRunner pid=845875) [validate_config] All configuration checks passed successfully!
(TaskRunner pid=845875) Size of train dataloader: 116, Size of val dataloader: 1
(TaskRunner pid=845875) Total training steps: 1740
(TaskRunner pid=845875) colocated worker base class <class 'verl.single_controller.base.worker.Worker'>
(pid=857940) [2025-05-23 18:01:25,188] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
(pid=859183) [2025-05-23 18:01:37,067] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
(pid=859184) [2025-05-23 18:01:38,124] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
(WorkerDict pid=859187) Monkey patch _flash_attention_forward in transformers.integrations.flash_attention
(WorkerDict pid=857940) Model config after override: Qwen2Config {
(WorkerDict pid=857940)   "architectures": [
(WorkerDict pid=857940)     "Qwen2ForCausalLM"
(WorkerDict pid=857940)   ],
(WorkerDict pid=857940)   "attention_dropout": 0.0,
(WorkerDict pid=857940)   "eos_token_id": 151645,
(WorkerDict pid=857940)   "hidden_act": "silu",
(WorkerDict pid=857940)   "hidden_size": 896,
(WorkerDict pid=857940)   "initializer_range": 0.02,
(WorkerDict pid=857940)   "intermediate_size": 4864,
(WorkerDict pid=857940)   "max_position_embeddings": 32768,
(WorkerDict pid=857940)   "max_window_layers": 21,
(WorkerDict pid=857940)   "model_type": "qwen2",
(WorkerDict pid=857940)   "num_attention_heads": 14,
(WorkerDict pid=857940)   "num_hidden_layers": 24,
(WorkerDict pid=857940)   "num_key_value_heads": 2,
(WorkerDict pid=857940)   "pad_token_id": 151643,
(WorkerDict pid=857940)   "rms_norm_eps": 1e-06,
(WorkerDict pid=857940)   "rope_scaling": null,
(WorkerDict pid=857940)   "rope_theta": 1000000.0,
(WorkerDict pid=857940)   "sliding_window": 32768,
(WorkerDict pid=857940)   "tie_word_embeddings": true,
(WorkerDict pid=857940)   "torch_dtype": "bfloat16",
(WorkerDict pid=857940)   "transformers_version": "4.51.1",
(WorkerDict pid=857940)   "use_cache": true,
(WorkerDict pid=857940)   "use_sliding_window": false,
(WorkerDict pid=857940)   "vocab_size": 151936
(WorkerDict pid=857940) }
(WorkerDict pid=857940) 
(WorkerDict pid=857940) Qwen2ForCausalLM contains 494.03M parameters
(WorkerDict pid=857940) wrap_policy: functools.partial(<function _or_policy at 0x7eece9dc9260>, policies=[functools.partial(<function transformer_auto_wrap_policy at 0x7eece9dc9120>, transformer_layer_cls={<class 'transformers.models.qwen2.modeling_qwen2.Qwen2DecoderLayer'>})])
(WorkerDict pid=857940) NCCL version 2.21.5+cuda12.4
(WorkerDict pid=857940) Actor use_remove_padding=True
(pid=859186) [2025-05-23 18:01:38,360] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) [repeated 5x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)
(WorkerDict pid=857940) Monkey patch _flash_attention_forward in transformers.integrations.flash_attention [repeated 7x across cluster]
(WorkerDict pid=859184) wrap_policy: functools.partial(<function _or_policy at 0x7f623caf5260>, policies=[functools.partial(<function transformer_auto_wrap_policy at 0x7f623caf5120>, transformer_layer_cls={<class 'transformers.models.qwen2.modeling_qwen2.Qwen2DecoderLayer'>})]) [repeated 7x across cluster]
(WorkerDict pid=857940) Model config after override: Qwen2Config {
(WorkerDict pid=857940)   "architectures": [
(WorkerDict pid=857940)     "Qwen2ForCausalLM"
(WorkerDict pid=857940)   ],
(WorkerDict pid=857940)   "attention_dropout": 0.0,
(WorkerDict pid=857940)   "eos_token_id": 151645,
(WorkerDict pid=857940)   "hidden_act": "silu",
(WorkerDict pid=857940)   "hidden_size": 896,
(WorkerDict pid=857940)   "initializer_range": 0.02,
(WorkerDict pid=857940)   "intermediate_size": 4864,
(WorkerDict pid=857940)   "max_position_embeddings": 32768,
(WorkerDict pid=857940)   "max_window_layers": 21,
(WorkerDict pid=857940)   "model_type": "qwen2",
(WorkerDict pid=857940)   "num_attention_heads": 14,
(WorkerDict pid=857940)   "num_hidden_layers": 24,
(WorkerDict pid=857940)   "num_key_value_heads": 2,
(WorkerDict pid=857940)   "pad_token_id": 151643,
(WorkerDict pid=857940)   "rms_norm_eps": 1e-06,
(WorkerDict pid=857940)   "rope_scaling": null,
(WorkerDict pid=857940)   "rope_theta": 1000000.0,
(WorkerDict pid=857940)   "sliding_window": 32768,
(WorkerDict pid=857940)   "tie_word_embeddings": true,
(WorkerDict pid=857940)   "torch_dtype": "bfloat16",
(WorkerDict pid=857940)   "transformers_version": "4.51.1",
(WorkerDict pid=857940)   "use_cache": true,
(WorkerDict pid=857940)   "use_sliding_window": false,
(WorkerDict pid=857940)   "vocab_size": 151936
(WorkerDict pid=857940) }
(WorkerDict pid=857940) 
(WorkerDict pid=857940) Qwen2ForCausalLM contains 494.03M parameters
(WorkerDict pid=859184) Actor use_remove_padding=True [repeated 7x across cluster]
(WorkerDict pid=857940) Monkey patch _flash_attention_forward in transformers.integrations.flash_attention [repeated 3x across cluster]
(WorkerDict pid=857940) wrap_policy: functools.partial(<function _or_policy at 0x7eece9dc9260>, policies=[functools.partial(<function transformer_auto_wrap_policy at 0x7eece9dc9120>, transformer_layer_cls={<class 'transformers.models.qwen2.modeling_qwen2.Qwen2DecoderLayer'>})])
(WorkerDict pid=859185) wrap_policy: functools.partial(<function _or_policy at 0x7f0572cf1260>, policies=[functools.partial(<function transformer_auto_wrap_policy at 0x7f0572cf1120>, transformer_layer_cls={<class 'transformers.models.qwen2.modeling_qwen2.Qwen2DecoderLayer'>})])(WorkerDict pid=859183) [2025-05-23 18:02:19] server_args=ServerArgs(model_path='/w/models/Qwen/Qwen2.5-0.5B-Instruct', tokenizer_path='/w/models/Qwen/Qwen2.5-0.5B-Instruct', tokenizer_mode='auto', skip_tokenizer_init=False, enable_tokenizer_batch_encode=False, load_format='dummy', trust_remote_code=False, dtype='bfloat16', kv_cache_dtype='auto', quantization=None, quantization_param_path=None, context_length=None, device='cuda', served_model_name='/w/models/Qwen/Qwen2.5-0.5B-Instruct', chat_template=None, completion_template=None, is_embedding=False, revision=None, host='127.0.0.1', port=30001, mem_fraction_static=0.6, max_running_requests=None, max_total_tokens=None, chunked_prefill_size=8192, max_prefill_tokens=16384, schedule_policy='fcfs', schedule_conservativeness=1.0, cpu_offload_gb=0, page_size=1, tp_size=1, pp_size=1, max_micro_batch_size=None, stream_interval=1, stream_output=False, random_seed=1058402991, constrained_json_whitespace_pattern=None, watchdog_timeout=300, dist_timeout=None, download_dir=None, base_gpu_id=0, gpu_id_step=1, log_level='info', log_level_http=None, log_requests=False, log_requests_level=0, show_time_cost=False, enable_metrics=False, decode_log_interval=40, enable_request_time_stats_logging=False, api_key=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, dp_size=1, load_balance_method='round_robin', ep_size=1, dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', lora_paths=None, max_loras_per_batch=8, lora_backend='triton', attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', speculative_algorithm=None, speculative_draft_model_path=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, disable_radix_cache=False, disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_nccl_nvls=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_multimodal=None, disable_overlap_schedule=False, enable_mixed_chunk=False, enable_dp_attention=False, enable_dp_lm_head=False, enable_ep_moe=False, enable_deepep_moe=False, deepep_mode='auto', enable_torch_compile=False, torch_compile_max_bs=32, cuda_graph_max_bs=None, cuda_graph_bs=None, torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=True, allow_auto_truncate=False, enable_custom_logit_processor=False, tool_call_parser=None, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through_selective', flashinfer_mla_disable_ragged=False, warmups=None, moe_dense_tp_size=None, n_share_experts_fusion=0, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, mm_attention_backend=None, debug_tensor_dump_output_folder=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, disaggregation_mode='null', disaggregation_bootstrap_port=8998, disaggregation_transfer_backend='mooncake', disaggregation_ib_device=None, pdlb_url=None)
(WorkerDict pid=859189) Flash Attention 2.0 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen2ForCausalLM is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `torch_dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", torch_dtype=torch.float16)` [repeated 7x across cluster]
(WorkerDict pid=859187) [2025-05-23 18:02:29] Attention backend not set. Use fa3 backend by default.
(WorkerDict pid=859187) [2025-05-23 18:02:29] Init torch distributed begin.
(WorkerDict pid=857940) [2025-05-23 18:02:20] server_args=ServerArgs(model_path='/w/models/Qwen/Qwen2.5-0.5B-Instruct', tokenizer_path='/w/models/Qwen/Qwen2.5-0.5B-Instruct', tokenizer_mode='auto', skip_tokenizer_init=False, enable_tokenizer_batch_encode=False, load_format='dummy', trust_remote_code=False, dtype='bfloat16', kv_cache_dtype='auto', quantization=None, quantization_param_path=None, context_length=None, device='cuda', served_model_name='/w/models/Qwen/Qwen2.5-0.5B-Instruct', chat_template=None, completion_template=None, is_embedding=False, revision=None, host='127.0.0.1', port=30000, mem_fraction_static=0.6, max_running_requests=None, max_total_tokens=None, chunked_prefill_size=8192, max_prefill_tokens=16384, schedule_policy='fcfs', schedule_conservativeness=1.0, cpu_offload_gb=0, page_size=1, tp_size=1, pp_size=1, max_micro_batch_size=None, stream_interval=1, stream_output=False, random_seed=547092864, constrained_json_whitespace_pattern=None, watchdog_timeout=300, dist_timeout=None, download_dir=None, base_gpu_id=0, gpu_id_step=1, log_level='info', log_level_http=None, log_requests=False, log_requests_level=0, show_time_cost=False, enable_metrics=False, decode_log_interval=40, enable_request_time_stats_logging=False, api_key=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, dp_size=1, load_balance_method='round_robin', ep_size=1, dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', lora_paths=None, max_loras_per_batch=8, lora_backend='triton', attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', speculative_algorithm=None, speculative_draft_model_path=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, disable_radix_cache=False, disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_nccl_nvls=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_multimodal=None, disable_overlap_schedule=False, enable_mixed_chunk=False, enable_dp_attention=False, enable_dp_lm_head=False, enable_ep_moe=False, enable_deepep_moe=False, deepep_mode='auto', enable_torch_compile=False, torch_compile_max_bs=32, cuda_graph_max_bs=None, cuda_graph_bs=None, torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=True, allow_auto_truncate=False, enable_custom_logit_processor=False, tool_call_parser=None, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through_selective', flashinfer_mla_disable_ragged=False, warmups=None, moe_dense_tp_size=None, n_share_experts_fusion=0, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, mm_attention_backend=None, debug_tensor_dump_output_folder=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, disaggregation_mode='null', disaggregation_bootstrap_port=8998, disaggregation_transfer_backend='mooncake', disaggregation_ib_device=None, pdlb_url=None) [repeated 7x across cluster]
(WorkerDict pid=859187) [2025-05-23 18:02:30] Init torch distributed ends. mem usage=0.00 GB
(WorkerDict pid=859187) [2025-05-23 18:02:30] Load weight begin. avail mem=89.16 GB
(WorkerDict pid=859187) [2025-05-23 18:02:32] Load weight end. type=Qwen2ForCausalLM, dtype=torch.bfloat16, avail mem=88.13 GB, mem usage=1.03 GB.
(WorkerDict pid=859187) [2025-05-23 18:02:32] KV Cache is allocated. #tokens: 4584520, K size: 26.23 GB, V size: 26.23 GB
(WorkerDict pid=859187) [2025-05-23 18:02:32] Memory pool end. avail mem=35.03 GB
(WorkerDict pid=859187) [2025-05-23 18:02:32] Capture cuda graph begin. This can take up to several minutes. avail mem=34.93 GB
(WorkerDict pid=859187) [2025-05-23 18:02:32] Capture cuda graph bs [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160]
Capturing batches (avail_mem=34.87 GB):   0%|          | 0/23 [00:00<?, ?it/s]
  0%|          | 0/23 [00:00<?, ?it/s]
Capturing batches (avail_mem=34.92 GB):   0%|          | 0/23 [00:00<?, ?it/s]
Capturing batches (avail_mem=34.65 GB):   4%|▍         | 1/23 [00:00<00:13,  1.60it/s]
Capturing batches (avail_mem=34.12 GB):   9%|▊         | 2/23 [00:00<00:09,  2.21it/s]
Capturing batches (avail_mem=34.01 GB):   9%|▊         | 2/23 [00:00<00:09,  2.21it/s]
(WorkerDict pid=859183) [2025-05-23 18:02:29] Attention backend not set. Use fa3 backend by default. [repeated 7x across cluster]
(WorkerDict pid=859183) [2025-05-23 18:02:29] Init torch distributed begin. [repeated 7x across cluster]
(WorkerDict pid=859189) [2025-05-23 18:02:30] Init torch distributed ends. mem usage=0.00 GB [repeated 7x across cluster]
(WorkerDict pid=859189) [2025-05-23 18:02:30] Load weight begin. avail mem=89.40 GB [repeated 7x across cluster]
(WorkerDict pid=859185) [2025-05-23 18:02:32] Load weight end. type=Qwen2ForCausalLM, dtype=torch.bfloat16, avail mem=86.57 GB, mem usage=1.03 GB. [repeated 7x across cluster]
(WorkerDict pid=859185) [2025-05-23 18:02:32] KV Cache is allocated. #tokens: 4502600, K size: 25.76 GB, V size: 25.76 GB [repeated 7x across cluster]
(WorkerDict pid=859185) [2025-05-23 18:02:32] Memory pool end. avail mem=34.40 GB [repeated 7x across cluster]
(WorkerDict pid=859185) [2025-05-23 18:02:32] Capture cuda graph begin. This can take up to several minutes. avail mem=34.31 GB [repeated 7x across cluster]
(WorkerDict pid=859185) [2025-05-23 18:02:33] Capture cuda graph bs [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160] [repeated 7x across cluster]
Capturing batches (avail_mem=34.24 GB):   0%|          | 0/23 [00:00<?, ?it/s] [repeated 6x across cluster]
Capturing batches (avail_mem=33.02 GB):  78%|███████▊  | 18/23 [00:05<00:01,  3.45it/s] [repeated 135x across cluster]
Capturing batches (avail_mem=33.52 GB):  91%|█████████▏| 21/23 [00:05<00:00,  4.16it/s]
Capturing batches (avail_mem=34.24 GB):  22%|██▏       | 5/23 [00:01<00:05,  3.38it/s] [repeated 2x across cluster]
(WorkerDict pid=859188) [2025-05-23 18:02:38] Capture cuda graph end. Time elapsed: 6.61 s. mem usage=1.42 GB. avail mem=33.51 GB.
(WorkerDict pid=859188) [2025-05-23 18:02:39] max_total_num_tokens=4584520, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=4097, context_len=32768
(WorkerDict pid=859187) [2025-05-23 18:02:39] max_total_num_tokens=4584520, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=4097, context_len=32768
(WorkerDict pid=859183) [2025-05-23 18:02:39] max_total_num_tokens=4532091, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=4097, context_len=32768
(WorkerDict pid=859189) [2025-05-23 18:02:39] max_total_num_tokens=4596808, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=4097, context_len=32768
(WorkerDict pid=859188) [2025-05-23 18:02:39] INFO:     Started server process [865038]
(WorkerDict pid=859188) [2025-05-23 18:02:39] INFO:     Waiting for application startup.
(WorkerDict pid=859188) [2025-05-23 18:02:39] INFO:     Application startup complete.
(WorkerDict pid=859188) [2025-05-23 18:02:39] INFO:     Uvicorn running on http://127.0.0.1:30006 (Press CTRL+C to quit)
(WorkerDict pid=857940) [2025-05-23 18:02:39] max_total_num_tokens=4522261, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=4097, context_len=32768
(WorkerDict pid=859188) [2025-05-23 18:02:39] Prefill batch. #new-seq: 1, #new-token: 1, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0
(WorkerDict pid=859184) [2025-05-23 18:02:40] max_total_num_tokens=4584520, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=4097, context_len=32768
(WorkerDict pid=859185) [2025-05-23 18:02:40] max_total_num_tokens=4502600, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=4097, context_len=32768
(WorkerDict pid=859186) [2025-05-23 18:02:41] max_total_num_tokens=3993387, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=4097, context_len=32768
(WorkerDict pid=859188) [2025-05-23 18:02:42] Cache not flushed because there are pending requests. #queue-req: 0, #running-req: 1
(WorkerDict pid=859188) /opt/ac2/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
(WorkerDict pid=859188)   warnings.warn(
(WorkerDict pid=859189) [2025-05-23 18:02:42] The server is fired up and ready to roll!
(WorkerDict pid=859188) [2025-05-23 18:02:42] Scheduler hit an exception: Traceback (most recent call last):
(WorkerDict pid=859188)   File "/opt/ac2/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 2269, in run_scheduler_process
(WorkerDict pid=859188)     scheduler.event_loop_overlap()
(WorkerDict pid=859188)   File "/opt/ac2/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(WorkerDict pid=859188)     return func(*args, **kwargs)
(WorkerDict pid=859188)            ^^^^^^^^^^^^^^^^^^^^^
(WorkerDict pid=859188)   File "/opt/ac2/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 663, in event_loop_overlap
(WorkerDict pid=859188)     result = self.run_batch(batch)
(WorkerDict pid=859188)              ^^^^^^^^^^^^^^^^^^^^^
(WorkerDict pid=859188)   File "/opt/ac2/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 1525, in run_batch
(WorkerDict pid=859188)     model_worker_batch = batch.get_model_worker_batch()
(WorkerDict pid=859188)                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(WorkerDict pid=859188)   File "/opt/ac2/lib/python3.12/site-packages/sglang/srt/managers/schedule_batch.py", line 1602, in get_model_worker_batch
(WorkerDict pid=859188)     seq_lens_cpu = self.seq_lens.cpu()
(WorkerDict pid=859188)                    ^^^^^^^^^^^^^^^^^^^
(WorkerDict pid=859188) RuntimeError: CUDA error: an illegal memory access was encountered
(WorkerDict pid=859188) CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(WorkerDict pid=859188) For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(WorkerDict pid=859188) Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(WorkerDict pid=859188) 
(WorkerDict pid=859188) 
(WorkerDict pid=859188) [2025-05-23 18:02:42] Received sigquit from a child process. It usually means the child failed.
(WorkerDict pid=859187) 
(WorkerDict pid=859187) 
(WorkerDict pid=857940) [2025-05-23 18:02:43] Cache flushed successfully!
Capturing batches (avail_mem=29.01 GB):  87%|████████▋ | 20/23 [00:07<00:00,  3.19it/s] [repeated 22x across cluster]
Capturing batches (avail_mem=28.99 GB): 100%|██████████| 23/23 [00:08<00:00,  2.81it/s] [repeated 23x across cluster]
(WorkerDict pid=859186) [2025-05-23 18:02:41] Capture cuda graph end. Time elapsed: 8.71 s. mem usage=1.42 GB. avail mem=28.99 GB. [repeated 7x across cluster]
(WorkerDict pid=859186) [2025-05-23 18:02:42] INFO:     Started server process [865039] [repeated 7x across cluster]
(WorkerDict pid=859186) [2025-05-23 18:02:42] INFO:     Waiting for application startup. [repeated 7x across cluster]
(WorkerDict pid=859186) [2025-05-23 18:02:42] INFO:     Application startup complete. [repeated 7x across cluster]
(WorkerDict pid=859186) [2025-05-23 18:02:42] INFO:     Uvicorn running on http://127.0.0.1:30004 (Press CTRL+C to quit) [repeated 7x across cluster]
(WorkerDict pid=859186) [2025-05-23 18:02:43] Prefill batch. #new-seq: 1, #new-token: 6, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0 [repeated 10x across cluster]
(WorkerDict pid=859187) [2025-05-23 18:02:42] Cache not flushed because there are pending requests. #queue-req: 0, #running-req: 1
(WorkerDict pid=859186) /opt/ac2/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html . [repeated 7x across cluster]
(WorkerDict pid=859186)   warnings.warn( [repeated 7x across cluster]
(WorkerDict pid=859186) [2025-05-23 18:02:44] The server is fired up and ready to roll! [repeated 5x across cluster]
(WorkerDict pid=859187) [2025-05-23 18:02:42] Scheduler hit an exception: Traceback (most recent call last):
(WorkerDict pid=859187)   File "/opt/ac2/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 2269, in run_scheduler_process
(WorkerDict pid=859187)     scheduler.event_loop_overlap()
(WorkerDict pid=859187)   File "/opt/ac2/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(WorkerDict pid=859187)     return func(*args, **kwargs)
(WorkerDict pid=859187)              ^^^^^^^^^^^^^^^^^^^^^ [repeated 2x across cluster]
(WorkerDict pid=859187)   File "/opt/ac2/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 663, in event_loop_overlap
(WorkerDict pid=859187)     result = self.run_batch(batch)
(WorkerDict pid=859187)   File "/opt/ac2/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 1525, in run_batch
(WorkerDict pid=859187)     model_worker_batch = batch.get_model_worker_batch()
(WorkerDict pid=859187)                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(WorkerDict pid=859187)   File "/opt/ac2/lib/python3.12/site-packages/sglang/srt/managers/schedule_batch.py", line 1602, in get_model_worker_batch
(WorkerDict pid=859187)     seq_lens_cpu = self.seq_lens.cpu()
(WorkerDict pid=859187)                    ^^^^^^^^^^^^^^^^^^^
(WorkerDict pid=859187) RuntimeError: CUDA error: an illegal memory access was encountered
(WorkerDict pid=859187) CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(WorkerDict pid=859187) For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(WorkerDict pid=859187) Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

[yuleiqin]

same error!!!!!

[yuleiqin]

same error!!!!!

:job_id:02000000
:actor_name:WorkerDict
Flash Attention 2.0 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen2ForCausalLM is torch.float32. You should run training or inference using Automatic Mixed-Precision via the with torch.autocast(device_type='torch_device'): decorator, or load the model with the torch_dtype argument. Example: model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", torch_dtype=torch.float16)
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with model.to('cuda').

Loading checkpoint shards: 0%| | 0/17 [00:00<?, ?it/s]
Loading checkpoint shards: 6%|▌ | 1/17 [00:41<11:08, 41.79s/it]
Loading checkpoint shards: 12%|█▏ | 2/17 [01:24<10:37, 42.49s/it]
Loading checkpoint shards: 18%|█▊ | 3/17 [02:04<09:35, 41.10s/it]
Loading checkpoint shards: 24%|██▎ | 4/17 [02:41<08:33, 39.47s/it]
Loading checkpoint shards: 29%|██▉ | 5/17 [03:14<07:25, 37.14s/it]
Loading checkpoint shards: 35%|███▌ | 6/17 [03:45<06:28, 35.28s/it]
Loading checkpoint shards: 41%|████ | 7/17 [04:18<05:45, 34.58s/it]
Loading checkpoint shards: 47%|████▋ | 8/17 [04:52<05:09, 34.38s/it]
Loading checkpoint shards: 53%|█████▎ | 9/17 [05:29<04:40, 35.04s/it]
Loading checkpoint shards: 59%|█████▉ | 10/17 [06:07<04:11, 35.99s/it]
Loading checkpoint shards: 65%|██████▍ | 11/17 [06:49<03:47, 37.88s/it]
Loading checkpoint shards: 71%|███████ | 12/17 [07:32<03:17, 39.48s/it]
Loading checkpoint shards: 76%|███████▋ | 13/17 [08:16<02:42, 40.64s/it]
Loading checkpoint shards: 82%|████████▏ | 14/17 [08:55<02:00, 40.30s/it]
Loading checkpoint shards: 88%|████████▊ | 15/17 [09:33<01:18, 39.49s/it]
Loading checkpoint shards: 94%|█████████▍| 16/17 [10:06<00:37, 37.45s/it]
Loading checkpoint shards: 100%|██████████| 17/17 [10:36<00:00, 35.37s/it]
Loading checkpoint shards: 100%|██████████| 17/17 [10:36<00:00, 37.44s/it]
[rank4]:[W523 07:36:31.835305812 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 4] using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.

0%| | 0/35 [00:00<?, ?it/s]
Capturing batches (avail_mem=19.66 GB): 0%| | 0/35 [00:00<?, ?it/s]
Capturing batches (avail_mem=19.66 GB): 3%|▎ | 1/35 [00:00<00:20, 1.63it/s]
Capturing batches (avail_mem=19.22 GB): 3%|▎ | 1/35 [00:00<00:20, 1.63it/s]
Capturing batches (avail_mem=19.22 GB): 6%|▌ | 2/35 [00:01<00:18, 1.83it/s]
Capturing batches (avail_mem=19.03 GB): 6%|▌ | 2/35 [00:01<00:18, 1.83it/s]
Capturing batches (avail_mem=19.03 GB): 9%|▊ | 3/35 [00:01<00:18, 1.72it/s]
Capturing batches (avail_mem=18.86 GB): 9%|▊ | 3/35 [00:01<00:18, 1.72it/s]
Capturing batches (avail_mem=18.86 GB): 11%|█▏ | 4/35 [00:02<00:16, 1.88it/s]
Capturing batches (avail_mem=18.68 GB): 11%|█▏ | 4/35 [00:02<00:16, 1.88it/s]
Capturing batches (avail_mem=18.68 GB): 14%|█▍ | 5/35 [00:02<00:15, 1.97it/s]
Capturing batches (avail_mem=18.52 GB): 14%|█▍ | 5/35 [00:02<00:15, 1.97it/s]
Capturing batches (avail_mem=18.52 GB): 17%|█▋ | 6/35 [00:03<00:14, 2.03it/s]
Capturing batches (avail_mem=18.36 GB): 17%|█▋ | 6/35 [00:03<00:14, 2.03it/s]
Capturing batches (avail_mem=18.36 GB): 20%|██ | 7/35 [00:03<00:13, 2.09it/s]
Capturing batches (avail_mem=18.20 GB): 20%|██ | 7/35 [00:03<00:13, 2.09it/s]
Capturing batches (avail_mem=18.20 GB): 23%|██▎ | 8/35 [00:04<00:12, 2.11it/s]
Capturing batches (avail_mem=18.05 GB): 23%|██▎ | 8/35 [00:04<00:12, 2.11it/s]
Capturing batches (avail_mem=18.05 GB): 26%|██▌ | 9/35 [00:04<00:12, 2.16it/s]
Capturing batches (avail_mem=17.91 GB): 26%|██▌ | 9/35 [00:04<00:12, 2.16it/s]
Capturing batches (avail_mem=17.91 GB): 29%|██▊ | 10/35 [00:04<00:11, 2.20it/s]
Capturing batches (avail_mem=17.77 GB): 29%|██▊ | 10/35 [00:04<00:11, 2.20it/s]
Capturing batches (avail_mem=17.77 GB): 31%|███▏ | 11/35 [00:05<00:10, 2.23it/s]
Capturing batches (avail_mem=17.63 GB): 31%|███▏ | 11/35 [00:05<00:10, 2.23it/s]
Capturing batches (avail_mem=17.63 GB): 34%|███▍ | 12/35 [00:05<00:10, 2.25it/s]
Capturing batches (avail_mem=17.51 GB): 34%|███▍ | 12/35 [00:05<00:10, 2.25it/s]
Capturing batches (avail_mem=17.51 GB): 37%|███▋ | 13/35 [00:06<00:09, 2.25it/s]
Capturing batches (avail_mem=17.38 GB): 37%|███▋ | 13/35 [00:06<00:09, 2.25it/s]
Capturing batches (avail_mem=17.38 GB): 40%|████ | 14/35 [00:06<00:09, 2.26it/s]
Capturing batches (avail_mem=17.27 GB): 40%|████ | 14/35 [00:06<00:09, 2.26it/s]
Capturing batches (avail_mem=17.27 GB): 43%|████▎ | 15/35 [00:07<00:08, 2.27it/s]
Capturing batches (avail_mem=17.16 GB): 43%|████▎ | 15/35 [00:07<00:08, 2.27it/s]
Capturing batches (avail_mem=17.16 GB): 46%|████▌ | 16/35 [00:07<00:08, 2.26it/s]
Capturing batches (avail_mem=17.05 GB): 46%|████▌ | 16/35 [00:07<00:08, 2.26it/s]
Capturing batches (avail_mem=17.05 GB): 49%|████▊ | 17/35 [00:07<00:07, 2.31it/s]
Capturing batches (avail_mem=17.02 GB): 49%|████▊ | 17/35 [00:07<00:07, 2.31it/s]
Capturing batches (avail_mem=17.02 GB): 51%|█████▏ | 18/35 [00:08<00:07, 2.33it/s]
Capturing batches (avail_mem=16.93 GB): 51%|█████▏ | 18/35 [00:08<00:07, 2.33it/s]
Capturing batches (avail_mem=16.93 GB): 54%|█████▍ | 19/35 [00:08<00:07, 2.26it/s]
Capturing batches (avail_mem=16.84 GB): 54%|█████▍ | 19/35 [00:08<00:07, 2.26it/s]
Capturing batches (avail_mem=16.84 GB): 57%|█████▋ | 20/35 [00:09<00:06, 2.30it/s]
Capturing batches (avail_mem=16.75 GB): 57%|█████▋ | 20/35 [00:09<00:06, 2.30it/s]
Capturing batches (avail_mem=16.75 GB): 60%|██████ | 21/35 [00:09<00:06, 2.27it/s]
Capturing batches (avail_mem=16.67 GB): 60%|██████ | 21/35 [00:09<00:06, 2.27it/s]
Capturing batches (avail_mem=16.67 GB): 63%|██████▎ | 22/35 [00:10<00:05, 2.28it/s]
Capturing batches (avail_mem=16.60 GB): 63%|██████▎ | 22/35 [00:10<00:05, 2.28it/s]
Capturing batches (avail_mem=16.60 GB): 66%|██████▌ | 23/35 [00:10<00:05, 2.31it/s]
Capturing batches (avail_mem=16.58 GB): 66%|██████▌ | 23/35 [00:10<00:05, 2.31it/s]
Capturing batches (avail_mem=16.58 GB): 69%|██████▊ | 24/35 [00:11<00:04, 2.32it/s]
Capturing batches (avail_mem=16.51 GB): 69%|██████▊ | 24/35 [00:11<00:04, 2.32it/s]
Capturing batches (avail_mem=16.51 GB): 71%|███████▏ | 25/35 [00:11<00:04, 2.33it/s]
Capturing batches (avail_mem=16.50 GB): 71%|███████▏ | 25/35 [00:11<00:04, 2.33it/s]
Capturing batches (avail_mem=16.50 GB): 74%|███████▍ | 26/35 [00:11<00:03, 2.36it/s]
Capturing batches (avail_mem=16.43 GB): 74%|███████▍ | 26/35 [00:11<00:03, 2.36it/s]
Capturing batches (avail_mem=16.43 GB): 77%|███████▋ | 27/35 [00:12<00:03, 2.25it/s]
Capturing batches (avail_mem=16.40 GB): 77%|███████▋ | 27/35 [00:12<00:03, 2.25it/s]
Capturing batches (avail_mem=16.40 GB): 80%|████████ | 28/35 [00:12<00:02, 2.34it/s]
Capturing batches (avail_mem=16.36 GB): 80%|████████ | 28/35 [00:12<00:02, 2.34it/s]
Capturing batches (avail_mem=16.36 GB): 83%|████████▎ | 29/35 [00:13<00:02, 2.35it/s]
Capturing batches (avail_mem=16.33 GB): 83%|████████▎ | 29/35 [00:13<00:02, 2.35it/s]
Capturing batches (avail_mem=16.33 GB): 86%|████████▌ | 30/35 [00:13<00:02, 2.37it/s]
Capturing batches (avail_mem=16.31 GB): 86%|████████▌ | 30/35 [00:13<00:02, 2.37it/s]
Capturing batches (avail_mem=16.31 GB): 89%|████████▊ | 31/35 [00:13<00:01, 2.39it/s]
Capturing batches (avail_mem=16.30 GB): 89%|████████▊ | 31/35 [00:13<00:01, 2.39it/s]
Capturing batches (avail_mem=16.30 GB): 91%|█████████▏| 32/35 [00:14<00:01, 2.33it/s]
Capturing batches (avail_mem=16.29 GB): 91%|█████████▏| 32/35 [00:14<00:01, 2.33it/s]
Capturing batches (avail_mem=16.29 GB): 94%|█████████▍| 33/35 [00:14<00:00, 2.34it/s]
Capturing batches (avail_mem=16.27 GB): 94%|█████████▍| 33/35 [00:14<00:00, 2.34it/s]
Capturing batches (avail_mem=16.27 GB): 97%|█████████▋| 34/35 [00:15<00:00, 2.35it/s]
Capturing batches (avail_mem=16.25 GB): 97%|█████████▋| 34/35 [00:15<00:00, 2.35it/s]
Capturing batches (avail_mem=16.25 GB): 100%|██████████| 35/35 [00:15<00:00, 2.33it/s]
Capturing batches (avail_mem=16.25 GB): 100%|██████████| 35/35 [00:15<00:00, 2.23it/s]
/usr/local/lib/python3.10/dist-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
warnings.warn(
[2025-05-23 07:45:49 TP3] Scheduler hit an exception: Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/scheduler.py", line 2026, in run_scheduler_process
scheduler.event_loop_overlap()
File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/scheduler.py", line 666, in event_loop_overlap
self.process_batch_result(
File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/scheduler.py", line 1426, in process_batch_result
self.process_batch_result_prefill(batch, result, launch_done)
File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/scheduler_output_processor_mixin.py", line 50, in process_batch_result_prefill
self.tp_worker.resolve_last_batch_result(
File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/tp_worker_overlap_thread.py", line 179, in resolve_last_batch_result
copy_done.synchronize()
File "/usr/local/lib/python3.10/dist-packages/torch/cuda/streams.py", line 224, in synchronize
super().synchronize()
RuntimeError: CUDA error: unspecified launch failure
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

terminate called after throwing an instance of 'c10::Error'
[2025-05-23 07:45:49] Received sigquit from a child process. It usually means the child failed.
[rank3]:[E523 07:45:49.024173609 ProcessGroupNCCL.cpp:1895] [PG ID 2 PG GUID 3 Rank 3] Process group watchdog thread terminated with exception: CUDA error: unspecified launch failure
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at /pytorch/c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fadad96c1b6 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7fadad915a76 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7fadadd86918 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7fad5bc1c556 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x7fad5bc298c0 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x617 (0x7fad5bc2b557 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7fad5bc2c6ed in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #7: + 0xdc253 (0x7fadb7627253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #8: + 0x94ac3 (0x7fadb9386ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #9: + 0x126850 (0x7fadb9418850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

terminate called recursively
Fatal Python error: Aborted

Thread 0x00007fa7fd4d2640 (most recent call first):
File "/usr/lib/python3.10/threading.py", line 324 in wait
File "/usr/lib/python3.10/threading.py", line 607 in wait
File "/usr/local/lib/python3.10/dist-packages/tqdm/_monitor.py", line 60 in run
File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007fa805750640 (most recent call first):
File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/scheduler.py", line 1599 in watchdog_thread
File "/usr/lib/python3.10/threading.py", line 953 in run
File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007fa8c0aa6640 (most recent call first):
File "/usr/lib/python3.10/threading.py", line 320 in wait
File "/usr/lib/python3.10/queue.py", line 171 in get
File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/tp_worker_overlap_thread.py", line 124 in forward_thread_func_
File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116 in decorate_context
File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/tp_worker_overlap_thread.py", line 112 in forward_thread_func
File "/usr/lib/python3.10/threading.py", line 953 in run
File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007fadb7518dc0 (most recent call first):
File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/scheduler.py", line 2043 in run_scheduler_process
File "/usr/lib/python3.10/multiprocessing/process.py", line 108 in run
File "/usr/lib/python3.10/multiprocessing/process.py", line 314 in _bootstrap
File "/usr/lib/python3.10/multiprocessing/spawn.py", line 129 in _main
File "/usr/lib/python3.10/multiprocessing/spawn.py", line 116 in spawn_main
File "", line 1 in

Extension modules: msgpack._cmsgpack, google._upb._message, psutil._psutil_linux, psutil._psutil_posix, setproctitle, yaml._yaml, _brotli, zstandard.backend_c, charset_normalizer.md, uvloop.loop, ray._raylet, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, multidict._multidict, yarl._quoting_c, aiohttp._helpers, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket, frozenlist._frozenlist, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, zmq.backend.cython.context, zmq.backend.cython.message, zmq.backend.cython.socket, zmq.backend.cython._device, zmq.backend.cython._poll, zmq.backend.cython._proxy_steerable, zmq.backend.cython._version, zmq.backend.cython.error, zmq.backend.cython.utils, markupsafe._speedups, PIL._imaging, PIL._imagingft, sklearn.__check_build._check_build, cython.cimports.libc.math, scipy._lib._ccallback_c, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg._matfuncs_expm, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, sklearn.utils.murmurhash, scipy.spatial._ckdtree, scipy._lib.messagestream, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.spatial.transform._rotation, scipy.optimize._group_columns, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize._cython_nnls, scipy._lib._uarray._uarray, scipy.linalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.optimize._direct, scipy.integrate._odepack what(): , CUDA error: unspecified launch failure
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at /pytorch/c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fadad96c1b6 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7fadad915a76 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7fadadd86918 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10_cuda.so)
frame #3: + 0x103aadc (0x7fad5ba65adc in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #4: + 0x10433c5 (0x7fad5ba6e3c5 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #5: + 0x6417b2 (0x7fada52d07b2 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so)
frame #6: + 0x6f30f (0x7fadad94d30f in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #7: c10::TensorImpl::~TensorImpl() + 0x21b (0x7fadad94633b in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #8: c10::TensorImpl::~TensorImpl() + 0x9 (0x7fadad9464e9 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #9: + 0x8fefb8 (0x7fada558dfb8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so)
frame #10: THPVariable_subclass_dealloc(_object*) + 0x2f6 (0x7fada558e306 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so)
frame #11: + 0x150660 (0x562ded273660 in sglang::scheduler_TP3)
frame #12: + 0x164518 (0x562ded287518 in sglang::scheduler_TP3)
frame #13: + 0x164545 (0x562ded287545 in sglang::scheduler_TP3)
frame #14: + 0x164545 (0x562ded287545 in sglang::scheduler_TP3)
frame #15: + 0x164545 (0x562ded287545 in sglang::scheduler_TP3)
frame #16: + 0x164545 (0x562ded287545 in sglang::scheduler_TP3)
frame #17: + 0x164545 (0x562ded287545 in sglang::scheduler_TP3)
frame #18: _PyEval_EvalFrameDefault + 0x293d (0x562ded2684ed in sglang::scheduler_TP3)
frame #19: _PyFunction_Vectorcall + 0x7c (0x562ded27d9fc in sglang::scheduler_TP3)
frame #20: _PyEval_EvalFrameDefault + 0x2a27 (0x562ded2685d7 in sglang::scheduler_TP3)
frame #21: _PyFunction_Vectorcall + 0x7c (0x562ded27d9fc in sglang::scheduler_TP3)
frame #22: _PyEval_EvalFrameDefault + 0x8ac (0x562ded26645c in sglang::scheduler_TP3)
frame #23: _PyFunction_Vectorcall + 0x7c (0x562ded27d9fc in sglang::scheduler_TP3)
frame #24: _PyEval_EvalFrameDefault + 0x8ac (0x562ded26645c in sglang::scheduler_TP3)
frame #25: _PyFunction_Vectorcall + 0x7c (0x562ded27d9fc in sglang::scheduler_TP3)
frame #26: _PyEval_EvalFrameDefault + 0x6bd (0x562ded26626d in sglang::scheduler_TP3)
frame #27: _PyFunction_Vectorcall + 0x7c (0x562ded27d9fc in sglang::scheduler_TP3)
frame #28: _PyEval_EvalFrameDefault + 0x198c (0x562ded26753c in sglang::scheduler_TP3)
frame #29: + 0x13f9c6 (0x562ded2629c6 in sglang::scheduler_TP3)
frame #30: PyEval_EvalCode + 0x86 (0x562ded358256 in sglang::scheduler_TP3)
frame #31: + 0x260108 (0x562ded383108 in sglang::scheduler_TP3)
frame #32: + 0x2599cb (0x562ded37c9cb in sglang::scheduler_TP3)
frame #33: PyRun_StringFlags + 0x81 (0x562ded375ab1 in sglang::scheduler_TP3)
frame #34: PyRun_SimpleStringFlags + 0x41 (0x562ded375961 in sglang::scheduler_TP3)
frame #35: Py_RunMain + 0x375 (0x562ded374b15 in sglang::scheduler_TP3)
frame #36: Py_BytesMain + 0x2d (0x562ded34b02d in sglang::scheduler_TP3)
frame #37: + 0x29d90 (0x7fadb931bd90 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #38: __libc_start_main + 0x80 (0x7fadb931be40 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #39: _start + 0x25 (0x562ded34af25 in sglang::scheduler_TP3)
scipy.integrate._quadpack
, scipy.integrate._vode, scipy.integrate._dop, scipy.integrate._lsoda, scipy.interpolate._fitpack, scipy.interpolate._dfitpack, scipy.interpolate._dierckx, scipy.interpolate._ppoly, scipy.interpolate._interpnd, scipy.interpolate._rbfinterp_pythran

[yuleiqin]

�[36m(TaskRunner pid=59362)�[0m Dumped generations to /cfs_turbo/yuleiqin/verl/checkpoints/maths_async_rl/Qwen2.5-32b_function_maths-async-sgl-multi-w-tool-verify-n4-real-32gpu_20250526_094300/rollouts/1.jsonl
�[36m(TaskRunner pid=59362)�[0m step:1 - global_seqlen/min:8124.000 - global_seqlen/max:22418.000 - global_seqlen/minmax_diff:14294.000 - global_seqlen/balanced_min:14394.000 - global_seqlen/balanced_max:15033.000 - global_seqlen/mean:14489.188 - actor/entropy_loss:0.475 - actor/pg_loss:0.046 - actor/pg_clipfrac:0.000 - actor/ppo_kl:0.000 - actor/pg_clipfrac_lower:0.000 - actor/grad_norm:1.739 - perf/mfu/actor:0.305 - perf/max_memory_allocated_gb:24.025 - perf/max_memory_reserved_gb:44.482 - perf/cpu_memory_used_gb:182.129 - actor/lr:0.000 - training/global_step:1.000 - training/epoch:0.000 - critic/score/mean:0.293 - critic/score/max:1.000 - critic/score/min:0.000 - critic/rewards/mean:0.293 - critic/rewards/max:1.000 - critic/rewards/min:0.000 - critic/advantages/mean:-0.033 - critic/advantages/max:1.500 - critic/advantages/min:-1.500 - critic/returns/mean:-0.033 - critic/returns/max:1.500 - critic/returns/min:-1.500 - response_length/mean:1362.789 - response_length/max:4417.000 - response_length/min:150.000 - response_length/clip_ratio:0.000 - prompt_length/mean:448.359 - prompt_length/max:611.000 - prompt_length/min:386.000 - prompt_length/clip_ratio:0.000 - timing_s/gen:299.410 - timing_s/reward:0.138 - timing_s/old_log_prob:10.123 - timing_s/adv:0.006 - timing_s/update_actor:65.845 - timing_s/dump_rollout_generations:0.532 - timing_s/step:376.189 - timing_per_token_ms/update_actor:0.142 - timing_per_token_ms/adv:0.000 - timing_per_token_ms/gen:0.858 - perf/total_num_tokens:463654.000 - perf/time_per_step:376.189 - perf/throughput:38.516
�[36m(TaskRunner pid=59362)�[0m
Training Progress: 0%!|(MISSING) | 1/12260 [06:16<1281:53:01, 376.44s/it]
�[36m(WorkerDict pid=1056, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m >>> maths verification answer='-0.5'
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m [2025-05-26 10:11:06 TP1] Scheduler hit an exception: Traceback (most recent call last):
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/scheduler.py", line 2026, in run_scheduler_process
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m scheduler.event_loop_overlap()
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m return func(args, kwargs)
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/scheduler.py", line 666, in event_loop_overlap
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m self.process_batch_result(
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/scheduler.py", line 1424, in process_batch_result
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m self.process_batch_result_decode(batch, result, launch_done)
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/scheduler_output_processor_mixin.py", line 198, in process_batch_result_decode
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m logits_output, next_token_ids = self.tp_worker.resolve_last_batch_result(
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/tp_worker_overlap_thread.py", line 179, in resolve_last_batch_result
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m copy_done.synchronize()
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m File "/usr/local/lib/python3.10/dist-packages/torch/cuda/streams.py", line 224, in synchronize
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m super().synchronize()
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m RuntimeError: CUDA error: unspecified launch failure
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m For debugging consider passing CUDA_LAUNCH_BLOCKING=1
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m terminate called after throwing an instance of 'c10::Error'
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m [2025-05-26 10:11:06] Received sigquit from a child process. It usually means the child failed.
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m what(): CUDA error: unspecified launch failure
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m For debugging consider passing CUDA_LAUNCH_BLOCKING=1
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m Exception raised from c10_cuda_check_implementation at /pytorch/c10/cuda/CUDAException.cpp:43 (most recent call first):
[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)[0m frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7ff2b6f6c1b6 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)[0m frame #1: c10::detail::torchCheckFail(char const, char const, unsigned int, std::string const&) + 0x64 (0x7ff2b6f15a76 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)[0m frame #2: c10::cuda::c10_cuda_check_implementation(int, char const, char const*, int, bool) + 0x118 (0x7ff2b73a3918 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10_cuda.so)
[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)[0m frame #3: + 0x103aadc (0x7ff265465adc in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)[0m frame #4: + 0x10433c5 (0x7ff26546e3c5 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)[0m frame #5: + 0x6417b2 (0x7ff2aecd07b2 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so)
[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)[0m frame #6: + 0x6f30f (0x7ff2b6f4d30f in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)[0m frame #7: c10::TensorImpl::~TensorImpl() + 0x21b (0x7ff2b6f4633b in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)[0m frame #8: c10::TensorImpl::~TensorImpl() + 0x9 (0x7ff2b6f464e9 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)[0m frame #9: + 0x8fefb8 (0x7ff2aef8dfb8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so)
[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)[0m frame #10: THPVariable_subclass_dealloc(_object*) + 0x2f6 (0x7ff2aef8e306 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so)
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m frame #11: + 0x150660 (0x55f5f43f6660 in sglang::scheduler_TP1)
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m frame #12: + 0x164518 (0x55f5f440a518 in sglang::scheduler_TP1)
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m frame #13: + 0x164545 (0x55f5f440a545 in sglang::scheduler_TP1)
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m frame #14: + 0x164545 (0x55f5f440a545 in sglang::scheduler_TP1)
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m frame #15: + 0x164545 (0x55f5f440a545 in sglang::scheduler_TP1)
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m frame #16: + 0x164545 (0x55f5f440a545 in sglang::scheduler_TP1)
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m frame #17: + 0x164545 (0x55f5f440a545 in sglang::scheduler_TP1)
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m frame #18: _PyEval_EvalFrameDefault + 0x293d (0x55f5f43eb4ed in sglang::scheduler_TP1)
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m frame #19: _PyFunction_Vectorcall + 0x7c (0x55f5f44009fc in sglang::scheduler_TP1)
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m frame #20: _PyEval_EvalFrameDefault + 0x2a27 (0x55f5f43eb5d7 in sglang::scheduler_TP1)
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m frame #21: _PyFunction_Vectorcall + 0x7c (0x55f5f44009fc in sglang::scheduler_TP1)
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m frame #22: _PyEval_EvalFrameDefault + 0x8ac (0x55f5f43e945c in sglang::scheduler_TP1)
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m frame #23: _PyFunction_Vectorcall + 0x7c (0x55f5f44009fc in sglang::scheduler_TP1)
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m frame #24: _PyEval_EvalFrameDefault + 0x8ac (0x55f5f43e945c in sglang::scheduler_TP1)
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m frame #25: _PyFunction_Vectorcall + 0x7c (0x55f5f44009fc in sglang::scheduler_TP1)
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m frame #26: _PyEval_EvalFrameDefault + 0x6bd (0x55f5f43e926d in sglang::scheduler_TP1)
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m frame #27: _PyFunction_Vectorcall + 0x7c (0x55f5f44009fc in sglang::scheduler_TP1)
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m frame #28: _PyEval_EvalFrameDefault + 0x198c (0x55f5f43ea53c in sglang::scheduler_TP1)
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m frame #29: + 0x13f9c6 (0x55f5f43e59c6 in sglang::scheduler_TP1)
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m frame #30: PyEval_EvalCode + 0x86 (0x55f5f44db256 in sglang::scheduler_TP1)
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m frame #31: + 0x260108 (0x55f5f4506108 in sglang::scheduler_TP1)
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m frame #32: + 0x2599cb (0x55f5f44ff9cb in sglang::scheduler_TP1)
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m frame #33: PyRun_StringFlags + 0x81 (0x55f5f44f8ab1 in sglang::scheduler_TP1)
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m frame #34: PyRun_SimpleStringFlags + 0x41 (0x55f5f44f8961 in sglang::scheduler_TP1)
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m frame #35: Py_RunMain + 0x375 (0x55f5f44f7b15 in sglang::scheduler_TP1)
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m frame #36: Py_BytesMain + 0x2d (0x55f5f44ce02d in sglang::scheduler_TP1)
[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)[0m frame #37: + 0x29d90 (0x7ff2c2c5bd90 in /usr/lib/x86_64-linux-gnu/libc.so.6)
[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)[0m frame #38: __libc_start_main + 0x80 (0x7ff2c2c5be40 in /usr/lib/x86_64-linux-gnu/libc.so.6)
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m frame #39: _start + 0x25 (0x55f5f44cdf25 in sglang::scheduler_TP1)
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m Fatal Python error: Aborted
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m Thread 0x00007fee009fe640 (most recent call first):
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m File "/usr/lib/python3.10/threading.py", line 324 in wait
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m File "/usr/lib/python3.10/threading.py", line 607 in wait
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m File "/usr/local/lib/python3.10/dist-packages/tqdm/_monitor.py", line 60 in run
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m Thread 0x00007fee01fff640 (most recent call first):
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/scheduler.py", line 1599 in watchdog_thread
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m File "/usr/lib/python3.10/threading.py", line 953 in run
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m File "/usr/lib/python3.10/threading.py", line 973 in bootstrap
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m Thread 0x00007fee04c49640 (most recent call first):
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m File "/usr/lib/python3.10/threading.py", line 320 in wait
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m File "/usr/lib/python3.10/queue.py", line 171 in get
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/tp_worker_overlap_thread.py", line 124 in forward_thread_func
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116 in decorate_context
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/tp_worker_overlap_thread.py", line 112 in forward_thread_func
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m File "/usr/lib/python3.10/threading.py", line 953 in run
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m Current thread 0x00007ff2c0e58dc0 (most recent call first):
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/scheduler.py", line 2043 in run_scheduler_process
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m File "/usr/lib/python3.10/multiprocessing/process.py", line 108 in run
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m File "/usr/lib/python3.10/multiprocessing/process.py", line 314 in _bootstrap
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m File "/usr/lib/python3.10/multiprocessing/spawn.py", line 129 in _main
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m File "/usr/lib/python3.10/multiprocessing/spawn.py", line 116 in spawn_main
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m File "", line 1 in
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m
�[36m(WorkerDict pid=1052, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m Extension modules: msgpack._cmsgpack, google._upb._message, psutil._psutil_linux, psutil._psutil_posix, setproctitle, yaml._yaml, _brotli, zstandard.backend_c, charset_normalizer.md, uvloop.loop, ray._raylet, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, multidict._multidict, yarl._quoting_c, aiohttp._helpers, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket, frozenlist._frozenlist, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, zmq.backend.cython.context, zmq.backend.cython.message, zmq.backend.cython.socket, zmq.backend.cython._device, zmq.backend.cython._poll, zmq.backend.cython._proxy_steerable, zmq.backend.cython._version, zmq.backend.cython.error, zmq.backend.cython.utils, markupsafe._speedups, PIL._imaging, PIL._imagingft, sklearn.__check_build._check_build, cython.cimports.libc.math, scipy._lib._ccallback_c, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg._matfuncs_expm, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, sklearn.utils.murmurhash, scipy.spatial._ckdtree, scipy._lib.messagestream, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.spatial.transform._rotation, scipy.optimize._group_columns, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize._cython_nnls, scipy._lib._uarray._uarray, scipy.linalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.optimize._direct, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, scipy.integrate._dop, scipy.integrate._lsoda, scipy.interpolate._fitpack, scipy.interpolate._dfitpack, scipy.interpolate._dierckx, scipy.interpolate._ppoly, scipy.interpolate._interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, scipy.interpolate._bspl, scipy.special.cython_special, scipy.stats._stats, scipy.stats._sobol, scipy.stats._qmc_cy, scipy.stats._biasedurn, scipy.stats._stats_pythran, scipy.stats._levy_stable.levyst, scipy.stats._ansari_swilk_statistics, scipy.stats._mvn, scipy.stats._rcont.rcont, scipy.ndimage._nd_image, scipy.ndimage._rank_filter_1d, _ni_label, scipy.ndimage._ni_label, sklearn.utils._isfinite, sklearn.utils._openmp_helpers, sklearn.utils._logistic_sigmoid, sklearn.utils.sparsefuncs_fast, sklearn.preprocessing._csr_polynomial_expansion, sklearn.utils._typedefs, sklearn.utils._readonly_array_wrapper, sklearn.metrics._dist_metrics, sklearn.metrics.cluster._expected_mutual_info_fast, sklearn.metrics._pairwise_distances_reduction._datasets_pair, sklearn.utils._cython_blas, sklearn.metrics._pairwise_distances_reduction._base, sklearn.metrics._pairwise_distances_reduction._middle_term_computer, sklearn.utils._heap, sklearn.utils._sorting, sklearn.metrics._pairwise_distances_reduction._argkmin, sklearn.utils._vector_sentinel, sklearn.metrics._pairwise_distances_reduction._radius_neighbors, sklearn.metrics._pairwise_fast, sentencepiece._sentencepiece, ujson, msgspec._core, _cffi_backend, Cython.Utils, Cython.Plex.Actions, Cython.Plex.Transitions, Cython.Plex.Machines, Cython.Plex.DFA, Cython.Plex.Scanners, Cython.Compiler.Scanning, Cython.StringIOTree, Cython.Compiler.Code, regex._regex, cuda_utils, pyarrow.lib, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pyarrow._compute, pandas._libs.ops, numexpr.interpreter, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, sklearn.utils._random, __triton_launcher (total: 219)
�[36m(WorkerDict pid=1061, ip=train-1328084434016812288-a38undppxmgw-worker-2)�[0m >>> maths verification answer='0'�[32m [repeated 6x across cluster]�[0m
�[36m(WorkerDict pid=1056, ip=train-1328084434016812288-a38undppxmgw-worker-0)�[0m >>> maths verification answer='Correct'
�[33m(raylet)�[0m A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: ffffffffffffffff02cc2f412e7b790ae1c9baec02000000 Worker ID: 787b352d889b62e425eeb39b1ce9e488e588fcc0d115e646f20cf967 Node ID: 955ee609a8008acbbb0b7e113f3b3cd78524c2b47be69db23714967f Worker IP address: train-1328084434016812288-a38undppxmgw-worker-0 Worker port: 10004 Worker PID: 1052 Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors.
Error executing job with overrides: ['algorithm.adv_estimator=grpo', 'data.train_batch_size=64', 'data.max_prompt_length=8192', 'data.max_response_length=8192', 'data.filter_overlong_prompts=False', 'data.truncation=error', 'data.return_raw_chat=True', 'actor_rollout_ref.model.path=/cfs/yuleiqin/models/Qwen2.5-32B-Instruct_Qwen', 'actor_rollout_ref.actor.optim.lr=1e-6', 'actor_rollout_ref.model.use_remove_padding=True', 'actor_rollout_ref.actor.ppo_mini_batch_size=8', 'actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=1', 'actor_rollout_ref.actor.use_kl_loss=False', 'actor_rollout_ref.actor.kl_loss_coef=0.001', 'actor_rollout_ref.actor.kl_loss_type=low_var_kl', 'actor_rollout_ref.actor.entropy_coeff=0', 'actor_rollout_ref.model.enable_gradient_checkpointing=True', 'actor_rollout_ref.actor.fsdp_config.param_offload=False', 'actor_rollout_ref.actor.fsdp_config.optimizer_offload=False', 'actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=32', 'actor_rollout_ref.rollout.tensor_model_parallel_size=2', 'actor_rollout_ref.rollout.name=sglang_async', 'actor_rollout_ref.rollout.gpu_memory_utilization=0.7', 'actor_rollout_ref.rollout.n=4', 'actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=32', 'actor_rollout_ref.ref.fsdp_config.param_offload=True', 'algorithm.use_kl_in_reward=False', 'trainer.val_before_train=False', 'trainer.critic_warmup=0', 'trainer.logger=[console,wandb]', 'trainer.project_name=maths_async_rl', 'trainer.experiment_name=Qwen2.5-32b_function_maths-async-sgl-multi-w-tool-verify-n4-real-32gpu_20250526_094300', 'trainer.n_gpus_per_node=8', 'trainer.nnodes=4', 'trainer.save_freq=5', 'trainer.test_freq=-1', 'trainer.total_epochs=20', 'custom_reward_function.path=/cfs_turbo/yuleiqin/verl/examples/grpo_trainer_tool/rewards/reward_score.py', 'custom_reward_function.name=default_compute_score', 'reward_model.reward_manager=agent', 'actor_rollout_ref.actor.ppo_max_token_len_per_gpu=16384', 'actor_rollout_ref.rollout.log_prob_max_token_len_per_gpu=16384', 'actor_rollout_ref.ref.log_prob_max_token_len_per_gpu=16384', 'critic.ppo_max_token_len_per_gpu=16384', 'critic.forward_max_token_len_per_gpu=16384', 'data.train_files=data/deepscaleR/train.parquet', 'data.val_files=data/deepscaleR/test.parquet', 'actor_rollout_ref.rollout.multi_turn.tool_config_path=/cfs_turbo/yuleiqin/verl/examples/sglang_multiturn/config/tool_config/gsm8k_tool_config.yaml']
Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/cfs_turbo/yuleiqin/verl/verl/trainer/main_ppo.py", line 246, in
main()
File "/usr/local/lib/python3.10/dist-packages/hydra/main.py", line 94, in decorated_main
_run_hydra(
File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 394, in _run_hydra
_run_app(
File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 457, in _run_app
run_and_report(
File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 223, in run_and_report
raise ex
File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 220, in run_and_report
return func()
File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 458, in
lambda: hydra.run(
File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/hydra.py", line 132, in run
_ = ret.return_value
File "/usr/local/lib/python3.10/dist-packages/hydra/core/utils.py", line 260, in return_value
raise self._return_value
File "/usr/local/lib/python3.10/dist-packages/hydra/core/utils.py", line 186, in run_job
ret.return_value = task_function(task_cfg)
File "/cfs_turbo/yuleiqin/verl/verl/trainer/main_ppo.py", line 64, in main
run_ppo(config)
File "/cfs_turbo/yuleiqin/verl/verl/trainer/main_ppo.py", line 76, in run_ppo
ray.get(runner.run.remote(config))
File "/usr/local/lib/python3.10/dist-packages/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
return fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/ray/_private/worker.py", line 2771, in get
values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
File "/usr/local/lib/python3.10/dist-packages/ray/_private/worker.py", line 919, in get_objects
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(ActorDiedError): �[36mray::TaskRunner.run()�[39m (pid=59362, ip=train-1328084434016812288-a38undppxmgw-master-0, actor_id=302a3d8eba009bb8c01b4e4f02000000, repr=<main_ppo.TaskRunner object at 0x7f919ebeb010>)
File "/cfs_turbo/yuleiqin/verl/verl/trainer/main_ppo.py", line 183, in run
trainer.fit()
File "/cfs_turbo/yuleiqin/verl/verl/trainer/ppo/ray_trainer.py", line 911, in fit
gen_batch_output = self.actor_rollout_wg.generate_sequences(gen_batch)
File "/cfs_turbo/yuleiqin/verl/verl/single_controller/ray/base.py", line 49, in func
output = ray.get(output)
ray.exceptions.ActorDiedError: The actor died unexpectedly before finishing this task.
class_name: create_colocated_worker_cls..WorkerDict
actor_id: 02cc2f412e7b790ae1c9baec02000000
pid: 1052
name: S9ujDkWorkerDict_1:0
namespace: 9eff1230-e66e-4016-9188-dbbcdae1a15a
ip: train-1328084434016812288-a38undppxmgw-worker-0
The actor is dead because its worker process has died. Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors.
�[36m(WorkerDict pid=59876)�[0m >>> maths verification answer='7'�[32m [repeated 9x across cluster]�[0m
2025-05-26 10:11:15,734 ERR cli.py:71 -- �[31m---------------------------------------�[39m
2025-05-26 10:11:15,734 ERR cli.py:72 -- �[31mJob 'raysubmit_yRQZE7NFFpqeytpt' failed�[39m
2025-05-26 10:11:15,734 ERR cli.py:73 -- �[31m---------------------------------------�[39m
2025-05-26 10:11:15,735 INFO cli.py:86 -- Status message: Job entrypoint command failed with exit code 1, last available logs (truncated to 20,000 chars):
output = ray.get(output)
ray.exceptions.ActorDiedError: The actor died unexpectedly before finishing this task.
class_name: create_colocated_worker_cls..WorkerDict
actor_id: 02cc2f412e7b790ae1c9baec02000000
pid: 1052
name: S9ujDkWorkerDict_1:0
namespace: 9eff1230-e66e-4016-9188-dbbcdae1a15a
ip: train-1328084434016812288-a38undppxmgw-worker-0
The actor is dead because its worker process has died. Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors.
�[36m(WorkerDict pid=59876)�[0m >>> maths verification answer='7'�[32m [repeated 9x across cluster]�[0m

[jybsuper]

@MrAta @yuleiqin @liulurong @kemuxiaozi000 I found that the workaround for the OOM issue also resolves this illegal memory access error. I’ve opened a PR #1686 . Could you give it a try and let me know if it works on your end?

[kemuxiaozi000]

@MrAta @yuleiqin @liulurong @kemuxiaozi000 I found that the workaround for the OOM issue also resolves this illegal memory access error. I’ve opened a PR #1686 . Could you give it a try and let me know if it works on your end?

@jybsuper It's not worked for my case. still have the error

[ysy-phoenix]

@MrAta @yuleiqin @liulurong @kemuxiaozi000 I found that the workaround for the OOM issue also resolves this illegal memory access error. I’ve opened a PR #1686 . Could you give it a try and let me know if it works on your end?

I tested the above patch, but it did not resolve the issue. After a few training steps, the same error as before reappeared.