Multi-Turn Rollout on a single GPU

[ltdemey]

TLDR;

Running into the assertion-error at /verl/workers/rollout/vllm_rollout/vllm_async_server.py:55
when running a single node variant of verl/tests/rollout/test_vllm_multi_turn.py. (See script in Appendix A)

Description

Given the exciting work on multi-turn rollouts introduced in #1138, we are trying to add multimodal environment interactions (similar idea as described in the VAGEN-paper)).

However, I run into an assertion error when running the AsyncLLMManager with a single AsyncLLMWorker. I tried going through the debugger for quite some while but cannot figure out why the Executor does not seem to find the Actors.

Reproduction

(see Appendix A for script)

python tests/rollout/test_vllm_multi_turn.py

2025-05-02 14:57:34,894 DEBUG worker.py:1576 -- Automatically increasing RLIMIT_NOFILE to max value of 65536
2025-05-02 14:57:34,897 DEBUG node.py:293 -- Setting node ID to d1cce7a158262a4f3567d99a4612737ffef92c57256a3bc1d555541b
2025-05-02 14:57:34,903 DEBUG node.py:1401 -- Process STDOUT and STDERR is being redirected to /tmp/ray/session_2025-05-02_14-57-34_895398_2793/logs.
2025-05-02 14:57:35,829 DEBUG node.py:1430 -- Process STDOUT and STDERR is being redirected to /tmp/ray/session_2025-05-02_14-57-34_895398_2793/logs.
2025-05-02 14:57:35,861 DEBUG npu.py:60 -- Could not import AscendCL: No module named 'acl'
2025-05-02 14:57:35,863 DEBUG tpu.py:115 -- Failed to detect number of TPUs: [Errno 2] No such file or directory: '/dev/vfio'
2025-05-02 14:57:35,864 DEBUG services.py:2140 -- Determine to start the Plasma object store with 9.49 GB memory using /dev/shm.
2025-05-02 14:57:35,912 INFO worker.py:1832 -- Started a local Ray instance. View the dashboard at 127.0.0.1:8265 
2025-05-02 14:57:36,852 WARNING __init__.py:161 -- DeprecationWarning: `ray.state.available_resources_per_node` is a private attribute and access will be removed in a future Ray version.
colocated worker base class <class 'verl.single_controller.base.worker.Worker'>
bind role actor_rollout method execute_method to class <class 'verl.single_controller.ray.base.create_colocated_worker_cls.<locals>.WorkerDict'>
WARNING:2025-05-02 14:57:37,950:Waiting for register center actor 5Y8l0k_register_center to be ready. Elapsed time: 0 seconds out of 300 seconds.
(WorkerDict pid=3433) Model config after override: Qwen2Config {
(WorkerDict pid=3433)   "architectures": [
(WorkerDict pid=3433)     "Qwen2ForCausalLM"
(WorkerDict pid=3433)   ],
(WorkerDict pid=3433)   "attention_dropout": 0.0,
(WorkerDict pid=3433)   "eos_token_id": 151645,
(WorkerDict pid=3433)   "hidden_act": "silu",
(WorkerDict pid=3433)   "hidden_size": 896,
(WorkerDict pid=3433)   "initializer_range": 0.02,
(WorkerDict pid=3433)   "intermediate_size": 4864,
(WorkerDict pid=3433)   "max_position_embeddings": 32768,
(WorkerDict pid=3433)   "max_window_layers": 21,
(WorkerDict pid=3433)   "model_type": "qwen2",
(WorkerDict pid=3433)   "num_attention_heads": 14,
(WorkerDict pid=3433)   "num_hidden_layers": 24,
(WorkerDict pid=3433)   "num_key_value_heads": 2,
(WorkerDict pid=3433)   "pad_token_id": 151643,
(WorkerDict pid=3433)   "rms_norm_eps": 1e-06,
(WorkerDict pid=3433)   "rope_scaling": null,
(WorkerDict pid=3433)   "rope_theta": 1000000.0,
(WorkerDict pid=3433)   "sliding_window": 32768,
(WorkerDict pid=3433)   "tie_word_embeddings": true,
(WorkerDict pid=3433)   "torch_dtype": "bfloat16",
(WorkerDict pid=3433)   "transformers_version": "4.51.1",
(WorkerDict pid=3433)   "use_cache": true,
(WorkerDict pid=3433)   "use_sliding_window": false,
(WorkerDict pid=3433)   "vocab_size": 151936
(WorkerDict pid=3433) }
(WorkerDict pid=3433) 
(WorkerDict pid=3433) Flash Attention 2.0 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen2ForCausalLM is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `torch_dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", torch_dtype=torch.float16)`
(WorkerDict pid=3433) You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
(WorkerDict pid=3433) NCCL version 2.21.5+cuda12.4
(WorkerDict pid=3433) [rank0]:[W502 14:57:46.146030738 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 0]  using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.
(WorkerDict pid=3433) Qwen2ForCausalLM contains 494.03M parameters
(WorkerDict pid=3433) wrap_policy: functools.partial(<function _or_policy at 0x7f8717a2e170>, policies=[functools.partial(<function transformer_auto_wrap_policy at 0x7f8717a2e050>, transformer_layer_cls={<class 'transformers.models.qwen2.modeling_qwen2.Qwen2DecoderLayer'>})])
(WorkerDict pid=3433) /usr/local/lib/python3.10/dist-packages/torch/distributed/fsdp/_init_utils.py:444: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1.
(WorkerDict pid=3433)   warnings.warn(
(WorkerDict pid=3433) Total steps: -1, num_warmup_steps: 0
(WorkerDict pid=3433) Actor use_remove_padding=False
(WorkerDict pid=3433) /usr/local/lib/python3.10/dist-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
(WorkerDict pid=3433)   warnings.warn(
INFO 05-02 14:57:49 [__init__.py:239] Automatically detected platform cuda.
(AsyncvLLMServer pid=3620) FastAPI startup
(AsyncvLLMServer pid=3620) override_generation_config: {'n': 1, 'logprobs': 0, 'max_tokens': 1024, 'temperature': 1.0, 'top_k': -1, 'top_p': 1, 'ignore_eos': False}
(AsyncvLLMServer pid=3620) WARNING 05-02 14:58:04 [arg_utils.py:1713] Detected VLLM_USE_V1=1 with Engine in background thread. Usage should be considered experimental. Please report any issues on Github.
(AsyncvLLMServer pid=3620) WARNING 05-02 14:58:04 [cuda.py:96] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used
(AsyncvLLMServer pid=3620) WARNING 05-02 14:58:04 [core_client.py:368] SIGUSR1 handler not installed because we are not running in the main thread. In this case the forked engine process may not be killed when an exception is raised, and you need to handle the engine process shutdown manually.
(AsyncvLLMServer pid=3620) 2025-05-02 14:58:11,322      INFO worker.py:1832 -- Started a local Ray instance. View the dashboard at 127.0.0.1:8266 
(AsyncvLLMServer pid=3620) EngineCoreProc.run_engine_core
(AsyncvLLMServer pid=3620) EngineCoreProc.__init__
(AsyncvLLMServer pid=3620) EngineCore.__init__ - Start
(AsyncvLLMServer pid=3620) EngineCore.__init__ - Before model executor initialization
(AsyncvLLMServer pid=3620) namespace: 39fff769-940c-4c88-bb08-0ad63fb5e321
(AsyncvLLMServer pid=3620) actor_names: []
(AsyncvLLMServer pid=3620) all actors: []
(AsyncvLLMServer pid=3620) ERROR 05-02 14:58:12 [core.py:406] EngineCore hit an exception: Traceback (most recent call last):
(AsyncvLLMServer pid=3620) ERROR 05-02 14:58:12 [core.py:406]   File "/home/ec2-user/vllm/vllm/v1/engine/core.py", line 394, in run_engine_core
(AsyncvLLMServer pid=3620) ERROR 05-02 14:58:12 [core.py:406]     engine_core = EngineCoreProc(*args, **kwargs)
(AsyncvLLMServer pid=3620) ERROR 05-02 14:58:12 [core.py:406]   File "/home/ec2-user/vllm/vllm/v1/engine/core.py", line 335, in __init__
(AsyncvLLMServer pid=3620) ERROR 05-02 14:58:12 [core.py:406]     super().__init__(vllm_config, executor_class, log_stats)
(AsyncvLLMServer pid=3620) ERROR 05-02 14:58:12 [core.py:406]   File "/home/ec2-user/vllm/vllm/v1/engine/core.py", line 68, in __init__
(AsyncvLLMServer pid=3620) ERROR 05-02 14:58:12 [core.py:406]     self.model_executor = executor_class(vllm_config)
(AsyncvLLMServer pid=3620) ERROR 05-02 14:58:12 [core.py:406]   File "/home/ec2-user/vllm/vllm/executor/executor_base.py", line 52, in __init__
(AsyncvLLMServer pid=3620) ERROR 05-02 14:58:12 [core.py:406]     self._init_executor()
(AsyncvLLMServer pid=3620) ERROR 05-02 14:58:12 [core.py:406]   File "/home/ec2-user/verl/verl/workers/rollout/vllm_rollout/vllm_async_server.py", line 58, in _init_executor
(AsyncvLLMServer pid=3620) ERROR 05-02 14:58:12 [core.py:406]     assert len(actor_names) == vllm_dp_size * vllm_tp_size, f"instance_id: {self.vllm_config.instance_id} has {len(actor_names)} actors, but vllm_dp_size: {vllm_dp_size} * vllm_tp_size: {vllm_tp_size} = {vllm_dp_size * vllm_tp_size} is expected."
(AsyncvLLMServer pid=3620) ERROR 05-02 14:58:12 [core.py:406] AssertionError: instance_id: 39fff769-940c-4c88-bb08-0ad63fb5e321:5Y8l0k:1:0 has 0 actors, but vllm_dp_size: 1 * vllm_tp_size: 1 = 1 is expected.
(AsyncvLLMServer pid=3620) ERROR 05-02 14:58:12 [core.py:406] 
(raylet) A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: ffffffffffffffff4d25a17abd5c9be7ad69593a01000000 Worker ID: 5e1994c85b7e38c75d974f2886346b5662a2e33db3c8f160d894f1fb Node ID: d1cce7a158262a4f3567d99a4612737ffef92c57256a3bc1d555541b Worker IP address: xxx Worker port: 44215 Worker PID: 3620 Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors.
Traceback (most recent call last):
  File "/home/ec2-user/verl/tests/rollout/test_vllm_multi_turn.py", line 179, in <module>
    test_vllm_multi_turn()
  File "/home/ec2-user/verl/tests/rollout/test_vllm_multi_turn.py", line 100, in test_vllm_multi_turn
    async_rollout_manager = AsyncLLMServerManager(
  File "/home/ec2-user/verl/verl/workers/rollout/async_server.py", line 271, in __init__
    ray.get([worker.init_engine.remote() for worker in self.async_llm_servers])
  File "/usr/local/lib/python3.10/dist-packages/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/ray/_private/worker.py", line 2771, in get
    values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
  File "/usr/local/lib/python3.10/dist-packages/ray/_private/worker.py", line 921, in get_objects
    raise value
ray.exceptions.ActorDiedError: The actor died unexpectedly before finishing this task.
        class_name: AsyncvLLMServer
        actor_id: 4d25a17abd5c9be7ad69593a01000000
        pid: 3620
        name: async_llm_worker_0
        namespace: 39fff769-940c-4c88-bb08-0ad63fb5e321
        ip: xxx
The actor is dead because its worker process has died. Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors.
root@ip-172-31-44-216:/home/ec2-user/verl# 

I correctly see the necessary Actors running in the Ray dashboard.

Appendix A

verl/tests/rollout/test_vllm_multi_turn.py

# FIRST PART OF FILE UNCHANGED 

def test_vllm_multi_turn():
    config = OmegaConf.load("verl/trainer/config/ppo_trainer.yaml")
    model_path = "Qwen/Qwen2.5-0.5B-Instruct"
    model_name = "/".join(model_path.split("/")[-2:])
    config.actor_rollout_ref.model.path = model_path
    config.actor_rollout_ref.rollout.mode = "async"
    config.actor_rollout_ref.rollout.chat_scheduler = "examples.ppo_trainer.naive_chat_scheduler.NaiveChatCompletionScheduler"
    config.actor_rollout_ref.rollout.prompt_length = 2048
    config.actor_rollout_ref.rollout.response_length = 1024


    # test sleep/wake_up with fsdp offload
    config.actor_rollout_ref.actor.fsdp_config.param_offload = False
    config.actor_rollout_ref.actor.fsdp_config.optimizer_offload = False

    # add vm specific config
    config.ray_init.num_cpus = 8
    config.trainer.n_gpus_per_node = int(os.environ.get("NGPUS_PER_NODE", 1))
    config.actor_rollout_ref.rollout.tensor_model_parallel_size = int(os.environ.get("GEN_TP", 1))
    
    # =========================== 1. Create hybrid ActorRollout workers ===========================
    # make openai client happy
    os.environ["no_proxy"] = ""
    os.environ["http_proxy"] = ""
    os.environ["https_proxy"] = ""

    ray.init(
        logging_level=logging.DEBUG,
        runtime_env={
            "env_vars": {
                "TOKENIZERS_PARALLELISM": "true",
                "NCCL_DEBUG": "WARN",
                "VLLM_LOGGING_LEVEL": "WARN",
                "VLLM_USE_V1": "1",
            }
        }
    )
   
## REST OF FILE UNCHANGED

[eric-haibin-lin]

cc @wuxibin89

[wuxibin89]

@ltdemey It's a little weird I can't reproduce the problem.

AssertionError: instance_id: 39fff769-940c-4c88-bb08-0ad63fb5e321:5Y8l0k:1:0 has 0 actors, but vllm_dp_size: 1 * vllm_tp_size: 1 = 1 is expected.

This assertion says that AsyncvLLMServer can't find named ray actors starts with 5Y8l0k(which should be launched by the actor_rollout_wg). Please add a debug message to print the named actor as below:

diff --git a/tests/rollout/test_vllm_multi_turn.py b/tests/rollout/test_vllm_multi_turn.py
index d37b8e8..5201416 100644
--- a/tests/rollout/test_vllm_multi_turn.py
+++ b/tests/rollout/test_vllm_multi_turn.py
@@ -42,6 +42,9 @@ def test_vllm_multi_turn():
     config.actor_rollout_ref.actor.fsdp_config.param_offload = True
     config.actor_rollout_ref.actor.fsdp_config.optimizer_offload = True
 
+    config.trainer.n_gpus_per_node = 1
+    config.actor_rollout_ref.rollout.tensor_model_parallel_size = 1
+
     # =========================== 1. Create hybrid ActorRollout workers ===========================
     # make openai client happy
     os.environ["no_proxy"] = ""
@@ -88,6 +91,8 @@ def test_vllm_multi_turn():
     actor_rollout_wg = all_wg["actor_rollout"]
     actor_rollout_wg.init_model()
 
+    actor_names = [actor_name for actor_name in ray.util.list_named_actors() if actor_name.startswith(f"{actor_rollout_wg.name_prefix}WorkerDict")]
+    print(f"[DEBUG]: actor_names: {actor_names}")
     # =========================== 2. Create AsyncLLMServerManager  ===========================
     async_rollout_manager = AsyncLLMServerManager(
         config=config.actor_rollout_ref,

Which should print message like

WARNING:2025-05-03 10:49:06,656:Waiting for register center actor PP9ggc_register_center to be ready. Elapsed time: 0 seconds out of 300 seconds.
...
[DEBUG]: actor_names: ['PP9ggcWorkerDict_0:0']
...

[ltdemey]

Hm, it's indeed unexpected that you cannot recreate the error. I'm running on a simple g6.2xlarge ec2 instance.

In any case, the log accurately identifies that a ray actor is launched ([DEBUG]: actor_names: ['misQrdWorkerDict_0:0']). I also tried to add a breakpoint() at verl/verl/workers/rollout/vllm_rollout/vllm_async_server.py:55 and notice that while the process of the worker is still running (see stack trace below), it is not listed under ray.util.list_named_actors().

Process 5480: ray::WorkerDict
Python v3.10.12 (/usr/bin/python3.10)

Thread 5480 (idle): "MainThread"
    main_loop (ray/_private/worker.py:935)
    <module> (ray/_private/workers/default_worker.py:297)
Thread 5636 (idle): "Thread-1 (_read_thread)"
    _recv_msg (torch/_inductor/compile_worker/subproc_pool.py:53)
    _read_thread (torch/_inductor/compile_worker/subproc_pool.py:161)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)

python tests/rollout/test_vllm_multi_turn.py

2025-05-04 12:00:33,561 INFO worker.py:1832 -- Started a local Ray instance. View the dashboard at 127.0.0.1:8265 
[DEBUG] Main script Ray namespace: cc8e0b67-d5f3-4301-b358-282d67db13f6
2025-05-04 12:00:34,508 WARNING __init__.py:161 -- DeprecationWarning: `ray.state.available_resources_per_node` is a private attribute and access will be removed in a future Ray version.
colocated worker base class <class 'verl.single_controller.base.worker.Worker'>
bind role actor_rollout method execute_method to class <class 'verl.single_controller.ray.base.create_colocated_worker_cls.<locals>.WorkerDict'>
WARNING:2025-05-04 12:00:35,609:Waiting for register center actor rZXzt6_register_center to be ready. Elapsed time: 0 seconds out of 300 seconds.
(WorkerDict pid=2374) Model config after override: Qwen2Config {
(WorkerDict pid=2374)   "architectures": [
(WorkerDict pid=2374)     "Qwen2ForCausalLM"
(WorkerDict pid=2374)   ],
(WorkerDict pid=2374)   "attention_dropout": 0.0,
(WorkerDict pid=2374)   "eos_token_id": 151645,
(WorkerDict pid=2374)   "hidden_act": "silu",
(WorkerDict pid=2374)   "hidden_size": 896,
(WorkerDict pid=2374)   "initializer_range": 0.02,
(WorkerDict pid=2374)   "intermediate_size": 4864,
(WorkerDict pid=2374)   "max_position_embeddings": 32768,
(WorkerDict pid=2374)   "max_window_layers": 21,
(WorkerDict pid=2374)   "model_type": "qwen2",
(WorkerDict pid=2374)   "num_attention_heads": 14,
(WorkerDict pid=2374)   "num_hidden_layers": 24,
(WorkerDict pid=2374)   "num_key_value_heads": 2,
(WorkerDict pid=2374)   "pad_token_id": 151643,
(WorkerDict pid=2374)   "rms_norm_eps": 1e-06,
(WorkerDict pid=2374)   "rope_scaling": null,
(WorkerDict pid=2374)   "rope_theta": 1000000.0,
(WorkerDict pid=2374)   "sliding_window": 32768,
(WorkerDict pid=2374)   "tie_word_embeddings": true,
(WorkerDict pid=2374)   "torch_dtype": "bfloat16",
(WorkerDict pid=2374)   "transformers_version": "4.51.1",
(WorkerDict pid=2374)   "use_cache": true,
(WorkerDict pid=2374)   "use_sliding_window": false,
(WorkerDict pid=2374)   "vocab_size": 151936
(WorkerDict pid=2374) }
(WorkerDict pid=2374) 
(WorkerDict pid=2374) Flash Attention 2.0 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen2ForCausalLM is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `torch_dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", torch_dtype=torch.float16)`
(WorkerDict pid=2374) You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
(WorkerDict pid=2374) [rank0]:[W504 12:00:43.714360945 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 0]  using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.
(WorkerDict pid=2374) NCCL version 2.21.5+cuda12.4
(WorkerDict pid=2374) Qwen2ForCausalLM contains 494.03M parameters
(WorkerDict pid=2374) wrap_policy: functools.partial(<function _or_policy at 0x7f32a915e170>, policies=[functools.partial(<function transformer_auto_wrap_policy at 0x7f32a915e050>, transformer_layer_cls={<class 'transformers.models.qwen2.modeling_qwen2.Qwen2DecoderLayer'>})])
(WorkerDict pid=2374) /usr/local/lib/python3.10/dist-packages/torch/distributed/fsdp/_init_utils.py:444: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1.
(WorkerDict pid=2374)   warnings.warn(
(WorkerDict pid=2374) Total steps: -1, num_warmup_steps: 0
(WorkerDict pid=2374) Actor use_remove_padding=False
[DEBUG]: actor_names: ['rZXzt6WorkerDict_0:0']
(WorkerDict pid=2374) /usr/local/lib/python3.10/dist-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
(WorkerDict pid=2374)   warnings.warn(
INFO 05-04 12:00:47 [__init__.py:239] Automatically detected platform cuda.
(AsyncvLLMServer pid=2576) FastAPI startup
(AsyncvLLMServer pid=2576) override_generation_config: {'n': 1, 'logprobs': 0, 'max_tokens': 1024, 'temperature': 1.0, 'top_k': -1, 'top_p': 1, 'ignore_eos': False}
(AsyncvLLMServer pid=2576) WARNING 05-04 12:01:02 [arg_utils.py:1713] Detected VLLM_USE_V1=1 with Engine in background thread. Usage should be considered experimental. Please report any issues on Github.
(AsyncvLLMServer pid=2576) WARNING 05-04 12:01:02 [cuda.py:96] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used
(AsyncvLLMServer pid=2576) WARNING 05-04 12:01:03 [core_client.py:368] SIGUSR1 handler not installed because we are not running in the main thread. In this case the forked engine process may not be killed when an exception is raised, and you need to handle the engine process shutdown manually.
(AsyncvLLMServer pid=2576) 2025-05-04 12:01:09,819      INFO worker.py:1832 -- Started a local Ray instance. View the dashboard at 127.0.0.1:8266 
(AsyncvLLMServer pid=2576) EngineCoreProc.run_engine_core
(AsyncvLLMServer pid=2576) EngineCoreProc.__init__
(AsyncvLLMServer pid=2576) EngineCore.__init__ - Start
(AsyncvLLMServer pid=2576) EngineCore.__init__ - Before model executor initialization
(AsyncvLLMServer pid=2576) namespace: cc8e0b67-d5f3-4301-b358-282d67db13f6
(AsyncvLLMServer pid=2576) actor_names: []
(AsyncvLLMServer pid=2576) [DEBUG]: all actors: []
(AsyncvLLMServer pid=2576) ERROR 05-04 12:01:10 [core.py:406] EngineCore hit an exception: Traceback (most recent call last):
(AsyncvLLMServer pid=2576) ERROR 05-04 12:01:10 [core.py:406]   File "/home/ec2-user/vllm/vllm/v1/engine/core.py", line 394, in run_engine_core
(AsyncvLLMServer pid=2576) ERROR 05-04 12:01:10 [core.py:406]     engine_core = EngineCoreProc(*args, **kwargs)
(AsyncvLLMServer pid=2576) ERROR 05-04 12:01:10 [core.py:406]   File "/home/ec2-user/vllm/vllm/v1/engine/core.py", line 335, in __init__
(AsyncvLLMServer pid=2576) ERROR 05-04 12:01:10 [core.py:406]     super().__init__(vllm_config, executor_class, log_stats)
(AsyncvLLMServer pid=2576) ERROR 05-04 12:01:10 [core.py:406]   File "/home/ec2-user/vllm/vllm/v1/engine/core.py", line 68, in __init__
(AsyncvLLMServer pid=2576) ERROR 05-04 12:01:10 [core.py:406]     self.model_executor = executor_class(vllm_config)
(AsyncvLLMServer pid=2576) ERROR 05-04 12:01:10 [core.py:406]   File "/home/ec2-user/vllm/vllm/executor/executor_base.py", line 52, in __init__
(AsyncvLLMServer pid=2576) ERROR 05-04 12:01:10 [core.py:406]     self._init_executor()
(AsyncvLLMServer pid=2576) ERROR 05-04 12:01:10 [core.py:406]   File "/home/ec2-user/verl/verl/workers/rollout/vllm_rollout/vllm_async_server.py", line 59, in _init_executor
(AsyncvLLMServer pid=2576) ERROR 05-04 12:01:10 [core.py:406]     assert len(actor_names) == vllm_dp_size * vllm_tp_size, f"instance_id: {self.vllm_config.instance_id} has {len(actor_names)} actors, but vllm_dp_size: {vllm_dp_size} * vllm_tp_size: {vllm_tp_size} = {vllm_dp_size * vllm_tp_size} is expected."
(AsyncvLLMServer pid=2576) ERROR 05-04 12:01:10 [core.py:406] AssertionError: instance_id: cc8e0b67-d5f3-4301-b358-282d67db13f6:rZXzt6:1:0 has 0 actors, but vllm_dp_size: 1 * vllm_tp_size: 1 = 1 is expected.
(AsyncvLLMServer pid=2576) ERROR 05-04 12:01:10 [core.py:406] 
(raylet) A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: ffffffffffffffff4002a1d9c4490b2eb5ecac7101000000 Worker ID: 482edc7ef16ece3f4e2b579296a684e36cdc0a098b8885b0cad01ffa Node ID: 67e823c9d9589801f9a45aa3731f818cd2e34862acf44d3ac73483f7 Worker IP address: 172.31.44.216 Worker port: 39343 Worker PID: 2576 Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors.
Traceback (most recent call last):
  File "/home/ec2-user/verl/tests/rollout/test_vllm_multi_turn.py", line 177, in <module>
    test_vllm_multi_turn()
  File "/home/ec2-user/verl/tests/rollout/test_vllm_multi_turn.py", line 98, in test_vllm_multi_turn
    async_rollout_manager = AsyncLLMServerManager(
  File "/home/ec2-user/verl/verl/workers/rollout/async_server.py", line 271, in __init__
    ray.get([worker.init_engine.remote() for worker in self.async_llm_servers])
  File "/usr/local/lib/python3.10/dist-packages/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/ray/_private/worker.py", line 2771, in get
    values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
  File "/usr/local/lib/python3.10/dist-packages/ray/_private/worker.py", line 921, in get_objects
    raise value
ray.exceptions.ActorDiedError: The actor died unexpectedly before finishing this task.
        class_name: AsyncvLLMServer
        actor_id: 4002a1d9c4490b2eb5ecac7101000000
        pid: 2576
        name: async_llm_worker_0
        namespace: cc8e0b67-d5f3-4301-b358-282d67db13f6
        ip: 172.31.44.216
The actor is dead because its worker process has died. Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors.

[wuxibin89]

@ltdemey Finally, I find the root cause of this problem. It's a bug in RayWorkerGroup holding weak references to these actors.

[ruiyiw]

Hi @wuxibin89 ! After merging PR #1443 , I still ran into the same assertion error (0 actors found) when using multi-turn rollout.

verl/tests/workers/rollout/test_vllm_multi_turn.py

# ...
# ==== Same as original file ====

def init_config() -> DictConfig:
    config = OmegaConf.load("verl/trainer/config/ppo_trainer.yaml")
    model_path = "Qwen/Qwen2-7B-Instruct"
    config.actor_rollout_ref.model.path = model_path
    config.actor_rollout_ref.rollout.mode = "async"
    config.actor_rollout_ref.rollout.chat_scheduler = "examples.ppo_trainer.naive_chat_scheduler.NaiveChatCompletionScheduler"
    config.actor_rollout_ref.rollout.prompt_length = 4096
    config.actor_rollout_ref.rollout.response_length = 4096

    # test sleep/wake_up with fsdp offload
    config.actor_rollout_ref.actor.fsdp_config.param_offload = True
    config.actor_rollout_ref.actor.fsdp_config.optimizer_offload = True

    config.ray_init.num_cpus = 8
    config.trainer.n_gpus_per_node = 8
    config.actor_rollout_ref.rollout.tensor_model_parallel_size = 1

# ==== Same as original file ====
# ...

Error message

(AsyncvLLMServer pid=936340) ERROR 05-12 21:14:48 [core.py:390]   File "/tmp/verl/verl/workers/rollout/vllm_rollout/vllm_async_server.py", line 57, in _init_executor
(AsyncvLLMServer pid=936340) ERROR 05-12 21:14:48 [core.py:390]     assert len(actor_names) == vllm_dp_size * vllm_tp_size, f"instance_id: {self.vllm_config.instance_id} has {len(actor_names)} actors, but vllm_dp_size: {vllm_dp_size} * vllm_tp_size: {vllm_tp_size} = {vllm_dp_size * vllm_tp_size} is expected."
(AsyncvLLMServer pid=936340) ERROR 05-12 21:14:48 [core.py:390] AssertionError: instance_id: 196789c2-789d-40cb-8253-9201fd4b087f:Wo5DL0:8:2 has 0 actors, but vllm_dp_size: 8 * vllm_tp_size: 1 = 8 is expected.

To help debugging, I printed out actor_names in verl/tests/workers/rollout/async_rollout_utils.py (line 61):

# ...
    all_wg = {}
    for resource_pool, class_dict in resource_pool_to_cls.items():
        worker_dict_cls = create_colocated_worker_cls(class_dict=class_dict)
        wg_dict = RayWorkerGroup(resource_pool=resource_pool, ray_cls_with_init=worker_dict_cls)
        spawn_wg = wg_dict.spawn(prefix_set=class_dict.keys())
        all_wg.update(spawn_wg)
    actor_rollout_wg = all_wg["actor_rollout"]
    actor_rollout_wg.init_model()

    actor_names = [actor_name for actor_name in ray.util.list_named_actors() if actor_name.startswith(f"{actor_rollout_wg.name_prefix}WorkerDict")]
    print(f"[DEBUG]: actor_names: {actor_names}")
# ...

which successfully refers to the 8 actors as expected.

[DEBUG]: actor_names: ['JXRqHqWorkerDict_0:3', 'JXRqHqWorkerDict_0:4', 'JXRqHqWorkerDict_0:6', 'JXRqHqWorkerDict_0:1', 'JXRqHqWorkerDict_0:2', 'JXRqHqWorkerDict_0:5', 'JXRqHqWorkerDict_0:7', 'JXRqHqWorkerDict_0:0']

However, when trying to print out actor workers in verl/workers/rollout/vllm_rollout/vllm_async_server.py (line 54)

# ...
        namespace, wg_prefix, vllm_dp_size, vllm_dp_rank = fields[0], fields[1], int(fields[2]), int(fields[3])

        # Make sure subprocess in same namespace as parent actor.
        # actor name format: {name_prefix}WorkerDict_{pg_idx}:{local_rank}
        ray.init(namespace=namespace)
        actor_names = [actor_name for actor_name in ray.util.list_named_actors() if actor_name.startswith(f"{wg_prefix}WorkerDict")]
        print(ray.util.list_named_actors())
# ...

It still returns an empty actor list.

@wuxibin89 Is there a way to fix this? Thank you!

[wuxibin89]

@ruiyiw I can't not reproduce this problem either. ExternalRayDistributedExecutor runs in the subprocess of AsyncvLLMServer, so it's not work to print. You can try save debug info in file like this, all debug info are expected to be same.

diff --git a/tests/rollout/test_vllm_multi_turn.py b/tests/rollout/test_vllm_multi_turn.py
index 2707722..f45e81e 100644
--- a/tests/rollout/test_vllm_multi_turn.py
+++ b/tests/rollout/test_vllm_multi_turn.py
@@ -37,6 +37,8 @@ def init_config() -> DictConfig:
     # test sleep/wake_up with fsdp offload
     config.actor_rollout_ref.actor.fsdp_config.param_offload = True
     config.actor_rollout_ref.actor.fsdp_config.optimizer_offload = True
+    config.trainer.n_gpus_per_node = 8
+    config.actor_rollout_ref.rollout.tensor_model_parallel_size = 1
 
     return config
 
@@ -56,6 +58,9 @@ def test_vllm_multi_turn(config):
     # =========================== 1. Init rollout manager ===========================
     model_name = "/".join(config.actor_rollout_ref.model.path.split("/")[-2:])
     async_rollout_manager = init_async_rollout_manager(config)
+    wg_prefix = async_rollout_manager.worker_group.name_prefix
+    actor_names = [actor_name for actor_name in ray.util.list_named_actors() if actor_name.startswith(f"{wg_prefix}WorkerDict")]
+    print(f"[DEBUG] wg_prefix: {wg_prefix}, actor_names: {actor_names}")
 
     # test sleep and wake_up
     async_rollout_manager.sleep()
@@ -192,4 +197,4 @@ async def test_vllm_streaming_response(config):
 if __name__ == "__main__":
     config = init_config()
     test_vllm_multi_turn(config)
-    asyncio.run(test_vllm_streaming_response(config))
+    # asyncio.run(test_vllm_streaming_response(config))
diff --git a/verl/workers/rollout/vllm_rollout/vllm_async_server.py b/verl/workers/rollout/vllm_rollout/vllm_async_server.py
index 3082292..451deae 100644
--- a/verl/workers/rollout/vllm_rollout/vllm_async_server.py
+++ b/verl/workers/rollout/vllm_rollout/vllm_async_server.py
@@ -52,6 +52,8 @@ class ExternalRayDistributedExecutor(Executor):
         # actor name format: {name_prefix}WorkerDict_{pg_idx}:{local_rank}
         ray.init(namespace=namespace)
         actor_names = [actor_name for actor_name in ray.util.list_named_actors() if actor_name.startswith(f"{wg_prefix}WorkerDict")]
+        with open(f"{self.vllm_config.instance_id}.log", "w") as f:
+            f.write(f"[DEBUG] wg_prefix: {wg_prefix}, actor_names: {actor_names}\n")
 
         vllm_tp_size = self.vllm_config.parallel_config.tensor_parallel_size
         assert len(actor_names) == vllm_dp_size * vllm_tp_size, f"instance_id: {self.vllm_config.instance_id} has {len(actor_names)} actors, but vllm_dp_size: {vllm_dp_size} * vllm_tp_size: {vllm_tp_size} = {vllm_dp_size * vllm_tp_size} is expected."

[ruiyiw]

Hi @wuxibin89! I appreciate your response!

The error still persists using the original code but I get it work by modifying the following lines:

In verl/workers/rollout/vllm_rollout/vllm_async_server.py, line 53:

# ==== MODIFICATION STARTS ====
# ray.init(namespace=namespace)

# Check if we can connect to existing Ray instance before initializing
try:
    ray.init(address="auto", namespace=namespace, ignore_reinit_error=True)

    actor_names = [actor_name for actor_name in ray.util.list_named_actors() if actor_name.startswith(f"{wg_prefix}WorkerDict")]
    
    with open(f"{self.vllm_config.instance_id}.log", "w") as f:
        f.write(f"[DEBUG] After connecting to Ray, found actors: {actor_names}\n")
    
except Exception as e:
    print(f"Error connecting to Ray: {e}")
    ray.init(namespace=namespace)

# ==== MODIFICATION ENDS ====

Here's the content of the log file:

[DEBUG] After connecting to Ray, found actors: ['Of865VWorkerDict_0:7', 'Of865VWorkerDict_0:0', 'Of865VWorkerDict_0:1', 'Of865VWorkerDict_0:3', 'Of865VWorkerDict_0:6', 'Of865VWorkerDict_0:5', 'Of865VWorkerDict_0:4', 'Of865VWorkerDict_0:2']

I am not sure the exact reason why the original initialization does not work for my environment. I hope this modification could be helpful to ppl experiencing the same error!

[mzmssg]

@wuxibin89
I can also reproduce this issue. It appears to be a bug in Ray: if ray.init() is called without explicitly passing address="auto", Ray fails to detect an already running local cluster and instead starts a new one. As a result, it cannot retrieve the existing workgroup.

Explicitly specifying address="auto" works as expected and successfully connects to the existing cluster. I recommend using this option as a workaround.

# A Ray cluster has already been launched in another process.

>>> import ray
>>> ray.__version__
'2.46.0'

# Implicit init — unexpectedly starts a new local cluster
>>> ray.init() 
2025-06-03 00:02:11,900 INFO worker.py:1888 -- Started a local Ray instance.
RayContext(dashboard_url='', python_version='3.12.7', ray_version='2.46.0', ray_commit='c3dd2ca0c2a24ddf327a213d2e936bd4eaa4ca0a')

>>> ray.shutdown()

# Explicit init — correctly connects to the existing cluster
>>> ray.init(address="auto")
2025-06-03 00:02:35,272 INFO worker.py:1694 -- Connecting to existing Ray cluster at address: xxxx:xxx...
2025-06-03 00:02:35,282 INFO worker.py:1888 -- Connected to Ray cluster.
RayContext(dashboard_url='', python_version='3.12.7', ray_version='2.46.0', ray_commit='c3dd2ca0c2a24ddf327a213d2e936bd4eaa4ca0a')

[Peng-YM]

Hi @wuxibin89! I appreciate your response!

The error still persists using the original code but I get it work by modifying the following lines:

In verl/workers/rollout/vllm_rollout/vllm_async_server.py, line 53:

==== MODIFICATION STARTS ====

ray.init(namespace=namespace)

Check if we can connect to existing Ray instance before initializing

try:
ray.init(address="auto", namespace=namespace, ignore_reinit_error=True)

actor_names = [actor_name for actor_name in ray.util.list_named_actors() if actor_name.startswith(f"{wg_prefix}WorkerDict")]

with open(f"{self.vllm_config.instance_id}.log", "w") as f:
    f.write(f"[DEBUG] After connecting to Ray, found actors: {actor_names}\n")

except Exception as e:
print(f"Error connecting to Ray: {e}")
ray.init(namespace=namespace)

==== MODIFICATION ENDS ====

Here's the content of the log file:

[DEBUG] After connecting to Ray, found actors: ['Of865VWorkerDict_0:7', 'Of865VWorkerDict_0:0', 'Of865VWorkerDict_0:1', 'Of865VWorkerDict_0:3', 'Of865VWorkerDict_0:6', 'Of865VWorkerDict_0:5', 'Of865VWorkerDict_0:4', 'Of865VWorkerDict_0:2']
I am not sure the exact reason why the original initialization does not work for my environment. I hope this modification could be helpful to ppl experiencing the same error!

I met the same problem on the latest main branch, this solution works for me.

cc: @wuxibin89