KeyError: 'visual.patch_embed.proj.weight

[HumanZhong]

Hi, thanks for sharing your great work. I have encountered this keyerror issue after following your installation steps. This reported error is as follows:

(WorkerDict pid=595461) [rank0]:   File "/mnt/anaconda3_new/envs/deepeyes/lib/python3.10/site-packages/ray/_private/function_manager.py", line 689, in actor_method_executor
(WorkerDict pid=595461) [rank0]:     return method(__ray_actor, *args, **kwargs)
(WorkerDict pid=595461) [rank0]:   File "/mnt/anaconda3_new/envs/deepeyes/lib/python3.10/site-packages/ray/util/tracing/tracing_helper.py", line 463, in _resume_span
(WorkerDict pid=595461) [rank0]:     return method(self, *_args, **_kwargs)
(WorkerDict pid=595461) [rank0]:   File "/mnt/DeepEyes/verl/single_controller/ray/base.py", line 446, in func
(WorkerDict pid=595461) [rank0]:     return getattr(self.worker_dict[key], name)(*args, **kwargs)
(WorkerDict pid=595461) [rank0]:   File "/mnt/DeepEyes/verl/single_controller/base/decorator.py", line 413, in inner
(WorkerDict pid=595461) [rank0]:     return func(*args, **kwargs)
(WorkerDict pid=595461) [rank0]:   File "/mnt/DeepEyes/verl/workers/fsdp_workers.py", line 560, in generate_sequences
(WorkerDict pid=595461) [rank0]:     with self.rollout_sharding_manager:
(WorkerDict pid=595461) [rank0]:   File "/mnt/DeepEyes/verl/utils/debug/performance.py", line 61, in f
(WorkerDict pid=595461) [rank0]:     return self.log(decorated_function, *args, **kwargs)
(WorkerDict pid=595461) [rank0]:   File "/mnt/DeepEyes/verl/utils/debug/performance.py", line 70, in log
(WorkerDict pid=595461) [rank0]:     output = func(*args, **kwargs)
(WorkerDict pid=595461) [rank0]:   File "/mnt/DeepEyes/verl/workers/sharding_manager/fsdp_vllm.py", line 119, in __enter__
(WorkerDict pid=595461) [rank0]:     self.update_params(params)
(WorkerDict pid=595461) [rank0]:   File "/mnt/DeepEyes/verl/workers/sharding_manager/fsdp_vllm.py", line 215, in update_params
(WorkerDict pid=595461) [rank0]:     loaded_params = model.load_weights(
(WorkerDict pid=595461) [rank0]:   File "/mnt/anaconda3_new/envs/deepeyes/lib/python3.10/site-packages/vllm/model_executor/models/qwen2_5_vl.py", line 1098, in load_weights
(WorkerDict pid=595461) [rank0]:     return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)
(WorkerDict pid=595461) [rank0]:   File "/mnt/anaconda3_new/envs/deepeyes/lib/python3.10/site-packages/vllm/model_executor/models/utils.py", line 235, in load_weights
(WorkerDict pid=595461) [rank0]:     autoloaded_weights = set(self._load_module("", self.module, weights))
(WorkerDict pid=595461) [rank0]:   File "/mnt/anaconda3_new/envs/deepeyes/lib/python3.10/site-packages/vllm/model_executor/models/utils.py", line 196, in _load_module
(WorkerDict pid=595461) [rank0]:     yield from self._load_module(prefix,
(WorkerDict pid=595461) [rank0]:   File "/mnt/anaconda3_new/envs/deepeyes/lib/python3.10/site-packages/vllm/model_executor/models/utils.py", line 173, in _load_module
(WorkerDict pid=595461) [rank0]:     loaded_params = module_load_weights(weights)
(WorkerDict pid=595461) [rank0]:   File "/mnt/anaconda3_new/envs/deepeyes/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 490, in load_weights
(WorkerDict pid=595461) [rank0]:     return loader.load_weights(weights)
(WorkerDict pid=595461) [rank0]:   File "/mnt/anaconda3_new/envs/deepeyes/lib/python3.10/site-packages/vllm/model_executor/models/utils.py", line 235, in load_weights
(WorkerDict pid=595461) [rank0]:     autoloaded_weights = set(self._load_module("", self.module, weights))
(WorkerDict pid=595461) [rank0]:   File "/mnt/anaconda3_new/envs/deepeyes/lib/python3.10/site-packages/vllm/model_executor/models/utils.py", line 196, in _load_module
(WorkerDict pid=595461) [rank0]:     yield from self._load_module(prefix,
(WorkerDict pid=595461) [rank0]:   File "/mnt/anaconda3_new/envs/deepeyes/lib/python3.10/site-packages/vllm/model_executor/models/utils.py", line 173, in _load_module
(WorkerDict pid=595461) [rank0]:     loaded_params = module_load_weights(weights)
(WorkerDict pid=595461) [rank0]:   File "/mnt/anaconda3_new/envs/deepeyes/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 400, in load_weights
(WorkerDict pid=595461) [rank0]:     param = params_dict[name]
(WorkerDict pid=595461) [rank0]: KeyError: 'visual.patch_embed.proj.weight

It seems the issue comes from verl/workers/sharding_manager/fsdp_vllm.py line109: self.update_params(params)
Further, it goes into vllm/model_executor/models/qwen2.py: load_weights(self, weights) of Qwen2Model.
I have checked the params_dict and weights_dict and they are indeed different.

I have tried these:

1. replacing the self.update_params with:

from verl.third_party.vllm import load_dtensor_weights
load_dtensor_weights(
    params, self.inference_engine.llm_engine.model_executor.driver_worker.worker.model_runner.model)

which was implemented in an older version of verl. After doing so, the keyerror is handled but the vllm_engine will generate random meaningless sequences so I guess the weight loading is still wrong.
2. change vllm==0.8.2 to vllm==0.8.0. It does not help.

Have you ever encountered similar issue and can you give some advice to deal with this issue?

[HumanZhong]

The launch scripts are mainly from your example: examples/agent/final_merged_v1v8_thinklite.sh but switch to a smaller batch_size.
The experiments are conducted on a machine with 4 gpus.
Details as follows:

set -x
# export VLLM_ATTENTION_BACKEND=XFORMERS

export PATH=/xxx/anaconda3_new/envs/deepeyes/bin:$PATH
# export PATH=/usr/bin:$PATH

export LLM_AS_A_JUDGE_BASE="http://xxxxx:xxxxx/v1"
export WORLD_SIZE=1
export GPUS_PER_NODE=${MLP_WORKER_GPU:-${KUBERNETES_CONTAINER_RESOURCE_GPU:-4}}
export NNODES=${MLP_WORKER_NUM:-${WORLD_SIZE:-1}}
export NODE_RANK=${MLP_WORKER_RACK_RANK_INDEX:-${MLP_ROLE_INDEX:-${RANK:-0}}}
export MASTER_ADDR=${MLP_WORKER_0_HOST:-${MASTER_ADDR:-127.0.0.1}}
export MASTER_PORT=${MLP_WORKER_0_PORT:-${MASTER_PORT:-1234}}


PROJECT_NAME="agent_vlagent_debug"
EXPERIMENT_NAME="debug_for_single_node"

export SAVE_CHECKPOINT_DIR="/xxx/hf-ckpts/verl_ckpt"
# export VLLM_ATTENTION_BACKEND=XFORMERS # vllm + qwen2-7b with flash_attn has some issues



BASE_MODEL=/xxx/Qwen/Qwen2.5-VL-7B-Instruct


BASEDIR=/xxx/DeepEyes-Datasets-47k
VISUAL_DATASET_TRAIN_0_6_2=${BASEDIR}/data_v0.6.2_reason.parquet
VISUAL_DATASET_TRAIN_0_1_2=${BASEDIR}/data_0.1.2_visual_toolbox_v2.parquet
VISUAL_DATASET_TRAIN_0_8=${BASEDIR}/data_v0.8_visual_toolbox_v2.parquet
VISUAL_DATASET_TEST=${BASEDIR}/seekworld_test.parquet
EUREKA_DATASET_TRAIN=${BASEDIR}/data_thinklite_reasoning_acc.parquet


BATCH_SIZE=32
MAX_GEN_LENGTH_K=20
PPO_MAX_TOKEN_LEN_PER_GPU=$((16*1024))
MAX_PROMPT_LENGTH=$((8*1024))
MAX_RESPONSE_LENGTH=$((MAX_GEN_LENGTH_K*1024))


export RAY_DEBUG_POST_MORTEM=1

###############
ray stop
rm -rv /tmp/ray/*
export RAY_num_server_call_thread=1
##############

if [ $NODE_RANK -eq 0 ]; then

ray start --block --head --port=6379 &

PYTHONUNBUFFERED=1 python3 -m verl.trainer.main_ppo \
    data.train_files=[${VISUAL_DATASET_TRAIN_0_1_2}] \
    data.val_files=[${EUREKA_DATASET_TRAIN}] \
    data.train_batch_size=${BATCH_SIZE} \
    data.max_prompt_length=${MAX_PROMPT_LENGTH} \
    data.max_response_length=${MAX_RESPONSE_LENGTH} \
    data.return_raw_chat=True \
    data.filter_overlong_prompts=True \
    algorithm.adv_estimator=grpo \
    algorithm.kl_ctrl.kl_coef=0.0 \
    actor_rollout_ref.model.path=${BASE_MODEL} \
    actor_rollout_ref.model.use_remove_padding=True \
    actor_rollout_ref.actor.optim.lr=1e-6 \
    actor_rollout_ref.actor.ppo_mini_batch_size=4 \
    actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=1 \
    actor_rollout_ref.actor.use_kl_loss=False \
    actor_rollout_ref.actor.kl_loss_coef=0.0 \
    actor_rollout_ref.actor.kl_loss_type=low_var_kl \
    actor_rollout_ref.actor.entropy_coeff=0.0 \
    actor_rollout_ref.actor.checkpoint.contents=['model','hf_model','optimizer','extra'] \
    actor_rollout_ref.actor.ulysses_sequence_parallel_size=1 \
    actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=8 \
    actor_rollout_ref.rollout.tensor_model_parallel_size=1 \
    actor_rollout_ref.rollout.name=vllm \
    actor_rollout_ref.rollout.n=16 \
    actor_rollout_ref.rollout.max_num_batched_tokens=32768 \
    actor_rollout_ref.rollout.gpu_memory_utilization=0.8 \
    actor_rollout_ref.rollout.enforce_eager=False \
    actor_rollout_ref.rollout.free_cache_engine=False \
    actor_rollout_ref.rollout.enable_chunked_prefill=False \
    actor_rollout_ref.actor.fsdp_config.param_offload=True \
    actor_rollout_ref.actor.fsdp_config.optimizer_offload=True \
    actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=8 \
    actor_rollout_ref.ref.fsdp_config.param_offload=True \
    actor_rollout_ref.rollout.agent.activate_agent=True \
    actor_rollout_ref.rollout.agent.tool_name_key=env_name \
    actor_rollout_ref.rollout.agent.single_response_max_tokens=10240 \
    actor_rollout_ref.rollout.agent.max_turns=5 \
    actor_rollout_ref.rollout.agent.concurrent_workers=1 \
    actor_rollout_ref.rollout.agent.show_tqdm=True \
    trainer.critic_warmup=0 \
    trainer.logger=['console','wandb','rl_logging_board'] \
    trainer.val_before_train=False \
    trainer.n_gpus_per_node=4 \
    trainer.nnodes=1 \
    trainer.save_freq=8 \
    trainer.test_freq=10000 \
    trainer.project_name=${PROJECT_NAME} \
    trainer.experiment_name=${EXPERIMENT_NAME} \
    trainer.default_local_dir=${SAVE_CHECKPOINT_DIR}/${PROJECT_NAME}/${EXPERIMENT_NAME} \
    +trainer.tensorboard_dir=${SAVE_CHECKPOINT_DIR}/logs/tensorboard \
    +trainer.rl_logging_board_dir=${SAVE_CHECKPOINT_DIR}/logs/rl_logging_board \
    trainer.total_epochs=32 2>&1 | tee ./logs/${EXPERIMENT_NAME}.log

else

ray start --block --address=${MASTER_ADDR}:6379

fi

[Qsingle]

Please try to use transformers==4.51.0. Maybe the newer version of transformers conflicts with Verl's VLLM for the Qwen-VL2.

[ChenShawn]

As mentioned by Qsingle and https://zhuanlan.zhihu.com/p/1908543355161417216. It seems like a version conflict issue.

The full requirements.txt has been provided in https://github.com/Visual-Agent/DeepEyes/blob/main/requirements_deepeyes.txt

[JLINEkai]

@Qsingle I met the same tissue. I have used transformers==4.51.0, and then it worked. Thanks.

[Qsingle]

@Qsingle I met the same tissue. I have used transformers==4.51.0, and then it worked. Thanks.

The method to resolve the problem is to add the weights mapper as mentioned at 19054.