[Bug]: assert len(indices) == len(inputs) with `Qwen/Qwen2-VL-2B-Instruct`

[sayakpaul]

Your current environment

The output of `python collect_env.py`

Collecting environment information...
WARNING 10-07 17:28:06 cuda.py:76] Detected different devices in the system: 
WARNING 10-07 17:28:06 cuda.py:76] NVIDIA A100-SXM4-80GB
WARNING 10-07 17:28:06 cuda.py:76] NVIDIA A100-SXM4-80GB
WARNING 10-07 17:28:06 cuda.py:76] NVIDIA A100-SXM4-80GB
WARNING 10-07 17:28:06 cuda.py:76] NVIDIA DGX Display
WARNING 10-07 17:28:06 cuda.py:76] NVIDIA A100-SXM4-80GB
WARNING 10-07 17:28:06 cuda.py:76] Please make sure to set `CUDA_DEVICE_ORDER=PCI_BUS_ID` to avoid unexpected behavior.
PyTorch version: 2.4.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.6 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
Clang version: 10.0.0-4ubuntu1 
CMake version: version 3.27.0
Libc version: glibc-2.31

Python version: 3.10.12 (main, Jul 17 2023, 11:18:22) [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.4.0-166-generic-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 12.3.52
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA A100-SXM4-80GB
GPU 1: NVIDIA A100-SXM4-80GB
GPU 2: NVIDIA A100-SXM4-80GB
GPU 3: NVIDIA DGX Display
GPU 4: NVIDIA A100-SXM4-80GB

Nvidia driver version: 535.129.03
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Byte Order:                         Little Endian
Address sizes:                      43 bits physical, 48 bits virtual
CPU(s):                             128
On-line CPU(s) list:                0-127
Thread(s) per core:                 2
Core(s) per socket:                 64
Socket(s):                          1
NUMA node(s):                       1
Vendor ID:                          AuthenticAMD
CPU family:                         23
Model:                              49
Model name:                         AMD EPYC 7742 64-Core Processor
Stepping:                           0
Frequency boost:                    enabled
CPU MHz:                            2608.221
CPU max MHz:                        2250,0000
CPU min MHz:                        1500,0000
BogoMIPS:                           4491.62
Virtualization:                     AMD-V
L1d cache:                          2 MiB
L1i cache:                          2 MiB
L2 cache:                           32 MiB
L3 cache:                           256 MiB
NUMA node0 CPU(s):                  0-127
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Vulnerable
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif umip rdpid overflow_recov succor smca sme sev sev_es

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.1.3.1
[pip3] nvidia-cuda-cupti-cu12==12.1.105
[pip3] nvidia-cuda-nvrtc-cu12==12.1.105
[pip3] nvidia-cuda-runtime-cu12==12.1.105
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.0.2.54
[pip3] nvidia-curand-cu12==10.3.2.106
[pip3] nvidia-cusolver-cu12==11.4.5.107
[pip3] nvidia-cusparse-cu12==12.1.0.106
[pip3] nvidia-ml-py==12.560.30
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] nvidia-nvjitlink-cu12==12.6.77
[pip3] nvidia-nvtx-cu12==12.1.105
[pip3] pyzmq==26.2.0
[pip3] torch==2.4.0
[pip3] torchvision==0.19.0
[pip3] transformers==4.45.1
[pip3] triton==3.0.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.6.3.dev116+g151ef4ef
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
�[4mGPU0	GPU1	GPU2	GPU3	GPU4	CPU Affinity	NUMA Affinity	GPU NUMA ID�[0m
GPU0	 X 	NV4	NV4	SYS	NV4	0-127	0		N/A
GPU1	NV4	 X 	NV4	SYS	NV4	0-127	0		N/A
GPU2	NV4	NV4	 X 	SYS	NV4	0-127	0		N/A
GPU3	SYS	SYS	SYS	 X 	PHB	0-127	0		N/A
GPU4	NV4	NV4	NV4	PHB	 X 	0-127	0		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

Model Input Dumps

No response

🐛 Describe the bug

Trying to run:

from vllm import LLM, SamplingParams
from PIL import Image


if __name__ == "__main__":
    vllm_engine = LLM("Qwen/Qwen2-VL-2B-Instruct")
    sampling_params = SamplingParams(max_tokens=120)
    
    prompt = "Describe this image."
    vllm_inputs = [{"prompt": prompt, "multi_modal_data": {"image": Image.new("RGB", (224, 224))}} for _ in range(4)]
    outputs = vllm_engine.generate(vllm_inputs, sampling_params)
    print(outputs)

Leads to:

[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/sayak/diffusers/check_video_vllm.py", line 23, in <module>
[rank0]:     outputs = vllm_engine.generate(vllm_inputs, sampling_params)
[rank0]:   File "/home/sayak/vllm/vllm/utils.py", line 1060, in inner
[rank0]:     return fn(*args, **kwargs)
[rank0]:   File "/home/sayak/vllm/vllm/entrypoints/llm.py", line 376, in generate
[rank0]:     self._validate_and_add_requests(
[rank0]:   File "/home/sayak/vllm/vllm/entrypoints/llm.py", line 831, in _validate_and_add_requests
[rank0]:     self._add_request(
[rank0]:   File "/home/sayak/vllm/vllm/entrypoints/llm.py", line 849, in _add_request
[rank0]:     self.llm_engine.add_request(
[rank0]:   File "/home/sayak/vllm/vllm/utils.py", line 1060, in inner
[rank0]:     return fn(*args, **kwargs)
[rank0]:   File "/home/sayak/vllm/vllm/engine/llm_engine.py", line 812, in add_request
[rank0]:     processed_inputs = self.input_processor(preprocessed_inputs)
[rank0]:   File "/home/sayak/vllm/vllm/inputs/registry.py", line 299, in process_input
[rank0]:     return processor(InputContext(model_config), inputs,
[rank0]:   File "/home/sayak/vllm/vllm/model_executor/models/qwen2_vl.py", line 857, in input_processor_for_qwen2_vl
[rank0]:     prompt_token_ids = _expand_pad_tokens(image_inputs,
[rank0]:   File "/home/sayak/vllm/vllm/model_executor/models/qwen2_vl.py", line 780, in _expand_pad_tokens
[rank0]:     assert len(indices) == len(inputs)
[rank0]: AssertionError

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

You should follow the prompt template as shown on their HuggingFace repo. The easiest way is to use LLM.chat method so you don't have to manually apply the chat template.

[sayakpaul]

If I were to use LLM.chat() interface, how should I use multi_modal_data?

[DarkLight1337]

Please read the examples on their HuggingFace repo. The format of messages in LLM.chat is the same as theirs.

[sayakpaul]

Modified my code:

from vllm import LLM, SamplingParams
from PIL import Image


if __name__ == "__main__":
    vllm_engine = LLM("Qwen/Qwen2-VL-2B-Instruct")
    sampling_params = SamplingParams(max_tokens=120)

    num_images = 3
    messages = [{"role": "user", "content": []}]
    for _ in range(num_images):
        new_image = {"type": "image", "image": Image.new("RGB", (224, 224))}
        messages[0]["content"].append(new_image)

    messages[0]["content"].append({"type": "text", "text": "Describe this image."})

    outputs = vllm_engine.chat(messages, sampling_params)
    print(outputs)

Now it leads to:

[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/sayak/diffusers/check_video_vllm.py", line 32, in <module>
[rank0]:     outputs = vllm_engine.chat(messages, sampling_params)
[rank0]:   File "/home/sayak/vllm/vllm/entrypoints/llm.py", line 556, in chat
[rank0]:     conversation, mm_data = parse_chat_messages(
[rank0]:   File "/home/sayak/vllm/vllm/entrypoints/chat_utils.py", line 487, in parse_chat_messages
[rank0]:     sub_messages = _parse_chat_message_content(msg, mm_tracker)
[rank0]:   File "/home/sayak/vllm/vllm/entrypoints/chat_utils.py", line 440, in _parse_chat_message_content
[rank0]:     result = _parse_chat_message_content_parts(
[rank0]:   File "/home/sayak/vllm/vllm/entrypoints/chat_utils.py", line 402, in _parse_chat_message_content_parts
[rank0]:     raise NotImplementedError(f"Unknown part type: {part_type}")
[rank0]: NotImplementedError: Unknown part type: image

[DarkLight1337]

    for _ in range(num_images):
        new_image = {"type": "image", "image": Image.new("RGB", (224, 224))}
        messages[0]["content"].append(new_image)

Instead of image, you should pass an image_url in the form of HTTP URL or base64 URL. This is similar to OpenAI API.

[sayakpaul]

Alright.

I will try that and report here but I think #9128 (comment) is misleading then. I did follow the exact "messages" formatting.

[DarkLight1337]

You should follow the prompt template as shown on their HuggingFace repo. The easiest way is to use LLM.chat method so you don't have to manually apply the chat template.

Sorry, I missed the slight difference in the image format. Hope that everything is cleared up now!

[exceedzhang]

I tested Qwen2-VL-7B-Instruct work fine, but tested Qwen2-VL-72B-Instruct-GPTQ-Int4 error!

[exceedzhang]

I tested Qwen2-VL-2B-Instruct working well!

[DarkLight1337]

I tested Qwen2-VL-7B-Instruct work fine, but tested Qwen2-VL-72B-Instruct-GPTQ-Int4 error!

Can you check whether both repos have a correctly defined chat template?

[sayakpaul]

With

def encode_image(image):
    buffered = io.BytesIO()
    image.save(buffered, format="JPEG")
    image_bytes = buffered.getvalue()
    return base64.b64encode(image_bytes).decode("utf-8")

num_images = 3
messages = [{"role": "user", "content": []}]
for _ in range(num_images):
    base64_image = encode_image(Image.new("RGB", (224, 224)))
    new_image = {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}}
    messages[0]["content"].append(new_image)

messages[0]["content"].append({"type": "text", "text": "Describe this image."})

I get:

[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/sayak/diffusers/check_video_vllm.py", line 41, in <module>
[rank0]:     outputs = vllm_engine.chat(messages, sampling_params)
[rank0]:   File "/home/sayak/vllm/vllm/entrypoints/llm.py", line 556, in chat
[rank0]:     conversation, mm_data = parse_chat_messages(
[rank0]:   File "/home/sayak/vllm/vllm/entrypoints/chat_utils.py", line 487, in parse_chat_messages
[rank0]:     sub_messages = _parse_chat_message_content(msg, mm_tracker)
[rank0]:   File "/home/sayak/vllm/vllm/entrypoints/chat_utils.py", line 440, in _parse_chat_message_content
[rank0]:     result = _parse_chat_message_content_parts(
[rank0]:   File "/home/sayak/vllm/vllm/entrypoints/chat_utils.py", line 392, in _parse_chat_message_content_parts
[rank0]:     mm_parser.parse_image(image_url["url"])
[rank0]:   File "/home/sayak/vllm/vllm/entrypoints/chat_utils.py", line 276, in parse_image
[rank0]:     placeholder = self._tracker.add("image", image)
[rank0]:   File "/home/sayak/vllm/vllm/entrypoints/chat_utils.py", line 205, in add
[rank0]:     raise ValueError(
[rank0]: ValueError: At most 1 image(s) may be provided in one request.

What am I missing out on?

[DarkLight1337]

With

def encode_image(image):
    buffered = io.BytesIO()
    image.save(buffered, format="JPEG")
    image_bytes = buffered.getvalue()
    return base64.b64encode(image_bytes).decode("utf-8")

num_images = 3
messages = [{"role": "user", "content": []}]
for _ in range(num_images):
    base64_image = encode_image(Image.new("RGB", (224, 224)))
    new_image = {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}}
    messages[0]["content"].append(new_image)

messages[0]["content"].append({"type": "text", "text": "Describe this image."})

I get:

[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/sayak/diffusers/check_video_vllm.py", line 41, in <module>
[rank0]:     outputs = vllm_engine.chat(messages, sampling_params)
[rank0]:   File "/home/sayak/vllm/vllm/entrypoints/llm.py", line 556, in chat
[rank0]:     conversation, mm_data = parse_chat_messages(
[rank0]:   File "/home/sayak/vllm/vllm/entrypoints/chat_utils.py", line 487, in parse_chat_messages
[rank0]:     sub_messages = _parse_chat_message_content(msg, mm_tracker)
[rank0]:   File "/home/sayak/vllm/vllm/entrypoints/chat_utils.py", line 440, in _parse_chat_message_content
[rank0]:     result = _parse_chat_message_content_parts(
[rank0]:   File "/home/sayak/vllm/vllm/entrypoints/chat_utils.py", line 392, in _parse_chat_message_content_parts
[rank0]:     mm_parser.parse_image(image_url["url"])
[rank0]:   File "/home/sayak/vllm/vllm/entrypoints/chat_utils.py", line 276, in parse_image
[rank0]:     placeholder = self._tracker.add("image", image)
[rank0]:   File "/home/sayak/vllm/vllm/entrypoints/chat_utils.py", line 205, in add
[rank0]:     raise ValueError(
[rank0]: ValueError: At most 1 image(s) may be provided in one request.

What am I missing out on?

Please check our docs on using VLMs. There is a section on how to input multiple images per prompt.

[sayakpaul]

I am following https://docs.vllm.ai/en/stable/models/vlm.html and I am still really not sure what I am missing out here. A bit more specificity in your replies would be appreciated.

[DarkLight1337]

I am following https://docs.vllm.ai/en/stable/models/vlm.html and I am still really not sure what I am missing out here. A bit more specificity in your replies would be appreciated.

You need to set limit_mm_per_prompt, as shown here