[Model][VLM] Add multi-video support for LLaVA-Onevision

[litianjian]

Add multi-video support for LLaVA-Onevision models.

Example

import av
import time
import numpy as np
from huggingface_hub import hf_hub_download
import vllm
from vllm import LLM, SamplingParams

MODEL="llava-hf/llava-onevision-qwen2-7b-ov-hf"

text_prompt = "<|im_start|>user <video> <video>\nWhat’s the difference between these two videos?.<|im_end|><|im_start|>assistant\n"

def read_video_pyav(container, indices):
    '''
    Decode the video with PyAV decoder.
    Args:
        container (`av.container.input.InputContainer`): PyAV container.
        indices (`List[int]`): List of frame indices to decode.
    Returns:
        result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3).
    '''
    frames = []
    container.seek(0)
    start_index = indices[0]
    end_index = indices[-1]
    for i, frame in enumerate(container.decode(video=0)):
        if i > end_index:
            break
        if i >= start_index and i in indices:
            frames.append(frame)
    return np.stack([x.to_ndarray(format="rgb24") for x in frames])


video_path = hf_hub_download(repo_id="raushan-testing-hf/videos-test", filename="sample_demo_1.mp4", repo_type="dataset")
container = av.open(video_path)
total_frames = container.streams.video[0].frames
indices = np.arange(0, total_frames, total_frames / 32).astype(int)
video = read_video_pyav(container, indices)

llm = LLM(model=MODEL, tensor_parallel_size=1)
sampling_params = SamplingParams(temperature=0.8,
                            top_p=0.95,
                            max_tokens=100)
outputs = llm.generate(
    {
        "prompt": text_prompt,
        "multi_modal_data": {
            "video": [video,video]
        }
    },
    sampling_params=sampling_params)

generated_text = ""
for o in outputs:
    generated_text += o.outputs[0].text
print(f"LLM output:{generated_text}")

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[DarkLight1337]

FYI I found some problems with the chat template earlier (see #8874). Please double check that you're using the correct chat templates for examples/tests in this PR.

[DarkLight1337]

Sorry for the long wait! Can you add a test with multi-video input so it is easy to verify that this works?

[litianjian]

Sorry for the long wait! Can you add a test with multi-video input so it is easy to verify that this works?

Sorry for late response. I added a test with multi-video input.

[litianjian]

I have updated the tests. Could you please review it when you're available? @DarkLight1337

[DarkLight1337]

Meanwhile let's see if the tests pass CI.

[DarkLight1337]

I ran the tests locally and they pass. Let's run the CI again.

Thanks for your patience!