Jingli Lin^1,2*, Runsen Xu^1,3*†, Shaohao Zhu^1,4, Sihan Yang¹, Peizhou Cao^1,5, Yunlong Ran^1,4, Miao Hu⁶, Chenming Zhu^1,7, Yiman Xie^1,4, Yilin Long^1,8, Wenbo Hu^1,9, Dahua Lin^1,3, Tai Wang^1✉, Jiangmiao Pang^1✉

1. Shanghai AI Laboratory    2. Shanghai Jiaotong University    3. The Chinese University of Hong Kong    4. Zhejiang University    5. Beihang University 6. Xi'an Jiaotong University    7. University of Hong Kong    8. Fudan University    9. University of California, Los Angeles

^*Equal Contribution    ^†Project Lead   

🔥[2025-12]: Our MMSI-Video-Bench has been integrated into VLMEvalKit.

🔥[2025-12]: We released our paper, benchmark, and evaluation codes.

MMSI-Video-Bench is a fully human-annotated benchmark for evaluating video-based spatial intelligence in MLLMs. It implements a four-level framework—Perception, Planning, Prediction, and Cross-Video Reasoning, through 1,106 questions grounded in 1,278 video clips sourced from 25 public datasets as well as in-house recordings.

• High Quality. All data are manually annotated by 11 domain experts in 3D vision, following a rigorous review and acceptance process to ensure annotation accuracy and reliability.

• Challenging. We evaluate 25 strong open-source and proprietary MLLMs, revealing a striking human–AI gap, even the best reasoning model trails human performance by nearly 60%.

• Diverse Video Sources. MMSI-Video-Bench includes videos from 25 public datasets and 1 in-house collection, spanning tabletop recordings, indoor and multi-floor environments, outdoor scenes, sports, and movie footage.

• Comprehensive and Hostile Task Coverage. The benchmark covers spatial layout reasoning, motion understanding, decision-making, and cross-video reasoning, providing a more holistic evaluation of video-based spatial intelligence.

The questions in MMSI-VIdeo-Bench span 5 major categories: (1).Spatial Construction：This category focuses on spatial attributes of instances and scenes, as well as spatial relationships among instances, scenes, and cameras (six subtypes in total); (2).Motion Understanding: This includes understanding camera motion, instance motion, and interactive motion between instances; (3).Planning based on spatiotemporal video information; (4).Prediction: Assessing a model’s ability to predict, anticipate, or imagine future states based on the observed video; (5).Cross-Video Reasoning: This involves memory update across temporally separated video segments and multi-view integration across videos captured from different viewpoints.

All of our data is available on Hugging Face and includes the following components:

🎥 Video Data (videos.zip): Contains the video clip file (.mp4) corresponding to each sample. This file is generally not required for most models.

🎥 Frame Data (frames.zip): Contains the frames (.jpg) extracted from each sample's video at the base sampling rate. This rate ensures no key information loss during sampling. Each frame file is named using the format {timestamp}_frame_{base_interval}_{image_id} (e.g., 00:06.00_frame_1.50_4), where the timestamp, also shown on the top-left corner of the frame, indicates its capture time in the original recording.

🖼️ Reference Image Data (ref_images.zip): Contains the auxiliary images referenced in the questions for each sample.

📝 Text Annotation (mmsivideo.json)：This file contains the annotation information for MMSI-Video-Bench. All time references in the questions correspond to the capture time in the original recording and align with the timestamp flag on each frame. Key fields include:

{
  "ref_images": [Paths to auxiliary images referenced in the question,...],
  "video_list": [
    {
      "path": Video clip file path,
      "start": Timestamp (in seconds) of the first frame of the video clip in the original recording,
      "end": Timestamp (in seconds) of the last frame of the video clip in the original recording,
      "base_fps": Base sampling rate
    },
    ...
  ],
  "frames_list": [[Paths to frames sampled at the base sampling rate,...],...],
  "system_prompt": "...",
  "task_prompt": Task-specific prompt,
  "user_prompt": Question text, with <video> as a placeholder for video and <image> for auxiliary images,
  "format_prompt": Output format requirements,
  "ground_truth": Correct answer
}

Unless otherwise specified, the model input generally consists of: system_prompt + task_prompt + user_prompt + format_prompt.

As outlined in our paper, we support two evaluation settings: Sufficient-Coverage ensures lossless sampling by preserving all essential information, while Uniform‑50 uniformly samples 50 frames and cannot guarantee full information retention. Although most models—due to API constraints or GPU memory limits—only support the Uniform‑50 setting, we still strongly recommend evaluating under the Sufficient‑Coverage setting whenever possible.

1. Data Preparation: After downloading the data, unzip the files and organize the directory as follows:

   data/
   ├── mmsivideo.json
   ├── frames/
   ├── ref_images/
   ├── videos/
   

2. Install Required Packages. If you plan to evaluate open-source models, please set up their corresponding environments.

   pip install -r requirements.txt
   

3. Run Inference & Evaluation: For proprietary models: Update the base_url and api_key in utils/openai_api.py with your own credentials. For open-source models: Modify the load_model function in inference.py to point to the corresponding model path. Run the following commands to perform inference for a specific model under a given setting, and evaluate the results on a specific benchmark (Main, Robot Bench, Indoor Scene Perception Bench or Grounding Bench):

   python inference.py --model_name {model_name} --setting Uniform-50/Sufficient-Coverage # inference
   python evaluation.py --eval_dir {path/to/results} --bench main/robot_bench/ground_bench/indoor_perception_bench/easy2hard_bench # evaluation

Our MMSI-Video-Bench has been integrated into VLMEvalKit. As VLMEvalKit currently does not support mixed video and image inputs, we have adapted a version that accepts only image inputs. Follow the QuickStart guide of VLMEvalKit and place the required data under the following structure:

LMUDATA/
├── MMSIVideo_SC.tsv
├── MMSIVideo_U50.tsv
├── images/
│   └── MMSIVideo/
│       ├── frames/
│       ├── ref_images/

When evaluating proprietary models, set img_detail = low; when evaluating QwenVL-series models, set max_pixels = 360*420. Run the evaluation with the following command:

python run.py --model Qwen2.5-VL-32B-Instruct --data MMSIVideo_SC/MMSIVideo_U50

Note: The results reported in our paper were generated through the Native Codebase Evaluation. While actual evaluation results may differ slightly from the paper, variations remain within an acceptable range. Potential contributing factors include:

(1) Variations in option ordering, runtime environments, or random seeds, as well as differences in inference configurations between VLMEvalKit and the Native Codebase Evaluation, may lead to fluctuations of 1–2% in the overall score.

(2) For certain models—such as the QwenVL series and the LLaVA-Video series—the input configuration in VLMEvalKit (image-only) differs from that in the Native Codebase (image + video), which may result in a variance of 2–3% in the overall score.

(3) The overall scores of thinking models (e.g., GPT-4o, Gemini 3 Pro, Gemini 2.5 Flash) may also fluctuate within a range of 2–3% across runs.

Note: For the three sub-benchmarks, we take the higher score of each model across the two settings for easier presentation.

@misc{lin2025mmsivideobenchholisticbenchmarkvideobased,
      title={MMSI-Video-Bench: A Holistic Benchmark for Video-Based Spatial Intelligence}, 
      author={Jingli Lin and Runsen Xu and Shaohao Zhu and Sihan Yang and Peizhou Cao and Yunlong Ran and Miao Hu and Chenming Zhu and Yiman Xie and Yilin Long and Wenbo Hu and Dahua Lin and Tai Wang and Jiangmiao Pang},
      year={2025},
      eprint={2512.10863},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2512.10863}, 
}

Shield:

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

MMSI-Video-Bench utilizes data from the following open-source datasets: Roomtour3d, ScanNet, ScanNet++, 3RScan, ARKitScenes, RealEstate10k, DL3DV, Waymo, NuScenes, OVIS, TrackingNet, LaSOT, UAV123, Ego4D, EPIC-KITCHENS, EgoExoLearn, MultiSports, charades, LEMMA, TF2023, CVMHT, AVA, DROID, RH20T, DTU. We sincerely thank the respective teams for their valuable contributions to the research community.

• Jingli Lin: [email protected]
• Runsen Xu: [email protected]