1. Introduction
2. Dataset
3. Results
4. Usage

VisFinEval is a large-scale Chinese benchmark platform designed to systematically evaluate the performance of multimodal large language models (MLLMs) in real-world financial business scenarios. The dataset includes 15,848 annotated question-answer pairs, covering eight typical financial image modalities (such as candlestick charts, financial statements, etc.), and is divided into three progressive difficulty levels: financial knowledge and data analysis, financial analysis and decision support, and financial risk control and asset optimization. We conducted zero-shot testing on 21 cutting-edge MLLMs, and the results showed that although the best-performing model, Qwen-VL-max, achieved an accuracy of 76.3%, surpassing non-specialized human performance, it still lagged significantly behind financial experts by more than 14 percentage points in advanced domain-specific tasks such as multi-step numerical reasoning and business process understanding.

Back to Top

This table compares multiple QA datasets across various dimensions, including question types, financial capability assessment, and scale.

VisFinEval comprises three scenario categories with 15 business scenario types:

Scenario Types:

1. Financial Knowledge and Data Analysis (Front Office): FDS, CCA, FIA, FERI, SSSB, FIE, FSR
2. Financial Analysis and Business Decision (Middle Office): FSA, IAI, IA, FMSA
3. Financial Risk Control and Asset Optimization (Back Office): FSO, FRPA, FDRI, AAA

Question Type Distribution

Back to Top

VisFinEval/
├── data/                                              # Raw datasets
│   ├── figure/                                        # Image assets
│   ├── markdown/                                      # Markdown-formatted data
│   ├── Financial Knowledge and Data Analysis          # Front Office Scenario Type data
│   ├── Financial Analysis and Business Decision       # Middle Office Scenario Type data
│   └── Financial Risk Control and Asset Optimization  # Back Office Scenario Type data
├── logs/                  						       # Runtime logs
├── output/                	              		       # Model outputs
├── scripts/             						       # Execution scripts
│   ├── visfineval.py    						       # test scripts
│   └── run_model.sh      						       # Unified evaluation scripts
├── README.md          						           # Project documentation
└── requirements.txt    					           # Dependency specifications

• Clone Repository:

  ​ Download VisFinEval from github.

• Download Dataset:

  Download the dataset from the following Google Drive link: https://drive.google.com/file/d/15DdloCn2GWRvyO-kieTJMiE2sMIfW4vF/view?usp=drive_link

  Extract the downloaded files and place them in the VisFinEval/data directory.

• Install Dependencies:

     pip install -r requirements.txt

• Using the run_model.sh Script to Run All Evaluation Question Types:

  bash run_model.sh

• Model Configuration for Evaluation:

  • API Model Configuration:

    Modify the model settings according to the comments:

  # API Model Configuration (Format: API_KEY, API_BASE_URL, Model Name)
  USE_API=true 
  API_MODEL_NAME="your_model_name"
  API_KEY="your_model_key"
  API_BASE_URL="your_url"

  • Local Model Configuration:

    Modify the model settings according to the comments:

  # Local Model Configuration (Format: Model Name, Model Path, Model Type, Additional Parameters)
  USE_API=false
  API_MODEL_NAME="your_model_name"
  MODEL_PATH="your_model_address"

• The evaluation results are output in the VisFinEval/output and VisFinEval/logs folders.
• The output results are organized by question type, with each folder further categorized by model.

Back to Top