Evaluations on ReVSI with corresponding VSI-Bench scores shown in parentheses. For ReVSI, each model is evaluated against frame-sampling-specific ground-truth answers corresponding to its inference frames (shown in Frames column), whereas VSI-Bench uses a single set of ground-truth answers shared across all frame settings. ReVSI scores that exceed their VSI-Bench counterparts are highlighted in green.

Performance of specialized 3D VLMs and their base models on ReVSI with corresponding VSI-Bench scores shown in parentheses. ReVSI evaluates each model under its native inference frame setting using frame-adaptive ground-truth answers. Scores lower than the base model are highlighted in red.

Object counting results on dummy videos where the ground-truth answer is always 0. Query-Drop removes frames containing the queried object while preserving the surrounding scene and other objects. First-Frame repeats the first frame of the Query-Drop video across all frames. Black uses pure black frames for the whole video. Exact Match accuracy is reported over 997 questions, with 0 as the only correct answer. Fine-tuned models tend to lean on strong priors learned from training data when answering, often overlooking the actual visual input.

Real Video

···

Query-Drop Video

···

First-Frame Video

···

Black Video

···

Three dummy-video constructions used to probe whether models rely on visual evidence when answering object-counting questions.

Method	Accuracy (Exact Match)
	Query-Drop	First-Frame	Black
Zero-shot
Human	100.0	100.0	100.0
GPT-5.2	74.0	89.6	99.2
Gemini 3 Pro	62.3	85.0	94.0
Qwen2.5-VL-7B-Instruct	55.8	79.9	99.8
Qwen3-VL-8B-Instruct	34.7	80.9	99.8
Qwen3-VL-32B-Instruct	50.5	92.7	100.0
InternVL3.5-8B	14.7	52.5	17.7
InternVL3.5-38B	9.1	45.0	1.2
LLaVA-Video-7B-Qwen2	47.7	49.9	0.0
LLaVA-Video-72B-Qwen2	45.0	65.6	15.2
Fine-tuned
Cambrian-S-7B	1.1	2.8	0.0
VST-7B-SFT	1.1	8.7	0.4
SpaceR-7B (SG-RLVR)	8.1	24.3	14.6
Spatial-MLLM-4B-135k	0.2	0.9	0.0
Spatial-MLLM-4B-820k	0.0	0.0	0.0
VLM3R-7B	4.1	2.5	0.2

Resources

ReVSI supports the following inference and evaluation frameworks:

ModelScope SWIFT

Scalable lightWeight Infrastructure for Fine-Tuning

LMMs-Eval

One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks

VLMEvalKit

A Toolkit for Evaluating Large Vision-Language Models

Citation

If you find this work useful, please consider citing our paper:

@article{zhang2026revsi,
  title={ReVSI: Rebuilding Visual Spatial Intelligence Evaluation for Accurate Assessment of VLM 3D Reasoning},
  author={Zhang, Yiming and Chen, Jiacheng and Tan, Jiaqi and Mao, Yongsen and Chen, Wenhu and Chang, Angel X.},
  journal={arXiv preprint arXiv:2604.24300},
  year={2026}
}

Acknowledgment

This work was funded in part by a CIFAR AI Chair and NSERC Discovery Grants, and enabled by support from Digital Research Alliance of Canada and a CFI/BCKDF JELF grant. We thank Jiayi Liu for help with benchmark data verification and discussion.