VLMEvalKit: An Open-Source ToolKit for Evaluating Large Multi-Modality Models

VLMEvalKit: An Open-Source ToolKit for Evaluating Large Multi-Modality Models

Haodong Duan, Xinyu Fang, Junming Yang, Xiangyu Zhao, Zerun Ma, Yuxuan Qiao, Mo Li, Tianhao Liang, Lin Zhu, Amit Agarwal, Xiaozhe Li, Shengyuan Ding, Jiazi Bu, Ziyu Liu, Zhangyang Qi, Yifei Li, Yuhang Zang, Zhe Chen, Lin Chen, Yuan Liu, Yubo Ma, Hailong Sun, Yifan Zhang, Shiyin Lu, Tack Hwa Wong, Weiyun Wang, Peiheng Zhou, Chaoyou Fu, Junbo Cui, Jixuan Chen, Enxin Song, Song Mao, Junming Lin, Xilin Wei, Jinsong Li, Zeyi Sun, Zhaowei Wang, Zicheng Zhang, Xiaoyi Dong, Junjun He, Pan Zhang, Jiaqi Wang, Dahua Lin, Kai Chen. arXiv (updated version v5) (2026).

Publisher X/Twitter LinkedIn Email

Summary. VLMEvalKit is a PyTorch toolkit for evaluating large multimodal models through a shared workflow across models and benchmarks. It automates common evaluation steps and includes the OpenVLM Leaderboard for tracking research progress.

Version note. This record represents the updated arXiv v5 preprint, dated July 6, 2026, whose author list includes Amit Agarwal. The original ACM Multimedia 2024 publication has a different author list and does not list Amit.

Versioned preprint, full abstract, and paper.

Related Papers

• Evaluate Generalisation & Robustness of Visual Features from Images to Video LJMU (2021)
• Enhancing Document AI Data Generation Through Graph-Based Synthetic Layouts IJERT 2024 (2024)
• LLM for Barcodes: Generating Diverse Synthetic Data for Identity Documents WCSC 2025 (2026)
• MVTamperBench: Evaluating Robustness of Vision-Language Models ACL Findings 2025 (2025)
• Survey of Large Multimodal Model Datasets, Application Categories and Taxonomy CVR 2025 (2026)