VQualA 2025 Challenge on Visual Quality Comparison for Large Multimodal Models: Methods and Results

Hanwei Zhu; Haoning Wu; Zicheng Zhang; Lingyu Zhu; Yixuan Li; Peilin Chen; Shiqi Wang; Chris Wei Zhou; Linhan Cao; Wei Sun; Xiangyang Zhu; Weixia Zhang; Yucheng Zhu; Jing Liu; Dandan Zhu; Guangtao Zhai; Xiongkuo Min; Zhichao Zhang; Xinyue Li; Shubo Xu; Anh Dao; Yifan Li; Hongyuan Yu; Jiaojiao Yi; Yiding Tian; Yupeng Wu; Feiran Sun; Lijuan Liao; Song Jiang

doi:10.1109/ICCVW69036.2025.00355

VQualA 2025 Challenge on Visual Quality Comparison for Large Multimodal Models: Methods and Results

Hanwei Zhu
, Haoning Wu
, Zicheng Zhang
, Lingyu Zhu
, Yixuan Li
, Peilin Chen
, Shiqi Wang^*
, Chris Wei Zhou
, Linhan Cao
, Wei Sun
, Xiangyang Zhu
, Weixia Zhang
, Yucheng Zhu
, Jing Liu
, Dandan Zhu
, Guangtao Zhai
, Xiongkuo Min
, Zhichao Zhang
, Xinyue Li
, Shubo Xu

Anh Dao, Yifan Li, Hongyuan Yu, Jiaojiao Yi, Yiding Tian, Yupeng Wu, Feiran Sun, Lijuan Liao, Song Jiang

^*Corresponding author for this work

Department of Computer Science

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

6 Citations (Scopus)

Abstract

This paper presents a summary of the VQualA 2025 Challenge on Visual Quality Comparison for Large Multimodal Models (LMMs), hosted as part of the ICCV 2025 Work-shop on Visual Quality Assessment. The challenge aims to evaluate and enhance the ability of state-of-the-art LMMs to perform open-ended and detailed reasoning about visual quality differences across multiple images. To this end, the competition introduces a novel benchmark comprising thousands of coarse-to-fine grained visual quality comparison tasks, spanning single images, pairs, and multi-image groups. Each task requires models to provide accurate quality judgments. The competition emphasizes holistic evaluation protocols, including 2AFC-based binary preference and multi-choice questions (MCQs). Around 100 participants submitted entries, with five models demonstrating the emerging capabilities of instruction-tuned LMMs on quality assessment. This challenge marks a significant step toward open-domain visual quality reasoning and comparison and serves as a catalyst for future research on inter-pretable and human-aligned quality evaluation systems. © 2025 IEEE.

Original language	English
Title of host publication	Proceedings - 2025 IEEE/CVF International Conference on Computer Vision Workshops
Subtitle of host publication	ICCV-W 2025
Publisher	IEEE
Pages	3383-3393
ISBN (Electronic)	9798331589882
ISBN (Print)	979-8-3315-8989-9
DOIs	https://doi.org/10.1109/ICCVW69036.2025.00355
Publication status	Published - Oct 2025
Event	2025 IEEE/CVF International Conference on Computer Vision Workshops (ICCV-W 2025) - Honolulu, United States Duration: 19 Oct 2025 → 23 Oct 2025 https://iccv.thecvf.com/

Publication series

Name	Proceedings - IEEE/CVF International Conference on Computer Vision Workshops, ICCV-W
ISSN (Print)	2473-9936
ISSN (Electronic)	2473-9944

Workshop

Workshop	2025 IEEE/CVF International Conference on Computer Vision Workshops (ICCV-W 2025)
Abbreviated title	ICCVW 2025
Place	United States
City	Honolulu
Period	19/10/25 → 23/10/25
Internet address	https://iccv.thecvf.com/

Funding

The research was partially supported by the RGC General Research Fund 11200323 and NSFC/RGC JRS Project N-CityU198/24.

Research Keywords

Image Quality Assessment
Large Multimodal Models

Access Output

10.1109/ICCVW69036.2025.00355

https://openaccess.thecvf.com/content/ICCV2025W/VQualA/html/Zhu_VQualA_2025_Challenge_on_Visual_Quality_Comparison_for_Large_Multimodal_ICCVW_2025_paper.html

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

Zhu, H., Wu, H., Zhang, Z., Zhu, L., Li, Y., Chen, P., Wang, S., Zhou, C. W., Cao, L., Sun, W., Zhu, X., Zhang, W., Zhu, Y., Liu, J., Zhu, D., Zhai, G., Min, X., Zhang, Z., Li, X., ... Jiang, S. (2025). VQualA 2025 Challenge on Visual Quality Comparison for Large Multimodal Models: Methods and Results. In Proceedings - 2025 IEEE/CVF International Conference on Computer Vision Workshops: ICCV-W 2025 (pp. 3383-3393). (Proceedings - IEEE/CVF International Conference on Computer Vision Workshops, ICCV-W). IEEE. https://doi.org/10.1109/ICCVW69036.2025.00355

@inproceedings{56081d3caaaa4f118b0d0dc0967124d0,

title = "VQualA 2025 Challenge on Visual Quality Comparison for Large Multimodal Models: Methods and Results",

abstract = "This paper presents a summary of the VQualA 2025 Challenge on Visual Quality Comparison for Large Multimodal Models (LMMs), hosted as part of the ICCV 2025 Work-shop on Visual Quality Assessment. The challenge aims to evaluate and enhance the ability of state-of-the-art LMMs to perform open-ended and detailed reasoning about visual quality differences across multiple images. To this end, the competition introduces a novel benchmark comprising thousands of coarse-to-fine grained visual quality comparison tasks, spanning single images, pairs, and multi-image groups. Each task requires models to provide accurate quality judgments. The competition emphasizes holistic evaluation protocols, including 2AFC-based binary preference and multi-choice questions (MCQs). Around 100 participants submitted entries, with five models demonstrating the emerging capabilities of instruction-tuned LMMs on quality assessment. This challenge marks a significant step toward open-domain visual quality reasoning and comparison and serves as a catalyst for future research on inter-pretable and human-aligned quality evaluation systems. {\textcopyright} 2025 IEEE.",

keywords = "Image Quality Assessment, Large Multimodal Models",

author = "Hanwei Zhu and Haoning Wu and Zicheng Zhang and Lingyu Zhu and Yixuan Li and Peilin Chen and Shiqi Wang and Zhou, \{Chris Wei\} and Linhan Cao and Wei Sun and Xiangyang Zhu and Weixia Zhang and Yucheng Zhu and Jing Liu and Dandan Zhu and Guangtao Zhai and Xiongkuo Min and Zhichao Zhang and Xinyue Li and Shubo Xu and Anh Dao and Yifan Li and Hongyuan Yu and Jiaojiao Yi and Yiding Tian and Yupeng Wu and Feiran Sun and Lijuan Liao and Song Jiang",

year = "2025",

month = oct,

doi = "10.1109/ICCVW69036.2025.00355",

language = "English",

isbn = "979-8-3315-8989-9",

series = "Proceedings - IEEE/CVF International Conference on Computer Vision Workshops, ICCV-W",

publisher = "IEEE",

pages = "3383--3393",

booktitle = "Proceedings - 2025 IEEE/CVF International Conference on Computer Vision Workshops",

address = "United States",

note = "2025 IEEE/CVF International Conference on Computer Vision Workshops (ICCV-W 2025) , ICCVW 2025 ; Conference date: 19-10-2025 Through 23-10-2025",

url = "https://iccv.thecvf.com/",

}

Zhu, H, Wu, H, Zhang, Z, Zhu, L, Li, Y, Chen, P , Wang, S, Zhou, CW, Cao, L, Sun, W, Zhu, X, Zhang, W, Zhu, Y, Liu, J, Zhu, D, Zhai, G, Min, X, Zhang, Z, Li, X, Xu, S, Dao, A, Li, Y, Yu, H, Yi, J, Tian, Y, Wu, Y, Sun, F, Liao, L & Jiang, S 2025, VQualA 2025 Challenge on Visual Quality Comparison for Large Multimodal Models: Methods and Results. in Proceedings - 2025 IEEE/CVF International Conference on Computer Vision Workshops: ICCV-W 2025. Proceedings - IEEE/CVF International Conference on Computer Vision Workshops, ICCV-W, IEEE, pp. 3383-3393, 2025 IEEE/CVF International Conference on Computer Vision Workshops (ICCV-W 2025) , Honolulu, Hawaii, United States, 19/10/25. https://doi.org/10.1109/ICCVW69036.2025.00355

VQualA 2025 Challenge on Visual Quality Comparison for Large Multimodal Models: Methods and Results. / Zhu, Hanwei; Wu, Haoning; Zhang, Zicheng et al.
Proceedings - 2025 IEEE/CVF International Conference on Computer Vision Workshops: ICCV-W 2025. IEEE, 2025. p. 3383-3393 (Proceedings - IEEE/CVF International Conference on Computer Vision Workshops, ICCV-W).

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - VQualA 2025 Challenge on Visual Quality Comparison for Large Multimodal Models

T2 - 2025 IEEE/CVF International Conference on Computer Vision Workshops (ICCV-W 2025)

AU - Zhu, Hanwei

AU - Wu, Haoning

AU - Zhang, Zicheng

AU - Zhu, Lingyu

AU - Li, Yixuan

AU - Chen, Peilin

AU - Wang, Shiqi

AU - Zhou, Chris Wei

AU - Cao, Linhan

AU - Sun, Wei

AU - Zhu, Xiangyang

AU - Zhang, Weixia

AU - Zhu, Yucheng

AU - Liu, Jing

AU - Zhu, Dandan

AU - Zhai, Guangtao

AU - Min, Xiongkuo

AU - Zhang, Zhichao

AU - Li, Xinyue

AU - Xu, Shubo

AU - Dao, Anh

AU - Li, Yifan

AU - Yu, Hongyuan

AU - Yi, Jiaojiao

AU - Tian, Yiding

AU - Wu, Yupeng

AU - Sun, Feiran

AU - Liao, Lijuan

AU - Jiang, Song

PY - 2025/10

Y1 - 2025/10

N2 - This paper presents a summary of the VQualA 2025 Challenge on Visual Quality Comparison for Large Multimodal Models (LMMs), hosted as part of the ICCV 2025 Work-shop on Visual Quality Assessment. The challenge aims to evaluate and enhance the ability of state-of-the-art LMMs to perform open-ended and detailed reasoning about visual quality differences across multiple images. To this end, the competition introduces a novel benchmark comprising thousands of coarse-to-fine grained visual quality comparison tasks, spanning single images, pairs, and multi-image groups. Each task requires models to provide accurate quality judgments. The competition emphasizes holistic evaluation protocols, including 2AFC-based binary preference and multi-choice questions (MCQs). Around 100 participants submitted entries, with five models demonstrating the emerging capabilities of instruction-tuned LMMs on quality assessment. This challenge marks a significant step toward open-domain visual quality reasoning and comparison and serves as a catalyst for future research on inter-pretable and human-aligned quality evaluation systems. © 2025 IEEE.

AB - This paper presents a summary of the VQualA 2025 Challenge on Visual Quality Comparison for Large Multimodal Models (LMMs), hosted as part of the ICCV 2025 Work-shop on Visual Quality Assessment. The challenge aims to evaluate and enhance the ability of state-of-the-art LMMs to perform open-ended and detailed reasoning about visual quality differences across multiple images. To this end, the competition introduces a novel benchmark comprising thousands of coarse-to-fine grained visual quality comparison tasks, spanning single images, pairs, and multi-image groups. Each task requires models to provide accurate quality judgments. The competition emphasizes holistic evaluation protocols, including 2AFC-based binary preference and multi-choice questions (MCQs). Around 100 participants submitted entries, with five models demonstrating the emerging capabilities of instruction-tuned LMMs on quality assessment. This challenge marks a significant step toward open-domain visual quality reasoning and comparison and serves as a catalyst for future research on inter-pretable and human-aligned quality evaluation systems. © 2025 IEEE.

KW - Image Quality Assessment

KW - Large Multimodal Models

UR - http://www.scopus.com/inward/record.url?scp=105035216246&partnerID=8YFLogxK

UR - https://www.scopus.com/record/pubmetrics.uri?eid=2-s2.0-105035216246&origin=recordpage

U2 - 10.1109/ICCVW69036.2025.00355

DO - 10.1109/ICCVW69036.2025.00355

M3 - RGC 32 - Refereed conference paper (with host publication)

SN - 979-8-3315-8989-9

T3 - Proceedings - IEEE/CVF International Conference on Computer Vision Workshops, ICCV-W

SP - 3383

EP - 3393

BT - Proceedings - 2025 IEEE/CVF International Conference on Computer Vision Workshops

PB - IEEE

Y2 - 19 October 2025 through 23 October 2025

ER -

Zhu H, Wu H, Zhang Z, Zhu L, Li Y, Chen P et al. VQualA 2025 Challenge on Visual Quality Comparison for Large Multimodal Models: Methods and Results. In Proceedings - 2025 IEEE/CVF International Conference on Computer Vision Workshops: ICCV-W 2025. IEEE. 2025. p. 3383-3393. (Proceedings - IEEE/CVF International Conference on Computer Vision Workshops, ICCV-W). doi: 10.1109/ICCVW69036.2025.00355

VQualA 2025 Challenge on Visual Quality Comparison for Large Multimodal Models: Methods and Results

Abstract

Publication series

Workshop

Funding

Research Keywords

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

Cite this