AesBiasBench: Evaluating Bias and Alignment in Multimodal Language Models for Personalized Image Aesthetic Assessment

Kun Li; Lai-Man Po; Hongzheng Yang; Xuyuan Xu; Kangcheng Liu; Yuzhi Zhao

doi:10.18653/v1/2025.emnlp-main.386

AesBiasBench: Evaluating Bias and Alignment in Multimodal Language Models for Personalized Image Aesthetic Assessment

Kun Li, Lai-Man Po, Hongzheng Yang, Xuyuan Xu, Kangcheng Liu, Yuzhi Zhao^*

^*Corresponding author for this work

Department of Electrical Engineering

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

1 Downloads (CityUHK Scholars)

Abstract

Multimodal Large Language Models (MLLMs)are increasingly applied in Personalized Image Aesthetic Assessment (PIAA) as a scalable alternative to expert evaluations. However, their predictions may reflect subtle biases influenced by demographic factors such as gender, age, and education. In this work, we propose AesBiasBench, a benchmark designed to evaluate MLLMs along two complementary dimensions: (1) stereotype bias, quantified by measuring variations in aesthetic evaluations across demographic groups; and (2) alignment between model outputs and genuine human aesthetic preferences. Our benchmark covers three subtasks (Aesthetic Perception, Assessment, Empathy) and introduces structured metrics (IFD, NRD, AAS) to assess both bias and alignment. We evaluate 19 MLLMs, including proprietary models (e.g., GPT-4o,Claude-3.5-Sonnet) and open-source models(e.g., InternVL-2.5, Qwen2.5-VL). Results indicate that smaller models exhibit stronger stereotype biases, whereas larger models align more closely with human preferences. Incorporating identity information often exacerbates bias, particularly in emotional judgments. These findings underscore the importance of identity-aware evaluation frameworks in subjective vision-language tasks. ©2025 Association for Computational Linguistics

Original language	English
Title of host publication	Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
Editors	Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, Violet Peng
Place of Publication	United States
Publisher	Association for Computational Linguistics
Pages	7607–7620
ISBN (Print)	979-8-89176-332-6
DOIs	https://doi.org/10.18653/v1/2025.emnlp-main.386
Publication status	Published - Nov 2025
Event	The 2025 Conference on Empirical Methods in Natural Language Processing - Suzhou, China Duration: 4 Nov 2025 → 9 Nov 2025 https://aclanthology.org/volumes/2025.emnlp-main/

Conference

Conference	The 2025 Conference on Empirical Methods in Natural Language Processing
Abbreviated title	EMNLP 2025
Place	China
City	Suzhou
Period	4/11/25 → 9/11/25
Internet address	https://aclanthology.org/volumes/2025.emnlp-main/

Bibliographical note

Research Unit(s) information for this publication is provided by the author(s) concerned.

Publisher's Copyright Statement

This full text is made available under CC-BY 4.0. https://creativecommons.org/licenses/by/4.0/

Access Output

10.18653/v1/2025.emnlp-main.386

394101722.pdfFinal Published version, 2.44 MBLicence: CC BY

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

Li, K., Po, L.-M., Yang, H., Xu, X., Liu, K., & Zhao, Y. (2025). AesBiasBench: Evaluating Bias and Alignment in Multimodal Language Models for Personalized Image Aesthetic Assessment. In C. Christodoulopoulos, T. Chakraborty, C. Rose, & V. Peng (Eds.), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (pp. 7607–7620). Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.emnlp-main.386

Li, Kun ; Po, Lai-Man ; Yang, Hongzheng et al. / AesBiasBench : Evaluating Bias and Alignment in Multimodal Language Models for Personalized Image Aesthetic Assessment. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. editor / Christos Christodoulopoulos ; Tanmoy Chakraborty ; Carolyn Rose ; Violet Peng. United States : Association for Computational Linguistics, 2025. pp. 7607–7620

@inproceedings{3641f449ed8e41f39b94b9c62869b496,

title = "AesBiasBench: Evaluating Bias and Alignment in Multimodal Language Models for Personalized Image Aesthetic Assessment",

abstract = "Multimodal Large Language Models (MLLMs)are increasingly applied in Personalized Image Aesthetic Assessment (PIAA) as a scalable alternative to expert evaluations. However, their predictions may reflect subtle biases influenced by demographic factors such as gender, age, and education. In this work, we propose AesBiasBench, a benchmark designed to evaluate MLLMs along two complementary dimensions: (1) stereotype bias, quantified by measuring variations in aesthetic evaluations across demographic groups; and (2) alignment between model outputs and genuine human aesthetic preferences. Our benchmark covers three subtasks (Aesthetic Perception, Assessment, Empathy) and introduces structured metrics (IFD, NRD, AAS) to assess both bias and alignment. We evaluate 19 MLLMs, including proprietary models (e.g., GPT-4o,Claude-3.5-Sonnet) and open-source models(e.g., InternVL-2.5, Qwen2.5-VL). Results indicate that smaller models exhibit stronger stereotype biases, whereas larger models align more closely with human preferences. Incorporating identity information often exacerbates bias, particularly in emotional judgments. These findings underscore the importance of identity-aware evaluation frameworks in subjective vision-language tasks. {\textcopyright}2025 Association for Computational Linguistics ",

author = "Kun Li and Lai-Man Po and Hongzheng Yang and Xuyuan Xu and Kangcheng Liu and Yuzhi Zhao",

note = "Research Unit(s) information for this publication is provided by the author(s) concerned.; The 2025 Conference on Empirical Methods in Natural Language Processing, EMNLP 2025 ; Conference date: 04-11-2025 Through 09-11-2025",

year = "2025",

month = nov,

doi = "10.18653/v1/2025.emnlp-main.386",

language = "English",

isbn = "979-8-89176-332-6",

pages = "7607–7620",

editor = "Christos Christodoulopoulos and Tanmoy Chakraborty and Carolyn Rose and Violet Peng",

booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",

publisher = "Association for Computational Linguistics",

address = "United States",

url = "https://aclanthology.org/volumes/2025.emnlp-main/",

}

Li, K , Po, L-M, Yang, H, Xu, X, Liu, K & Zhao, Y 2025, AesBiasBench: Evaluating Bias and Alignment in Multimodal Language Models for Personalized Image Aesthetic Assessment. in C Christodoulopoulos, T Chakraborty, C Rose & V Peng (eds), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, United States , pp. 7607–7620, The 2025 Conference on Empirical Methods in Natural Language Processing, Suzhou, China, 4/11/25. https://doi.org/10.18653/v1/2025.emnlp-main.386

AesBiasBench: Evaluating Bias and Alignment in Multimodal Language Models for Personalized Image Aesthetic Assessment. / Li, Kun ; Po, Lai-Man; Yang, Hongzheng et al.
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. ed. / Christos Christodoulopoulos; Tanmoy Chakraborty; Carolyn Rose; Violet Peng. United States : Association for Computational Linguistics, 2025. p. 7607–7620.

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - AesBiasBench

T2 - The 2025 Conference on Empirical Methods in Natural Language Processing

AU - Li, Kun

AU - Po, Lai-Man

AU - Yang, Hongzheng

AU - Xu, Xuyuan

AU - Liu, Kangcheng

AU - Zhao, Yuzhi

N1 - Research Unit(s) information for this publication is provided by the author(s) concerned.

PY - 2025/11

Y1 - 2025/11

N2 - Multimodal Large Language Models (MLLMs)are increasingly applied in Personalized Image Aesthetic Assessment (PIAA) as a scalable alternative to expert evaluations. However, their predictions may reflect subtle biases influenced by demographic factors such as gender, age, and education. In this work, we propose AesBiasBench, a benchmark designed to evaluate MLLMs along two complementary dimensions: (1) stereotype bias, quantified by measuring variations in aesthetic evaluations across demographic groups; and (2) alignment between model outputs and genuine human aesthetic preferences. Our benchmark covers three subtasks (Aesthetic Perception, Assessment, Empathy) and introduces structured metrics (IFD, NRD, AAS) to assess both bias and alignment. We evaluate 19 MLLMs, including proprietary models (e.g., GPT-4o,Claude-3.5-Sonnet) and open-source models(e.g., InternVL-2.5, Qwen2.5-VL). Results indicate that smaller models exhibit stronger stereotype biases, whereas larger models align more closely with human preferences. Incorporating identity information often exacerbates bias, particularly in emotional judgments. These findings underscore the importance of identity-aware evaluation frameworks in subjective vision-language tasks. ©2025 Association for Computational Linguistics

AB - Multimodal Large Language Models (MLLMs)are increasingly applied in Personalized Image Aesthetic Assessment (PIAA) as a scalable alternative to expert evaluations. However, their predictions may reflect subtle biases influenced by demographic factors such as gender, age, and education. In this work, we propose AesBiasBench, a benchmark designed to evaluate MLLMs along two complementary dimensions: (1) stereotype bias, quantified by measuring variations in aesthetic evaluations across demographic groups; and (2) alignment between model outputs and genuine human aesthetic preferences. Our benchmark covers three subtasks (Aesthetic Perception, Assessment, Empathy) and introduces structured metrics (IFD, NRD, AAS) to assess both bias and alignment. We evaluate 19 MLLMs, including proprietary models (e.g., GPT-4o,Claude-3.5-Sonnet) and open-source models(e.g., InternVL-2.5, Qwen2.5-VL). Results indicate that smaller models exhibit stronger stereotype biases, whereas larger models align more closely with human preferences. Incorporating identity information often exacerbates bias, particularly in emotional judgments. These findings underscore the importance of identity-aware evaluation frameworks in subjective vision-language tasks. ©2025 Association for Computational Linguistics

U2 - 10.18653/v1/2025.emnlp-main.386

DO - 10.18653/v1/2025.emnlp-main.386

M3 - RGC 32 - Refereed conference paper (with host publication)

SN - 979-8-89176-332-6

SP - 7607

EP - 7620

BT - Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing

A2 - Christodoulopoulos, Christos

A2 - Chakraborty, Tanmoy

A2 - Rose, Carolyn

A2 - Peng, Violet

PB - Association for Computational Linguistics

CY - United States

Y2 - 4 November 2025 through 9 November 2025

ER -

Li K , Po LM, Yang H, Xu X, Liu K, Zhao Y. AesBiasBench: Evaluating Bias and Alignment in Multimodal Language Models for Personalized Image Aesthetic Assessment. In Christodoulopoulos C, Chakraborty T, Rose C, Peng V, editors, Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. United States : Association for Computational Linguistics. 2025. p. 7607–7620 doi: 10.18653/v1/2025.emnlp-main.386

AesBiasBench: Evaluating Bias and Alignment in Multimodal Language Models for Personalized Image Aesthetic Assessment

Abstract

Conference

Bibliographical note

Publisher's Copyright Statement

Access Output

Other Access Options

Permanent Link

Fingerprint

Cite this