Automatic Evaluation of Multiple-Choice Items for Reading Comprehension: Effects of Question and Distractor Categories

John S. Y. Lee; Yin Poon; Shunjie Wang; Kai Wah Samuel Chu

Automatic Evaluation of Multiple-Choice Items for Reading Comprehension: Effects of Question and Distractor Categories

John S. Y. Lee
, Yin Poon
, Shunjie Wang
, Kai Wah Samuel Chu

Department of Linguistics and Translation

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

Abstract

Automatic generation of multiple-choice (MC) items for reading comprehension can support language learning by providing large amounts of practice materials. To enable rapid development of MC generation models, automatic assessment is essential since it is time-consuming to manually evaluate question and distractor quality. Although Text Informativity (TI) has been adopted as an automatic evaluation metric, the ability of Large Language Models (LLMs) to estimate the TI scores of different categories of questions and distractors has not yet been thoroughly analyzed. This paper investigates LLM performance in calculating TI scores for the range of questions and distractors defined in the PIRLS (Progress in International Reading Literacy Study) and STARC (Structured Annotations for Reading Comprehension) frameworks. We show that automatically estimated TI scores may result in systematic preferences for some question and distractor categories, and recommend that TI scores be used for within-category comparisons only.
©ELRA Language Resources Association (ELRA), 2026

Original language	English
Title of host publication	Proceedings of Shaping Multilingual, Multimodal AI for the Social Sciences and Humanities (LLMs4SSH) @ LREC 2026
Editors	Arturo Montejo-Ráez, Cristina Grisot, Joanna Blochowiak
Publisher	European Language Resources Association (ELRA)
Pages	170-174
ISBN (Electronic)	978-2-493814-85-2
Publication status	Published - 11 May 2026
Event	15th International Conference on Language Resources and Evaluation - Palau de Congressos de Palma, Palma, Spain Duration: 11 May 2026 → 16 May 2026 https://lrec2026.info/

Conference

Conference	15th International Conference on Language Resources and Evaluation
Abbreviated title	LREC 2026
Place	Spain
City	Palma
Period	11/05/26 → 16/05/26
Internet address	https://lrec2026.info/

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 4 Quality Education

Research Keywords

multiple-choice items
text informativity
question generation
distractor generation

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

Lee, J. S. Y., Poon, Y., Wang, S., & Chu, K. W. S. (2026). Automatic Evaluation of Multiple-Choice Items for Reading Comprehension: Effects of Question and Distractor Categories. In A. Montejo-Ráez, C. Grisot, & J. Blochowiak (Eds.), Proceedings of Shaping Multilingual, Multimodal AI for the Social Sciences and Humanities (LLMs4SSH) @ LREC 2026 (pp. 170-174). European Language Resources Association (ELRA).

Lee, John S. Y. ; Poon, Yin ; Wang, Shunjie et al. / Automatic Evaluation of Multiple-Choice Items for Reading Comprehension : Effects of Question and Distractor Categories. Proceedings of Shaping Multilingual, Multimodal AI for the Social Sciences and Humanities (LLMs4SSH) @ LREC 2026. editor / Arturo Montejo-Ráez ; Cristina Grisot ; Joanna Blochowiak. European Language Resources Association (ELRA), 2026. pp. 170-174

@inproceedings{7c10e3df4fd5461fa2d7457a8d6cfac8,

title = "Automatic Evaluation of Multiple-Choice Items for Reading Comprehension: Effects of Question and Distractor Categories",

abstract = "Automatic generation of multiple-choice (MC) items for reading comprehension can support language learning by providing large amounts of practice materials. To enable rapid development of MC generation models, automatic assessment is essential since it is time-consuming to manually evaluate question and distractor quality. Although Text Informativity (TI) has been adopted as an automatic evaluation metric, the ability of Large Language Models (LLMs) to estimate the TI scores of different categories of questions and distractors has not yet been thoroughly analyzed. This paper investigates LLM performance in calculating TI scores for the range of questions and distractors defined in the PIRLS (Progress in International Reading Literacy Study) and STARC (Structured Annotations for Reading Comprehension) frameworks. We show that automatically estimated TI scores may result in systematic preferences for some question and distractor categories, and recommend that TI scores be used for within-category comparisons only. {\textcopyright}ELRA Language Resources Association (ELRA), 2026",

keywords = "multiple-choice items, text informativity, question generation, distractor generation",

author = "Lee, \{John S. Y.\} and Yin Poon and Shunjie Wang and Chu, \{Kai Wah Samuel\}",

year = "2026",

month = may,

day = "11",

language = "English",

pages = "170--174",

editor = "Arturo Montejo-R{\'a}ez and Cristina Grisot and Joanna Blochowiak",

booktitle = "Proceedings of Shaping Multilingual, Multimodal AI for the Social Sciences and Humanities (LLMs4SSH) @ LREC 2026",

publisher = "European Language Resources Association (ELRA)",

note = "15th International Conference on Language Resources and Evaluation, LREC 2026 ; Conference date: 11-05-2026 Through 16-05-2026",

url = "https://lrec2026.info/",

}

Lee, JSY, Poon, Y, Wang, S & Chu, KWS 2026, Automatic Evaluation of Multiple-Choice Items for Reading Comprehension: Effects of Question and Distractor Categories. in A Montejo-Ráez, C Grisot & J Blochowiak (eds), Proceedings of Shaping Multilingual, Multimodal AI for the Social Sciences and Humanities (LLMs4SSH) @ LREC 2026. European Language Resources Association (ELRA), pp. 170-174, 15th International Conference on Language Resources and Evaluation, Palma, Spain, 11/05/26.

Automatic Evaluation of Multiple-Choice Items for Reading Comprehension: Effects of Question and Distractor Categories. / Lee, John S. Y.; Poon, Yin; Wang, Shunjie et al.
Proceedings of Shaping Multilingual, Multimodal AI for the Social Sciences and Humanities (LLMs4SSH) @ LREC 2026. ed. / Arturo Montejo-Ráez; Cristina Grisot; Joanna Blochowiak. European Language Resources Association (ELRA), 2026. p. 170-174.

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - Automatic Evaluation of Multiple-Choice Items for Reading Comprehension

T2 - 15th International Conference on Language Resources and Evaluation

AU - Lee, John S. Y.

AU - Poon, Yin

AU - Wang, Shunjie

AU - Chu, Kai Wah Samuel

PY - 2026/5/11

Y1 - 2026/5/11

N2 - Automatic generation of multiple-choice (MC) items for reading comprehension can support language learning by providing large amounts of practice materials. To enable rapid development of MC generation models, automatic assessment is essential since it is time-consuming to manually evaluate question and distractor quality. Although Text Informativity (TI) has been adopted as an automatic evaluation metric, the ability of Large Language Models (LLMs) to estimate the TI scores of different categories of questions and distractors has not yet been thoroughly analyzed. This paper investigates LLM performance in calculating TI scores for the range of questions and distractors defined in the PIRLS (Progress in International Reading Literacy Study) and STARC (Structured Annotations for Reading Comprehension) frameworks. We show that automatically estimated TI scores may result in systematic preferences for some question and distractor categories, and recommend that TI scores be used for within-category comparisons only. ©ELRA Language Resources Association (ELRA), 2026

AB - Automatic generation of multiple-choice (MC) items for reading comprehension can support language learning by providing large amounts of practice materials. To enable rapid development of MC generation models, automatic assessment is essential since it is time-consuming to manually evaluate question and distractor quality. Although Text Informativity (TI) has been adopted as an automatic evaluation metric, the ability of Large Language Models (LLMs) to estimate the TI scores of different categories of questions and distractors has not yet been thoroughly analyzed. This paper investigates LLM performance in calculating TI scores for the range of questions and distractors defined in the PIRLS (Progress in International Reading Literacy Study) and STARC (Structured Annotations for Reading Comprehension) frameworks. We show that automatically estimated TI scores may result in systematic preferences for some question and distractor categories, and recommend that TI scores be used for within-category comparisons only. ©ELRA Language Resources Association (ELRA), 2026

KW - multiple-choice items

KW - text informativity

KW - question generation

KW - distractor generation

M3 - RGC 32 - Refereed conference paper (with host publication)

SP - 170

EP - 174

BT - Proceedings of Shaping Multilingual, Multimodal AI for the Social Sciences and Humanities (LLMs4SSH) @ LREC 2026

A2 - Montejo-Ráez, Arturo

A2 - Grisot, Cristina

A2 - Blochowiak, Joanna

PB - European Language Resources Association (ELRA)

Y2 - 11 May 2026 through 16 May 2026

ER -

Lee JSY, Poon Y, Wang S, Chu KWS. Automatic Evaluation of Multiple-Choice Items for Reading Comprehension: Effects of Question and Distractor Categories. In Montejo-Ráez A, Grisot C, Blochowiak J, editors, Proceedings of Shaping Multilingual, Multimodal AI for the Social Sciences and Humanities (LLMs4SSH) @ LREC 2026. European Language Resources Association (ELRA). 2026. p. 170-174

Automatic Evaluation of Multiple-Choice Items for Reading Comprehension: Effects of Question and Distractor Categories

Abstract

Conference

UN SDGs

Research Keywords

Other Access Options

Permanent Link

Fingerprint

Cite this