Improving Readability Assessment with Ordinal Log-Loss

Ho Hung Lim; John S. Y. Lee

Improving Readability Assessment with Ordinal Log-Loss

Department of Linguistics and Translation

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

39 Downloads (CityUHK Scholars)

Abstract

Automatic Readability Assessment (ARA) predicts the level of difficulty of a text, e.g. at Grade 1 to Grade 12. ARA is an ordinal classification task since the predicted levels follow an underlying order, from easy to difficult. However, most neural ARA models ignore the distance between the gold level and predicted level, treating all levels as independent labels. This paper investigates whether distance-sensitive loss functions can improve ARA performance. We evaluate a variety of loss functions on neural ARA models, and show that ordinal log-loss can produce statistically significant improvement over the standard cross-entropy loss in terms of adjacent accuracy in a majority of our datasets. © 2024 Association for Computational Linguistics

Original language	English
Title of host publication	Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024)
Publisher	Association for Computational Linguistics
Pages	343–350
Publication status	Published - Jun 2024
Event	19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024) - Mexico City, Mexico Duration: 20 Jun 2024 → 21 Jun 2024 https://www.aclweb.org/portal/content/19th-workshop-innovative-use-nlp-building-educational-applications

Conference

Conference	19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024)
Country/Territory	Mexico
City	Mexico City
Period	20/06/24 → 21/06/24
Internet address	https://www.aclweb.org/portal/content/19th-workshop-innovative-use-nlp-building-educational-applications

Funding

This work was partly supported by the Language Fund from the Standing Committee on Language Education and Research (project EDB(LE)/P&R/EL/203/14) and by a Teaching Development Grant from City University of Hong Kong (project 6000834).

Publisher's Copyright Statement

This full text is made available under CC-BY 4.0. https://creativecommons.org/licenses/by/4.0/

Access Output

224845821.pdfFinal Published version, 192 KBLicence: CC BY

https://aclanthology.org/2024.bea-1.28

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

TDG(CityU): Chatbots as Pedagogical Tools: A Feasibility Study
LEE, J. S. Y. (Principal Investigator / Project Coordinator)
17/07/23 → …
Project: Research

Cite this

@inproceedings{821d70755e4c4ae3a0bffc073f55142a,

title = "Improving Readability Assessment with Ordinal Log-Loss",

abstract = "Automatic Readability Assessment (ARA) predicts the level of difficulty of a text, e.g. at Grade 1 to Grade 12. ARA is an ordinal classification task since the predicted levels follow an underlying order, from easy to difficult. However, most neural ARA models ignore the distance between the gold level and predicted level, treating all levels as independent labels. This paper investigates whether distance-sensitive loss functions can improve ARA performance. We evaluate a variety of loss functions on neural ARA models, and show that ordinal log-loss can produce statistically significant improvement over the standard cross-entropy loss in terms of adjacent accuracy in a majority of our datasets. {\textcopyright} 2024 Association for Computational Linguistics",

author = "Lim, {Ho Hung} and Lee, {John S. Y.}",

year = "2024",

month = jun,

language = "English",

pages = "343–350",

booktitle = "Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024)",

publisher = "Association for Computational Linguistics",

address = "United States",

note = "19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024) ; Conference date: 20-06-2024 Through 21-06-2024",

url = "https://www.aclweb.org/portal/content/19th-workshop-innovative-use-nlp-building-educational-applications",

}

Lim, HH & Lee, JSY 2024, Improving Readability Assessment with Ordinal Log-Loss. in Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024). Association for Computational Linguistics, pp. 343–350, 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), Mexico City, Mexico, 20/06/24. <https://aclanthology.org/2024.bea-1.28>

Improving Readability Assessment with Ordinal Log-Loss. / Lim, Ho Hung; Lee, John S. Y.
Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024). Association for Computational Linguistics, 2024. p. 343–350.

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - Improving Readability Assessment with Ordinal Log-Loss

AU - Lim, Ho Hung

AU - Lee, John S. Y.

PY - 2024/6

Y1 - 2024/6

N2 - Automatic Readability Assessment (ARA) predicts the level of difficulty of a text, e.g. at Grade 1 to Grade 12. ARA is an ordinal classification task since the predicted levels follow an underlying order, from easy to difficult. However, most neural ARA models ignore the distance between the gold level and predicted level, treating all levels as independent labels. This paper investigates whether distance-sensitive loss functions can improve ARA performance. We evaluate a variety of loss functions on neural ARA models, and show that ordinal log-loss can produce statistically significant improvement over the standard cross-entropy loss in terms of adjacent accuracy in a majority of our datasets. © 2024 Association for Computational Linguistics

AB - Automatic Readability Assessment (ARA) predicts the level of difficulty of a text, e.g. at Grade 1 to Grade 12. ARA is an ordinal classification task since the predicted levels follow an underlying order, from easy to difficult. However, most neural ARA models ignore the distance between the gold level and predicted level, treating all levels as independent labels. This paper investigates whether distance-sensitive loss functions can improve ARA performance. We evaluate a variety of loss functions on neural ARA models, and show that ordinal log-loss can produce statistically significant improvement over the standard cross-entropy loss in terms of adjacent accuracy in a majority of our datasets. © 2024 Association for Computational Linguistics

M3 - RGC 32 - Refereed conference paper (with host publication)

SP - 343

EP - 350

BT - Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024)

PB - Association for Computational Linguistics

T2 - 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024)

Y2 - 20 June 2024 through 21 June 2024

ER -

Improving Readability Assessment with Ordinal Log-Loss

Abstract

Conference

Funding

Publisher's Copyright Statement

Access Output

Other Access Options

Permanent Link

Fingerprint

Projects

TDG(CityU): Chatbots as Pedagogical Tools: A Feasibility Study

Cite this