Automatic readability assessment for sentences: neural, hybrid and large language models

Automatic readability assessment (ARA) aims to determine the cognitive load of a reader to comprehend a given text. ARA research has been mostly conducted at the text level, with numerous studies showing strong performance of neural and hybrid models. ARA at the sentence level, however, has received less attention, even though many applications in natural language processing (NLP) require assessment of the difficulty of individual sentences. This article compares the performance of neural models, hybrid models and large language models (LLMs) for sentence-level ARA, making three main contributions. First, we construct the first Chinese sentence-level ARA datasets, with nearly 70K sentences, to facilitate evaluation on Chinese. Second, we present the first experimental results on applying LLMs to sentence-level ARA. Finally, while previous work focused mostly on English data, we show that hybrid models outperform traditional classifiers, neural models, and LLMs in both English and Chinese data. The best hybrid model obtained state-of-the-art results on the Wall Street Journal dataset, surpassing the previous best result by almost 15% absolute. It also achieved competitive results on the CEFR-SP dataset. In detailed analyses, we identify the linguistic features that most significantly contributed to the performance of the hybrid model. We show that 10 linguistic features are correlated to readability across all datasets, and models trained on this reduced feature set achieve performance that rivals the full set. These results not only yield new insights into hybrid models for sentence-level ARA, but also set new benchmarks for future research in both English and Chinese.

© The Author(s), under exclusive licence to Springer Nature B.V. 2025