Abstract
Online self-learning platforms, such as question-answering (Q&A) websites, are popular technology for learning, and utilising Pretrained Language Models (PLMs) to maintain their content qualities is a common practice. However, it is challenging to identify the cause that affects the performance of PLMs. In this study, we propose using machine common sense to identify the cause affecting the performance of PLMs. We conducted an empirical experiment with three PLMs using a publicly available dataset. We select 45000 data points as the training data and 1000 as the testing data. We first train the PLMs with training data, then run machine common sense tests to examine their reasoning abilities. We define the causal relationship between content quality and reasoning ability, and use the cause to derive Metamorphic Relations (MR) of Metamorphic Testing (MT) for creating follow-up testing datasets. We analyse the changes between source and follow-up outputs to see whether the identified cause affects the performance. Results show that the reasonableness of the content is the cause that affects the performance of PLM, which has reasoning abilities. In addition, the proposed approach in this study is effective for identifying and validating the cause that affects the performance of PLMs, even on devices with limited computer resources. Future research can apply our approach and seek different machine common sense tests and counterfactual analysing techniques to identify different causes of performance issues of different PLMs.
Original language | English |
---|---|
Publication status | Accepted/In press/Filed - 14 May 2025 |
Event | 11th International Symposium on Educational Technology - Shangri-La Bangkok 89 Soi Wat Suan Plu, New Road, Bangrak, Bangkok 10500 Thailand, Bangkok, Thailand Duration: 22 Jul 2025 → 25 Jul 2025 https://hksmic.org.hk/iset/2025/ |
Conference
Conference | 11th International Symposium on Educational Technology |
---|---|
Abbreviated title | ISET |
Country/Territory | Thailand |
City | Bangkok |
Period | 22/07/25 → 25/07/25 |
Internet address |
Bibliographical note
Since this conference is yet to commence, the information for this record is subject to revision.Research Keywords
- Content quality prediction
- causality
- machine common sense
- metamorphic testing
- self-learning platform
- pre-trained language model