Identifying the cause of performance issues of Pretrained Language Model for Educational Technology

Research output: Conference PapersRGC 32 - Refereed conference paper (without host publication)peer-review

Abstract

Online self-learning platforms, such as question-answering (Q&A) websites, are popular technology for learning, and utilising Pretrained Language Models (PLMs) to maintain their content qualities is a common practice. However, it is challenging to identify the cause that affects the performance of PLMs. In this study, we propose using machine common sense to identify the cause affecting the performance of PLMs. We conducted an empirical experiment with three PLMs using a publicly available dataset. We select 45000 data points as the training data and 1000 as the testing data. We first train the PLMs with training data, then run machine common sense tests to examine their reasoning abilities. We define the causal relationship between content quality and reasoning ability, and use the cause to derive Metamorphic Relations (MR) of Metamorphic Testing (MT) for creating follow-up testing datasets. We analyse the changes between source and follow-up outputs to see whether the identified cause affects the performance. Results show that the reasonableness of the content is the cause that affects the performance of PLM, which has reasoning abilities. In addition, the proposed approach in this study is effective for identifying and validating the cause that affects the performance of PLMs, even on devices with limited computer resources. Future research can apply our approach and seek different machine common sense tests and counterfactual analysing techniques to identify different causes of performance issues of different PLMs.
Original languageEnglish
Publication statusAccepted/In press/Filed - 14 May 2025
Event11th International Symposium on Educational Technology - Shangri-La Bangkok 89 Soi Wat Suan Plu, New Road, Bangrak, Bangkok 10500 Thailand, Bangkok, Thailand
Duration: 22 Jul 202525 Jul 2025
https://hksmic.org.hk/iset/2025/

Conference

Conference11th International Symposium on Educational Technology
Abbreviated titleISET
Country/TerritoryThailand
CityBangkok
Period22/07/2525/07/25
Internet address

Bibliographical note

Since this conference is yet to commence, the information for this record is subject to revision.

Research Keywords

  • Content quality prediction
  • causality
  • machine common sense
  • metamorphic testing
  • self-learning platform
  • pre-trained language model

Fingerprint

Dive into the research topics of 'Identifying the cause of performance issues of Pretrained Language Model for Educational Technology'. Together they form a unique fingerprint.

Cite this