Identifying the cause of performance issues of Pretrained Language Model for Educational Technology

Online self-learning platforms, such as question-answering (Q&A) websites, are popular technology for learning, and utilising Pretrained Language Models (PLMs) to maintain their content qualities is a common practice. However, it is challenging to identify the cause that affects the performance of PLMs. In this study, we propose using machine common sense to identify the cause affecting the performance of PLMs. We conducted an empirical experiment with three PLMs using a publicly available dataset. We select 45000 data points as the training data and 1000 as the testing data. We first train the PLMs with training data, then run machine common sense tests to examine their reasoning abilities. We define the causal relationship between content quality and reasoning ability, and use the cause to derive Metamorphic Relations (MR) of Metamorphic Testing (MT) for creating follow-up testing datasets. We analyse the changes between source and follow-up outputs to see whether the identified cause affects the performance. Results show that the reasonableness of the content is the cause that affects the performance of PLM, which has reasoning abilities. In addition, the proposed approach in this study is effective for identifying and validating the cause that affects the performance of PLMs, even on devices with limited computer resources. Future research can apply our approach and seek different machine common sense tests and counterfactual analysing techniques to identify different causes of performance issues of different PLMs.