Investigating the potential of a customized AI chatbot to automate writing assessment in a compulsory English course at a Hong Kong university

Since the release of ChatGPT in 2022, Generative AI has brought a large influence on multiple aspects of language education, such as teaching, learning, and assessment. This study aims to explore whether ChatGPT can be used as an assessment tool in a compulsory English course at a Hong Kong university. To achieve high ecological validity in this teaching context, an AI-powered Chatbot was built to replicate the exact training process that English teachers need to undergo before assessing students’ writing tasks, such as reading assignment prompts, familiarizing themselves with the assessment rubric, and reviewing standardization samples. This Chatbot was applied to automatically score 100 narrative essays written by undergraduate students, who speak Chinese as their first language. The findings show a slight level of agreement (Kappa value of κ = 0.126) between three general grade levels (A, B, and C) and a positive, moderate correlation (r = 0.446) between Chatbot scores and teacher scores. These findings reveal both the opportunities and challenges of using GenAI to automate writing assessment in English classrooms. The study concludes with implications for future research and language teaching practices.