Automatic testing of student programs using token patterns

Abstract

Assessing students' programming exercises manually can be tedious and error-prone. To relieve these problems, many universities have developed software systems to automatically assess students' programs, usually by executing them to test their functional correctness, typically through simple matching of the program output with expected output. This approach requires that the precise and detailed form of the output be given in the specification of the exercise, because even a small and insignificant deviation (such as an extra whitespace) from the expected output will cause the matching to fail. An undesirable pedagogical consequence of this technical limitation is that it may unnecessarily confine the way of implementation, perhaps inhibit creativity, and distract students from the essentials of the exercise. To handle slightly deviated but acceptable program outputs, some automated systems employ tailored output comparison techniques that match certain kinds of output variants. However, existing techniques are mostly ad hoc and limited in capabilities. There is a need for more sophisticated techniques that can tolerate reasonably acceptable output variants. Towards this goal, we propose a novel approach, based on token patterns, that splits the output texts into smaller units for more fine-grained output comparison. As a result, the program output comparison can be more flexible and the accompanying undesirable side effects on teaching and learning can be reduced. An experimental prototype was initially built to evaluate our proposed approach. The prototype was then further integrated into a production system used in a programming course for empirical evaluation. The evaluation results are encouraging, showing that our proposed new approach is feasible and promising in improving student satisfaction in the use of the automated system, as well as producing better outcomes in terms of teaching and learning.

Date of Award	15 Jul 2011
Original language	English
Awarding Institution	City University of Hong Kong
Supervisor	Y.T. Yu (Supervisor) & Chung Keung POON (Co-supervisor)