Adoption of Computer Programming Exercises for Automatic Assessment — Issues and Caution

Computational thinking is an interdisciplinary core skill to be acquired in STEM education, while computer program coding is a concrete manifestation of such a skill. In response to the increasing size of computer programming classes and rapidly growing number of learners, particularly in massive open online courses (MOOCs), many instructors nowadays heavily rely on the use of automated systems to assess the programming work of students. However, these automated assessment systems typically perform black box testing to determine the correctness of student programs, which limits the type of programming exercises that can be automatically assessed. This paper reports a case study on the adoption of programming exercises from textbook and online resources, and categorises some difficulties and issues of caution due to the technical limitation of typical automated assessment systems. The identified issues are mainly related to the input/output and non-deterministic nature of the programs or the intended learning outcomes of some of the exercises. The paper concludes with a brief outline of recent research directions to alleviate these problems for improvement of learning.