Replication of Experimental Research: Implications for the Study of Public Management

Introduction Replication is increasingly recognised as an important part of the production of knowledge in not only the natural sciences, but also the social sciences (Francis 2012; Freese 2007; Nosek and Lakens 2014; Sargent 1981; Schmidt 2009). This chapter concentrates on replication in the context of public management experiments. There are multiple definitions of replication and many social science journals now have a standard for replication of published work in the sense of requiring authors to make their data, programs, and relevant details available to permit other researchers to reproduce the results that appear in the published paper. The checking of analysis is one important type of replication, and the issue of replicability is important for most kinds of method. Replication of this kind is important; however, less discussed but also important is replication that goes beyond the checking of data and rerunning of code. Replications of experiments can be divided into two categories. Direct replications implement the same experimental procedure as previous research, and conceptual replications attempt to replicate findings including tests of hypotheses by using different experimental procedures or materials (Schmidt 2009). Direct replication is often taken as the heart of experimental replication. This chapter sets out a set of distinctions discussed by Tsang and Kwan (1999) that unpack different forms of direct replication based on whether researchers use the same or different samples and same or different populations and extends into conceptual replication based on whether researchers use different experimental procedures or materials in their research. Replication has theory-based and practical benefits for the study of public management. This chapter illustrates these benefits using experiments about red tape as an example of their application to a core public management topic. Direct replications in particular can help uncover errors (and even fraud) and can help researchers understand if previous experimental findings have simply emerged by chance (Ioannidis 2005), decreasing false discovery rates. Replications can help generalise findings to contexts or subject populations beyond those where the initial experiment was conducted, helping to address the criticism sometimes made of experiments that their findings can lack generalisability (Cartwright and Hardie 2012). Relatedly, replications of experiments to other contexts can help assess the expectations of theory about how and why findings vary between settings.