Implementation analysis
9.5 Evaluation designs for implementation analysis
Implementation analysis provides an explanation of what works under specific conditions and identifies what does not. It’s important to gather as much information about the intervention as possible prior to doing fieldwork. Having a sense of the information already available will be essential for orienting the evaluation questions and for planning for the data collection. In particular, it is helpful to have a good description of the intervention, information about the broader context, detailed information on what was implemented and what was not, and achievement of results (King et al., 1987). Background documents or insiders’ views on implementation difficulties and dynamics will also be helpful. Implementation analysis often relies on complementary information provided by the monitoring to document discrepancies between what was planned and what has been implemented.
Assessment of program implementation requires taking into account the surrounding social systems, including characteristics of the organization in which the program is being implemented, characteristics of persons delivering the program (Viadro et al., 1997; Zapka et al., 2004), existing structures of the organizations and groups, organizational social system characteristics (e.g., interorganizational linkages, community agency partnerships; (Scheirer et al., 1995; Zapka et al., 2004), and factors in the external environment (e.g., competing events, controversy about the program, external political factors, and history and events that happen concurrently with the program (Scheirer et al., 1995; Viadro et al., 1997; Zapka et al., 2004). (as cited in Saunders et al., 2005, pp. 136-137)
Different designs can be used, according to the evaluation project objective and to the intervention implementation context.
- Process evaluation. Process evaluation aims at documenting implementation steps and dynamics. This type of evaluation “focusses on the internal dynamics and actual operations of a program in an attempt to understand its strengths and weaknesses” (Patton, 1997, p. 206). “Process evaluations search for explanations of the successes, failures, and changes in a program” including “unanticipated consequences in the full context of program implementation and development” (Patton, 1997, p. 206). The work of Langley and colleagues on process analysis and Brousselle & Champagne’s evaluation of the UNAIDS initiative in Chile can both provide insights and examples of how to conduct such analysis (Brousselle & Champagne, 2004; Langley, 1999; Langley et al., 2013).
- Comparative study. A comparative study across different implementation sites can provide information on the contextual determinants of the intervention’s implementation both in terms of its integrality and level of implementation (Champagne et al., 2011b, p. 246). Such a comparative study may benefit from existing monitoring data on process such as on fidelity and reach.
- Realist evaluation. Realist evaluation can be interpreted as a theory-based implementation analysis (Brousselle & Buregeya, 2018). Realist evaluation assesses complex programs by analyzing what works, for whom, and under what circumstances (Pawson & Tilley, 1997, 2004).
It is a logic of inquiry that illuminates the program theory underlying the inherent characteristics of program implementation (Hewitt et al., 2012; Pawson & Tilley, 1997, 2004) to investigate the generative mechanisms associated with the program (M), the contexts under which the pathways operate (C), and the ways in which outcomes occur (O) (Salter & Kothari, 2014). Context–mechanism–outcome (CMO) configurations, as outlined by Pawson and Tilley (1997, 2004), foster the examination of recurrent patterns in the midst of complex social reality through in-depth explanations of causal pathways. This helps the evaluator articulate the program theory to be investigated and test hypotheses to produce transferable advice based on that theory and to inform decisions as well as evidence-based policy-making processes (Hewitt et al., 2012). (as cited in Brousselle & Buregeya, 2018, p. 157)
Realist evaluation involves four core steps: 1) articulating the program theories to be tested; 2) collecting data to test the hypotheses; 3) testing the hypotheses; and 4) interpreting and refining them (Mehdipanah et al., 2015; Pawson & Tilley, 1997, 2004; Ranmuthugala et al., 2011; Salter & Kothari, 2014). It uncovers underlying implicit or explanatory theory leading to the program and its multiple components, and it identifies contextual factors that spearhead pathways of change to produce expected outcomes (Jagosh et al., 2015; Pawson, 2002; Pawson & Tilley, 1997, 2004; Ridde et al., 2012; Salter & Kothari, 2014). (as cited in Brousselle & Buregeya, 2018, p. 157)
