9.4 Implementation analysis questions

Implementation analysis can cover areas such as (King et al., 1987; Love, 2004):

• Implementation context. Does the implementation context favor the implementation of the program as planned? How? What are the characteristics of the intervention implementation site, of the people involved in the program, of the clientele? What are the main differences between the different implementation contexts and how do they interact with the intervention and the results production? How does the implementation vary across different implementation sites? These last two questions will be important when the same program is implemented in different contexts.
• Intervention history and purpose. What’s the program’s implementation history? Consider its inception, key milestones, and implementation steps. What is the rationale for its implementation and its objectives? Is the intervention implemented as planned (e.g. intensity of activities, unexpected events, etc.)? What aspects of the program design needs to (or should) be changed to obtain the expected results in this implementation context? What are the facilitators and obstacles to its implementation and success? What problems have been encountered, and how do they differ across implementation sites? How are activities meeting expectations? Were beneficiaries satisfied?
• Stakeholders and implementors. Who are the main groups involved? Have they changed? What’s their level of engagement/support? Any issues with staff recruitment and retention? Are professionals well trained and qualified? Are there important external actors supporting or impeding the implementation of the intervention?
• Participants. What are the characteristics of the participants? How are they selected? Is there any issue with the selection process? With retention? Does the program reach the targeted clientele? Are potential clients rejecting or leaving the program?
• Budget and Funding. Is the program well-funded? Are the costs as planned? Is the intervention sustainable in the longer term and financially healthy? What are the risks and threats to its sustainability?
• Results. What are the successes, weaknesses, strengths? Is the intervention leading to expected results? Are elements missing for achieving results? Which components seems the most important for reaching the expected results? Do the results vary from one site to the other? Has any factor prevented the production of effects? Which components seem the most important for reaching the expected results? How do results differ according to variations of the context, such as participants, professionals, contextual dynamics?

Implementation analysis can explain why interventions may not work in some contexts yet work well in other contexts. Effect analysis is often conducted in conditions where researchers try to control the context to isolate the causal effect, usually before intervention implementation. However, many contextual elements can impede or facilitate the production of effects. For example, culture may become a factor with a major influence when it comes to implementation. Such elements can be documented when conducting an implementation analysis and show how the specific contextual characteristics may influence the production of effects, even when the intervention is fully implemented.