Effect analysis and related approaches
8.6 Impact evaluation
Impact evaluation, as presented by Stern and colleagues, is an umbrella term used to present a diversity of designs focusing on identifying effects and impacts of interventions. According to these authors, impact evaluation aims at measuring effects but also answering the “how” and “why” questions, shifting the focus of the evaluation from causality to explanation (Stern, 2015; Stern et al., 2012).
Stern opposes the narrowly defined focus of traditional effect analysis to the consideration of effects in impact evaluation.
Experimental methods are concerned with intended rather than unintended effects; assume direct links between interventions and outcomes; address primary rather than secondary effects; and usually look to evidence in the short-term rather than the long-term. This latter is especially important as in many development settings effects are not known when programme funding ends, only becoming clear over a much more extended timescale. Most counterfactual methods on the other hand focus on the short-term, which is likely to capture only a sub-set of programme results. (Stern, 2015, p. 5)
Impact evaluation aims to answer the four following questions:
1) To what extent can a specific (net) impact be attributed to the intervention?
2) Did the intervention make a difference?
3) How has the intervention made a difference?
4) Will the intervention work elsewhere?” (Stern et al., 2012, p. 37)
A diversity of designs can be considered for conducting impact evaluations. Stern identifies four main approaches to impact evaluations (IE):
Regularity frameworks that depend on the frequency of association between cause and effect – the inference basis for statistical approaches to IE.
Counterfactual frameworks that depend on the difference between two otherwise identical cases – the inference basis for experimental and quasi-experimental approaches to IE.
Multiple causation that depends on combinations of causes that lead to an effect – the inference basis for ‘configurational’ approaches to IE including qualitative comparative analysis (QCA) and contribution analysis.
Generative causation that depends on identifying the ‘mechanisms’ that explain effects – the inference basis for ‘theory based’ and ‘realist’ approaches to IE. (Stern, 2015, p. 17)
Table 8.2 represents different available options. As you can see, the conceptualization of impact evaluation, as proposed by Stern (2015), encompasses experimental research, quasi-experimental research, natural experiments, and contribution analysis.
Table 8.3 elaborates on the choice of the evaluation design for impact evaluations based on the key evaluation questions and the contextual research conditions.
Table 8.2 Design Approaches, Variants and Causal Inference
| Design approaches | Specific variants | Basis for causal inference |
| Experimental | RCTs
Quasi experiments Natural experiments |
Counterfactuals: The difference between two otherwise identical cases – the manipulated and the controlled; the co-presence of cause and effects. |
| Statistical | Statistical modelling
Longitudinal studies Econometrics |
Regularity: Correlation between cause and effect or between variables, influence of (usually) isalatable multiple cuases on a single effect. Control of ‘confounders’. |
| Theory-based | Causal process designs: Theory of change, process tracing, contribution analysis, impact pathways,
Causal mechanism designs: Realist evaluations, congruence analysis |
Generative causation: Identification and confirmation of causal processes or ‘chains’.
Supporting factors and mechanisms at work in context. |
| Case-based | Interpretative: Naturalistic, grounded theory, ethnography
Structured: Configurations, QCA, within-case-analysis, simulations and network analysis |
Multiple causation: Comparison across and within cases of combinations of causal factors.
Analytic generalization based on theory. |
| Participatory | Normative designs: Participatory or democratic evaluation, empowerment evaluation.
Agency designs: learning by doing, policy dialogue, collaborative action research. |
Actor Agency: Validation by participants that their actions and experienced effects are ‘caused’ by programme.
Adoption, customization and commitment to a goal |
| Synthesis studies | Meta-analysis, narrative synthesis, realist-based synthesis | Accumulation and aggregation within a number of perspectives (statistical, theory-based, ethnographic). |
Source: Stern, E. (2015). Impact Evaluation. A Guide for Commisioners and Managers, p.18: https://www.bond.org.uk/wp-content/uploads/2022/08/impact_evaluation_guide_0515.pdf
Table 8.3 Summarizing the Design Implications of Different Impact Evaluation Questions
| Key evaluation questions | Related evaluation questions | Underlying assumption | Requirements | Suitable designs |
| To what extent can a specific (net) impact be attributed to the intervention? | What is the net effect of the intervention?
How much of the impact can be attributed to the intervention? What would have happened without the intervention? |
Expected outcomes and the intervention itself clearly understood and specifiable
Likelihood of primary cause and primary effect Interest in particular intervention rather than generalisation |
Can manipulate interventions
Sufficient numbers (beneficiaries, households, etc.) for statistical analysis |
Experiments
Statistical studies Hybrids with case-based and participatory designs |
| Has the intervention made a difference? | What causes are necessary or sufficient for the effect?
Was the intervention needed to produce the effect? Would these impacts have happened anyhow? |
There are several relevant causes that need to be disentangled
Interventions are just one part of a causal package
|
Comparable cases where a common set of causes are present and evidence exists as to their potency
|
Experiments
Theory-based evaluation, eg contribution analysis Case-based designs, eg QCA
|
| How has the intervention made a difference? | How and why have the impacts come about?
What causal factors have resulted in the observed impacts? Has the intervention resulted in any unintended impacts? For whom has the intervention made a difference? |
Interventions interact with other causal factors
It is possible to clearly represent the causal process through which the intervention made a difference – may require ‘theory development’
|
Understanding how supporting and contextual factors that connect intervention with effects
Theory that allows for the identification of supporting factors (proximate, contextual and historical)
|
Theory-based evaluation especially ‘realist’ variants and
Contribution Analysis Participatory approaches
|
| Can this be expected to work elsewhere? | Can this ‘pilot’ be transferred elsewhere and scaled up?
Is the intervention sustainable? What generalizable lessons have we learned about impact? |
What has worked in one place can work somewhere else
Stakeholders will cooperate in joint donor/beneficiary evaluations
|
Generic understanding of contexts eg typologies of context
Clusters of causal packages Innovation diffusion mechanisms
|
Participatory approaches and some Experimental and Theory-based approaches
Natural experiments Realist evaluation Synthesis studies |
Source: Stern, E. (2015). Impact Evaluation. A Guide for Commisioners and Managers, p.22: https://www.bond.org.uk/wp-content/uploads/2022/08/impact_evaluation_guide_0515.pdf
Impact evaluations and contribution analysis have expanded the methodological landscape for analyzing the effects and impacts of interventions. This diversity of approaches enables analysis in contexts where experiments are not always feasible. Each approach will have its advantages, strengths, and potential validity threats. This overview touches briefly on a significant domain of evaluation; further reading is strongly recommended for those intending to lead evaluations on this critical topic.
