{"id":242,"date":"2025-10-03T18:16:47","date_gmt":"2025-10-03T22:16:47","guid":{"rendered":"https:\/\/pressbooks.bccampus.ca\/foundationsofevaluation\/?post_type=chapter&p=242"},"modified":"2026-05-26T14:40:53","modified_gmt":"2026-05-26T18:40:53","slug":"impact-evaluation","status":"publish","type":"chapter","link":"https:\/\/pressbooks.bccampus.ca\/foundationsofevaluation\/chapter\/impact-evaluation\/","title":{"raw":"8.6 Impact evaluation","rendered":"8.6 Impact evaluation"},"content":{"raw":"Impact evaluation, as presented by Stern and colleagues, is an umbrella term used to present a diversity of designs focusing on identifying effects and impacts of interventions. According to these authors, impact evaluation aims at measuring effects but also answering the \u201chow\u201d and \u201cwhy\u201d questions, shifting the focus of the evaluation from causality to explanation (Stern, 2015; Stern et al., 2012).\r\n\r\nStern opposes the narrowly defined focus of traditional effect analysis to the consideration of effects in impact evaluation.\r\n
Experimental methods are concerned with intended<\/strong> rather than unintended effects; assume direct<\/strong> links between interventions and outcomes; address primary<\/strong> rather than secondary effects; and usually look to evidence in the short-term<\/strong> rather than the long-term. This latter is especially important as in many development settings effects are not known when programme funding ends, only becoming clear over a much more extended timescale. Most counterfactual methods on the other hand focus on the short-term, which is likely to capture only a sub-set of programme results. (Stern, 2015, p. 5)<\/blockquote>\r\nImpact evaluation aims to answer the four following questions:\r\n
\r\n

1) To what extent can a specific (net) impact be attributed to the intervention?<\/p>\r\n

2) Did the intervention make a difference?<\/p>\r\n

3) How has the intervention made a difference?<\/p>\r\n

4) Will the intervention work elsewhere?\u201d (Stern et al., 2012, p. 37)<\/p>\r\n<\/blockquote>\r\nA diversity of designs can be considered for conducting impact evaluations. Stern identifies four main approaches to impact evaluations (IE):\r\n

\r\n
Regularity frameworks<\/strong> that depend on the frequency of association between cause and effect \u2013 the inference basis for statistical approaches to IE.\r\n\r\nCounterfactual frameworks<\/strong> that depend on the difference between two otherwise identical cases \u2013 the inference basis for experimental and quasi-experimental approaches to IE.\r\n\r\nMultiple causation<\/strong> that depends on combinations of causes that lead to an effect \u2013 the inference basis for \u2018configurational\u2019 approaches to IE including qualitative comparative analysis (QCA) and contribution analysis.\r\n\r\nGenerative causation<\/strong> that depends on identifying the \u2018mechanisms\u2019 that explain effects \u2013 the inference basis for \u2018theory based\u2019 and \u2018realist\u2019 approaches to IE. (Stern, 2015, p. 17)<\/blockquote>\r\n<\/div>\r\nTable 8.2 represents different available options. As you can see, the conceptualization of impact evaluation, as proposed by Stern (2015), encompasses experimental research, quasi-experimental research, natural experiments, and contribution analysis.\r\n\r\nTable 8.3 elaborates on the choice of the evaluation design for impact evaluations based on the key evaluation questions and the contextual research conditions.\r\n\r\nTable <\/strong>8<\/strong>.<\/strong>2<\/strong> Design Approaches, Variants and Causal Inference<\/em>\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n
Design approaches<\/strong><\/td>\r\nSpecific variants<\/strong><\/td>\r\nBasis for causal inference<\/strong><\/td>\r\n<\/tr>\r\n
Experimental<\/td>\r\nRCTs\r\n\r\nQuasi experiments\r\n\r\nNatural experiments<\/td>\r\nCounterfactuals<\/strong>: The difference between two otherwise identical cases \u2013 the manipulated and the controlled; the co-presence of cause and effects.<\/td>\r\n<\/tr>\r\n
Statistical<\/td>\r\nStatistical modelling\r\n\r\nLongitudinal studies\r\n\r\nEconometrics<\/td>\r\nRegularity<\/strong>: Correlation between cause and effect or between variables, influence of (usually) isalatable multiple cuases on a single effect. Control of \u2018confounders\u2019.<\/td>\r\n<\/tr>\r\n
Theory-based<\/td>\r\nCausal process designs<\/strong>: Theory of change, process tracing, contribution analysis, impact pathways,\r\n\r\nCausal mechanism designs: Realist evaluations, congruence analysis<\/td>\r\nGenerative causation<\/strong>: Identification and confirmation of causal processes or \u2018chains\u2019.\r\n\r\nSupporting factors and mechanisms at work in context.<\/td>\r\n<\/tr>\r\n
Case-based<\/td>\r\nInterpretative<\/strong>: Naturalistic, grounded theory, ethnography\r\n\r\nStructured: Configurations, QCA, within-case-analysis, simulations and network analysis<\/td>\r\nMultiple causation<\/strong>: Comparison across and within cases of combinations of causal factors.\r\n\r\nAnalytic generalization based on theory.<\/td>\r\n<\/tr>\r\n
Participatory<\/td>\r\nNormative designs<\/strong>: Participatory or democratic evaluation, empowerment evaluation.\r\n\r\nAgency designs:<\/strong> learning by doing, policy dialogue, collaborative action research.<\/td>\r\nActor Agency:<\/strong> Validation by participants that their actions and experienced effects are \u2018caused\u2019 by programme.\r\n\r\nAdoption, customization and commitment to a goal<\/td>\r\n<\/tr>\r\n
Synthesis studies<\/td>\r\nMeta-analysis, narrative synthesis, realist-based synthesis<\/td>\r\nAccumulation<\/strong> and aggregation within a number of perspectives (statistical, theory-based, ethnographic).<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\nSource: Stern, E. (2015). Impact Evaluation. A Guide for Commisioners and Managers<\/em>, p.18: https:\/\/www.bond.org.uk\/wp-content\/uploads\/2022\/08\/impact_evaluation_guide_0515.pdf<\/a><\/sub>\r\n\r\nTable <\/strong>8<\/strong>.<\/strong>3<\/strong> Summarizing the Design Implications of Different Impact Evaluation Questions<\/em>\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n
Key evaluation questions<\/strong><\/td>\r\nRelated evaluation questions<\/strong><\/td>\r\nUnderlying assumption<\/strong><\/td>\r\nRequirements<\/strong><\/td>\r\nSuitable designs<\/strong><\/td>\r\n<\/tr>\r\n
To what extent can a specific (net) impact be attributed to the intervention?<\/td>\r\nWhat is the net effect of the intervention?\r\n\r\nHow much of the impact can be attributed to the intervention?\r\n\r\nWhat would have happened without the intervention?<\/td>\r\nExpected outcomes and the intervention itself clearly understood and specifiable\r\n\r\nLikelihood of primary cause and primary effect\r\n\r\nInterest in particular intervention rather than generalisation<\/td>\r\nCan manipulate interventions\r\n\r\nSufficient numbers (beneficiaries, households, etc.) for statistical analysis<\/td>\r\nExperiments\r\n\r\nStatistical studies\r\n\r\nHybrids with case-based and participatory designs<\/td>\r\n<\/tr>\r\n
Has the intervention made a difference?<\/td>\r\nWhat causes are necessary or sufficient for the effect?\r\n\r\nWas the intervention needed to produce the effect?\r\n\r\nWould these impacts have happened anyhow?<\/td>\r\nThere are several relevant causes that need to be disentangled\r\n\r\nInterventions are just one part of a causal package\r\n\r\n <\/td>\r\nComparable cases where a common set of causes are present and evidence exists as to their potency\r\n\r\n <\/td>\r\nExperiments\r\n\r\nTheory-based evaluation, eg contribution analysis\r\n\r\nCase-based designs, eg QCA\r\n\r\n <\/td>\r\n<\/tr>\r\n
How has the intervention made a difference?<\/td>\r\nHow and why have the impacts come about?\r\n\r\nWhat causal factors have resulted in the observed impacts?\r\n\r\nHas the intervention resulted in any unintended impacts?\r\n\r\nFor whom has the intervention made a difference?<\/td>\r\nInterventions interact with other causal factors\r\n\r\nIt is possible to clearly represent the causal process through which the intervention made a difference \u2013 may require \u2018theory development\u2019\r\n\r\n <\/td>\r\nUnderstanding how supporting and contextual factors that connect intervention with effects\r\n\r\nTheory that allows for the identification of supporting factors (proximate, contextual and historical)\r\n\r\n <\/td>\r\nTheory-based evaluation especially \u2018realist\u2019 variants and\r\n\r\nContribution Analysis\r\n\r\nParticipatory approaches\r\n\r\n <\/td>\r\n<\/tr>\r\n
Can this be expected to work elsewhere?<\/td>\r\nCan this \u2018pilot\u2019 be transferred elsewhere and scaled up?\r\n\r\nIs the intervention sustainable?\r\n\r\nWhat generalizable lessons have we learned about impact?<\/td>\r\nWhat has worked in one place can work somewhere else\r\n\r\nStakeholders will cooperate in joint donor\/beneficiary evaluations\r\n\r\n <\/td>\r\nGeneric understanding of contexts eg typologies of context\r\n\r\nClusters of causal packages\r\n\r\nInnovation diffusion mechanisms\r\n\r\n <\/td>\r\nParticipatory approaches and some Experimental and Theory-based approaches\r\n\r\nNatural experiments\r\n\r\nRealist evaluation\r\n\r\nSynthesis studies<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\nSource: Stern, E. (2015). Impact Evaluation. A Guide for Commisioners and Managers<\/em>, p.22: <\/sub>https:\/\/www.bond.org.uk\/wp-content\/uploads\/2022\/08\/impact_evaluation_guide_0515.pdf<\/sub><\/a>\r\n\r\nImpact evaluations and contribution analysis have expanded the methodological landscape for analyzing the effects and impacts of interventions. This diversity of approaches enables analysis in contexts where experiments are not always feasible. Each approach will have its advantages, strengths, and potential validity threats. This overview touches briefly on a significant domain of evaluation; further reading is strongly recommended for those intending to lead evaluations on this critical topic.","rendered":"

Impact evaluation, as presented by Stern and colleagues, is an umbrella term used to present a diversity of designs focusing on identifying effects and impacts of interventions. According to these authors, impact evaluation aims at measuring effects but also answering the \u201chow\u201d and \u201cwhy\u201d questions, shifting the focus of the evaluation from causality to explanation (Stern, 2015; Stern et al., 2012).<\/p>\n

Stern opposes the narrowly defined focus of traditional effect analysis to the consideration of effects in impact evaluation.<\/p>\n

Experimental methods are concerned with intended<\/strong> rather than unintended effects; assume direct<\/strong> links between interventions and outcomes; address primary<\/strong> rather than secondary effects; and usually look to evidence in the short-term<\/strong> rather than the long-term. This latter is especially important as in many development settings effects are not known when programme funding ends, only becoming clear over a much more extended timescale. Most counterfactual methods on the other hand focus on the short-term, which is likely to capture only a sub-set of programme results. (Stern, 2015, p. 5)<\/p><\/blockquote>\n

Impact evaluation aims to answer the four following questions:<\/p>\n

\n

1) To what extent can a specific (net) impact be attributed to the intervention?<\/p>\n

2) Did the intervention make a difference?<\/p>\n

3) How has the intervention made a difference?<\/p>\n

4) Will the intervention work elsewhere?\u201d (Stern et al., 2012, p. 37)<\/p>\n<\/blockquote>\n

A diversity of designs can be considered for conducting impact evaluations. Stern identifies four main approaches to impact evaluations (IE):<\/p>\n

\n

Regularity frameworks<\/strong> that depend on the frequency of association between cause and effect \u2013 the inference basis for statistical approaches to IE.<\/p>\n

Counterfactual frameworks<\/strong> that depend on the difference between two otherwise identical cases \u2013 the inference basis for experimental and quasi-experimental approaches to IE.<\/p>\n

Multiple causation<\/strong> that depends on combinations of causes that lead to an effect \u2013 the inference basis for \u2018configurational\u2019 approaches to IE including qualitative comparative analysis (QCA) and contribution analysis.<\/p>\n

Generative causation<\/strong> that depends on identifying the \u2018mechanisms\u2019 that explain effects \u2013 the inference basis for \u2018theory based\u2019 and \u2018realist\u2019 approaches to IE. (Stern, 2015, p. 17)<\/p><\/blockquote>\n<\/div>\n

Table 8.2 represents different available options. As you can see, the conceptualization of impact evaluation, as proposed by Stern (2015), encompasses experimental research, quasi-experimental research, natural experiments, and contribution analysis.<\/p>\n

Table 8.3 elaborates on the choice of the evaluation design for impact evaluations based on the key evaluation questions and the contextual research conditions.<\/p>\n

Table <\/strong>8<\/strong>.<\/strong>2<\/strong> Design Approaches, Variants and Causal Inference<\/em><\/p>\n\n\n\n\n\n\n\n\n\n
Design approaches<\/strong><\/td>\nSpecific variants<\/strong><\/td>\nBasis for causal inference<\/strong><\/td>\n<\/tr>\n
Experimental<\/td>\nRCTs<\/p>\n

Quasi experiments<\/p>\n

Natural experiments<\/td>\n

Counterfactuals<\/strong>: The difference between two otherwise identical cases \u2013 the manipulated and the controlled; the co-presence of cause and effects.<\/td>\n<\/tr>\n
Statistical<\/td>\nStatistical modelling<\/p>\n

Longitudinal studies<\/p>\n

Econometrics<\/td>\n

Regularity<\/strong>: Correlation between cause and effect or between variables, influence of (usually) isalatable multiple cuases on a single effect. Control of \u2018confounders\u2019.<\/td>\n<\/tr>\n
Theory-based<\/td>\nCausal process designs<\/strong>: Theory of change, process tracing, contribution analysis, impact pathways,<\/p>\n

Causal mechanism designs: Realist evaluations, congruence analysis<\/td>\n

Generative causation<\/strong>: Identification and confirmation of causal processes or \u2018chains\u2019.<\/p>\n

Supporting factors and mechanisms at work in context.<\/td>\n<\/tr>\n

Case-based<\/td>\nInterpretative<\/strong>: Naturalistic, grounded theory, ethnography<\/p>\n

Structured: Configurations, QCA, within-case-analysis, simulations and network analysis<\/td>\n

Multiple causation<\/strong>: Comparison across and within cases of combinations of causal factors.<\/p>\n

Analytic generalization based on theory.<\/td>\n<\/tr>\n

Participatory<\/td>\nNormative designs<\/strong>: Participatory or democratic evaluation, empowerment evaluation.<\/p>\n

Agency designs:<\/strong> learning by doing, policy dialogue, collaborative action research.<\/td>\n

Actor Agency:<\/strong> Validation by participants that their actions and experienced effects are \u2018caused\u2019 by programme.<\/p>\n

Adoption, customization and commitment to a goal<\/td>\n<\/tr>\n

Synthesis studies<\/td>\nMeta-analysis, narrative synthesis, realist-based synthesis<\/td>\nAccumulation<\/strong> and aggregation within a number of perspectives (statistical, theory-based, ethnographic).<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Source: Stern, E. (2015). Impact Evaluation. A Guide for Commisioners and Managers<\/em>, p.18: https:\/\/www.bond.org.uk\/wp-content\/uploads\/2022\/08\/impact_evaluation_guide_0515.pdf<\/a><\/sub><\/p>\n

Table <\/strong>8<\/strong>.<\/strong>3<\/strong> Summarizing the Design Implications of Different Impact Evaluation Questions<\/em><\/p>\n\n\n\n\n\n\n\n
Key evaluation questions<\/strong><\/td>\nRelated evaluation questions<\/strong><\/td>\nUnderlying assumption<\/strong><\/td>\nRequirements<\/strong><\/td>\nSuitable designs<\/strong><\/td>\n<\/tr>\n
To what extent can a specific (net) impact be attributed to the intervention?<\/td>\nWhat is the net effect of the intervention?<\/p>\n

How much of the impact can be attributed to the intervention?<\/p>\n

What would have happened without the intervention?<\/td>\n

Expected outcomes and the intervention itself clearly understood and specifiable<\/p>\n

Likelihood of primary cause and primary effect<\/p>\n

Interest in particular intervention rather than generalisation<\/td>\n

Can manipulate interventions<\/p>\n

Sufficient numbers (beneficiaries, households, etc.) for statistical analysis<\/td>\n

Experiments<\/p>\n

Statistical studies<\/p>\n

Hybrids with case-based and participatory designs<\/td>\n<\/tr>\n

Has the intervention made a difference?<\/td>\nWhat causes are necessary or sufficient for the effect?<\/p>\n

Was the intervention needed to produce the effect?<\/p>\n

Would these impacts have happened anyhow?<\/td>\n

There are several relevant causes that need to be disentangled<\/p>\n

Interventions are just one part of a causal package<\/p>\n

 <\/td>\n

Comparable cases where a common set of causes are present and evidence exists as to their potency<\/p>\n

 <\/td>\n

Experiments<\/p>\n

Theory-based evaluation, eg contribution analysis<\/p>\n

Case-based designs, eg QCA<\/p>\n

 <\/td>\n<\/tr>\n

How has the intervention made a difference?<\/td>\nHow and why have the impacts come about?<\/p>\n

What causal factors have resulted in the observed impacts?<\/p>\n

Has the intervention resulted in any unintended impacts?<\/p>\n

For whom has the intervention made a difference?<\/td>\n

Interventions interact with other causal factors<\/p>\n

It is possible to clearly represent the causal process through which the intervention made a difference \u2013 may require \u2018theory development\u2019<\/p>\n

 <\/td>\n

Understanding how supporting and contextual factors that connect intervention with effects<\/p>\n

Theory that allows for the identification of supporting factors (proximate, contextual and historical)<\/p>\n

 <\/td>\n

Theory-based evaluation especially \u2018realist\u2019 variants and<\/p>\n

Contribution Analysis<\/p>\n

Participatory approaches<\/p>\n

 <\/td>\n<\/tr>\n

Can this be expected to work elsewhere?<\/td>\nCan this \u2018pilot\u2019 be transferred elsewhere and scaled up?<\/p>\n

Is the intervention sustainable?<\/p>\n

What generalizable lessons have we learned about impact?<\/td>\n

What has worked in one place can work somewhere else<\/p>\n

Stakeholders will cooperate in joint donor\/beneficiary evaluations<\/p>\n

 <\/td>\n

Generic understanding of contexts eg typologies of context<\/p>\n

Clusters of causal packages<\/p>\n

Innovation diffusion mechanisms<\/p>\n

 <\/td>\n

Participatory approaches and some Experimental and Theory-based approaches<\/p>\n

Natural experiments<\/p>\n

Realist evaluation<\/p>\n

Synthesis studies<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Source: Stern, E. (2015). Impact Evaluation. A Guide for Commisioners and Managers<\/em>, p.22: <\/sub>https:\/\/www.bond.org.uk\/wp-content\/uploads\/2022\/08\/impact_evaluation_guide_0515.pdf<\/sub><\/a><\/p>\n

Impact evaluations and contribution analysis have expanded the methodological landscape for analyzing the effects and impacts of interventions. This diversity of approaches enables analysis in contexts where experiments are not always feasible. Each approach will have its advantages, strengths, and potential validity threats. This overview touches briefly on a significant domain of evaluation; further reading is strongly recommended for those intending to lead evaluations on this critical topic.<\/p>\n","protected":false},"author":20,"menu_order":6,"template":"","meta":{"pb_show_title":"on","pb_short_title":"","pb_subtitle":"","pb_authors":[],"pb_section_license":""},"chapter-type":[49],"contributor":[],"license":[],"class_list":["post-242","chapter","type-chapter","status-publish","hentry","chapter-type-numberless"],"part":228,"_links":{"self":[{"href":"https:\/\/pressbooks.bccampus.ca\/foundationsofevaluation\/wp-json\/pressbooks\/v2\/chapters\/242","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/pressbooks.bccampus.ca\/foundationsofevaluation\/wp-json\/pressbooks\/v2\/chapters"}],"about":[{"href":"https:\/\/pressbooks.bccampus.ca\/foundationsofevaluation\/wp-json\/wp\/v2\/types\/chapter"}],"author":[{"embeddable":true,"href":"https:\/\/pressbooks.bccampus.ca\/foundationsofevaluation\/wp-json\/wp\/v2\/users\/20"}],"version-history":[{"count":5,"href":"https:\/\/pressbooks.bccampus.ca\/foundationsofevaluation\/wp-json\/pressbooks\/v2\/chapters\/242\/revisions"}],"predecessor-version":[{"id":647,"href":"https:\/\/pressbooks.bccampus.ca\/foundationsofevaluation\/wp-json\/pressbooks\/v2\/chapters\/242\/revisions\/647"}],"part":[{"href":"https:\/\/pressbooks.bccampus.ca\/foundationsofevaluation\/wp-json\/pressbooks\/v2\/parts\/228"}],"metadata":[{"href":"https:\/\/pressbooks.bccampus.ca\/foundationsofevaluation\/wp-json\/pressbooks\/v2\/chapters\/242\/metadata\/"}],"wp:attachment":[{"href":"https:\/\/pressbooks.bccampus.ca\/foundationsofevaluation\/wp-json\/wp\/v2\/media?parent=242"}],"wp:term":[{"taxonomy":"chapter-type","embeddable":true,"href":"https:\/\/pressbooks.bccampus.ca\/foundationsofevaluation\/wp-json\/pressbooks\/v2\/chapter-type?post=242"},{"taxonomy":"contributor","embeddable":true,"href":"https:\/\/pressbooks.bccampus.ca\/foundationsofevaluation\/wp-json\/wp\/v2\/contributor?post=242"},{"taxonomy":"license","embeddable":true,"href":"https:\/\/pressbooks.bccampus.ca\/foundationsofevaluation\/wp-json\/wp\/v2\/license?post=242"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}