Effect analysis and related approaches
8.8 Outcome Harvesting
Outcome harvesting is an approach designed to identify effects in changing contexts. This approach is useful to understand social change processes (Wilson-Grau, 2019). It doesn’t try to assess causality. It can become more relevant as we experience quickly evolving and unpredictable contexts due to environmental change, post-truth, and disinformation.
Outcome Harvesting is not intended for projects or project components that are well-supported by evidence, or that closely follow a theory of change. Outcome Harvesting is appropriate for identifying and verifying outcomes, including unintended outcomes. It is a useful, commonsense approach that easily engages informants and is designed to generate concrete evidence to inform decisions about future actions. (Wilson-Grau et al., 2024)
It can be used when an organization is experiencing constant change and “unexpected and unforeseeable actors and factors in their programming environment” (Wilson-Grau, 2019, p. 1).
For example, these are observable changes in societal actors targeted by policy advocacy interventions: a government minister publicly declares she will restrict untendered contracts to under 5% (an action); a civil society organization launches a campaign for governmental transparency (an activity); two political parties join forces to collaborate rather than compete in proposing transparency legislation (relationship); a senior government official for the first time acknowledges the need for off-grid, sustainable energy production in rural areas (agenda); a legislature passes a new anti-corruption law (policy); or a government implements norms and procedures for publishing all procurement records (practice). Thus, the definition of “outcome” in Outcome Harvesting contrasts with the definition of an outcome as observed changes in the intended program beneficiaries’ wellbeing (learning, health, employment). Those changes would be the impact of these policy changes. (Wilson-Grau, 2019, p. 2).
Outcome harvesting focuses on the identification of outcomes defined as “a change in the behavior, relationships, actions, activities, policies, or practices of an individual, group, community, organization, or institution” (Wilson-Grau, 2019, p. 2). This approach is not meant to identify impacts.
The bottom line of social change is societal actors changing the way they do things. It is only when individuals, groups, communities, organizations, and institutions change their actions, activities, relationships, policies, and practices that a society changes, for good or bad. The essence of an outcome in Outcome Harvesting is those societal actors demonstrably changing their behaviors in those ways. (Wilson-Grau, 2019, p. 170).
Outcome Harvesting collects (“harvests”) evidence of what has changed (“outcomes”) and, working backwards, determines whether and how an intervention has contributed to these changes. (Wilson-Grau et al., 2024).
This approach includes 6 steps (see Exhibit 8.2).
Exhibit 8.2: 6-step approach to Outcome Harvesting
- Design the Outcome Harvest based on the principal uses of the primary users. Come to agreement with the people who will use the results of the Outcome Harvest on priority Outcome Harvesting questions to guide the harvest. Users and harvesters should also agree on the process: what information is to be collected, how, from whom, when, and with what resources in order to credibly answer the questions.
- Review documentation by identifying and formulating draft, potential outcome statements contained in secondary sources of information: reports, evaluations, press releases, and other documentation. These statements should comprise (a) changes in individuals, groups, communities, organizations, or institutions; and (b) how the intervention plausibly influenced them. They may contain other useful information such as the significance of each outcome.
- Engage with human sources. The harvester facilitates conversations with the people who have the most knowledge about what the intervention has achieved and how. They usually are the authors of the documentation, the intervention’s other field staff, allies, and others closest to the action. They review and fill gaps in the potential outcome statements extracted from documentation, identify and formulate additional ones, and together with the harvester agree on a set of robust outcome statements that are sufficiently precise to be verifiable.
- Substantiate with external sources a select number of outcome statements. Substantiators are one or more persons knowledgeable about the change but independent from the organization to ensure accuracy, or deepen understanding, or both. For example, they may be the societal actors who changed their behavior or allies who collaborated in the intervention, unless of course they already served as a primary source in the third step. This fourth step ensures that the whole set of outcome statements is sufficiently credible for the intended uses. These outcome statements are the evidence used in the next step.
- Analyze and interpret by first organizing outcome statements so they are manageable, for example, categorising them by evaluation question, and then using this to provide evidence-based answers to the prime evaluation questions, where the evidence is the information contained in the outcome statements generated in the previous three steps.
- Support use of findings after the evaluation questions are answered so the users make better use of the process and findings.
Source: Wilson-Grau, R. (2019). Outcome Harvesting: Principles, Steps, and Evaluation Applications. Information Age Publishing: 8-9.
