4.6 Evaluative questions

Evaluation is at the intersection of research and policy making/decision-making. A distinction is generally made between evaluative research and monitoring (Brousselle et al., 2011b). Evaluative research analyzes causal relations, specifically the links between the components of the intervention, including the results. Monitoring compares components and aspects of the intervention to some objectives or targets and to record progress. Monitoring provides information on the resources dedicated to the intervention and its structure, the activities (also called process), and the results.

Several questions exist in evaluation. Although authors present and regroup evaluation questions in different ways, most models include the same core questions. The typology and definitions used in this book are inspired and adapted from what was suggested in L’Évaluation: Concepts et Méthodes (Brousselle et al., 2011b). Planetary health considerations have been introduced in each one of them (see Figure 4.1 and the following chapters).

Figure 4.1 Evaluation Questions

Evaluative research questions

Evaluative research includes questioning in five categories (see Figure 2):

1.  Relevance: needs assessment
   • What is needed to meet the needs of the community (ex-ante)? Is the intervention fitting with the needs and commitments of the community (ex-post)?
   • Are the objectives of the intervention pertinent particularly when considering the socio-ecological context? Are the right causes targeted? Is the right population targeted?
   • Are the right partners involved, or should other partners be involved?
2.  Coherence: logic analysis
   • Is the program intervention plausible, given what is known from theory and practice to achieve the intended effects (direct logic analysis)
   • What are plausible interventions or alternatives to achieve the intended effects?
3. Effectiveness: effect analysis, contribution analysis and impact evaluation
   • What are the results of the intervention? Did the intervention make a difference and for whom?
   • Are the observed effects attributable to the intervention?
   • What are the impacts of the intervention on planetary health?
4. Efficiency: economic evaluation
   • Is the intervention efficient?
5. Implementation: implementation analysis

• • How does the context influence the implementation of the intervention and the production of the results?

Figure 4.2 Evaluative Research Questions

Monitoring questions

Monitoring is used to assess the fidelity of the intervention in comparison to the original plans (which includes the costs), the reach, the quality of the intervention, whether the goals were achieved, and the impacts on planetary health. Table 4.1 summarizes the different options for monitoring the intervention.

_{Table 4.1: Possible Focus for Monitoring}

_{Source: Adapted from Champagne, F., Hartz, Z., Brousselle, A., & Contandriopoulos, A. P. (2011d). L’appréciation normative. In A. Brousselle, F. Champagne, A. P. Contandriopoulos, & Z. Hartz (Eds.), L’évaluation: concepts et méthodes. Deuxième édition mise à jour. Les Presses de l’Université de Montréal: 98.}

Note that monitoring questions focus on the intervention’s components, whereas evaluative research questions examine the relationships or links between those components (see Figures 4.2 and 4.3).

Figure 4.3 Focus of monitoring

Be aware that assessing the effects in a monitoring exercise is different from an effect analysis. An effect analysis will analyze the causal relationship between the intervention and the observed effects. In monitoring, no causal relationship is assessed; effects are monitored but could potentially be due to another element than the intervention, such as another competing intervention or a change in the characteristics of the context.

Understanding the distinction between evaluative research and monitoring is important. However, in real life evaluators will often mix the use of the two approaches as they are different but complementary ways of analyzing an intervention. For example, in implementation analysis, the first stage may include an assessment of the program’s implementation (fidelity and reach) before explaining the implementation with a process analysis (Patton, 1997; Scheirer, 1994). The distinction between evaluative research and monitoring is mostly a heuristic approach to learning evaluation.