Monitoring
12.4 Establishing a monitoring system
Once the focus is determined, the next step consists of identifying indicators, targets, and sources of data. Table 12.2 provides an illustration of the different focal points and the corresponding criteria, indicators, targets, data sources, and collection processes.
Choosing targets to monitor is important. According to Champagne et al. (2011d), targets can be identified from different sources such as best practice guidelines, institutional norms, actual practices in similar organizations, and experts’ views. The source will be identified according to the monitoring system’s purpose. The evaluator may choose to select representative targets, such as the mean, or opt for elitist ones (Champagne et al., 2011d). For example, an education program aimed at excellence may choose not to compare its programs to the domain’s average achievement but instead compare itself to the best ones or specific targets of excellence in sub-areas. Targets can be ambitious but should be achievable (Markiewicz & Patrick, 2022/2016).
Monitoring can adopt a longitudinal perspective and track progress over time, using the initial data collected as a baseline for monitoring evolution over time. Monitoring can also be used to compare performance across programs or similar organizations or units (McDavid et al., 2019).
When designing a monitoring system, the evaluator should be specific about the areas that will be monitored and ensure measurable indicators are selected, data is accessible, and the organization can collect and analyze it (Markiewicz & Patrick, 2022/2016).
Table 12.2 Developing the Monitoring Plan
| Focus | Criteria | Sample Questions | Indicator | Target | Data Source | Whose responsibility and timing |
| Structure | Fidelity | Is the intervention implemented as planned? | Investments
Recruitment of professional Qualifications |
As set in the plan | ||
| Process | Fidelity | Are all activities taking place?
Are the costs respected? |
Amount and number of activities delivered
Operation costs |
As set in the plan | ||
| Reach | Is the target population participating as expected? | Target population versus participants | As set in the plan | |||
| Quality | Are the activities offered according to best practices? | Wait times
Survival rates |
According to best practices | |||
| Are clients satisfied (interpersonal dimensions of quality)? | Satisfaction rates
Dissatisfaction |
Agreed on levels
Nb of complaints |
||||
| Results | Goal achievements | Determined according to the logic model, for example, are students succeeding? | Graduate rates
Post-graduate employment Studies duration |
Best results in similar programs | ||
| Environmental impacts: pollution, biodiversity, land and waters | Is the intervention meeting its environmental commitments? | Greenhouse gas emissions
Use of plastic Greening of parking lots |
Established targets | |||
| Impacts on human systems | Is the intervention contributing to better health, greater equity and more prosperity? | Mental health indicators
Rates of burn out Diversity of beneficiaries Proportion of spending in local business |
According to best practices or agreed pre-set targets |
Source: Adapted from Markiewicz, A., & Patrick, I. (2022). Developing Monitoring and Evaluation Frameworks. SAGE Publications, and from Champagne, F., Hartz, Z., Brousselle, A., & Contandriopoulos, A. P. (2011d). L’appréciation normative. In A. Brousselle, F. Champagne, A. P. Contandriopoulos, & Z. Hartz (Eds.), L’évaluation: concepts et méthodes. Deuxième édition mise à jour. Les Presses de l’Université de Montréal.
