{"id":29,"date":"2019-11-05T16:17:03","date_gmt":"2019-11-05T21:17:03","guid":{"rendered":"https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/?post_type=chapter&#038;p=29"},"modified":"2021-06-18T18:31:08","modified_gmt":"2021-06-18T22:31:08","slug":"1-2-goal-setting-and-evaluation","status":"publish","type":"chapter","link":"https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/chapter\/1-2-goal-setting-and-evaluation\/","title":{"raw":"1.2 Goal Setting and Evaluation","rendered":"1.2 Goal Setting and Evaluation"},"content":{"raw":"<div class=\"textbox textbox--learning-objectives\"><header class=\"textbox__header\">\r\n<p class=\"textbox__title\">Learning Objectives<\/p>\r\n\r\n<\/header>\r\n<div class=\"textbox__content\">\r\n<ul>\r\n \t<li>To describe the importance of having process and outcome goals and the differences between them.<\/li>\r\n \t<li>To develop process and outcome goals in conjunction with evaluative criteria for those goals.<\/li>\r\n \t<li>To provide examples of ways in which process and outcome goals could be evaluated.<\/li>\r\n<\/ul>\r\n<img class=\"wp-image-583 aligncenter\" src=\"https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Standing-rat-e1575498393869-989x1024.png\" alt=\"\" width=\"150\" height=\"155\" \/>\r\n\r\n<\/div>\r\n<\/div>\r\n<div class=\"textbox textbox--sidebar textbox--exercises\"><header class=\"textbox__header\">\r\n<p class=\"textbox__title\">Discussion<\/p>\r\n\r\n<\/header>\r\n<div class=\"textbox__content\"><strong>Purpose vs. Goals<\/strong>: You can think of your purpose as overarching reason that you are communicating or the type of change you are hoping to accomplish. Goals are more specific, they outline what your destination looks like in practical terms or how you define success.<\/div>\r\n<div class=\"textbox__content\"><strong>Goals vs. Evaluative Criteria<\/strong>: If goals are how you define success, the evaluative criteria are objective measures that will help you to determine whether or not you have achieved your goals.\u00a0 A goal without a measure of success is not a goal worth having.<\/div>\r\n<div><img class=\"wp-image-581 alignright\" src=\"https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Pondering-rat-e1575581547259-300x281.png\" alt=\"\" width=\"175\" height=\"163\" \/><\/div>\r\n<\/div>\r\nNow that you have identified the general purpose for your communication campaign, it is time to set specific goals. A tried and true tool for goal setting is the <a href=\"https:\/\/www.ucop.edu\/local-human-resources\/_files\/performance-appraisal\/How%20to%20write%20SMART%20Goals%20v2.pdf\">SMART<\/a> acronym. Specifically, good goals are:\r\n<ul>\r\n \t<li><strong>S<\/strong>pecific (What will you do? What will you accomplish?)<\/li>\r\n \t<li><strong>M<\/strong>easurable (How will you know you have achieved your goals?)<\/li>\r\n \t<li><strong>A<\/strong>ttainable (Are you reasonably certain of success?)<\/li>\r\n \t<li><strong>R<\/strong>elevant (Why is the goal even important?)<\/li>\r\n \t<li><strong>T<\/strong>ime bound (What is the time frame for completion?)<\/li>\r\n<\/ul>\r\nBut how can you know for sure your goals are SMART? The best way to go about goal setting is actually to start with the end \u2014 the evaluation. What?!? That seems a little backward. Are we really asking you to think about how you will evaluate your communication program before you have even designed it? The answer is\u2026yes! This is because goals and evaluation criteria are best developed in conjunction with one another. Having a good idea in mind about how your program could and should be evaluated actually helps you to design a better program \u2014 just like a grading rubric helps to focus you and keep you on track for a class assignment!\r\n<h1>Developing goals and evaluative criteria<\/h1>\r\n<h2><span style=\"font-family: 'Times New Roman', Georgia, 'SBL Greek', serif;font-size: 18.6667px\">You should have goals and evaluative criteria related to both the process and the outcome of your communication campaign.\u00a0<\/span><\/h2>\r\n<h2>The process<\/h2>\r\n<span class=\"pullquote-right\">Ultimately, the success of your communication plan can be measured by answering one simple question: Did you achieve your goals? This illustrates why it is important to spend time thinking about and verbalizing your goals and evaluation from the outset.<\/span>Process goals and evaluations are focused on the mechanics of the plan and its implementation. For example, you might want to set goals and evaluative criteria around whether your message reached the target audience, the consistency and timeliness of messaging across different platforms, how information flowed within the communications team, etc. These areas of focus will be highly dependent on the scenario but could include components that you view as particularly critical to the success of the program and\/or components where you anticipate having problems. As much as possible your process goals\/evaluation criteria should be mapped out in advance, but you should also make sure you revisit them throughout the communication process, because chances are that you are going to encounter barriers (or opportunities!) that you did not anticipate, and you may even have to adjust or change your goals accordingly. One component of process evaluation that is often ignored is efficiency. Did you manage to implement your program and achieve your goal with the least amount of wasted time, energy, money, etc.?\r\n<h2>The outcome<\/h2>\r\n<div class=\"textbox textbox--sidebar textbox--exercises\"><header class=\"textbox__header\">\r\n<p class=\"textbox__title\">Discussion<\/p>\r\n\r\n<\/header>\r\n<div class=\"textbox__content\">\r\n\r\n<strong>Think About Baselines!<\/strong>\r\n\r\nMost outcomes involve some sort of a change \u2014 change in awareness, change in knowledge, change in behavior, etc. However, you can\u2019t detect or evaluate that change if you don\u2019t know where your audience was at in the first place (i.e., your baseline). For this reason, it may be prudent to define and measure your baseline before your start the communication process.\r\n\r\n<\/div>\r\n<img class=\"wp-image-581 aligncenter\" src=\"https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Pondering-rat-e1575581547259-300x281.png\" alt=\"\" width=\"187\" height=\"175\" \/>\r\n\r\n<\/div>\r\nOutcome goals and evaluations are focused on the impact of your plan on its target audience. If your goal was to inform, did people understand more about the issue in question after versus before your communication effort? If your goal was to change behavior, did the behavior of your target audience actually change? <span class=\"pullquote-left\">'Unmonitored for outcome, risk communication consumes and wastes valuable resources, are ineffective, and create a false sense of achievement on those who are responsible.' \u2013 Gaya Gamhewage, World Health Organization<\/span> That being said, these are lofty goals and can be very difficult to measure. That does not mean that you should not strive for them, but it might be wise to come up with some creative but still impactful outcome aims. For example, did your communication effort increase your audience\u2019s access to information about the issue in question? Did people feel empowered as a result of your communication? If you are involving stakeholders in the communication process, an aim could be whether those stakeholders felt that they were engaged in a positive and meaningful way.\r\n\r\n<span style=\"font-size: 14pt\">Once you have decided on your goals and evaluative criteria, you will need to outline how you will conduct the evaluation. Again, this is important to do at the beginning because it will help you to assess the feasibility of your goals and evaluative criteria.<\/span>\r\n<h1>Communication program evaluation<\/h1>\r\n<h2>Timing<\/h2>\r\nThere are three key times when an evaluation can take place:\r\n<h3 style=\"padding-left: 30px\">Before<\/h3>\r\n<p style=\"padding-left: 30px\">This is called pre-testing. This involves implementing all or part of your communication plan on representatives from your intended audience (ideally). For example, perhaps you might have a focus group with stakeholders to review your risk communication materials and make sure that everyone understands the message and finds it compelling.<\/p>\r\n\r\n<h3 style=\"padding-left: 30px\">During<\/h3>\r\n<p style=\"padding-left: 30px\">It can be helpful to have a mid-point review. You often can\u2019t know what works and what does not until you have started to implement. Additionally, there may be unforeseen barriers and opportunities. This is usually more of a process-focused check-up than a full evaluation \u2014 the purpose is to make sure that everything is on track. For programs that take a long time to bear fruit, you may need several mid-point check-ins.<\/p>\r\n\r\n<h3 style=\"padding-left: 30px\">After<\/h3>\r\n<p style=\"padding-left: 30px\">Once there has been time for your communication effort to bear fruit, it is time for a more formal and in-depth evaluation. This is often focused on outcomes, but, don\u2019t forget the process component because the two go hand in hand. Without a process review, it will be hard to identify the root causes of your success or failure.<\/p>\r\n\r\n<div class=\"textbox textbox--exercises\"><header class=\"textbox__header\">\r\n<p class=\"textbox__title\">Supplementary Material<\/p>\r\n\r\n<\/header>\r\n<div class=\"textbox__content\">\r\n<h1>The importance of audience and stakeholder pre-testing<\/h1>\r\n<em>by Taneille Johnson (SPPH 552 2020W1)<\/em>\r\n\r\nFor today's blog post, I'm looking at a campaign run by the BC Government on overdose prevention and de-stigmatization. The reception of this campaign by an important advocacy group is an example of the importance of identifying stakeholders and pre-testing communication messages.\r\n\r\n<img class=\"size-medium wp-image-152 alignright\" src=\"http:\/\/blogs.ubc.ca\/coursec26705f4ac47f2ed7197be5c698caee895b81f1e\/files\/2020\/09\/38205949_1066886990131334_8598558525553115136_o-300x166.png\" alt=\"\" width=\"300\" height=\"166\" \/>\r\n\r\nThis campaign originally ran in 2018. To me, the target audience was likely upper-middle class individuals and aimed to raise awareness that anyone could be a drug user (i.e. drug users are NOT only lower income individuals, and those who use drugs are \"real people too.\"\r\n\r\nShortly after this campaign aired, there was outcry from the Canadian Association of People who Use Drugs. This group argued that the government's campaign shifted the focus onto individual drug users and left out many of the complex systems issues that arguably play a larger role in the opioid epidemic.\r\n\r\nThis advocacy group quickly released remixed posters onto social media (see below, taken from Facebook: https:\/\/www.facebook.com\/notes\/canadian-association-of-people-who-use-drugs\/capud-launches-new-anti-stigma-campaign-aimed-at-bc-provincial-government-\/1063640897122610\/)\r\n\r\n<img class=\"alignnone size-medium wp-image-153\" src=\"http:\/\/blogs.ubc.ca\/coursec26705f4ac47f2ed7197be5c698caee895b81f1e\/files\/2020\/09\/38218385_1066886536798046_9027589388640452608_o-300x169.jpg\" alt=\"\" width=\"300\" height=\"169\" \/> <img class=\"alignnone size-medium wp-image-154\" src=\"http:\/\/blogs.ubc.ca\/coursec26705f4ac47f2ed7197be5c698caee895b81f1e\/files\/2020\/09\/38065468_1066891203464246_3726360054606069760_o-300x169.jpg\" alt=\"\" width=\"300\" height=\"169\" \/>\r\n\r\nCould this reaction and outcry have been prevented by consulting this advocacy group ahead of time? Was audience and potential audience reaction considered? \u00a0Did the government consider partnering with advocacy groups already working on de-stigmatization?\u00a0The government responded by saying that they stood by their original ad campaign and that the goal was to \"bring humanism\" into the opioid crisis (see https:\/\/www.cbc.ca\/news\/canada\/british-columbia\/advocacy-group-takes-issue-with-b-c-government-s-ad-campaign-to-fight-opioid-crisis-1.4773781).\r\n\r\nFor me, this leads to a larger question of the duty of government to engage with stakeholders prior to communication campaigns. The sticky point is that it is impossible to expect the government (or any communicator) to anticipate every hurdle and consult with every actor.\r\n\r\nWhile the advocacy group may have disagreed with the overall message, perhaps there could have been a middle ground. Perhaps the government could have run the original ad as the first in a series and then followed up with a 2nd ad drawing attention to the systems issues. Regardless, I think this ad (and the ensuing outcry) serves as a reminder to identify your stakeholders or key advocacy groups and pre-test the message beforehand!\r\n\r\n<\/div>\r\n<\/div>\r\n<h2>Data sources<\/h2>\r\nA process evaluation relies largely on internal data sources, for example risk communication plans, messages, and communication products (i.e., media interviews, written and visual materials, internet posts, etc.), and interviews with those involved in the communication process.\r\n\r\nAn outcome evaluation, on the other hand, relies largely on external data sources such as surveys, interviews, and focus groups with segments of your target audience, usage tracking for websites and social media, and data to identify a change in behavior.\r\n<h2>Process<\/h2>\r\nThis is not rocket science! Program evaluations are often omitted from communication plans and we think it is because people make them more complicated than they need to be. For some reason you use the word \u2018evaluation\u2019 and the tendency is to think that is needs some complicated, energy-draining, and time-consuming process. In reality, if you need this sort of process, your goals and evaluative criteria are probably way too complex. The goal of your evaluation is not to completely, utterly, and undeniably prove that you have achieved your goal, but rather to provide reasonable evidence to support that you made steps in the right direction.\r\n<p style=\"padding-left: 30px\">Step 1: Identify the goals for the evaluation. These will be derived from your process\/outcome goals and evaluative criteria.<\/p>\r\n<p style=\"padding-left: 30px\">Step 2: Determine what data you will need to perform the evaluation.<\/p>\r\n<p style=\"padding-left: 30px\">Step 3: Collect those data.<\/p>\r\n<p style=\"padding-left: 30px\">Step 4: Analyze those data.<\/p>\r\n<p style=\"padding-left: 30px\">Step 5: Draw conclusions and act on those conclusions by modifying your plan.<\/p>\r\n&nbsp;\r\n<div class=\"textbox textbox--examples\"><header class=\"textbox__header\">\r\n<p class=\"textbox__title\">Example<\/p>\r\n\r\n<\/header>\r\n<div class=\"textbox__content\">\r\n\r\nWe developed a communication campaign for rural Sri Lankans in communities with a high incidence of dog and human rabies.\r\n\r\n[caption id=\"attachment_514\" align=\"alignright\" width=\"500\"]<a href=\"https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Sri-Lanka-July-2010.jpg\"><img class=\"wp-image-514\" src=\"https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Sri-Lanka-July-2010.jpg\" alt=\"\" width=\"500\" height=\"281\" \/><\/a> Feral dogs are the main source of rabies in Sri Lanka.[\/caption]\r\n\r\nThe purpose of this campaign was to <strong>educate<\/strong> adult community members about rabies. A <strong>process goal<\/strong> was that 75% of adults in the target communities had seen and read the poster. An <strong>outcome goal<\/strong> was that people who had read the poster demonstrated an increased understanding of the risks associated with rabies and how to mitigate those risks. The <strong>process evaluative criteria<\/strong> was self-reported rates of seeing reading the poster using one-on-one interviews conducted <em>after<\/em> the communication campaign. The <strong>outcome evaluative criteria<\/strong> was a demonstrated increase in understanding regarding rabies risk and risk mitigation between one-on-one interviews conducted <em>before and after<\/em> the communication campaign.\r\n\r\nThe <strong>evaluation process<\/strong> was as follows:\r\n\r\nStep 1. The goal of the evaluation was to estimate the proportion of the target audience who had seen and read the posters and to assess whether or not those who had read the posters demonstrated an increased understanding of rabies.\r\n\r\nStep 2. Data were collected through one-on-one interviews with a subset of adults identified by a community leader.\r\n\r\nStep 3. Participants were interviewed before and after the educational campaign.\r\n\r\nStep 4. Data were analyzed quantitatively and qualitatively.\r\n\r\nStep 5. Data were used to determine whether goals had been met and to identify whether important gaps remained.\r\n\r\n<\/div>\r\n<\/div>\r\n<h1>Takeaways<\/h1>\r\n<div class=\"textbox textbox--key-takeaways\"><header class=\"textbox__header\">\r\n<p class=\"textbox__title\">Key Takeaways<\/p>\r\n\r\n<\/header>\r\n<div class=\"textbox__content\">\r\n<ul>\r\n \t<li>Your goal is to clearly define what success will 'look like' in the context of your communication campaign. Goals should be specific, measurable, attainable, relevant, and time-bound (SMART).<\/li>\r\n \t<li>Goals should be developed in conjunction with evaluative criteria, which are objective measures that will help you determine if (and prove that) you have achieved your goals.<img class=\"wp-image-582 alignright\" style=\"font-size: 16.8px\" src=\"https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Seated-rat-e1575568791861-300x224.png\" alt=\"\" width=\"150\" height=\"112\" \/><\/li>\r\n \t<li>You should have goals and evaluative criteria related to both the process and outcome of your communication plan. Process goals and evaluations are focused on the mechanics of the\u00a0plan and its implementation. Outcome goals and evaluations are focused on the impact of your plan on its target audience.<\/li>\r\n \t<li>Once you decide on your goals and evaluative criteria you should outline how you will conduct the evaluation, including the timing, data sources, and process.<\/li>\r\n<\/ul>\r\n<\/div>\r\n<\/div>","rendered":"<div class=\"textbox textbox--learning-objectives\">\n<header class=\"textbox__header\">\n<p class=\"textbox__title\">Learning Objectives<\/p>\n<\/header>\n<div class=\"textbox__content\">\n<ul>\n<li>To describe the importance of having process and outcome goals and the differences between them.<\/li>\n<li>To develop process and outcome goals in conjunction with evaluative criteria for those goals.<\/li>\n<li>To provide examples of ways in which process and outcome goals could be evaluated.<\/li>\n<\/ul>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-583 aligncenter\" src=\"https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Standing-rat-e1575498393869-989x1024.png\" alt=\"\" width=\"150\" height=\"155\" srcset=\"https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Standing-rat-e1575498393869-989x1024.png 989w, https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Standing-rat-e1575498393869-290x300.png 290w, https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Standing-rat-e1575498393869-768x795.png 768w, https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Standing-rat-e1575498393869-65x67.png 65w, https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Standing-rat-e1575498393869-225x233.png 225w, https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Standing-rat-e1575498393869-350x362.png 350w, https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Standing-rat-e1575498393869.png 1401w\" sizes=\"auto, (max-width: 150px) 100vw, 150px\" \/><\/p>\n<\/div>\n<\/div>\n<div class=\"textbox textbox--sidebar textbox--exercises\">\n<header class=\"textbox__header\">\n<p class=\"textbox__title\">Discussion<\/p>\n<\/header>\n<div class=\"textbox__content\"><strong>Purpose vs. Goals<\/strong>: You can think of your purpose as overarching reason that you are communicating or the type of change you are hoping to accomplish. Goals are more specific, they outline what your destination looks like in practical terms or how you define success.<\/div>\n<div class=\"textbox__content\"><strong>Goals vs. Evaluative Criteria<\/strong>: If goals are how you define success, the evaluative criteria are objective measures that will help you to determine whether or not you have achieved your goals.\u00a0 A goal without a measure of success is not a goal worth having.<\/div>\n<div><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-581 alignright\" src=\"https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Pondering-rat-e1575581547259-300x281.png\" alt=\"\" width=\"175\" height=\"163\" srcset=\"https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Pondering-rat-e1575581547259-300x281.png 300w, https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Pondering-rat-e1575581547259-768x720.png 768w, https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Pondering-rat-e1575581547259-1024x960.png 1024w, https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Pondering-rat-e1575581547259-65x61.png 65w, https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Pondering-rat-e1575581547259-225x211.png 225w, https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Pondering-rat-e1575581547259-350x328.png 350w, https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Pondering-rat-e1575581547259.png 1606w\" sizes=\"auto, (max-width: 175px) 100vw, 175px\" \/><\/div>\n<\/div>\n<p>Now that you have identified the general purpose for your communication campaign, it is time to set specific goals. A tried and true tool for goal setting is the <a href=\"https:\/\/www.ucop.edu\/local-human-resources\/_files\/performance-appraisal\/How%20to%20write%20SMART%20Goals%20v2.pdf\">SMART<\/a> acronym. Specifically, good goals are:<\/p>\n<ul>\n<li><strong>S<\/strong>pecific (What will you do? What will you accomplish?)<\/li>\n<li><strong>M<\/strong>easurable (How will you know you have achieved your goals?)<\/li>\n<li><strong>A<\/strong>ttainable (Are you reasonably certain of success?)<\/li>\n<li><strong>R<\/strong>elevant (Why is the goal even important?)<\/li>\n<li><strong>T<\/strong>ime bound (What is the time frame for completion?)<\/li>\n<\/ul>\n<p>But how can you know for sure your goals are SMART? The best way to go about goal setting is actually to start with the end \u2014 the evaluation. What?!? That seems a little backward. Are we really asking you to think about how you will evaluate your communication program before you have even designed it? The answer is\u2026yes! This is because goals and evaluation criteria are best developed in conjunction with one another. Having a good idea in mind about how your program could and should be evaluated actually helps you to design a better program \u2014 just like a grading rubric helps to focus you and keep you on track for a class assignment!<\/p>\n<h1>Developing goals and evaluative criteria<\/h1>\n<h2><span style=\"font-family: 'Times New Roman', Georgia, 'SBL Greek', serif;font-size: 18.6667px\">You should have goals and evaluative criteria related to both the process and the outcome of your communication campaign.\u00a0<\/span><\/h2>\n<h2>The process<\/h2>\n<p><span class=\"pullquote-right\">Ultimately, the success of your communication plan can be measured by answering one simple question: Did you achieve your goals? This illustrates why it is important to spend time thinking about and verbalizing your goals and evaluation from the outset.<\/span>Process goals and evaluations are focused on the mechanics of the plan and its implementation. For example, you might want to set goals and evaluative criteria around whether your message reached the target audience, the consistency and timeliness of messaging across different platforms, how information flowed within the communications team, etc. These areas of focus will be highly dependent on the scenario but could include components that you view as particularly critical to the success of the program and\/or components where you anticipate having problems. As much as possible your process goals\/evaluation criteria should be mapped out in advance, but you should also make sure you revisit them throughout the communication process, because chances are that you are going to encounter barriers (or opportunities!) that you did not anticipate, and you may even have to adjust or change your goals accordingly. One component of process evaluation that is often ignored is efficiency. Did you manage to implement your program and achieve your goal with the least amount of wasted time, energy, money, etc.?<\/p>\n<h2>The outcome<\/h2>\n<div class=\"textbox textbox--sidebar textbox--exercises\">\n<header class=\"textbox__header\">\n<p class=\"textbox__title\">Discussion<\/p>\n<\/header>\n<div class=\"textbox__content\">\n<p><strong>Think About Baselines!<\/strong><\/p>\n<p>Most outcomes involve some sort of a change \u2014 change in awareness, change in knowledge, change in behavior, etc. However, you can\u2019t detect or evaluate that change if you don\u2019t know where your audience was at in the first place (i.e., your baseline). For this reason, it may be prudent to define and measure your baseline before your start the communication process.<\/p>\n<\/div>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-581 aligncenter\" src=\"https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Pondering-rat-e1575581547259-300x281.png\" alt=\"\" width=\"187\" height=\"175\" srcset=\"https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Pondering-rat-e1575581547259-300x281.png 300w, https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Pondering-rat-e1575581547259-768x720.png 768w, https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Pondering-rat-e1575581547259-1024x960.png 1024w, https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Pondering-rat-e1575581547259-65x61.png 65w, https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Pondering-rat-e1575581547259-225x211.png 225w, https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Pondering-rat-e1575581547259-350x328.png 350w, https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Pondering-rat-e1575581547259.png 1606w\" sizes=\"auto, (max-width: 187px) 100vw, 187px\" \/><\/p>\n<\/div>\n<p>Outcome goals and evaluations are focused on the impact of your plan on its target audience. If your goal was to inform, did people understand more about the issue in question after versus before your communication effort? If your goal was to change behavior, did the behavior of your target audience actually change? <span class=\"pullquote-left\">&#8216;Unmonitored for outcome, risk communication consumes and wastes valuable resources, are ineffective, and create a false sense of achievement on those who are responsible.&#8217; \u2013 Gaya Gamhewage, World Health Organization<\/span> That being said, these are lofty goals and can be very difficult to measure. That does not mean that you should not strive for them, but it might be wise to come up with some creative but still impactful outcome aims. For example, did your communication effort increase your audience\u2019s access to information about the issue in question? Did people feel empowered as a result of your communication? If you are involving stakeholders in the communication process, an aim could be whether those stakeholders felt that they were engaged in a positive and meaningful way.<\/p>\n<p><span style=\"font-size: 14pt\">Once you have decided on your goals and evaluative criteria, you will need to outline how you will conduct the evaluation. Again, this is important to do at the beginning because it will help you to assess the feasibility of your goals and evaluative criteria.<\/span><\/p>\n<h1>Communication program evaluation<\/h1>\n<h2>Timing<\/h2>\n<p>There are three key times when an evaluation can take place:<\/p>\n<h3 style=\"padding-left: 30px\">Before<\/h3>\n<p style=\"padding-left: 30px\">This is called pre-testing. This involves implementing all or part of your communication plan on representatives from your intended audience (ideally). For example, perhaps you might have a focus group with stakeholders to review your risk communication materials and make sure that everyone understands the message and finds it compelling.<\/p>\n<h3 style=\"padding-left: 30px\">During<\/h3>\n<p style=\"padding-left: 30px\">It can be helpful to have a mid-point review. You often can\u2019t know what works and what does not until you have started to implement. Additionally, there may be unforeseen barriers and opportunities. This is usually more of a process-focused check-up than a full evaluation \u2014 the purpose is to make sure that everything is on track. For programs that take a long time to bear fruit, you may need several mid-point check-ins.<\/p>\n<h3 style=\"padding-left: 30px\">After<\/h3>\n<p style=\"padding-left: 30px\">Once there has been time for your communication effort to bear fruit, it is time for a more formal and in-depth evaluation. This is often focused on outcomes, but, don\u2019t forget the process component because the two go hand in hand. Without a process review, it will be hard to identify the root causes of your success or failure.<\/p>\n<div class=\"textbox textbox--exercises\">\n<header class=\"textbox__header\">\n<p class=\"textbox__title\">Supplementary Material<\/p>\n<\/header>\n<div class=\"textbox__content\">\n<h1>The importance of audience and stakeholder pre-testing<\/h1>\n<p><em>by Taneille Johnson (SPPH 552 2020W1)<\/em><\/p>\n<p>For today&#8217;s blog post, I&#8217;m looking at a campaign run by the BC Government on overdose prevention and de-stigmatization. The reception of this campaign by an important advocacy group is an example of the importance of identifying stakeholders and pre-testing communication messages.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"size-medium wp-image-152 alignright\" src=\"http:\/\/blogs.ubc.ca\/coursec26705f4ac47f2ed7197be5c698caee895b81f1e\/files\/2020\/09\/38205949_1066886990131334_8598558525553115136_o-300x166.png\" alt=\"\" width=\"300\" height=\"166\" \/><\/p>\n<p>This campaign originally ran in 2018. To me, the target audience was likely upper-middle class individuals and aimed to raise awareness that anyone could be a drug user (i.e. drug users are NOT only lower income individuals, and those who use drugs are &#8220;real people too.&#8221;<\/p>\n<p>Shortly after this campaign aired, there was outcry from the Canadian Association of People who Use Drugs. This group argued that the government&#8217;s campaign shifted the focus onto individual drug users and left out many of the complex systems issues that arguably play a larger role in the opioid epidemic.<\/p>\n<p>This advocacy group quickly released remixed posters onto social media (see below, taken from Facebook: https:\/\/www.facebook.com\/notes\/canadian-association-of-people-who-use-drugs\/capud-launches-new-anti-stigma-campaign-aimed-at-bc-provincial-government-\/1063640897122610\/)<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-medium wp-image-153\" src=\"http:\/\/blogs.ubc.ca\/coursec26705f4ac47f2ed7197be5c698caee895b81f1e\/files\/2020\/09\/38218385_1066886536798046_9027589388640452608_o-300x169.jpg\" alt=\"\" width=\"300\" height=\"169\" \/> <img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-medium wp-image-154\" src=\"http:\/\/blogs.ubc.ca\/coursec26705f4ac47f2ed7197be5c698caee895b81f1e\/files\/2020\/09\/38065468_1066891203464246_3726360054606069760_o-300x169.jpg\" alt=\"\" width=\"300\" height=\"169\" \/><\/p>\n<p>Could this reaction and outcry have been prevented by consulting this advocacy group ahead of time? Was audience and potential audience reaction considered? \u00a0Did the government consider partnering with advocacy groups already working on de-stigmatization?\u00a0The government responded by saying that they stood by their original ad campaign and that the goal was to &#8220;bring humanism&#8221; into the opioid crisis (see https:\/\/www.cbc.ca\/news\/canada\/british-columbia\/advocacy-group-takes-issue-with-b-c-government-s-ad-campaign-to-fight-opioid-crisis-1.4773781).<\/p>\n<p>For me, this leads to a larger question of the duty of government to engage with stakeholders prior to communication campaigns. The sticky point is that it is impossible to expect the government (or any communicator) to anticipate every hurdle and consult with every actor.<\/p>\n<p>While the advocacy group may have disagreed with the overall message, perhaps there could have been a middle ground. Perhaps the government could have run the original ad as the first in a series and then followed up with a 2nd ad drawing attention to the systems issues. Regardless, I think this ad (and the ensuing outcry) serves as a reminder to identify your stakeholders or key advocacy groups and pre-test the message beforehand!<\/p>\n<\/div>\n<\/div>\n<h2>Data sources<\/h2>\n<p>A process evaluation relies largely on internal data sources, for example risk communication plans, messages, and communication products (i.e., media interviews, written and visual materials, internet posts, etc.), and interviews with those involved in the communication process.<\/p>\n<p>An outcome evaluation, on the other hand, relies largely on external data sources such as surveys, interviews, and focus groups with segments of your target audience, usage tracking for websites and social media, and data to identify a change in behavior.<\/p>\n<h2>Process<\/h2>\n<p>This is not rocket science! Program evaluations are often omitted from communication plans and we think it is because people make them more complicated than they need to be. For some reason you use the word \u2018evaluation\u2019 and the tendency is to think that is needs some complicated, energy-draining, and time-consuming process. In reality, if you need this sort of process, your goals and evaluative criteria are probably way too complex. The goal of your evaluation is not to completely, utterly, and undeniably prove that you have achieved your goal, but rather to provide reasonable evidence to support that you made steps in the right direction.<\/p>\n<p style=\"padding-left: 30px\">Step 1: Identify the goals for the evaluation. These will be derived from your process\/outcome goals and evaluative criteria.<\/p>\n<p style=\"padding-left: 30px\">Step 2: Determine what data you will need to perform the evaluation.<\/p>\n<p style=\"padding-left: 30px\">Step 3: Collect those data.<\/p>\n<p style=\"padding-left: 30px\">Step 4: Analyze those data.<\/p>\n<p style=\"padding-left: 30px\">Step 5: Draw conclusions and act on those conclusions by modifying your plan.<\/p>\n<p>&nbsp;<\/p>\n<div class=\"textbox textbox--examples\">\n<header class=\"textbox__header\">\n<p class=\"textbox__title\">Example<\/p>\n<\/header>\n<div class=\"textbox__content\">\n<p>We developed a communication campaign for rural Sri Lankans in communities with a high incidence of dog and human rabies.<\/p>\n<figure id=\"attachment_514\" aria-describedby=\"caption-attachment-514\" style=\"width: 500px\" class=\"wp-caption alignright\"><a href=\"https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Sri-Lanka-July-2010.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-514\" src=\"https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Sri-Lanka-July-2010.jpg\" alt=\"\" width=\"500\" height=\"281\" srcset=\"https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Sri-Lanka-July-2010.jpg 3968w, https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Sri-Lanka-July-2010-300x169.jpg 300w, https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Sri-Lanka-July-2010-768x432.jpg 768w, https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Sri-Lanka-July-2010-1024x576.jpg 1024w, https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Sri-Lanka-July-2010-65x37.jpg 65w, https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Sri-Lanka-July-2010-225x127.jpg 225w, https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Sri-Lanka-July-2010-350x197.jpg 350w\" sizes=\"auto, (max-width: 500px) 100vw, 500px\" \/><\/a><figcaption id=\"caption-attachment-514\" class=\"wp-caption-text\">Feral dogs are the main source of rabies in Sri Lanka.<\/figcaption><\/figure>\n<p>The purpose of this campaign was to <strong>educate<\/strong> adult community members about rabies. A <strong>process goal<\/strong> was that 75% of adults in the target communities had seen and read the poster. An <strong>outcome goal<\/strong> was that people who had read the poster demonstrated an increased understanding of the risks associated with rabies and how to mitigate those risks. The <strong>process evaluative criteria<\/strong> was self-reported rates of seeing reading the poster using one-on-one interviews conducted <em>after<\/em> the communication campaign. The <strong>outcome evaluative criteria<\/strong> was a demonstrated increase in understanding regarding rabies risk and risk mitigation between one-on-one interviews conducted <em>before and after<\/em> the communication campaign.<\/p>\n<p>The <strong>evaluation process<\/strong> was as follows:<\/p>\n<p>Step 1. The goal of the evaluation was to estimate the proportion of the target audience who had seen and read the posters and to assess whether or not those who had read the posters demonstrated an increased understanding of rabies.<\/p>\n<p>Step 2. Data were collected through one-on-one interviews with a subset of adults identified by a community leader.<\/p>\n<p>Step 3. Participants were interviewed before and after the educational campaign.<\/p>\n<p>Step 4. Data were analyzed quantitatively and qualitatively.<\/p>\n<p>Step 5. Data were used to determine whether goals had been met and to identify whether important gaps remained.<\/p>\n<\/div>\n<\/div>\n<h1>Takeaways<\/h1>\n<div class=\"textbox textbox--key-takeaways\">\n<header class=\"textbox__header\">\n<p class=\"textbox__title\">Key Takeaways<\/p>\n<\/header>\n<div class=\"textbox__content\">\n<ul>\n<li>Your goal is to clearly define what success will &#8216;look like&#8217; in the context of your communication campaign. Goals should be specific, measurable, attainable, relevant, and time-bound (SMART).<\/li>\n<li>Goals should be developed in conjunction with evaluative criteria, which are objective measures that will help you determine if (and prove that) you have achieved your goals.<img loading=\"lazy\" decoding=\"async\" class=\"wp-image-582 alignright\" style=\"font-size: 16.8px\" src=\"https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Seated-rat-e1575568791861-300x224.png\" alt=\"\" width=\"150\" height=\"112\" srcset=\"https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Seated-rat-e1575568791861-300x224.png 300w, https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Seated-rat-e1575568791861-768x573.png 768w, https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Seated-rat-e1575568791861-1024x764.png 1024w, https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Seated-rat-e1575568791861-65x49.png 65w, https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Seated-rat-e1575568791861-225x168.png 225w, https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Seated-rat-e1575568791861-350x261.png 350w, https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-content\/uploads\/sites\/855\/2019\/11\/Seated-rat-e1575568791861.png 1580w\" sizes=\"auto, (max-width: 150px) 100vw, 150px\" \/><\/li>\n<li>You should have goals and evaluative criteria related to both the process and outcome of your communication plan. Process goals and evaluations are focused on the mechanics of the\u00a0plan and its implementation. Outcome goals and evaluations are focused on the impact of your plan on its target audience.<\/li>\n<li>Once you decide on your goals and evaluative criteria you should outline how you will conduct the evaluation, including the timing, data sources, and process.<\/li>\n<\/ul>\n<\/div>\n<\/div>\n<div class=\"media-attributions clear\" prefix:cc=\"http:\/\/creativecommons.org\/ns#\" prefix:dc=\"http:\/\/purl.org\/dc\/terms\/\"><h2>Media Attributions<\/h2><ul><li about=\"https:\/\/www.flickr.com\/photos\/21903270@N00\/4846953345\"><a rel=\"cc:attributionURL\" href=\"https:\/\/www.flickr.com\/photos\/21903270@N00\/4846953345\" property=\"dc:title\">Fig 1.2.1 Sri Lanka July 2010<\/a>  &copy;  <a rel=\"dc:creator\" href=\"https:\/\/www.flickr.com\/photos\/nousis\/\" property=\"cc:attributionName\">john.nousis<\/a>    is licensed under a  <a rel=\"license\" href=\"https:\/\/creativecommons.org\/licenses\/by-nc-sa\/4.0\/\">CC BY-NC-SA (Attribution NonCommercial ShareAlike)<\/a> license<\/li><\/ul><\/div>","protected":false},"author":801,"menu_order":2,"template":"","meta":{"pb_show_title":"on","pb_short_title":"","pb_subtitle":"","pb_authors":[],"pb_section_license":""},"chapter-type":[],"contributor":[],"license":[],"class_list":["post-29","chapter","type-chapter","status-publish","hentry"],"part":3,"_links":{"self":[{"href":"https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-json\/pressbooks\/v2\/chapters\/29","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-json\/pressbooks\/v2\/chapters"}],"about":[{"href":"https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-json\/wp\/v2\/types\/chapter"}],"author":[{"embeddable":true,"href":"https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-json\/wp\/v2\/users\/801"}],"version-history":[{"count":25,"href":"https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-json\/pressbooks\/v2\/chapters\/29\/revisions"}],"predecessor-version":[{"id":2273,"href":"https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-json\/pressbooks\/v2\/chapters\/29\/revisions\/2273"}],"part":[{"href":"https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-json\/pressbooks\/v2\/parts\/3"}],"metadata":[{"href":"https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-json\/pressbooks\/v2\/chapters\/29\/metadata\/"}],"wp:attachment":[{"href":"https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-json\/wp\/v2\/media?parent=29"}],"wp:term":[{"taxonomy":"chapter-type","embeddable":true,"href":"https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-json\/pressbooks\/v2\/chapter-type?post=29"},{"taxonomy":"contributor","embeddable":true,"href":"https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-json\/wp\/v2\/contributor?post=29"},{"taxonomy":"license","embeddable":true,"href":"https:\/\/pressbooks.bccampus.ca\/missionmessagemedium\/wp-json\/wp\/v2\/license?post=29"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}