Evaluate the Plan
It is important to design the evaluation process with the Plan (namely the activities and their goals), and together with the community to ensure an appropriate and responsive approach to data collection and use. For each goal the work is striving to achieve, identify how you will determine success by asking yourself: How will you know when you’ve reached the goal?
You can think of evaluation in terms of process and outcomes. The purpose of process evaluation is to understand how the program is working and if it is being implemented as intended. Outcome evaluation is to understand the difference activities are making.
Outputs of activities are the easiest element to measure when assessing progress towards your goals. Measure outputs by counting the volume/numbers of activities and participants of those activities.
For example: How many people attended the Facebook live event?
Answer by looking at your individual activity data, for example, by counting the attendees including number of people who: stayed the whole time, stayed for more than half the time, viewed for even a couple seconds.
When measuring change, you’re measuring outcomes. Outcomes can be short-term, mid-term, and long- term.
Outcome measurement is most commonly done by conducting pre and post surveys at different time points, for example, during the activities and 3 months after the activity. In these pre and post surveys, participants indicate their knowledge, skill and behaviour before and after the activity has taken place.
Data may also be gathered through storytelling or dialogue such as interviews, focus groups, or talking circles.
Short-term outcome measurement tracks what happens immediately as a result of what you’re doing, for example, increasing knowledge about a topic. Mid or long-term outcome measurement tracks changes after an activity has happened, for example, by tracking a change in behaviour.
Short-term outcome measurement
Let’s suppose the Facebook live event from above is a webinar of sorts, that talks about how to have a conversation with someone who is considering suicide. The goal of offering the webinar is to help people become more confident having that conversation. Is the webinar having that effect? You can measure this outcome by asking participants how they feel before and after the webinar. Using pre and post surveys, ask participants to rate their confidence levels, on a scale of 1-10. For example, ‘How confident do you feel to talk to someone considering suicide?’ Then, compare the before and after ratings. This gives you an outcome measure for this question.
Compiling these data from many participants can give you a picture of how well the webinar is accomplishing the goal.
This approach can be expanded to ask other questions pertaining to this specific activity, or scaled to ask questions of the overarching Plan, too. You can also measure medium-term outcomes by asking the same questions again a few months later. Now you are trying to determine if the immediate learning ‘stuck’ – do people report the same level of confidence they did immediately following the webinar? This gives you insight into how participants integrated the change into their behaviour.
Long-term outcome measurement
Measuring long-term outcomes (or impacts) is important, too! In this example, we are gathering qualitative data instead of using rating scales.
Let’s say you want to know if, after attending the webinar from the example above, participants are more likely to engage with people considering suicide.
Set up interview times where you can talk with a sample of participants, individually. Ask them questions, such as, ‘How has what you learned in the workshop: shifted your beliefs about suicide?; changed your ability to help others?; deepened your compassion for others?’
When you gather all of the responses, compare them to identify themes that emerge. Do the themes match your goal? Are people more likely to engage with someone considering suicide?
Qualitative data can be more difficult to interpret but it can also provide information that is difficult to capture in a rating scale. Using both approaches can give a fuller picture than using only one.
Evaluating your process involves tracking the way the Plan (or activity!) is being implemented. By measuring it as you go along, you can use this information to make real-time changes.
Data can be gathered through administrative data, surveys, storytelling, and/or dialogue such as interviews, focus groups, or talking circles.
Survey or focus group questions could include: What can be improved in the process of promoting our Facebook live sessions? Are there groups of people we have missed? How else could it be promoted?
Pose the questions to a wide group of stakeholders, including people involved in planning the activity, such as committee members, activity participants, and people who didn’t participate in the activity.
National Collaborating Centre for Methods and Tools
Evaluation and health-focused resources.