The first thing you have to do when you think about assessing impact is to ask why you want to do it, what specifically you expect of that process.
There are in general three main reasons why organizations or collectives work on assessing impact.
- You may want to feed your strategy – assessing impact is a way to have feedback and measure whether you are achieving your goals and to rework your strategy accordingly.
- The second reason has to do with the quality of your work. Whatever your goals are, how do you know that you are doing well and creating value for your stakeholders?
- Finally, you might want to tell an impact-oriented story about what you are doing, and move from explaining actions to explaining what impact it creates. It might be a matter of internal communication – knowing your impact helps people on your team, your partners and supporters be engaged and have positive feedback on the value they contribute. It also helps to move a message across to external parties – to communicate on the value you create to your external audience. This can also be about “selling” what you do by showing the value you create. It makes a better case for stakeholders to “buy in” or contribute funds to what you are doing.
Let us take the example of an afterschool program for kids from disadvantaged backgrounds, that mixes homework tutoring and a hands on initiation to the arts. They have been asked repeatedly by their funders to assess their impact, which is their main motivation to embark on an evaluation journey. They also want to increase the motivation of their teams, who are not paid much and can get discouraged because of the behaviour of some children.
A key determinant in your assessment is figuring out which of the three matters most. This will help you define your priorities and identify the main target audience of your impact assessment: who are you doing this for? This will define what you want to measure and how you are going to do it. In an ideal world we would measure everything and do it in the most scientific, structured way possible. But with this would come a cost and a time commitment that is probably higher than what you can afford. So you have to strike a balance and clarify your goals.
This is particularly true when it comes to external communication. If you want to lobby the government, you need a very solid, nearly scientific impact assessment to make your case and convince them of a change in legislation. Whereas if you just want small contributions from donors, you can simply tell short stories of who you are impacting. But even internally, if you want to motivate your teams, you have to identify the key motivators and arguments that will be useful.
If you want to feed your strategy with impact information, you have to consider who is involved in that strategy – both in framing the strategy and in turning it into an action plan. The expectations of a steering committee or a board of directors will be different than those of the operational team, who will need specific actionables rather than a broad overview of impact.
And finally if you aim at quality, your main audience is your target audience group. Most of your assessment is going to consist in organizing a dialogue with them and get feedback on the experience they are having and the value they perceive.
Let us come back to our example. The above mentioned organization having a clear focus on communication, is looking at two main external audiences: funders, who expect a justification of the money they spend on the program, and team members, who need success stories to remember why they have decided to do that work.
Time
The next thing to do is to consider time. How far are you in your project? When will you be able to collect data about the impact of your work?
Your ability to assess impact will indeed be different depending on how far along you are in your journey. When you are just starting, you can collect baseline information that you can later use as a basis of comparison. If you are five years in and have not done any previous data collection, your ability to compare will be much more limited. Your assessment will be subjective, as the only way to proceed will be to ask people about their perceived difference over time.
Our daycare program has started collecting data about their beneficiaries at the beginning of the program. But they have improved their data collection and the nature of the information they are interested in. So while they have strong, relevant informations about their last 3 years of operation, they only have basic contact informations and data about previous years. Their ability to analyze data over time is hence limited. They now have a robust information system that they will be able to rely on moving forward.
Resources
Another important question is the identification of available resources to do the impact assessment. Do you have a budget at hand, and internal skills to do that work? Or will you need to hire skills? Do you have an information system that already tracks valuable information? The more resources, information and knowledge management, the more data and the more ambitious you can be in your impact assessment. If your volunteers don’t like using computers and are not used to tracking information, you will be much more limited than having paid staff that is data savvy and handles a sophisticated information system.
Impact
Everything we have covered so far lays the ground for the key question of an impact assessment: what do you want to measure? What will serve you and your goals? And what impacts will you be able to assess in my work?
The first step in defining what to measure is to lay out your impact hypotheses. You are doing this work because you intend some impact. These are your strategic impact hypotheses, those that match your theory of change. There might be some other impacts that you have not intended but are still produced and perceived by your stakeholders. You want to list as many impacts as possible, and group them in a logical way.
In that process, the more stakeholders you ask to formulate hypotheses, the more robust your impact map is going to be:
- Ask your teams what are you seeing when you do your work? When colleagues do their work?
- Ask your target groups: When we work with you, what value do you derive? What impact does that have on your life? On the life of those around you?
- Ask your operational and financial partners: what impact do you perceive? Why do you decide to work with us?
All these impact hypotheses may regard your target groups, or other indirect beneficiaries. At that stage, you have no evidence that these impacts effectively occur, but you have strong intuitions.
You may have a list of 40-100 impacts. Obviously this would be too much work – you need to narrow that down to 5-10 impacts you will be able to assess. To do that prioritization work, you can first start by identifying those about which you already have information – these are some that won’t require any extra work, so independently from their importance you will be able to keep tracking them.
Prioritize
But besides these few impacts you are already tracking, you want to prioritize. The best way is to ask yourself and your stakeholders to look at your impact map and to identify which of these impacts are the most commonly witnessed in your work; which can be the most directly attributed to your work; which create the most value to each stakeholder; which are key to your strategy; which are key to convincing financial and operational partners to support you. These series of questions should highlight 5 to 10 impacts, which you will then be able to place your attention on.
To frame their impact assessment strategy, our daycare program brought together a group of 15 former beneficiaries and parents. They asked them to express the main things they remembered from the program and how it had changed their lives and trajectories. They then worked with their staff to identify the impacts that were occurring the most and that were the most aligned with the work of the organization. Finally, they invited their main financial partners to express the impacts that were most meaningful to them. From that they pulled out a list of 12 impacts that they would like to document.
Then, and only when you have those, can you develop your strategy for impact assessment. Take each of the impacts on your shortlist. You can first put aside those for which you already collect data and have evidence. On those impacts, ask yourself whether the evidence is convincing enough or if you need to improve your metrics. For the other impacts, you have to define metrics.
Indicator
The most effective way to build an indicator is to describe as precisely as possible what that impact means – what traits can you objectively perceive. “This child has created three pieces of art over the course of the schoolyear.” “His grades have increased steadily over the past 4 months.”
And then from that description, you are poised to choose 1-3 indicators that are going to feed information. These indicators will let you say that at the end of the year 70% of these kids have developed their creativity and 60% of them have increased their overall school performance.
An indicator is not enough. You now need a method of collecting that information. How will you know that they child is improving his school performance in a systematic and objective way? Are you going to ask the teachers, the parents? Can you afford the time and money to collect that data? Will teachers play along? Will the information be reliable? Are all the informators sharing the same definition of your indicator? So you may end up with 7 impacts you want to measure, identify 10 metrics you think are relevant, but only manage to select 4 you can obtain objectively.
That is for the time being, as moving forward you shall be able to improve your ability to collect more data and inform more metrics, by training your volunteers or working more closely with partners. So the impact assessment plan also includes how you will evolve your ability to track impact over time.
Then comes the really hard part, the implementation.
At the end of the day, our afterschool care program came up with a list of 15 metrics and a plan to increase their data collection for the next three years. They also decided to conduct retrospective interviews with all their graduates from 10 years ago to learn about their trajectory and identify the impacts of their program. In order to do so, they called on a funder for support and worked with a sociologist. That report was a great way to satisfy their financial partners, and a great opportunity for their staff to look back on their work and be hopeful for the kids they are taking care of.