Theories of Change and Common Metrics Drive Search for Big Impact in Global Health
August 18, 2014 | 1854 views
In trying to solve big, complex problems, how do we figure out which approaches work and which ones don't? This series presents reflections from experts in the field on how we should think about evaluation, once we acknowledge that the world and its problems are complex, dynamic and ever-evolving. How do we get beyond simplistic models of cause and effect, inputs and outputs, and ground our approach to evaluation in a more sophisticated understanding of the world we work in?
Associate Director, Center for Evaluation Innovation
Innovation is a long game. What starts as a bold idea with the promise of tremendous social gains can take years to transform the status quo and achieve a big impact. To date, Grand Challenges Canada has supported over 500 innovations in our four short years of operation. We have come to appreciate the power of early results that can indicate if and when a big impact is on the horizon.
Grand Challenges Canada funds the full scope of innovations – scientific, technological, social and business – that are needed to address the enormous health challenges that burden low- and middle-income countries.
We support portfolios of bold ideas that promise to have big impact in three areas: improving maternal and newborn survival around the time of birth (Saving Lives at Birth), increasing human capital through early child development (Saving Brains), and increasing access to care and effectiveness of treatments for mental health conditions (Global Mental Health).
We have had to figure out not only how to evaluate the progress of a wide variety of individual innovations, but also how to track how well each portfolio of innovations contributes to solving these challenges.
To help focus resources and attention towards the end goal of saving and improving lives (the big impact) and – importantly – to help identify meaningful results along the path to impact at scale, we have adopted theories of change and common metrics.
Theory of Change
A theory of change is a road map that shows how the proposed innovation will make change in the world. To create a theory of change, it is necessary to understand the current situation and the desired impact with enough detail that a logical series of steps to get from the current state to the desired impact can be articulated. Drafting a theory of change can save a great deal of time and resources in the long run.
In addition to project-level theories of change we are refining a common theory of change for each of the challenges we target (e.g. the theory of change for the portfolio of Global Mental Health innovations). The common theory of change can provide a helpful starting point by drawing attention to steps that are important along the path to scale and sustainability.
To assess whether the necessary steps in a theory of change are being achieved, we define a set of measurable indicators. Reporting these project-specific metrics provides evidence that progress is being made towards the desired impact and that the assumptions underlying the theory of change are valid.
At the portfolio level, each common theory of change also has a set of metrics. These common metrics are a language that can be used to express collective progress towards a specific challenge across a set of projects.
For example, different projects may be working to diagnose common mental disorders using very different innovations (e.g. a smart-phone EEG to diagnose seizure disorders or a way to integrate depression screening into primary care). Long before the ultimate outcome of improved lives can be reported, each project can report on the number of people who have been screened or accurately diagnosed. As the innovation process unfolds, these results confirm that the innovation can be used effectively for diagnosis in the target population.
Such reporting also allows for a comparison between projects that have different approaches, which may illuminate potential “bright spots” or transformative strategies that could accelerate the progress of other innovations. Finally, we can aggregate the total number of people diagnosed across all the projects, which helps to indicate how well the portfolio as a whole is performing.
Looking across all the innovations supported by Grand Challenges Canada, we defined the highest level of common metrics that allow us to estimate our total impact. This framework proceeds from outputs, such as the number of service delivery models developed, through intermediate outcomes (e.g. the number of people who access the services), to ultimate outcomes: the number of lives saved and the number of people with improved health outcomes.
Tell the story of progress
Even as the innovation process unfolds, results matter. Resources are precious and thousands of bold ideas are proposed each year, so it is important for those who have the opportunity to develop their innovations to demonstrate progress.
Effectively applying the theory of change and common metrics requires effort – but the ability to see the path ahead and track progress along the way it is worth it. A theory of change and common metrics help to tell a story of progress that is backed by evidence.