How do you measure impact? CARTA Graduate contributes to research on improving evaluations of research capacity strengthening programs
Anne Khisa our Cohort Three Fellow in partnership with the Centre for Capacity Research at the Liverpool School of Tropical Medicine, have developed new guidance on improving research capacity strengthening (RCS) evaluation practice.
The project seeks to address the gap of very little evidence to help funders and program implementers to design ways of measuring RCS impact, particularly in sub-Saharan Africa.
The paper titled, A Framework and Indicators to Improve Research Capacity Strengthening Evaluation Practice, was published on June 14, 2019. The other authors are Evelyn Gitau, Director Research Capacity Strengthening at APHRC, Justin Pulford and Imelda Bates both of Liverpool School of Tropical Medicine Centre for Capacity Research.
Anne, who is a doctoral graduate from the University of Nairobi, was part of the seven-month project that also identified indicators to better direct current and future research capacity strengthening investments. It was commissioned by the UK’s Department for International Development and funded by the Department for International Development’s Strategic Evaluation Fund.
Professor Imelda Bates, Head of the Centre for Capacity Research, said: “Despite significant investment in RCS programs in low and middle-income countries, frameworks and robust indicators to determine the impact of these programs are lacking. This DFID funded project was designed to address this gap, identifying indicators to better direct current and future RCS investments.”
The team identified existing RCS frameworks and indicators from published literature along with indicators from 35 programs (including 31 from DFID) that had a strong RCS and Africa focus. The indicators were mapped to an RCS framework through consultative workshops with funders, implementers, university faculty and research management staff, and through key informant interviews with senior individuals from a variety of RCS funding organizations.
The paper comes months after Anne co-facilitated a one-day workshop at the African Population and Health Research Center (where she is a postdoctoral fellow) that discussed common indicators used to measure the impact of RCS initiatives as well as to elicit and collate additional indicators being used to measure the impact of RCS initiatives. The other co-facilitator was Professor Bates.
Two novel concepts emerged from the project which has practical implications for commissioning, designing and evaluating RCS programs. Firstly, over-arching theories of change that describe how schemes will achieve impact, and the theories of change for each funded capacity strengthening project within a scheme, should be aligned. Secondly, program funders can maximize the impact of evaluations by explicitly capturing the capacity strengthening ‘ripple benefits’ that occur between individuals, institutions, and societies. These concepts address the current problems faced by funders created by the lack of a unifying, evidence-based approach to underpin their RCS efforts. Combined with the validated framework and indicators, incorporation of these concepts into new and existing RCS schemes will facilitate comparisons and enable a much more rigorous, harmonized and effective evaluation.
Other outputs from the project include practical recommendations about designing and conducting RCS evaluations, generic indicators to compare progress among projects and schemes and understanding of the need to design and prioritize RCS activities as standalone projects.
“The recommendations and guidance developed through the project are aimed at funders of RCS schemes, programs and projects” continued Professor Bates: “and will facilitate real-time learning and tracking along a trajectory to achieve RCS impact. They will help to moderate unrealistic expectations that investments in individuals and should have a high-level impact.”
Additional information from the Liverpool School of Tropical Medicine.