The UK’s Independent Commission on Aid Impact (ICAI) published another fascinating review of DFID’s work this month. It’s well worth a look, for their methods as much as their findings.
This one was on DFID’s Contribution to the Reduction of Child Mortality in Kenya. (Good summary here from the Guardian.)
It builds on an impressive few years work. ICAI have developed a sensitive approach to systematically assessing a wide range of programmes. And some practical tools others in the sector may be able to learn from. Their work is not perfect. But it’s the result of a lot of thoughtful investment and looks pretty good from the outside.
I’m sorry this blog has stretched so long – it hit my sweet spot! The summary is that ICAI seem to have created a powerful and practical framework for assessing, reporting & helping to improve DFID’s work by applying five lessons:
- Creating a simple assessment framework
- Focusing on performance, rather than impact or plans
- Using a range of assessment methods
- Coming to practical conclusions and recommendations
- Using summary traffic lights
1. Simple assessment framework
ICAI’s assessment framework is short and simple, fitting on one page of paper. It can be applied to all DFID’s programmes around the world. How about that as a standard for management frameworks?
It is made up of 21 questions, in four sections: objectives, delivery, impact & learning. For example:
- Are the programme’s design and objectives responsive to intended beneficiary needs and to the context?
- Is robust programme management in place, ensuring the efficiency and effectiveness of the delivery chain?
- Are the results and impact of the programme likely to be long term and sustained?
- Are appropriate amendments made to the programme to take account of the lessons learnt?
I particularly like how this balances two sets of concerns:
On the one hand, it emphasises robust programme management, efficiency and results. On the other it focuses on adapting to changes in the context, responding to beneficiaries, effects on women & girls, and learning.
The framework lets DFID managers know exactly how they will be assessed. And it allows consistent judgements to be made across programmes. We all judge performance when we visit programmes, but seldom on this kind of consistent, transparent basis.
2. Focus on performance, rather than impact or plans
ICAI use the framework to assess how well each programme “performs against ICAI’s criteria for effectiveness and value for money”.
This allows a range of factors to be considered, listed in the 21 questions, above. They include aspects of the process that DFID managers should follow as well as the results they achieve.
This all adds up to a practical and effective approach, as promoted on this site. We know from hard won experience that focusing only on impact or pre-determined plans is not.
3. Range of assessment methods
The Kenya report used a mix of methods that any thoughtful NGO practitioner would be proud of:
… the review included a literature review, a methodology review, head office interviews, a beneficiary survey, key informant interviews in Kenya, site visits and a review of bed nets and immunisation programmes, including an analysis of delivery chains and their cost effectiveness.”
Not a hint of inappropriate randomised control trials!
ICAI’s methods have had their critics. As Professor Robert Picciotto concluded: “In a nutshell, ICAI delivers competent comprehensive audit reports. But auditing is not evaluation.”
I’m OK with that. These reviews don’t aim to be evaluations. Evaluations play a crucial role in development work, in particular to inform future policy. But DFID has its own evaluation unit. And evaluations are expensive and a different kettle of fish to assessing performance, which can create conditions to drive high quality work today.
And I’m really impressed by the Chief Commissioner, Graham Ward’s comment that “We will continue to deepen our focus on beneficiaries in the coming year, because we believe that, if their needs are being met in an effective and sustainable way, the UK taxpayer can be confident that their contribution to the aid budget has been worthwhile.” (Opening letter in Annual Report 2012/13)
This is a wonderful breath of fresh air, compared to focusing on the supply-side of whether pre-determined activities have been completed or objectives achieved.
4. Practical conclusions and recommendations
ICAI summarises the Kenya report:
DFID has been effective in reducing under-five mortality in Kenya through its wider influence in the international system and through its own bilateral work. It has implemented proven interventions, identified by global research and incorporating cross-country learning, particularly for malaria reduction.”
“Sustaining the gains of child mortality reduction is essential. This requires continued DFID funding in the short term and a clear plan for further engaging with – and transferring responsibility to – the Government of Kenya in the medium term. The core of sustainability lies in strengthening basic health systems. This is an area of DFID expertise and should be an increasing focus of its work.”
The review makes three headline conclusions for improvement:
- Specify DFID’s policy on equity and services for hard-to-reach populations.
- Develop a clear exit strategy and criteria for resuming direct financing to the Kenyan government.
- Help strengthen Kenyan health structures & systems.
ICAI’s 2012 – 13 Annual Report gives a thoughtful overview of findings and conclusions across all the reviews they have completed. The issues will be familiar to any development agency, such as:
- The importance of establishing a clear theory of change for each programme.
- The importance of local teams & difficulty in getting HR consistently right.
- How best to define & measure admin costs.
- The importance of beneficiary involvement.
- What is the right level of DFID oversight for the wide range of activities undertaken?
- The issue of maintaining institutional memory & learning lessons.
5. Use of summary traffic lights
Each ICAI report summarises its findings using one of four traffic lights: green, green-amber, amber-red and red. The Kenya report gives the programmes under review a “green-amber”.
These are inevitably controversial, but powerful ratings. I’m a fan. Mark Lowcock, DFID’s Permanent Secretary, says they are not the first thing he looks at. But they must surely command a great deal of senior management attention, as well as attention from the public in Britain (who foot the bill) and recipient countries (who are meant to gain from it).
And that’s desperately needed: a practical way to focus senior management attention on the components of programme performance that matter most. Without that, there are no incentives to improve. And conversations about quality remain just that: words not action.
Last year the British government confirmed that it wants ICAI to continue its work.
It would be fascinating to understand more about what has really changed in DFID as a result of ICAI’s work – both for good and ill. DFID have mentioned that reviews take up a lot of staff time. Lessons from ICAI’s reviews may not be shared and followed up as widely as they might.
Could anyone share any experience on this?
And how about someone bringing together an overview of the whole approach? It would be a wonderful contribution to the urgent question we all face of how to assess & manage development programmes better.
Finally, other major development agencies might want to to consider whether any of these lessons apply to their own work.