In just over four years, UK’s Independent Commission for Aid Impact (ICAI) has made quite a mark. ICAI is a body worth taking seriously – not just by DFID and the recipients of DFID funds, but by the wider development sector. For instance, in its latest report, ICAI discussed the limitations of the ‘results agenda’, particularly in the DFID context, but which in reality is an issue that cuts across many big donors. It is creditable that the UK has encouraged an institution such as the ICAI to flourish.
ICAI has a new Chief Commissioner: Alison Evans. In her first message, Evans lays out the commission’s priorities:
We are keen to take a strategic look at how UK aid is responding to global and local development challenges. We will be introducing different styles of review so that our work reflects better the evolving context and ambition for UK development assistance; but we will still look in depth at single programmes and business processes from time to time.
In terms of method we will seek to test DFID’s claims to beneficiary impact (by testing theories of change), and assess how far UK aid is investing in and building on evidence of what works. We will not conduct our own impact evaluations, but rather use and assess the quality of existing evidence, including evaluations funded by UK aid, while drawing on third party evidence where we see fit.
Taken together with Evans’ interview in the Guardian, where she says, “we’re not a think-tank – we’re a scrutiny body”, we can expect some important and insightful reviews to come out of the ICAI in the coming months and years. From the sound of things, it appears we will see a new revamped ICAI. ICAI will continue to grapple with a limited budget (a pittance compared to some of the mega evaluation/research consortia that DFID continues to fund) and in being careful not to duplicate the work of the National Audit Office (NAO), external evaluations and now increasingly, systematic reviews…
ICAI’s role as an ‘independent’ evaluator is interesting in itself. It is a model that I looked up when establishing the Research Evaluation & Learning Unit in Somalia. The two obvious models were the World Bank’s Independent Evaluation Group (IEG) and ICAI. Determining the balance between ‘independence’ and being embedded in an organisation is key, as with evaluations of individual projects. Being external usually has a cost in terms of how well evaluators understand the implementation context and this could lead to reviews that are off-the-mark in terms of the lessons they offer.
For example, take the ICAI review of DFID’s anti-corruption work which got a severe thrashing from reviewers, well-deserved in my opinion. One, the review had a botched up methodology to gather evidence; two, it made recommendations that did not seem to consider the weight (and debates around) of existing global evidence. Also, the view seemed to have misunderstood the specific context in which some of the anti-corruption programmes were designed and implemented. Here is an excerpt from a review by Matthew Stephenson
ICAI’s report suggests — perhaps intentionally, perhaps not — that DFID should shift its focus from things like extractive industry transparency, improving courts and prosecutors, and fighting money laundering to, instead, focusing on youth education and campaigns to end petty bribery. That’s not obviously correct, and indeed the weight of opinion among anticorruption experts seems to be moving in the opposite direction: toward the view that petty corruption, though destructive, is often only a symptom of deeper systemic problems, and that grand corruption–say, the theft of public resources from government programs–is much more harmful to the poor than the petty bribes that people may need to pay in everyday transactions, even if the latter are more visible
Considering this, hearing Evans talk about using and assessing the quality of existing evidence makes a lot of sense. The other challenge that the commission will face is to mobilise the level of objective expertise required to review the wide portfolio of programmes that DFID implements across the developing world. Surely, another challenge would be bridging high-level strategic approaches and the granular context-specific ‘theories of change’. Determining this balance in reviews – both in terms of the approach and methodology they choose, and in the programmes they pick up – will be critical.
In sum – eagerly waiting for the next review!