The Independent Commission for Aid Impact (ICAI), UK’s aid watch dog, today, released its review of DFID’s programming and results in water sanitation and hygiene (WASH). – In this impact review, they take a close look at the results DFID reported in its 2015 Annual Report; results that cost £ 713 million between 2010 – 2014.
Do read the full report here.
Some thoughts on the areas of concern in the report:
- The focus on ‘leaving no one behind’ is spot on. It is easy to stack up impressive WASH numbers if one ignores the poorest and the most vulnerable in communities. Safe sanitation and hygiene need to be universal for health benefits to accrue to communities. Within WASH, sanitation is specifically complex, sometimes also called a ‘wicked problem’ – a challenge foremost, of inducing lasting behaviour change. The very nature of careful social engineering required to bring about this behaviour change seems to run contrary to some of the factors that make an intervention scalable – an ability to standardise inputs and break programme components down to easily replicable bits.
- Within the broad basket of ‘service delivery’ interventions, WASH is one of the trickier sectors when it comes to measuring sustained impact, especially at scale. Naturally then, ICAI find that while DFID’s claims of having reached 62.9 million people are broadly correct, it is very hard to establish if the benefits are sustained. Therefore, the results reported remain at the ‘output’ level and that is what ICAI ends up assessing, even though what they set out to do is an ‘impact’ review. While the report speculates on sustaining benefits beyond the 2011-15 period, I wonder whether those that accessed the programme in 2011-12 continued to experience any benefits in 2015.
- The link with government systems, in terms of implementation, monitoring and sustenance remains unclear: another typical WASH issue. Barring say, India, (and this is true especially in sub-Saharan countries, government WASH budgets are highly inadequate. A lot of the work that happens is funded by donors and this implies that monitoring and maintenance happens outside the official system. Achieving local ownership in such a context is a challenge.
- ICAI finds it difficult to assess value for money (VfM) in DFID’s WASH programmes. On one hand, it finds that there isn’t enough competitive procurement, but also there is a lack of established metrics and benchmarks to analyse VfM. Following DFID’s own 3Es framework, an Economy and Efficiency analysis should be possile across the portfolio, and as far as I can tell, is rapidly being developed in the sector, and within DFID. However, partly as a consequence of the lack of ‘outcome/impact’ data, cost-effectiveness studies are likely to remain a challenge. This work by an OPM-led consortium should be particularly relevant in improving VfM analysis across the sector
How could this review have been better?
- We could do with a better feel for DFID’s global portfolio – what the variationa in context-specific experiences, challenges and results are. But probably fair to concede that these boundaries to scope must have been considered when the methodology was being finalised.
- Use administrative data: The review states that several DFID programmes contributed towards larger national programmes, or attempted to strengthen national/local data management systems. It would be great to hear more about experiences from different countries – with varying state capacities – on the extent to which it was feasible to use administrative data as an effective monitoring tool.
- While in absolute terms, 62.9 million is an impressive figure, we are looking at it against the backdrop of a staggering 2.4 billion who lack access to reliable WASH services. In the countries where DFID made significant investments, was there significant progress made towards reaching the MDG targets? Yes, the MDGs were output-based too, but it would be great to get a sense of the scale of impact achieved at the national-level.
- Finally, on learning, the report is critical of DFID, saying that not enough is being done to share lessons across programmes, or from related sectors such as health and education. But surely, there are lessons programmes have learnt over the five years (2010-15)? Further, in the absence of good impact data, was there at least good quality process data? Have we learnt how to implement a WASH programme better?
As I have written previously, ICAI reports are a great addition to the sector, both in terms their scrutiny function of one of the biggest donors in play, as well as because of the questions they ask. So do keep an eye on this space.