Mo money, mo problems? AMF does not make Givewell’s top-three for 2013 #giving season

(This is a joint post with Heather Lanthorn, also posted on her blog)

***

Recently, Givewell has revised its recommendation on one of its previously top-ranked ‘charities,’ the Against Malaria Foundation (AMF), which focuses on well-tracked distributions of bednets.  Givewell “find[s] outstanding giving opportunities and publish the full details of our analysis to help donors decide where to give.” This approach seems to have succeeded in moving donors beyond using tragic stories and heart-wrenching images to raise funds, looking rather at effectiveness and funding gaps.

In their latest list, AMF does not rank amongst the top three recommended charities.  Here, based on the experience with AMF, we outline the seeming result of Givewell’s attention on AMF, consider the possible lessons and ask whether Givewell seems to have learnt from this episode, taking clear steps towards changing their ranking methods to avoid similar mishaps in future. As it stands, around US$ 10m now lie parked (transparently and hopefully temporarily) with AMF as a result of its stalled distributions, a fact for which Givewell shares some responsibility.

Givewell lays out its thinking on revising AMF’s recommendation in detail.  As a quick re-cap of that blog post: when Givewell looked at AMF two years ago, AMF was successfully delivering bednets at the small- to medium-scale (up to hundreds of thousands in some cases) through partnerships with NGOs (only the delivery of health products such as bednets and cash transfers meet Givewell’s current eligibility criteria). Following Givewell’s rating, a whole bunch of money came in, bumping AMF into a new scale, with new stakeholders and constraints. The big time hasn’t been going quite so well (as yet).

This is slippery ground for a rating service seeking credibility in the eyes of its donors. Currently, Givewell ranks charities on several rating criteria, including: strong evidence of the intervention’s effectiveness and cost-effectiveness of intervention; whether a funding gap exists and resources can be absorbed; and the transparency of activities and accountability to donors.

In its younger/happier days, AMF particularly shone on transparency and accountability. Recognizing that supplies of bednets are often diverted and don’t reach the intended beneficiaries, AMF is vigilant about providing information on ‘distribution verification’ as well as household continued use and upkeep of nets.

These information requirements – shiny at the small scale – create a glare at large-scale, which is part of the problem AMF now faces. ‘Scale’ generally means ‘government’ unless you are discussing a country like Bangladesh with nationwide NGO networks. The first hurdle between information and governments is that the required data can be politically sensitive.  Distribution and use of information is great for donors’ accountability but it can be threatening  to government officials, who want to appear to be doing a good job (and/or may benefit from distributing nets to particular constituents or adding a positive price, etc).

As a second, equally important, hurdle: even if government agencies intend to carry out the distribution as intended (proper targeting etc), data collection has high costs (monetary, personnel, and otherwise)  – especially when carried out country-wide. AMF doesn’t actually fund or support collection of the data on distribution and use that they require of the implementing agencies. AMF is probably doing this to keep its own costs low, instead passing collection costs and burdens on to the local National Malaria Control Programmes (NMCP), which is definitely not the best way make friends with the government. Many government bureaucracies in Sub-Saharan Africa are constrained not only by funds but also capacity to collect and manage data about their own activities.

What do these data needs mean for donors and what do they mean for implementers? For donors, whose resources are scarce, information on transparency and delivery can guide where to allocate money they wish to give. Givewell, by grading on transparency of funding flows and activities, encourages NGOs to compete on these grounds. Donors feel they have made a wise investment and the NGOs that have invested in transparency and accountability benefit from increased visibility.

At issue is that there seems to exist a tension between focusing on transparency and the ability to achieve impact on the ground. If the donor, and possibly Givewell, do not fully take into account institutions (formal and informal), organizational relationships and bureaucratic politics, the problem of a small organization not being able to replicate their own successful results at scale may resurface. Givewell says that it vets a given charity but it is not clear what role potential implementing partners play in this process. Givewell likely needs to account for the views of stakeholders critical to implementation, including those people and organizations that may become more important stakeholders given a scale-up. The fact that NMCPs (or the relevant counterpart) as well as bilaterals and multilaterals are hesitant to work with AMF could have been weighed into Givewell’s algorithm.

Givewell seems to be listening and recognizing these challenges, first by its publicly reasoned response to AMF’s performance, second by posting reviews (in particular, this recent review by Dr. de Savigny) and third, updating its selection criteria for 2013, including a consideration of scalability. de Savingny’s review raises AMF’s strategies in working with governments, both coordinating with donor governments and supporting ‘recipient’ governments with determining data needs and collecting data.

What else can Givewell do now? Expand the criteria beyond need, evidence-base (intervention and organization) and commitment to transparency by also including:

1. Feedback from previous implementing partners.

2. Specific project proposals from applicants, in which they lay out a plan to implement their activity in a specific country. Potential funding recipients should think through and detail their government engagement strategy and gain statements of buy-in from likely implementing partners  – global and local – in that context.

3. Givewell should more carefully calibrate how much money goes to organizations for proposed projects. Funding based on engagement in a particular country can help avoid problems of getting too much too fast: funding can be pegged to the requirements of the specific project that has been put up, for which the organization has need and absorptive capacity.