Defending Liberia’s right to experiment, and a few questions

Liberia continues to attract criticism for its Partnership Schools for Liberia (PSL) pilot. Here is a recent news report  already pronouncing the verdict on the programme:

Coalition for Transparency and Accountability in Education (COTAE) said in its report released last week Wednesday that the PPP is gradually but emphatically proving to be a failure and the education sector further weakening, presenting a vague future for a nation of impoverished and mostly illiterate citizens.

I have earlier written about how we must support Liberia in experimenting with this model of partnership schools. To recap, Liberia’s Ministry of Education acknowledged that “42 percent of primary age children remain out of school. And most of those who are enrolled are simply not receiving the quality of education they deserve and need” – commendably referring to the problems of both access and quality of education in the country. Conventional education systems have failed to deliver, and research from across the globe supports the view that just having higher paid, or qualified permanent civil service teachers do not yield results. In this context, PSL seeks to generate evidence, and provide decision-makers in Liberia with the tools to iterate reforms to their largely dysfunctional schooling system. Liberia’s education system is not working, and it needs to test out bold new ideas. I therefore fully defend the government’s right to experiment.

It is sometimes hard to disentangle the criticism of the concept of a Public-Private Partnership (PPP) from that of specific providers. Much of the criticism of the PSL seems to be directly targeting the for-profit education company, Bridge International Academies (BIA). But BIA are only one of the eight service providers, running 24 out of the 93 schools (fewer than the originally intended 120) under the PSL.

It is no secret that BIA’s classroom cap (maximum 55 students) is denying students access to education by denying them access to the Bridge schools. These students unfortunately end up not enrolling in school at all—a situation that is counterproductive to government’s compulsory primary education policy. Some of those that are rejected end up in an overcrowded class in another nearby school that tries to accommodate them…

…Schools accommodating students who were denied access to BIA schools are overcrowded and face serious logistical challenges. In some instances, parents have hurriedly erected makeshift structures to accommodate students rejected by Bridge, but lack of teachers and other logistical challenges are still affecting the quality of education in these schools…

…COTAE also accused BIA of breaching the MOU with the government. “Some schools close before the stipulated time due to lack of or inconsistency of the feeding program for students. This breach has serious implications for the curriculum as all materials may not be covered. Students, mostly children, are expected to be in school from 7:30 a.m. to 3:45 p.m., but without food,” the report noted.

In recent months, critics, led prominently by Action Aid International, have sharpened their attacks. The Economist writes about the main concerns being voiced: one, that PSL operators have limited class sizes and are pushing out poor-performing students, and more broadly, will look to game the system to suit their methods; two, that operators are raising and spending philanthropic funds in these schools in addition to the government’s capitation (computed annually, per-child) grant of $50; and three, the business model operated by operators like BIA end up channelling a significant proportion of philanthropic funds raised such programmes on people and systems located outside the recipient country.

There are clearly two separate sets of issues here. One, the question of legality of practices followed in PSL schools. It is important to remember that as in any Public-Private-Partnership, the government needs to play an active oversight and regulatory role. It will not be up to the researchers (however independent they might be) to bring to light, cases of operational deficiencies, or even malfeasance. If BIA and other providers are indulging in practices that violate the commitments made by Government of Liberia to its citizens (and indeed, commitments made by the PSL providers to the GoL), those have to be addressed through the education system, and law enforcement. Admittedly, it is easy to sit outside and demand that a government, already suffering from capacity constraints, play an active role and stand up to powerful donors and donor-funded multinational corporates/NGOs when there are instances of wrongdoing.  But that’s where critics and activists should focus their efforts – in supporting the government to monitor better, and enforce standards.

The second set of issues that The Economist raised are related to the success/failure of the pilot, and its replicability. These are weighty criticisms, and are being addressed to varying degrees by the independent evaluation led by Innovations for Poverty Action (IPA). The researchers have set up a randomised control trial, where intervention schools were assigned randomly to the operators from a set of schools chosen for the evaluation. Critics however argue that the independent evaluation will not provide clear evidence on the PSL. See here and here for this debate that will be fought out in the months and years to come.

The question of additional philanthropic funds being pumped by the operators into PSL schools is a tricky one for the evaluation. Different operators, to their ability and intent, will bring in varying amounts and types of additional investments. These investments are helping them overcome the terms of their agreement with the government that stipulate that they cannot charge any school fees. This could make the programme entirely un-viable even if it comes out successful in the evaluation. Providers like BIA might argue that unit costs would fall with scale, but there are obviously no assurances that will happen. This will also be partially determined by the extent to which the government eventually wants to regulate private providers in the education sector. This is of course a secondary question – first, PSL has to deliver improvements in learning – but one that the government and donors should already be thinking about.

Are we being too quick to judge Liberia’s ‘partnership schools’ pilot?

Liberia has provoked serious outrage from some quarters with its decision to enter into an agreement with private providers to run its primary schools. An official release on Liberia’s Ministry of Education website sets out the problem, saying that

42 percent of primary age children remain out of school. And most of those who are enrolled are simply not receiving the quality of education they deserve and need”.

One must commend George Kronnisanyon Werner, the Minister of Education for focusing his attention on what is really critical – not only are many children out of school, but even the ones that are part of the system are not receiving quality education. Children enrolled in Liberian schools are not learning, and when asked about the state of the education sector, are quite critical.

Meanwhile, the United Nations Special Rapporteur on the right to education, Kishore Singh is scathing in his criticism of the Liberian government’s decision. He asserted that

“Provision of public education of good quality is a core function of the State. Abandoning this to the commercial benefit of a private company constitutes a gross violation of the right to education,”, and “it is ironic that Liberia does not have resources to meet its core obligations to provide a free primary education to every child, but it can find huge sums of money to subcontract a private company to do so on its behalf,”.

This criticism is worrying on two counts – the unyielding stance on engagement of private providers in schooling systems; and more importantly, the statement seems to limit the definition of the state’s implementation capacity to its availability of financial resources. More funds in the hands of governments (or for that matter, donors and NGOs) do not automatically translate into better outcomes.

This is particularly true of primary education. Over the last decade, governments and donors have poured in enormous sums of money into education systems, into securing higher levels of access for children in poor countries. In the process, globally, we have taken our eye off quality and learning outcomes. Year after year now, globally recognised learning surveys tell us that the vast majority of children might be in school, but they aren’t learning. This is true in Liberia’s case too. The debate over Liberia’s policy decision fits right into the global debate over the role of private providers, and to do justice to the enormous education challenge facing us, the least we should insist on, is a nuanced discussion.

In recent years, low-cost private schools have become an important feature on the policy agenda to promote education for all. These schools are not free, but they are low-cost in comparison to the standard understanding of what ‘private’ schools charge. Even so, the $5 – 7 per month that is charged can be quite a burden on the pockets of poor families. Others have countered this by pointing out that education expenditures make up a significant proportion of what families with children normally spend (ranking just behind food and health). Therefore, private schools are springing up, especially in rural and peri-urban areas of low- and middle-income countries. Parents appear to be voting with their feet and private school enrolment is increasing in absolute numbers and as a percentage of gross enrolment figures in countries such as India, Pakistan, Kenya.

Evidence suggests that private providers have a role to play. They deliver learning outcomes that are as good, or slightly better than in public schools at lower per-capita costs, according to evaluations in India and Kenya. Of course, the absolute levels of learning in both types of schools seem abysmally poor, so there is really very little to choose from there. And the costs are kept low by paying teachers lower, which is yet another hot potato. For one, teacher absenteeism and time on task in public schools is very poor in comparison to private schools. More importantly, there is little evidence to show that having higher paid, or qualified permanent civil service teachers yield results in terms of learning outcomes for children.

Other independent studies have found that management systems that are able to hold teachers to account make a huge difference, and many private providers follow models that are more accountable than public systems.  But this debate is far from over. We often complain about the lack of sufficient evidence-informed policymaking—but it is also true that policymakers do not always have the best evidence possible at a given decision-point, where best means both rigorously collected and analysed as well as relevant to the context. This is where the ‘partnership schools’ experiment will provide Liberia with an excellent opportunity – to generate evidence, and make their own decisions based on clear, transparent decision criteria. The official statement says as much: “an independent body will be commissioned to evaluate the outcomes of the pilot program.

Many critics are up in arms over the fact that Liberia is partnering with Bridge International Academies (BIA), an American firm that has investors such as Mark Zuckerberg, Bill Gates, and the World Bank Group’s International Finance Corporation (IFC). Engaging a single provider in such a large experiment is risky. But, at this point, it is not even clear that is the case. Liberia says they will run a competitive procurement process to identify partners, so one would hope that there would be multiple partners in the mix, and an opportunity to test out more than one approach. The eco-system of private providers is a large one. There are the likes of BIA, which are large multinational chains, and then, there are the single-entrepreneur schools, community/NGO schools, church-run schools, etc. One could argue that local private players are better rooted in the communities they work in and therefore, are more responsive and accountable. But these are assumptions that this pilot should help test out.

When Liberia decides to initiate a pilot of up to 120 schools (3% of the all public schools in the country), it is not abdicating its responsibility towards its children. By starting this pilot, the Liberian government is neither off the hook for improving quality of the remaining public schools in the country, nor can it give up its oversight of private schools that would function under its watch. The criticism of Liberia on the lines that it is “outsourcing its entire education system” just isn’t right. As a country in the middle of a post-conflict recovery process, Liberia needed to do something different to revive its education sector. This is a bold gamble, and promises to be an interesting public policy experiment. On this occasion, we must not be too quick to judge Liberia.

***

This is also my livemint column

Do phone surveys work?

A question we frequently encounter – for reasons of access, cost and reliability of data.

Interesting findings therefore, in this paper studying micro-enterprises in the township of Soweto, South Africa:

We randomly assign micro-enterprises to three groups, who receive are interviewed face-to-face at monthly intervals (mimicking a standard method of collecting data from micro-enterprises), face-to-face at weekly intervals, and by mobile phone at weekly intervals. We find high frequency data collection is useful: it captures extensive volatility in a number of measures not visible in less frequent data collection. We also find is viable: on most measures and at most quantiles of the distribution, data patterns are indistinguishable in interviews conducted weekly or monthly and face-to-face or by phone.

High frequency surveys (which are easier when conducted over the phone) provide valuable data; and the authors do not find any significant difference in attrition rates between phone and face-to-face surveys. Also, variable costs are about 40% lower for phone surveys – the cost savings would likely be higher if conducted over widely dispersed rural areas instead of an urban township.

Recap: Delhi’s #OddEven plan, in hindsight

Late last year, Delhi’s Chief Minister, Arvind Kejriwal, announced a measure to tackle the severe air pollution crisis in the city. The proposal was to implement an odd-even plan for private cars on Delhi roads: cars with odd numbered registration plates would be allowed to ply on odd dates and those with even numbered registration plates allowed on the other days. There was an exemption list that included single women (or with children), public vehicles, medical emergencies, etc. This was to be piloted for a period of fifteen days, starting on 1st January 2016.

For a detailed account of how the city dealt with this rule, see here.  An excerpt:

During the odd-even period, the use of cars fells by 30 per cent while those car-pooling went up by a whopping 387.7 per cent, indicating the success of the government’s push towards that option. Delhiites using private auto-rickshaws went up by 156.3 per cent compared to the period before odd-even, while Metro use went up by 58.4 per cent.

On average, the respondents’ took 12 minutes less to commute from home to work during the odd-even period. Car and bus users reached their workplaces 13 and 14 minutes faster during the 15-day period

I will come to the outcomes of this pilot in just a moment. Outcomes aside, the Delhi government’s Odd-Even plan has yielded a rich bounty. It sets the template for citizen engagement with a public policy reform experiment: heightened awareness regarding the core issue, mass participation, intense public scrutiny, and a data-driven discourse. Let’s take these one-by-one.

Heightened awareness: From the Delhi High Court downwards, it is widely acknowledged that this government move has made pollution a talking point, and increased general awareness. Odd-Even has consistently trended on Twitter (the barometer of our times of how important a topic is) and has sparked numerous prime-time slanging matches. No doubt, there is also a haze of misinformation, but mostly, people are now hearing more about air pollution, how it affects their health, and what the various ways to deal with this problem are.

Mass participation: Largely as a result of the above, citizens in Delhi have demonstrated an overwhelming level of compliance with this experiment. I do not believe that just the fear of fines was sufficient to bring about this level of compliance. People have participated in solidarity with the scheme, partially out of their concern for the levels of air pollution, and also possibly, a curiosity to see if the experiment will yield any results. Either way, this experiment would have been a non-starter without this level of mass participation.

Intense public scrutiny: Just as much as commuters in Delhi have participated in the scheme, a wider population has actively dissected this experiment and have come out on both sides of the divide. This set of observers and analysts have highlighted implementation challenges, bringing out data on current and historical pollution levels, gathered and presented from multiple sources – government and non-government – this has been a public policy enthusiast’s dream come true. Heavy public participation, accompanied with this level of public scrutiny, makes for an ideal public policy reform experiment.

Data-driven: The resolution of the debate over the effectiveness of the Odd-Even plan has to rely on data, and what is available currently is not all robust, or scientifically well-informed. But in order to measure impact, hard data will have to be collected, and analysed, before a conclusion can be reached regarding the effectiveness of the scheme. One could also look at data on the numerous indirect and unintended benefits – decongestion is a prominent one, for example. To the quantitative data on pollution readings, one can add the qualitative data on people’s perceptions, attitudes and behaviours.

The reality of policy experimentation

Resistance to public policy reform, and experimentation ranges from the gentle to vitriolic. Politically, one can argue both sides of the coin – either that this just a political gimmick, or that the success in implementing the experiment reveals how strong public support for Aam Aadmi Party and Arvind Kejriwal continues to be. Reactions to the Odd-Even policy also reflect entrenched positions, many of which are political and motivated, but importantly, not necessarily just so. This is important – a public policy reform will be subject to serious criticism, not just motivated by ‘interests’, but stemming from genuinely opposing ‘ideas’. Those initiating a reform must be prepared to ‘learn’ – trial, review, tweak, and trial again.

Evidence trickles in

Over the last fortnight, researchers Michael Greenstone, Santosh Harish and Anant Sudarshan were collecting hard data on how the pilot fared. They find that the Odd-Even plan reduced pollution by significant levels in Delhi. The headline: this study finds there was an 18% reduction in PM 2.5 due to the pilot during the hours that the rule was in effect. The effect size is truly staggering, and is quite unusual for studies that use such rigorous methodology to look at the impact of policy interventions.

Starting January 1, while absolute pollution levels increased both inside and outside Delhi (for atmospheric reasons, as noted by other commentators), the increase in fine particle levels in Delhi was significantly less than in the surrounding region. Overall, there was a 10-13 per cent relative decline in Delhi.

…Around 8 am, the gap between Delhi’s pollution and that in neighbouring regions begins to form and steadily increases until mid afternoon. As temperatures begin to fall, and pollution is less likely to disperse, this gap starts to close. We see another small gap emerge between 9-11 pm, which probably reflects the new limits on truck traffic in Delhi, which also came into force on January 1. Soon after midnight, the gap closes, and Delhi and neighbouring areas show similar pollution patterns until 8 am comes around again. When focusing just on the hours that the odd-even policy was in effect, our estimates suggest that particulates pollution declined by 18 per cent due to the pilot. 

The methodology and analysis is set out in greater detail here:

We compare the changes in PM 2.5 concentration levels before and after the program in Delhi monitors and outside Delhi in the NCR: Faridabad, Gurgaon and Noida. While the odd-even program was in place for commuters from these cities to Delhi, the odd-even program was not directly implemented there. If anything, the impact on the commuters makes our estimates lower bounds to the true impact.

In doing this, they addressed a glaring oversight that journalists were earlier making – of taking into account, what would have happened in the absence of this pilot and comparing that with Delhi. Using a counterfactual and a difference-in-difference approach, the researchers are able to conclude that the levels of pollution in Delhi were indeed lower during the fortnight that the Odd-Even plan was implemented. Thus, while debates raged in the mainstream media, social media, and street-corners over the success and failings of this scheme over the two weeks of implementation, with the last piece – hard data – coming in, we truly have a case worth studying! Delightfully, the study has also silenced those who had waded in on the backs of unfounded methods and faulty data.

What next?

What one needs to consider is also whether this arrangement is sustainable in the long-term? Perhaps not, as has been the experience in other cities, unless Delhi sees collective action at an unprecedented level (phasing out old cars, people agreeing not to buy second-hand cars, etc). But while you cannot change the weather, and it may be difficult to stop farmers from burning crop waste in the short-term, the success of Odd-Even presents a new and effective tool – an option that can be implemented to bring down peak air pollution, or neutralise at times when the weather is most unfavourable.

The Delhi government should remember though that without serious efforts to expand and improve public transport, and introducing measures such as congestion pricing, or other forms of pollution taxes, vehicular pollution will not be controlled. Governments or leaders who claim to have all the answers probably aren’t the ones who would be open to trial and learning. As the researchers say:

More generally speaking, governments need to accept that we don’t have all the answers to policy problems and adopt a culture of trying out new ideas, testing them carefully, and then deciding which ones to adopt at scale

So what next?

It is often said that we get the government we deserve. Citizens in Delhi have participated honestly in a worthwhile public policy experiment. While the final outcomes will be ascertained as more data is collected and analysed, it is clear that problems like pollution can only be tackled with a critical mass of people coming together; collective action that can look beyond personal inconvenience. So the biggest gain from the Odd-Even plan would be a willingness to participate in experiments in collective action – a citizenry that is aware, engaged, and willing to work together with its government to find solutions to its problems.

In return, what they deserve is a government that is committed to finding answers to difficult questions – in this case, a government that is willing to explore all possibilities to devise a set of interventions that can tackle the scourge of air pollution in Delhi. Here’s hoping we see many more of such ideas…and in the meantime, the Delhi government deserves a wide round of applause.

Doing research? Spend more time with programme staff…

From Duncan’s blog covering the paper ‘Using Participatory Process Evaluation to Understand the Dynamics of Change in a Nutrition Education Programme’, by Andrea Cornwall

While it is always interesting to hear about interesting approaches to data collection and analysis – and a lot of this feeds into thoughts/reactions I had while attending (by complete happenstance) the first day of the Doing Development Differently workshop at the Kennedy school, in this blog, I wanted to quickly highlight something that caught my eye in Duncan’s blog

Its this bit in particular:

“What we did next was not to go to the program sites, but spend some more time in the headquarters. We were interested in the perspectives of a variety of people involved with implementation, from the managers to those involved in everyday activities on the ground. And what we sought to understand was not only what the project had achieved, but also what people had thought it might do at the outset and what had surprised them”

The focus on measuring impact on the ground often obscures the focus on understanding how projects are delivered. That’s one of the reasons why researchers have so little in their papers on the context and implementation of the interventions they seem to be studying (which is one of the many good points Heather makes in this blog when she laments the lack of ‘intervention description’). Often implementation staff perspectives are not only considered extraneous to the key research questions, but they are also sometimes actively blocked out so that research findings are not biased: guess only the biases of the researcher get to make it to the final paper/report. Talking to programme staff can not only provide key insights and lessons to implementation, but also elevate their status from being foot soldiers that executed a project logframe to leaders and innovators in their own right, who came up with creative solutions to complex and emergent situations on the ground (or not!).

Look forward to reading the full paper; and more thoughts on the DDD workshop, coming up soon (hopefully).

How Bharat squats

In this column for livemint, I disagree that in our sanitation policy, all we need to worry is about behaviour change communication. If we dont get our infrastructure right, little else is going to make sense. Many other poorer countries may have succeeded with little funds and pit-latrines, but the same has not worked for India and is unlikely to, irrespective of what we do by way of motivating those squatting in the open to use a toilet instead.

I peg my argument on the findings from the recent SQUAT survey by RICE.

Read the column here

Enforcing accountability in decision-making

This is a joint post with Heather and the sixth and last post in the series on decisionmaking, continuing from the last one

***

A recent episode reminded us of why we began this series of posts, of which is this is the last. We recently saw our guiding scenario for this series play out: a donor was funding a pilot project accompanied by a rigorous evaluation, which was intended to inform further funding decisions.

In this specific episode, a group of donors discussed an on-going pilot programme in Country X, part of which was evaluated using a randomized-control trial. The full results and analyses were not yet in; the preliminary results, marginally significant, suggested that there ought to be a larger pilot taking into account lessons learnt.

Along with X’s government, the donors decided to scale-up. The donors secured a significant funding contribution from the Government of X — before the evaluation yielded results. Indeed, securing government funding for the scale-up and a few innovations in the operational model had already given this project a sort-of superstar status, in the eyes of both the donor as well as the government. It appeared the donors in question had committed to the government that the pilot would be scaled-up before the results were in. Moreover, a little inquiry revealed that the donors did not have clear benchmarks or decision-criteria going into the pilot about key impacts and magnitudes — that is, the types of evidence and results — that would inform whether to take the project forward.

There was evidence (at least it was on the way) and there was a decision but it is not clear how they were linked or how one informed the other.

Reminder: scenario

We started this series of posts by admitting the limited role evidence plays in decision-making — even when an agency commissions evidence specifically to inform a decision. The above episode illustrates this, as well as the complex and, sometimes, messy way that (some) agencies, like (some) donors, approach decision-making. We have suggested that, given that resources to improve welfare are scarcer than needs, this approach to decision-making is troubling at best and irresponsible at worst. Note that it is the lack of expectations and a plan for decision-making that are troublesome as the limited use of outcome and impact evidence.

In response to this type of decision-making, we have had two guiding goals in this series of posts. First, are there ways to design evaluations that will make the resultant outcomes more useable and useful (link to posts 1 & 2)? Second, given all the factors that influence decisions, including evidence, can the decision-making process be made more fair and consistent across time and space?

To address the second question, we have drawn primarily on the work of Norm Daniels, to consider whether and how decisions can be made through a fair, deliberative process that, under certain conditions, can generate outcomes that a wide range of stakeholders can accept as ‘fair’.

Daniels suggests that achieving four key criteria, these “certain conditions” for fair deliberation can be met, including deliberation about which programs to scale after receiving rigorous evidence and other forms of politically relevant feedback.

Closing the loop: enforceability

So far, we have reviewed three of these conditions: relevant reasons, publicity, and revisibility. In this post, we examine the final condition, enforceability (or regulation).

Meeting the enforceability criterion means providing mechanisms to ensure that the processes set by the other criteria are adhered to. This is, of course, easier said than done. In particular, it is unclear who should do the enforcing.*

We identify two key questions about enforcement:

First, should enforcement be external to or strictly internal to the funding and decision-making agency?

Second, should enforcement rely on top-down or bottom-up mechanisms?

Underlying these questions is a more basic, normative question: In which country should these mechanisms reside — the donor or the recipient? The difficulty of answer this question is compounded by the fact that many donors are not nation-states.

We don’t have clear answers to these questions, which themselves likely need to be subjected to a fair, deliberative process. Here, we lay out some of our own internal debates on two key questions, in hopes that they point to topics for productive conversation.

Should enforcement of agency decision making be internal or external to the agency?

This is a normative question but it links with a positive one: can we rely on donors to self-regulate when it comes to adopted decision-making criteria and transparency commitments?

Internal, self-regulation is the most common model we see around us, in the form of internal commitments such as multi-year strategies, requests for funds made to the treasury, etc. In addition, most agencies have an internal but-independent ‘results’ or ‘evaluation’ cell, intended to make sure that M&E is carried out. In the case of DFID for instance, the Independent Commission for Aid Impact (ICAI) seems to have a significant impact on DFID’s policies and programming. It also empowers the British parliament to hold DFID to account over a variety of funding decisions, as well as future strategy.

Outside the agency, oversight and enforcement of achieving relevancy, transparency, and revisibility could come from multiple sources. From above, it could be a multi-lateral agency/agreement or a global INGO, similar to a Publish What You Pay(?). Laterally, the government in which a programme is being piloted could play an enforcing role. Finally, oversight and enforcement could come from below, through citizens or civic society organisations, both in donor and recipient countries. This brings us to our next question.

Should enforcement come top-down or bottom-up?

While this question could be answered about internal agency functioning and hierarchy, we focus on the potential for external enforcement from one direction or the other. And, again, the question is a normative one but there are positive aspects related to capacity to monitor and capacity to enforce.

Enforcement from ‘above’ could come through multilateral agencies or through multi- or bi-lateral agreements. One possible external mechanisms is where more than one donor come together to make a conditional funding pledge to a programme – contingent on the achievement of pre-determined targets. However, as we infer from the opening example, it is important that such commitments should be based on a clear vision of success, not just on political imperatives or project visibility.

Enforcement from below can come from citizens in donor and/or recipient countries, including through CSOs and the media. One way in which to introduce bottom-up pressure is if donors adhere to the steps we have covered in our previous posts – agreement on relevant reasons, transparency and revisibility – and thereby involve a variety of external stakeholders, including media, citizens, CSOs. These can contribute to a mechanism where there is pressure from the ground on donors in living up to their own commitments.

Media is obviously a very important player in these times. Extensive media reporting of donor commitments is a strong mechanism for informing and involving citizens – in both donor and recipient countries; media is also relevant to helping citizens understand limits and how decisions are made in face of resource constraints.

Our gut feeling though is that in the current system of global aid and development, the most workable approach will probably include a mixture of formal top-down and informal bottom-up pressure. From a country-ownership point of view, we feel that recipient country decision-makers should have a (strong) role to play here (more than they seem to have currently), as well as citizens in those countries.

However, bilateral donors, will probably continue to be more accountable to their own citizens (directly and via representative legislatures) and, therefore, a key task is to consider how to bolster their capacity to ensure ‘accountability for reasonableness’ in the use of evidence and decision-making more generally. At the same time multilateral donors may have more flexibility to consider other means of enforcement, since they don’t have a narrow constituency of citizens and politicians to be answerable to. However, we worry that the prominent multilateral agencies we know are also bloated bureaucracies with unclear chains of accountability (as well as a typical sense of self-perpetuation).

While there is no clear blueprint for moving forward, we hope the above debate has gone a small step towards asking the right questions.

In sum

In this final post, we have considered how to enforce decision-making and priority-setting processes that are ideally informed by rigorous and relevant evidence but also, more importantly, in line with principles of fairness and accountability for reasonableness. These are not fully evident in the episode that opened this post.

Through this series of posts, we have considered how planning for decision-making can help in the production of more useful evidence and can set up processes to make fairer decisions. For the latter, we have relied on Norm Daniel’s framework for ensuring ‘accountability for reasonableness’ in decision-making. This is, of course, only one guide to decision-making, but one that we have found useful in broaching questions of not only how decisions are made but how they should be made.

In it, Daniels proposes that deliberative processes should be based on relevant reasons and commitments to transparency and revisibility that are set ex ante to the decision-point. We have focused specifically on decision-making relating to continuing, scaling, altering, or scrapping pilot programs, particularly those for which putatively informative evidence has been commissioned.

We hope that through these posts, we have been able to make a case for designing evaluations to generate evidence useful decision-making as well as for facilitating fair, deliberative processes for decision-making that can take account of evidence generated. At the very least, we hope that evaluators will recognise the importance of a fair process and will not stymie them in the pursuit of the perfect research design.

***
*In Daniels’s work, which primarily focuses on national or large private health insurance plans, the regulative role of the state is clear. In cases of global development, involving several states and agencies, governance and regulation become less clear. Noting this lack of clarity in global governance is hardly a new point; however, the idea of needing to enforce the conditions of fair processes and accountability for reasonableness provides a concrete example of the problem.