Have evidence, will… um, erm?

This is a joint post with Heather Lanthorn, first in a two-part series. Find this post also on the World Bank PSD blog

***

Commissioning evidence

Among those who talk about development & welfare policy/programs/projects, it is tres chic to talk about evidence-informed decision-making (including the evidence on evidence-informed decision-making and the evidence on the evidence on…[insert infinite recursion]).

This concept — formerly best-known as evidence-based policy-making — is contrasted with faith-based or we-thought-really-really-hard-about-this-and-mean-well-based decision-making. It is also contrasted with the (sneaky) strategy of policy-based evidence-making. Using these approaches may lead to not-optimal decision-making, adoption of not-optimal policies and subsequent not-optimal outcomes.

In contrast, proponents of the evidence-informed decision-making approach believe that through approach, decision-makers are able to make more sound judgments between those policies that will provide the best way forward, those that may not and/or those that should maybe be repealed or revised. This may lead them to make decisions on policies according to these judgments, which, if properly implemented or rolled-back may, in turn, improve development and welfare outcomes. It is also important to bear in mind however that it is not evidence alone that drives policymaking. We discuss this idea in more detail in our next post.

In this post, we work with a scenario where evidence is accepted as an important determinant of decision-making and this is acknowledged at least broadly by stakeholders who make explicit (or implicit) commitments to ‘use’ the evidence generated to drive their decisions. As good as this may sound, there are barriers to making decisions informed by evidence. One is the stock of accessible well-considered data and rigorous analyses, including the stock in readable-yet-appropriately-nuanced, relevant, timely forms. Several organizations’ raison d’etre is to increase this supply of ‘much needed’ evidence. Another barrier is lack of demand among decision-makers for (certain types of rigorous) evidence (not just for per diems that come with listening about evidence) – including evidence that could have positive or negative outcomes.

We don’t disagree that both supply and demand for high-quality evidence are important issues. But these two posts are not about those scenarios. Rather, we focus on a scenario in which there is, at least, the demand for commissioning evidence.

Key examples are donor agencies, big (I)ngos (bingos, if we must) or even government ministries that engage in evidence-generating activities, particularly when the stated goal is to make decisions about piloted programs (continue funding, scale-up, scrap, etc) or make significant tweaks to on-going programs. This should be the ‘easiest’ case of using evidence to inform a decision, where demand for evidence leads to the generation of a supply of by-definition-relevant evidence.

And yet, from what we have seen and experienced, even agencies that have made it to this seeming enlightened precipice of evidence-informed decision-making don’t know, at a practical level, what to do with that evidence once they’ve got it. We are not suggesting that those inside such agencies are not skilled at reading and interpreting evidence. Rather, we suggest that so much attention has been given to supplying and demanding evidence that use has been overlooked.

Absent attention on use, how generated evidence informs decision-making, if it does at all, is something of a mystery. Absent a plan for use, it can also be mysterious (or, at least, not transparent) as to why the agency bothered to commission the evidence-generation at all. We suspect that better considered evidence and better plans for use can improve the use of evidence. Our hunches drive these two blog posts.

In this post, we make two main points.

• One, we hold that that a careful formative stage during which stakeholders are engaged to help develop research questions while remaining mindful of the policy process can help generate evidence that those stakeholders will know how to use. There is overlap and complementarity between our suggestions and the recent ideas of Monitoring, Structured experiential Learning & Evaluation (mee; Pritchett, Samji & Hammer) and Problem-Driven Iterative Adaptation (PDIA; Andrews, Pritchett & Woolcock). However, here, we remain focused on planning for evaluation and setting the questions.
• Two, and relatedly, we advocate for more careful planning of how the generated evidence will be used in decision-making, regardless of the outcomes. In our next post, we take seriously that evidence is far from the only decision-making criterion. We discuss how evidence might be fit into a fair, deliberative process of decision-making by agencies and what such a process might entail.

At the outset, we recognize that there is a poor one-to-one mapping of the results of a single rigorous study or paper with policy changes (e.g. And also fun). In these two posts, however, we stay focused on studies that are set up specifically to guide future decisions and thus *should*, by definition, be immediately relevant to policy/programmatic funding/scaling decisions.

Formative work: assessing needs and interests of decision-makers and other stakeholders

An early and wise step, we think, in planning evaluation that is not only policy-associated (we looked at a real, live policy!) But explicitly policy-relevant in terms of decision-making is to identify what kinds of decisions may be made at the end of the evaluation (i.e. What will be informed) and who may be involved. ‘Involved’ includes elite decision-makers and possible policy champions and heroes; it also includes middle- and street-level bureaucrats who will implement the policy/program if that is the decision taken (see, e.g. Here and here on getting buy-in beyond visible leaders).

Among those who talk about demand-generation for evidence, there’s increasing recognition that stakeholder buy-in for the process of using evidence (not just for the policy under investigation) is required early on. But there seems to be less talk on actually asking stakeholders what they want to know to make decisions. We don’t suggest that what stakeholders deem most interesting should define the limits of what will be collected, analysed and presented. Many decision-makers won’t spontaneously crave rigorous impact evaluation.

There is plenty of evidence that decision-makers are heavily influenced by stories, images, even immersive experiences. This is not categorically bad and it certainly should not be ignored or discounted. Rather, in addition to the types of data and analyses readily labelled as rigorous in the impact evaluation arena, we can be creative about collecting and analysing additional types of data in more rigorous and positioned within a counterfactual framework. Because, in the end, incorporating stakeholder preference for the kinds of evidence they need to drive policy change would enhance the quality of the evidence generation process.

Another consideration relates to asking what magnitude of impacts decision-makers feel they need to see to be confident in making their decisions. We don’t suggest this is an easy question to ask — nor to answer. We only suggest that it could be a useful exercise to undertake (as with all our suggestions, empirical evidence from process data about decision-making would be very helpful).

A related exercise is to honestly assess reasoned expectations for the elapsed time between introducing an intervention and the potential expression of relevant impacts. The evaluation should be planned accordingly, as a shorter evaluation period may not generate outcomes related to the issues.

Planning to use evidence

It often seems that commissioners of evidence (and even those who generate the evidence) don’t actively consider how the evidence will actually be used in design or funding or whatever decisions will be made. There seems to be that there seems to be even less consideration of how the evidence will be used regardless of what the outcome is – positive, negative, mixed, null (a point made by, among others in other fora, Jeannie Annan, here). This may be one reason null and negative results seem to go unaddressed.

If there is a (potentially imposed) desire to commission rigorous evidence, one might assume there is genuine equipoise (or uncertainty, also here) about the efficacy, effectiveness or cost-effectiveness of a policy/program. Yet many talks about early buy-in are actually about the program and the potential to validate a flagship programme and justify related spending through evaluation — not about the value of the evaluation process itself for learning. We don’t think this represents the best use of evaluation resources.

An exercise early in the formative phase during which decision-makers consider how the evidence help them make a decision may be useful – if they are asked to consider scenarios in which the evidence is clearly positive, clearly negative or null, mixed, fuzzy or indeterminant. This might also help to clarify research questions that should be asked as part of an evaluation.

In a recent blog post, Dr. Ian Goldman suggests getting decision-maker buy-in by asking “departments to submit proposals for evaluations so that they will want to use the findings.” This is an important step. But it does not mean that proposal-submitters have considered how they will use the evidence if it comes back anything but unequivocally positive for the policy/program/project in question.

Dr. Goldman also proposes asking departments to design “improvement plans” after their evaluations are complete. We’d like to hear more about this process. But we suspect that drafting such a plan early in the formative stage might actually inform some of the research questions, thus better linking the evaluation to action plans for improvement. For example, Sophie (Oxfam) has written about IE results that left them with an “evidence puzzle” rather than a clear idea of how to improve the program. We don’t know if an early exercise in drafting an “improvement plan” would have yielded less puzzling outcomes – but that is an empirical question.

We hope that agencies doing such formative work will document and share the processes and their experiences.

Be honest about the full theory of change for using evidence

In a good evaluation, positive validation is not the only possible outcome. Therefore, the commissioning agency should honestly consider whether, if the results come back null or negative, the agency would actually be willing to pull or roll-back the policy. In many cases, programs have political cache and entitlement value regardless of objective welfare benefits delivered. Rolling-back will not be a politically viable option in such cases. While it is important to build the general evidence base about policy/program cost/effectiveness, when an agency asks for evidence towards a particular decision that it isn’t actually willing to make, we are not sure the evaluation should go forward.

Or, at least, we are uncertain if it should go forward as a yes/no question, where a negative result implies stopping the program. We suspect that evaluation will start to be more appreciated by decision-makers if designed to compare the effectiveness of option A or option B in delivering the favoured program, rather than only examining whether option A works (and why). The former set-up provides ways forward regardless of the outcome; the latter may, in the political sense, not.

Moving forward

In sum, we think that careful formative and needs-assessment work on what decision-makers (and potential implementers) want to see to be convinced and what types of evidence will inform decision-making may lead to the generation of evidence that is not only policy-related but genuinely policy-relevant.  When an agency or ministry specifically commissions an evaluation with the stated goal of using it in decision-making, this seems particularly important. Doing this work well will require collaboration between commissioners, implementers and evaluators.

In the next post, we humbly consider the overall role evidence plays in decision-making and consider how it might fit into an overall fair and deliberative process.