Do Experiments Change Policy?

Evidence from Brazil

The use of randomized controlled trials in development economics has been a genuine revolution. If we want to see what policy works, and what doesn’t, we can just go out and test it. Policies guided by RCTs have saved literally millions of lives – we now know better how to distribute medical care and aid. RCTs have shown us the best way to distribute bednets to prevent malaria, plausibly saving millions of lives, and that the price elasticity of healthcare which is clearly beneficial in the long-run is often extremely high. Programs guided by J-PAL, the largest and best of the academic institutions doing RCTs, have affected (as of 2016) over 200 million people. (See page 21).

RCTs can also provide strong evidence that things we thought were good are ineffective. Most famously, RCTs led to us moving away from the fad of microcredit – we simply couldn’t find enough positive effects to justify it. (See, for example, this paper from Banerjee, Duflo, Glennerster, and Kinnan).

But in order for RCTs to be effective, we need for them to actually influence policy. If RCTs are called simply to confirm whatever policymakers planned to do all along, they have but limited effectiveness. Since RCTs have a random element, a biased policymaker who has control over how often studies are run can use them to confirm their priors. Worst of all, the policymaker could simply ignore what tests show, and do what they want anyway.

Banerjee, Duflo and Glennerster could not overcome this, for example, in trying to improve attendance by midwives at Indian medical clinics. The primary reason that people didn’t use the free government healthcare, and instead opted for more expensive private facilities, is that the people employed by the government simply did not show up for work. They were supposed to be open 6 hours a day, 6 hours a week, yet when surveyed were closed 56% of the time. Why bother going?

The intervention was to install timeclocks to punch in and out, and to have a schedule of punishments for absence. At first, all went well – absences were substantially reduced, and people were more likely to use the clinics. 18 months later, though, the intervention did nothing. They were back to the same level of absences as before. As it turned out, the local political authorities lacked the will to actually enforce the rules. The nurses would simply destroy the timeclocks, and go unpunished. Thus, all the evidence in the world matters for naught if governments are unwilling to enforce it.

So, do experiments change policy? Naturally, we ran an experiment. Hjort, Moreira, Rao, and Santini (2020) use a large dataset on the mayors of Brazilian municipalities to find how much they value information from studies, and how much they change policy in response. There are 5,570 municipalities in Brazil, and their study involved 2,150 of them. First, they offered experimental findings to mayors, and elicited their willingness to pay. Mayors were willing to update their beliefs, and preferred larger studies (although they did not update as much as they should have).

They then test whether providing information actually changes policy directly, by providing information at a convention for the heads of Brazilian local government. They invited a random sample of the mayors there, and gave a 45 minute talk on how sending out a taxpayer reminder letter substantially increased compliance with taxes. Checking back two years later, municipalities which received the talk were 33% more likely to send out a letter compared to those who did not.

I am confident that RCTs matter. They are not a panacea, of course, but they are often the best method to test the things we care about most.

And now for some various and sundries. The sheer number of Brazilian mayors has made them a useful source of data for other papers, most notably two from Claudio Ferraz and Frederico Finan. Since mayors are restricted to two terms, they can show that not being up for election makes mayors more prone to corruption. This makes sense, as being exposed for corruption makes reelection less likely, especially when there is local radio to spread the news. This time including local legislators, they found that paying politicians selected for better-educated and wealthier candidates, and that they legislators were arguably more productive. They were also able to show that better paid were more likely to be reelected.

I should also note that for those interested, the AEA maintains a registry of experiments here, with many thousands of experiments registered.