We Should Test Programs

Not take success for granted.

We take a lot of things for granted when the government does something. When we pass a law, we hardly ever set aside funds to run an experiment on whether the law is having the intended effects. As of 2018, there have only been 13 randomized control trials at the federal level studying a whole program. Of these, 11 of them showed no positive long term effects. We should take care to remember what Peter H. Rossi called the “Zinc law” of social programs - the programs most likely to fail are the ones most likely to be tested. Nevertheless, this should shake us out of any sort of complacency that the effects of a program are what we intend them to be. Even the most obviously positive interventions, such as universal pre-K, might actually have negative effects. This study looked at a pre-K program in Tennessee - when the program was oversubscribed, the state used a lottery to allocate spots. What the researchers found was that being assigned to free pre-K, while having good results in the very short term, actually led to worse results than not getting it by the third grade. You were worse off for getting the program! Surely we would want to know this?

I propose we pass a law - or perhaps a constitutional amendment - committing ourselves to funding experiments in the future. For any intervention, unless it has been tested multiple times, we shall spend 1% (or another appropriate proportion) of whatever was allocated to the program to set up a randomized controlled trial. We do not bind ourselves to any particular policy decisions based on the trials, as that would be too inflexible, but they should have a major impact on what we choose to fund - ineffective interventions should be cut without sentimentality. 

The precise method of implementation is not the important part. It can be done by any level of government, on any sort of program. Neither is the law per se the important part - it would only be passed by a government with an ethos of accountability and cost effectiveness, and that’s what we want.