Barro’s “Democracy and Growth” is a Bad Paper
Just running regressions across countries is not a serious method
I read a paper recently which boggled my mind. Unfortunately, I was not astonished by its conclusions. Rather, I was astonished that an otherwise bright professor would write this, and an otherwise reputable journal would publish it. I discuss it now only to show how far we have come as a profession, and the danger of not taking causal reasoning seriously.
The paper is “Democracy and Growth”, by Robert Barro. He is, naturally, interested in the relationship between democracy and growth. Does it tend to increase or reduce growth? And in reverse, does growth tend to lead toward more democracy, or less growth to less? These are very important questions, and I do not begrudge anyone trying to look into it. I will begrudge one not looking at it very seriously, though!
The method is simply cross country regressions. Suppose you want to know the relationship between democracy and growth. You can rate the strength of democracy in all countries, and have that be your x-axis, and have growth rates be your y-axis, and draw a line of best fit. Obviously, you cannot simply naively do this — democracy might itself be caused by economic growth. Barro’s strategy is to control for other things which might cause growth and affect the level of democracy, like rule of law or corruption or market freedom. Trying to control on observables, though, is a big ask. There’s a lot of ways in which countries differ, and you cannot expect to pick up everything. This comes up in the medical field a lot — if you’re looking for the effect of some food on health, it isn’t enough just to control for income. There might still be unobserved correlates which are actually responsible.
The even bigger problem is that the variables affect each other, or indeed might cause a change in the dependent variables through one of the controlled variables. Democracy might cause better rule of law, which then causes growth. If you control for better rule of law, then you might spuriously claim that democracy has no effect. You have no way of knowing, simply from cross-sectional analysis, which way the causal arrow points. You simply cannot tell without theory or some source of exogeneity. Since there is no attempt at the latter, we cannot use cross-country regressions to find the effects.
I will also note that the presence of measurement errors in the strength of democracy biases the slope downwards. Democracy is the treatment variable, which produces change in the outcome variable GDP; uncertainty in the strength of the treatment variable makes the slope of the line of best fit flatter. This is a bit unintuitive, and best explained with a picture, taken from Andrew Gelman.
He refers to the starting level of GDP in 1960 as an “instrumental variable”, which confuses me, as no matter what you call it, it isn’t an instrumental variable. The idea is that places with lower levels of starting GDP will experience faster growth, because it is cheaper to imitate than to innovate. We don’t have a sound theoretical reason to think this, though — other, equally plausible models, which include investment into finding ideas, will give us opposite results.
If you want to investigate the effects of democracy on growth, you need something other than simply running regressions. For example, you could use instances where democracy was imposed from the outside, or there was a narrow election which led to, or did not lead to, stronger democracy. This is a really tough question to answer, and I’m not fully convinced by anyone else’s work. Acemoglu, Naidu, Restrepo, and Robinson are, at least, taking the question of causality more seriously. Their work isn’t the final word, but looking at the question with multiple different methods, in hopes they accord with each other, seems better to me.
I have not yet mentioned the results from the study, nor will I. I do not think the information from it will improve our beliefs. Nor am I rejecting it because I disagree with the conclusions. My quarrel is solely with the methods. We should judge papers on this basis primarily — the results should be almost an afterthought.
Is there any role for cross-country regressions? Yes, but with caution. I think the use of them in Sala-i-martin’s “I Just Ran Two Million Regressions” is good in that it acknowledges it is not creating dispositive evidence, but simply generating ideas. He wants to find what variables robustly explain economic growth, even when controlling for many different combinations of other variables. The explanations are still vulnerable for the same reasons as before, but it can give us an idea of what variables we should investigate further, in a way which is not obvious from casual observation of the data set.