How Much Should We Trust Synthetic Control Estimates?

My notes on learning a literature

Economists are often concerned with causality. We would like to know the effect of X on Y, and we would like to know what causes what. The ideal case for discovering causality is one where the receipt of treatment X is totally uncorrelated with their attributes. If it is correlated, then we would be unable to say whether it was really the treatment X which caused the change in Y, or the attributes which they had. We could try and control for attributes – which is to say, we compare only the effect within certain types – but this means we are unable to rule out unobservable attributes affecting the results.

In a perfect world, we randomize. Economists like randomization because, conditional upon being in the sample which is randomized, attributes both observed and unobserved are uncorrelated with treatment. If you want to know the effect of a medication, then you just give medicine to some and a placebo to others. Nice, clean, easy. Alternatively, we can have some variable Z which is correlated with X and uncorrelated with Y conditional on X – an instrumental variable.

We often do not have randomization available, however. Suppose that we are studying the effect of an increase in the minimum wage on unemployment in the hypothetical state of “New Jersey”. There are two basic ways to study this. The first is to construct a structural model of workers, specify their utility functions, the conduct with which firms compete, and so on, and then simulate various counterfactuals. This just kicks identification down the road, though – you now need exogenous shocks to identify all these primitives.

The other is to just look at what happened. Card and Krueger (1994) studied this exact scenario in New Jersey. Their argument for identification is that the areas of Philadelphia in Pennsylvania are affected by the same trends as the areas in New Jersey, with the sole exception that one saw a change in the minimum wage. If you are willing to buy that assumption, then they have found the true effect of minimum wage laws.

However, it is often not clear that the proper control is. Suppose that Pennsylvania had different trends or levels in unemployment before. Can we compare it anyway? Can we compare it to any other state in the union? Differences-in-differences is also generally more believable when there is a clean and plausibly exogenous break between treated and untreated units. The regions that Card and Krueger studied bordered each other, and I’m inclined to believe a study which examines places on either side of border like Kansas-Nebraska, but what should we do if the affected regions are far away from any border? If you are considering the effect of a minimum wage hike in California, with the vast majority of its workers far away from any state boundary, what can you use as the control?

What’s more, it’s not even clear that bordering is a good thing. In order for the local average treatment effect to equal the “true” effect, we have to assume no spillovers between units. One can imagine a world in which employment plummets in New Jersey, but it’s only because people go work across the border in Pennsylvania. A minimum wage hike in both places would not have the same effect.

To deal with this, Alberto Abadie and Javier Gardeazabal (2003) devised a method called synthetic control. They are interested in what the effect of the ETA’s campaign of terrorism in the Basque country was on economic development. Rather than take any one particular place to be the control unit, they instead took a weighted average of Spanish provinces that most closely matched the behavior of the Basque province before. Specifically, they said the Basque country, out of 16 provinces, was closest to 85% Catalonia and 15% Madrid. Computing the weights is actually a computationally difficult problem, which entails a nested fixed-point procedure that you may have seen before if you are familiar with the industrial organization literature. This “synthetic” control produces a more realistic control than any particular region. The key reference is Alberto Abadie writing in the Journal of Economic Literature.

That is what I knew coming into writing this. I have always been a bit intensely suspicious of synthetic control, because of the degrees of freedom it affords the researcher. If at first you don’t have a result, you can try, try again. The point of writing this, then, was to document my process of learning about it.

I shall number my concerns so that we can refer back to how an advocate for synthetic control would address them, starting with the ones I was concerned about going in.

1. You can control which units go into your donor pool of possible control units, and how you divide up those units. In the case of the effect of terrorism in the Basque country, you can choose whether you use provinces or local municipalities, or whether foreign provinces should be included.

2. If there are many different combinations of units which arrive similarly matched pre-trends, you have the choice of which combination to use. Ideally, you would arrive at a single combination of best fit, but this actually needn’t happen. With enough potential control units, it is possible for there to be an infinite number of combinations of control units which produce the pre-event pattern, all of which will have different after event trends.

3. You have discretion over what pre-trends you care about. Suppose you’re examining the economic development of a country. Do you weight on years of education? Test scores? GDP? The commodities they export? Or any of a dozen possible other factors, some of which may be causally related to each other. The more factors you include, the more units in the donor pool you need.

4. If the control units are affected by transitory shocks, then matching on pre-trends doesn’t do anything at all. You observe Z, which are the realized values, but don’t observe mu, which are the shocks. So, if you’re matching on GDP, but don’t include type of industry, you will get entirely spurious differences. Basically there’s no reason for pretrends to continue unless they were caused by the same factors. Note that including a longer panel need not improve this at all. Differences-in-differences faces the same problem, but the point of comparing areas right next to each other is to account for all of the shared shocks.

5. Related to objection 4, and to be honest just a clarification of what it means, there’s no guarantee that the closest fit is actually the best comparison. Imagine a room with a dimmer that is slowly being slid from 0 to 1 brightness, and on average the room is ½ bright. With a large enough pool of rooms which have their dimmers being randomly flipped up and down between 0 and 1, we should be able to construct a sample which exactly mirrors the tested room. Let’s suppose the treatment happens and there’s no effect, but the room continues increasing in brightness from 1 to 2. Meanwhile the randomly flipping rooms stay between 0 and 1. We have gotten a totally spurious result despite a perfect fit of pretrends.

Objection 2 is, I think, the easiest one to reject. If researchers do not meddle with the algorithm to find the best fit, and there is a unique solution, then there is no room for meddling besides the usual publication bias reasons. I sincerely doubt that many researchers are tweaking how the algorithm computes in order to get significant results; and if they are, they really should be doing something better than reg x, y (complicated).

There are also some basic robustness checks which you see in every paper using it. First, you change the year in which the shock took place. Your results should now show spurious results for any year you test. Second, you can “leave one out” of your donor pool – essentially testing for outlier having outsized effects. You need to pass these hurdles to get published.

There are still problems with researcher degrees of freedom, which have been tested with Monte Carlo simulations. Ferman, Pinto, and Possebom (2020) tests how much cherrypicking is possible in the choice of variables, which is objection 3. First, the good news. The more periods to fit onto, the less scope there is for cherry-picking. This also answers objection 4 – as periods go to infinity, you can abstract away from transitory shocks. Now the bad news: there is still considerable scope for cherry-picking in the choice of predictor variables. They run simulations on data in the Current Population Survey, rather in the spirit of Bertrand-Duflo-Mullainathan (2004). If no cherry-picking, they should reject the null 5% of the time – they actually reject it 14%. Worse still, there will be serial correlation in your data over time, which has to be addressed ad hoc. (They set the parameter equal to .5). The more correlated with itself it is, the less able you are to get to minimize bias. And above all, the bias from objection 4 is what it converges too asymptotically as the number of periods goes to infinity, but in almost all use cases you are quite far from that happening.

Pickett, Hill, and Cowan (2022) is in a similar vein, and aims to dispel certain myths about synthetic control. The first problem is that the optimization procedure is actually different on Stata vs R, which leads to finding different results. (!!) The covariates of objection 4 are also empirically relevant, given the number of periods we have to work with. And objection five (their myth no. 3) really does matter. If you include too much, you end up fitting on noise and getting spurious differences. I’m not entirely sure I understand the solutions to overfitting on noise. My understanding is that they are essentially taking an average of the data they have. I don’t know enough to comment.

But enough of work with simulations. How about the robustness of published studies? Unfortunately, there is not nearly as much as I would have hoped for, but what results there are are hopeful. The best one I found was McClelland and Gault (2017), which throws a bunch of stuff at Abadie, Diamond, and Hainmueller (2010) and finds it comes out alright. Otherwise, though, I was unable to find something systematically assessing the trustworthiness

To sum up: I think objection 1 is a serious one, but afflicts differences-in-differences too. Your choice of donors should be intuitively reasonable, and ideally you should show that your results are robust to taking in a different bunch of donors. Objection 2 is not, in practice, a serious one. Objection 3 and 4 are two sides of the same coin, and are a fundamental weakness. To the extent that you keep things transparent by only including one outcome variable, the more vulnerable you are to transient shocks; but the more you try to account for transient shocks, the more you can get pointless curvefitting. It seems solvable only by good judgement, just the same as differences-in-differences.

I think synthetic controls do have extraordinary potential. I entirely agree with people like Susan Athey and Guido Imbens when they describe it as one of the most important innovations in econometrics in this century. However, it does not replace the hard work of arguing for exogeneity that you find in differences-in-differences papers. Whatever your weights are, they should be intuitive and sensible in the same way that differences-in-differences should be.

Further, taking the results requires an element of trust, and quite a lot of work in trying out all the different possible specifications. For that reason, I think we should not use this for trivial questions. It would be much better for society to have a few, very good and incredibly rigorous synthetic control papers on the very most pressing questions, than a firehose of synthetic control papers on every event, no matter how trivial. Results should jump off the page, and you turn to synthetic control to figure out by how much, rather than to see if there is any result at all.