We really need better data

How else are we supposed to answer the big questions?

Anyone familiar with the literature on early-life educational interventions will have heard of ABC/CARE, and the Perry Preschool Project. And with good reason — the studies have long crossed into having more studies about them than they’ve had participants in the studies! The Perry Preschool Project had 123 participants — restricting my search to articles authored by James Heckman alone returns 148 papers written on the subject. The data is indeed incredibly detailed over a long period, which other studies don’t have — hence its over-representation. But isn’t it a bit silly we don’t have more?

Or consider our knowledge on the effect of additional health insurance coverage on health. We have had three big studies, one of which is quite old (the RAND healthcare experiment) and the other of which was too short-lasting to draw positive or negative conclusions (The Oregon Medicaid experiment). We also have the Karnataka Health Insurance experiment, which is also short-lasting and insufficiently powered to detect much. Otherwise, we have to rely upon quasi-experimental evidence.

Why must we do this to ourselves? The optimal levels of education and healthcare are enormous questions, accounting for 6 and 17 percent of our GDP respectively. Should we not test our guesses of the optimal quantities rigorously? We are perfectly happy to do this in developing countries. We have randomized controlled trials for all sorts of things — the effects of free bednet distribution, deworming, building more schools, stricter attendance standards in medical clinics, and on and on. Why not here?

It seems there are some questions we would prefer not be answered. We may secretly value the intervention for its own sake, regardless of whether or not it works. Later looks at preschool interventions find that they are less effective than the first wave was. While their data is not as detailed, and does not span as long a time period, they find considerably smaller positive effects. But to argue that, perhaps, on the margin, we consume too much education is unutterable in polite circles. 

Ah well. I always have hope, even when it is senseless. Government should aim to test the programs they implement. It should never, ever, just roll out programs without some provision to measure its effects. Data collection should be long-term, and detailed. It would be absolutely fantastic if we had a national ID system, and tied it into standardized national tests. I envy the data quality of some of the Scandinavian countries, which administer IQ tests to the entire population, and tracks income tied back to the person throughout life. This is not merely me, the economist, lamenting a lack of beautiful datasets. Trillions of dollars depend on knowing what works.

What I say about the first world goes doubly in the developing world. It is embarrassing the extent to which we do not know even the most elementary facts, despite substantial investments by the World Bank which have improved the statistical situation. In order to say what causes living standards to improve, we first must know what living standards are, were, and how much they are changing by. We do not even know, within tens of millions, how many people live in some countries. Nigeria, for example, has not had an honest census since 1962. The North, which had previously been predominant, lost its position, and so 8.5 million people were found in the next census, in 1963. Since then, in order not to disturb the detente between Muslim North and Christian South, the censuses since just added a fixed proportion to all areas. They haven’t even bothered pretending to have a census since 2006 — the one scheduled for 2023 was delayed by Buhari with no date to actually do it in site.

Now, it is plausible that the difficulties with cross-country living standards are intractable. Deaton’s 2010 AEA address is the best resource on this, beginning in section II (page 11). I would not say that it provides nothing at all, but figuring out the relative level of consumption when people in countries consume completely different goods requires some rather optimistic imputation. There are hardly ever identical goods between countries. If you did focus on matching identical goods, you’d likely overstate poverty too. Deaton gives the example of something like Frosted Flakes being sold in specialty stores at unrealistically high prices. The data collected has to be a random sample, and often isn’t — China’s data collection being biased toward urban areas probably meant they could revise GDP estimates up 10% (p. 35). (Prices are systematically higher in more developed places, owing to the Balassa-Samuelson effect). Large informal economies mean that you miss a lot of economic activity if you look only at what is reported to the authorities. The thing optimistic to come out of that, at least, is that estimates likely lead to an overestimate of poverty in developing countries. Alwyn Young (2012) finds that estimates based on actually going out and asking people what they consume lead to much higher estimates of growth, on the order of 3-4 times higher. (And this to growth rates! Not levels, rates! So it will compound!). This “going out and asking” is something which can be improved, and we must.

This reminds me of the literature on living standards in pre-industrial countries. Greg Clark estimates no changes in real wages in England from medieval times to just before the Industrial Revolution; Joel Mokyr estimates a six-fold increase between Hastings and 1800. Neither of them has fudged their numbers — Clark has simply chosen a narrower consumption basket, focusing only on the bare necessities. Every difficulty with modern day living standards is in economic history, but doubled — we can’t even ask people, but must rely on what people happened to write down. Heck, we’ve had times (see Stephenson 2018) where we’ve found that what we thought were wages weren’t wages at all! They were the payments to contractors, who then went out and hired laborers at wage rates entirely unknown to us. I am satisfied that for the biggest questions, we don’t need to know wage rates (see here for prior blog posts on the subject) but it’d still be nice to know when we started growing, and by how much, for the same reasons we want to know now.

There need be nothing inevitable about bad data. We can make getting better a priority. I am also optimistic about satellite data allowing far better estimates. (Reading Donaldson and Storeygard 2016 filled me with a hum of excitement, and a sense that I had seen the future). We need only make it a priority.