Africa is Doing Worse Than You Think

Why harmonized learning outcome measures overstate education in Africa

I think everybody believes that education matters, at least a little, for the economic prospects of a country. If people don’t have skills, they can’t do things. How can someone be an engineer if they don’t know calculus? What we still don’t know is how important it is, really, and what we can do to improve. I do not have a solution to this, so I will instead address one tiny corner – the datasets we use are kinda just bad, and bad to the point that we will believe false things about the world.

Specifically, in order to convert the achievement tests in Africa to something internationally comparable, we need to make assumptions which are almost certainly not true. The result is that the educational performance of every Francophone African nation is much too high, and that education is more important than we think. Further, countries like India, while performing much worse than the developed world, are not actually on par with sub-Saharan Africa, and it is reasonable to be more optimistic about India’s growth prospects.

The first attempt to try and link “human capital” to growth was Mankiw, Romer, and Weil (1992), who propose adding it as a factor of production in a Solow model, and seeing how much of growth they can attribute it to the variable. (For more detail on the Solow model, see here). They find it is surprisingly effective at explaining growth, although attributing causality is difficult. (Do countries get rich because they school, or do they have school because they are rich?). However, their measure of education was really bad. It’s just the average percentage of the population 15-19 that was in secondary school between 1960 and 1985, which completely misses variation in primary school. So, later, better data sets would use years of education as the variable.

But of course, even this is inadequate. The point of school is to learn things. Countries can vary in the number of years they spend in school without varying in what they teach in the end, and they can spend the same number of years for vastly different academic outcomes. The ideal measure of skills in a country is measuring skill itself.

Countries do not all take the same tests, unfortunately, although there have been serious attempts to have many countries take the same tests. The OECD sponsors PISA, which is the gold standard for how to do this. The test is written to be the same for each country despite the language changes, with each student receiving a random draw from a bank of questions, allowing them to adjust for bias in a particular question. The tests are linked from year to year by the use of common test items, and the scores are normed such that 500 was the mean when first administered, and 100 the standard deviation. Other measures of learning will also adopt this norm, so please remember that that is what the scores mean.

Not everybody takes this test, though, and countries have been known to withdraw rather than face the music. Consider India, who has only participated in one round of internationally comparable testing, PISA in 2009. It was a disaster and a national embarrassment. India performed much worse than expected, worse than every other country except Kyrgyzstan. They were so embarrassed by this, in fact, that they have pulled out of every subsequent round of testing, pretending to want to come back for 2020 only to claim “Covid disruption” and cancel it again. Still other countries, including basically all of Africa, have never taken PISA at all. Instead, they have regional tests – PASEC for Francophone countries, and SACHEM for Anglophone ones. What to do?

Patrinos and Angrist (2019) try to construct a set of Harmonized Learning Outcomes for the entire world, and certainly make a brave attempt of it. You can see a map of their learning outcomes through Our World In Data. The key to harmonizing the different tests is to look at countries in which the same tests were administered, and then compare the relative performance of the students on the different tests. This then allows them use these “doubloon” countries to convert from one test to another.

So what does this look like in practice? Take Gabon. (Gabon is a Francophone country mostly on the equator. While it notionally has a relatively high-GDP for the region, this is entirely absorbed by the elites, and the country is very poor. We would have no reason to expect their performance to be exceptional). Gabon took PASEC in 2014, so first we need to convert backwards in time to 2006. This is done through Togo. Then, we take their score and use Mauritius to convert from PASEC to SACHEM. That’s still not enough, though, because to make internationally comparable we need to go through Botswana (which did not take PISA, but does take TIMSS, which is similar).

Things get worse than just pure noise, though. Mauritius, while French is common, uses English as the language of instruction. If they do better on the English test than the French test, then the results of every single Francophone country will be biased upwards. And, well…

What possible common factor might lead Senegal to overperform The Gambia, Burkina Faso and Cote D’Ivoire to outperform Ghana, and Cameroon to outperform Nigeria? Who knows, it’s a mystery. If you take this seriously, you would think that Gabon is outperforming not only India, but also China and most of Eastern Europe.

One of those countries (Senegal) did take a variant of PISA, PISA-D, which is designed to be easier so that we can get more differentiation at the very bottom of the testing scale. By this scale, they scored 412 on normalized outcomes; by PISA-D, they got 304, a full standard deviation worse. (And it’s worth noting that the mean for PISA is a bit higher than TIMSS, which calibrates its mean to a worse sample of countries, so it’s actually a bit too high). When you get down this low, the test becomes unable to distinguish how bad learning outcomes are. You start to bump into the floor of just guessing.

I do not think that this approach of converting from country to country using such tenuous links is useful.The best way to do it is from Sandefur and Patel (2020), who give students in Bihar questions from multiple tests in order to directly compare performance with the same sample at the same time. This is excellent, and we should do more of it.

The other takeaway is that countries in sub-Saharan Africa are doing much worse on educational outcomes than you think. India might be doing poorly, but sub-Saharan Africa is barely on the same scale. This biases our measurement of the influence of learning on economic outcomes through two channels. First, since the African nations are largely at the bottom end of both education and economic outcomes, overstating their education will bias the slope of a linear regression down. Second, even pure noise in the x variable, education, will also bias the slope of an ordinary least squares regression down. (This is because OLS is trying to minimize the squared vertical distance from each datapoint to the line. If you shift a data point a little to the right or a little to the left, these do not have equal effects because the distance is then squared. Measurement error in the Y axis does not have this same effect).

This leads me to have a very pessimistic view of the growth prospects of sub-Saharan Africa. I do not think we can expect them to have the same takeoff into sustained growth as the East Asian miracles, or the post-Soviet countries. I think that they are still afflicted by the long reach of geography.