Sunday, January 31, 2021

How not to calculate excess mortality

I got in a twitter argument with someone about COVID19 and they threw a surprising stat at me. South Korea had over 20k excess deaths this year. This made no sense to me. SK is maniacal about testing, their official COVID19 death toll for 2020 was 917. Did they miss 20x that many? Was there some other big killer? Is it a statistical blip? The answer is "none of the above".

The source was this WSJ article. It's pay-walled but the key information is this info-graphic which purports to show that many countries have vast quantities of excess death, above their official COVID numbers.


So the world has massively under-counted COVID19 deaths? Probably but the other key information is how they calculated excess deaths

Methodology: To analyze the pandemic’s toll, the Journal compiled weekly or monthly death data for 2020 and for 2015-19, where available. Most of the data was collected from national statistical agencies, either directly or indirectly through inter-governmental or academic groups. In a handful of nations, data was collected by health data organizations or local analysts. Epidemiologists use several methods to calculate excess deaths, adjusting for age composition, incomplete data and other factors. The Journal used a straightforward method, summing deaths for the portion of 2020 available and subtracting from that total the average number of deaths that occurred in the same span of each year from 2015-19. When the result falls below zero—when the 2020 death total fell below the average—some countries adjust the result to zero, boosting excess death totals. The Journal did not adjust in those cases. All totals are based on actual counts and comparisons. For some nations, the average was based on three or four recent years, typically 2016-19.

I have bolded the important part.

Unfortunately this straight-forward method is a fundamentally flawed methodology (did they not talk to an epidemiologist before publishing?). It ignores the fact that most countries have underlying mortality trends due to their demographics. Using their methodology South Korea has +24k excess death in 2020 but guess what, it had +21k excess death in 2019! This is what SK's recent excess deaths look like with WSJ's methodology. Here is the sheet if you want to explore.


As you can see, SK's mortality is rising pretty rapidly, presumably due to a population explosion in the 50s and 60s. I believe most countries are similar. This makes all of the numbers in the article somewhere between questionable and meaningless.

Applying their methodology to the whole world, we get an excess of 1.7M in 2020 and 1.3M in 2019. They did a subset of the world and got 2.8M which is also interesting, I don't know where that discrepancy comes from.

So while the world has surely under-counted official COVID death, WSJ's figures could almost be anything, an over-count or an under-count. What's bizarre is that they said "Epidemiologists use several methods to calculate excess deaths, adjusting for age composition, incomplete data and other factors" and then proceeded to just do it wrong anyway.


Addendum: New Zealand is quite similar. Here's the data



No comments: