Epidemiologists, government failure, and COVID-19

by
Eric Kramer
My articles My booksMy videos
Follow on:

Jason Brennan has a new post up doubling down on his earlier criticism of epidemiologists and government policy in response to the COVID crisis.  I responded to his earlier blog posts here.  I am still not convinced, but there are useful lessons to be learned from going through his argument.

Brennan continues to claim that epidemiologists produced bad statistical analysis, and that we should not take their advice seriously (all bolding is mine):

I’ve been criticizing epidemiologists–including the ones publishing in JAMA, the Lancet, NEJM, etc., and the famous ones who were making apocalyptic predictions on TV last month–for doing what is clearly bad work. My main complaint is, again, that their estimates about the danger of the virus are based on the wrong data (current infections) collected the wrong way (non-random testing of people who present themselves as sick). We all know better than that. You don’t sample on the dependent variable. You don’t sample in ways that suffer from severe selection bias. If you mostly test people who show up saying they are sick, and 3.4% of them die, it doesn’t tell you how many people have the infection, nor does it tell you what percent of people who have the infection will die.
Now, while many economists and others trained in stats have been saying the same thing, it’s surprising how many untrained people say we should instead defer to epidemiologists. You can see some of their arguments on Facebook and others in the comments to previous posts.

He does not repeat the suggestion in his earlier post that decisions based on analysis from epidemiologists “presumptively lack authority and legitimacy”, but he doesn’t withdraw it either.

Here is one way to think about the claims Brennan is making.  Suppose the following statements are true:

• Epidemiologists overestimated the risk of COVID-19 by applying bad statistical methods to poor data
• Politicians adopted costly lockdown policies based on flawed risk estimates produced by epidemiologists
• Lockdown policies were unjustified at the time they were adopted, and this could have been known if better statistical analysis had been undertaken
• Lockdown policies are unjustified now, given currently available information, and should be replaced by a different policy

If these things are true, we have grounds to

• Criticize epidemiologists for using bad statistical methods
• Criticize politicians for listening to epidemiologists
• Criticize the initial decision to impose the lockdown policies, and attribute that bad decision to poor research by epidemiologists
• Advocate for repeal of the lockdown policy now (and perhaps its replacement with a different policy)

The problem is that the empirical and moral claims in this argument are at least debatable, and some seem to be clearly wrong.

Did epidemiologists overestimate the risk of COVID 19? 

I’m sure some did.  I’m sure many didn’t.  Brennan himself admits that “we all know better” than to estimate case fatality rates by dividing deaths by diagnosed cases.  Every epidemiologist I know is aware of this.  If Brennan wants to claim that poor statistical thinking led key epidemiologists – either leading academics or government advisors – to overestimate the risk of COVID-19 or to rely on papers that did this, he should roll up his sleeves and produce some evidence for his claim.

In fact, he does cite one piece of evidence, obliquely:

Yes, if you go digging back through the testimony, you see a small number of people saying, “Oh, sure, 3.4% of people diagnosed with COVID died, but we have little to no idea what the infection fatality rate is because of our poor testing procedures. We’ve been testing for the purpose of helping the sick rather than for getting data to estimate the dangers.” But what you mostly see with the Imperial College people, Fauci, and others, is making mass projections based on the crude case fatality rates calculated from bad data.

This is very misleading.  Go look at the Imperial College Study.  They did not use a crude case fatality rate of 3.4%.  They used a case fatality rate of 0.9%.  Based on, you know, a study by epidemiologists.  (The Imperial College Study was quite influential – it was the main source of the claim that 2 million might die in an uncontrolled epidemic.)

Did politicians listen to epidemiologists and adopt the current lockdown policies based on their advice?

My sense is that epidemiologists – many of them, at any rate – wanted governments to act faster than they did, to limit the prevalence of the disease so that it would be manageable with test/trace/isolate policies.  Governments messed up because they did not listen to epidemiologists.

Brennan is also completely ignoring the actual political world we live in, viz., one in which Donald Trump is president.  The reason so many people are saying “Listen to the experts/epidemiologists” is because they are worried that Trump will re-open the economy too soon.  They hope that the public will be inclined to trust the epidemiologists and resist pre-mature re-opening.  We can debate the wisdom of this political strategy, but Brennan simply ignores what is actually happening.

Were lockdown policies unjustified at the time they were adopted? 

I don’t think so, and Brennan provides no evidence for this claim.  (Criticizing the data and statistical work available at the time the decision to lockdown was made is not the same thing as showing the policy was unjustified.)

Are lockdown policies unjustified now?

This is the key question.  We need to decide what policies to pursue today.  Brennan contributes nothing our understanding of this question.  Instead, he focuses on the “fact” that epidemiologists are bad at math and the “fact” that politicians relied on epidemiologists to decide what to do in response to the epidemic to argue that the initial decision was based on poor information.  Both of these “facts” seem not to exist, but even if these claims are correct, so what?  The initial decision to institute a lockdown is is a sunk cost.  We need to focus on what to do now.

In his original post, Brennan claimed that the lockdown policy is presumptively illegitimate and lacks authority.  What sensible people care about, however, is whether the policy makes sense now, regardless of whether the decision-making process leading to the adoption of the policy was imperfect (which of course it was, as it is with almost every policy).

Finally, Brennan believes that there has been a massive government failure in connection with COVID-19:

I can understand shutting down everything temporarily in an abundance of caution.  [ed note:  his earlier post seemed to suggest that the lockdown decision was not “legitimate” or “authoritative”]  But states are immediately obligated to collect the right kind of data the right way, so we can get a proper estimate of the real dangers and make decisions competently. They haven’t done so. The past month has seen government failure on a mass scale.

Sure, but so what?  Every academic observer I am familiar with agrees with the need for better data, and the government is indeed preparing to do population studies.  And most observers believe that most governments have performed poorly in connection with COVID-19.  The practical question this raises is what to do about this government failure.  Brennan has nothing to offer here.  Having a moral philosopher criticize epidemiologists (unfairly, in large part) for not understanding statistics will not help us make better decisions in this epidemic, and it will not help us make better decisions in the next epidemic because bad statistical analysis by epidemiologists was at most a tiny contributing factor the failures that have allowed the epidemic to get out of control and have hampered our ability to respond effectively.