How to understand the basic statistical concept the FDA chief flubbed

At issue is the difference between absolute and relative effects. Imagine a headline that reads, “New drug reduces covid-19 deaths by 50 percent.” That sounds like a huge effect and might lead one to expect, for example, that if 100 covid-19 patients took the drug, it would save the lives of 50 of them.

But that’s not right, because covid-19 doesn’t kill everybody to begin with. Let’s say, purely for the sake of illustration, that the coronavirus has a mortality rate of 10 percent: If 100 people are diagnosed with the disease, 10 succumb to it.

So if the drug reduces covid-19 mortality by 50 percent, the number of fatalities among the 100 patients taking it drops from 10 to five. In other words, five lives are saved, not 50.

“New drug lowers covid-19 mortality by 50 percent” is a statement of relative effect: Mortality is reduced by half relative to the baseline. To know what the absolute effect is, you need to know that baseline.

Unfortunately, that information is usually not included in news coverage of scientific results, in large part because the researchers writing the underlying reports rarely attempt to characterize absolute effects. Academic studies are written primarily for academics, who are often, and understandably, more interested in the narrow effects of a given intervention, rather than the much broader implications for an entire population.

Taken to an extreme, however, mistaking relative for absolute effects can lead to falsehoods, as when Hahn claimed on Monday that among “100 people who are sick with covid-19, 35 would have been saved because of the administration of plasma.” Hahn took a statement of relative effect — “Plasma treatment reduces covid-19 mortality by 35 percent” — and attempted to recast it as a statement of absolute effect.

Hahn forgot to ask the critical question: 35 percent of what? If he had done that, he probably could have avoided misinforming his audience.