How Good Is the Research on Natural Immunity?

Ari Schulman: Brendan, in our last discussion, we saw that there is no clear agreement in the research that’s been done on whether natural immunity or vaccination offers stronger protection. A couple of studies show a big advantage for one or the other, and a handful show they’re very roughly similar. Today I’d like to think about how to make sense of this uncertainty — why it exists, and how we should act in light (or dark?) of it.

Let’s start by looking at that CDC study, which seemingly shows that vaccination is 5 times more effective than natural immunity. I want to pick on this not because of the result, but because it’s been leaned on so heavily by the CDC, and the study design seems deeply flawed.

To recap: The study looked at people who were hospitalized with Covid-like illness in the U.S. They were tested for Covid, and among people who had had a previous infection (meaning they were naturally immune), the positive rate was 5 times higher than the vaccinated group.

But the oddity is that the study isn’t looking at a random sample of the population. It’s looking at people who are already hospitalized — and with Covid-like illness, no less. That seems like a huge selection bias, doesn’t it?

Brendan Foht: It clearly is a big selection bias. So if you’re inclined to dislike the outcome of this study, you have ample room to pick apart its methodology. But that’s true of pretty much all of these studies.

If we wanted to be totally sure about how well natural immunity works, the “gold standard” of evidence would be a randomized controlled trial. But how would you set up an RCT for natural immunity? You’d have to randomly assign people to be either infected with Covid and recover from it, giving them natural immunity — or not. Then you’d monitor the two groups for weeks or months and compare their rates of infection, serious illness, hospitalization, and death from Covid.

But, well, there are really big hurdles to doing this. The biggest is the ethical one of deliberately infecting people.

Ari: Right — human challenge trials. I’ve heard proposals to do these to accelerate vaccine testing. So just for the sake of argument, why not do human challenge trials to study natural immunity?

Brendan: Well, it’s already controversial for vaccine testing. But at least there, what you’re learning about is a medical treatment that could help a lot of people. But that just isn’t the case for natural immunity. Asking people to risk their lives to help develop life-saving therapies is one thing — but nothing we find out about natural immunity from these studies will make it into a medical treatment.

Ari: So it’s just practically harder to understand how effective natural immunity is against Covid than it is for vaccines.

Brendan: Right. That’s because without RCTs, we’re left with observational studies of various kinds. These involve a lot of decisions about which groups of vaccinated and naturally immune people you are looking at and comparing. And those decisions will introduce room for selection effects and bias.

For instance, you might want to account for the effects of waning immunity by comparing people who have been vaccinated or recovered for the same amount of time. But in most countries, the first people to be vaccinated were particularly vulnerable to Covid infection, whether because they work in hospitals or because they are older or have some medical comorbidities. So the people who were vaccinated in January 2021 might be at higher risk for infection than the people who recovered from Covid in January 2021. You can try to control for this in various ways, but this just involves more slicing of the data in ways that could introduce other kinds of bias.

And because there are so many decisions to make about how to compare the vaccinated and naturally immune, it’s going to be very hard to compare different studies that ask these questions. Because there isn’t a single, clearcut way to get the right answer here — or even to ask the question — these studies are not comparing like to like.

And that’s how you get studies from Israel that find that natural immunity is 13 times better than vaccination and studies from the CDC that find that vaccination is 5 times better than natural immunity.

Ari: So how should we handle a situation where the science is highly uncertain?

This is a question we’ve focused on for a long time at The New Atlantis, so let me note a couple of the possible frameworks for answering it.

One answer is just to say that we should all do a better job at understanding the uncertainty itself. Scientists should not just tell us “what the research says” — they should go into more detail about the limitations of the methodologies. And (as I’ve proposed before), it should be standard practice to give the public “low, medium, or high” qualitative ratings of the confidence we should attach to a given body of research — sort of like weather forecasters do.

Or we could observe how divergent research creates a “choose your own adventure” situation, where the pro-vax side has its favorite studies, the pro-natural immunity side has its favorite studies, and everyone is convinced they’re right. We could say that this is where science actually makes disagreement harder by “scientizing” the debate, creating answers that diverge. Instead we should just focus on a policy solution that can get broad buy-in.

Brendan: Science might make disagreement harder, but I don’t think there’s any way to avoid it. How can you ask people to debate questions about natural immunity and vaccination without bringing up their beliefs about how well these two forms of immunity work at preventing Covid? And it doesn’t really make sense to get those beliefs from anywhere but science, broadly speaking.

One thing we can say confidently here is that the quality of the science we do have is lacking. Instead of settling for “choose your own adventure,” why not demand better answers? Research funders could be doing a lot more to set up high-quality studies on this question.

This is one of the cases where you want to go with the Dan Sarewitz approach to science rather than the Vannevar Bush approach. This isn’t a “basic research” question to be solved by “the free play of free intellects,” where you allow very smart people to investigate whatever appears interesting to them in the expectation that some of them will hit on some kind of unexpected breakthrough. Instead, an answer here is going to require well-organized collection of data at a large scale over an extended period of time. You want the government to define the question and then provide scientists with the resources and organizational capacity to answer it.

The UK study that follows a randomly selected set of households, sending survey questions and asking people to be tested regularly for Covid, seems like a good example of what we should be doing more of in the United States. Conducting that kind of study doesn’t take brilliant scientific insights, but it does require a lot of organization and planning on the part of the researchers and a lot of trust and willingness to participate on the part of the public. Arranging for all that is much more like a political task than a scientific one, and we should be demanding more effort on the part of our political leaders to get it done.

Ari: Okay, enough dancing around, no more angels on the head of a pin. I’m President Biden and I’m trying to deal with this politically contentious question, and you’re my chief science advisor.

Channeling James Gandolfini in Zero Dark Thirty: Tell me simply, is natural immunity from Covid as effective as vaccination or isn’t it?

Brendan: If we’re looking to craft policy right now, without further evidence, I think we should treat them as being similar instead of saying we don’t know.

When you look at the evidence we have, there are some widely diverging estimates suggesting that vaccines are much better than natural immunity or that natural immunity is much better than vaccines, and a larger set of studies suggesting that they are pretty similar. Given all the uncertainties around all of this research, it doesn’t seem like we have any really convincing evidence that either vaccines or natural immunity are superior, and so the most reasonable position to take for now is that neither provides markedly stronger immunity than the other.

There are important limits to this conclusion, and their relevance depends on which particular policy question you’re trying to answer.

Ari: We’ll look at those policy questions next time.