One of the most vexing questions of the coronavirus pandemic has been how many people have actually been infected. We know that testing has been inadequate, and that many cases of the disease are mild or even asymptomatic, making them less likely to be detected. So how many cases have slipped under the radar?
One way to find out the true prevalence of the disease is to test random samples of the population using a blood test that detects antibodies produced by the immune system against the virus. This is different from the swab tests that have been used worldwide throughout the pandemic, which detect the genetic material of the virus itself.
Blood tests, which require just a pinprick, can be done at the point of care, and so are faster and easier to process than the swab tests, where samples need to be shipped to a lab. Blood tests are also helpful because the prevalence of the disease can be assessed from a larger population, which can include people who have recovered from the disease, as the antibodies it detects remain in the blood even after recovery. However, because antibodies take several days after infection to be produced, these tests cannot replace swab tests for the very important purpose of detecting new cases of the illness before it can be spread to others.
Much of the attention on antibody tests has focused on how they can help us estimate just how deadly Covid-19 really is by giving us a better sense of the total number of infected people, and thus of the true rate of infected people who died. If the number of infected people is much larger than expected because there are many undiagnosed infections, that means the probability of dying from an infection is much lower than it would be if we looked only at the number of diagnosed cases. Some have hoped that antibody tests could show us that the virus “isn’t as deadly as we thought,” and may therefore inform “better policy decisions” about restrictive social distancing policies.
There have been several recent attempts to use these blood tests to determine the prevalence of the disease, and with it its true fatality rate. One that has garnered a great deal of attention has been a study conducted by Stanford researchers in Santa Clara County, the metropolitan region around Silicon Valley. By taking blood tests on subjects recruited from a Facebook survey, researchers estimated the prevalence of infection in the community to be much higher — and the fatality rate to be much lower — than confirmed diagnoses would indicate.
In the initial preprint version of the paper from April 17, they estimated that between 2.5 and 4.1 percent of the county had been infected, 50 to 85 times the roughly 1,000 cases that had been diagnosed at the time. Based on those numbers, they estimated the infection fatality rate of the disease to be between 0.12 and 0.2 percent, far lower than the roughly 5 percent fatality rate for diagnosed cases in the United States, and not much higher than the 0.1 percent fatality rate for seasonal influenza.
The study was widely criticized at the time, and although the researchers published a revised version of the paper addressing some of these criticisms on April 30, their topline results have not changed much: Their estimate for the prevalence of the disease in Santa Clara County is now between 1.3 and 4.7 percent (extending their earlier range), while their estimate for the infection fatality rate is now 0.17 percent (within their earlier range).
Criticisms of the paper centered on two major weaknesses. First, the authors reported a relatively high false positive rate, which is particularly problematic in a place like Santa Clara that has not seen a major outbreak. Second, critics voiced concerns that people who suspected they may have had undiagnosed coronavirus infections may have been more likely to participate in the survey, biasing the results. Let’s look at each of these in turn, and at how the researchers responded to these concerns in their revised paper.
The researchers originally reported that the lab tests they ran to try to validate their test kits prior to use had a 0.5 percent false positive rate. This may not seem like much. But in their actual survey, just 1.5 percent of tests came back positive. (Their higher final prevalence estimate was made by adjusting for demographic factors in their sample.) This would suggest that a third of their positive results may have been erroneous. But 0.5 percent is just a “point estimate” of their false positive rate, based on how many errors they observed in the limited number of validation tests they performed, and the actual false positive rate could be higher or lower.
Critics argued that it’s statistically plausible that the false positive rate could have been even higher than the number of positive results the researchers found. This would mean that the positive results they found would be consistent with there being zero true positive cases. Of course, no one believed that there were literally no true positive cases of coronavirus in Santa Clara County. But when the false positive rate is statistically compatible with the number of positive results detected, this casts doubt on the quality of the evidence provided by the test.
In their updated paper, however, the authors ran a larger number of tests to validate the performance of their kits to provide a more precise estimate of the false positive rate; their point estimate for the false positive rate is still 0.5 percent, but with an upper bound of just 0.8 percent and a lower bound of 0.3 percent, which can give us greater confidence of the accuracy of their results.
The second area of concern is about sample bias in the volunteers that participated in the study. The subjects were recruited through a Facebook survey, and then drove to testing sites. It is plausible that many of those who decided to volunteer for a free coronavirus test were people who had some suspicion that they had already had the illness without being tested. In the updated draft of the paper, the authors report that 3 percent of their subjects had experienced cough and fever within the past two weeks, while 20 percent had symptoms within the past two months, and that those who reported symptoms were more likely to receive positive antibody tests than those who did not.
The authors speculate that “the proportion of participants with cough + fever may actually be lower than what is typical in California, and, if so, the bias in the estimation of prevalence may be even in the opposite direction.” But they give no indication of what is the typical proportion of the Californian population who report symptoms of fever and cough. Their source for the claim is the CDC’s map of the historical levels of influenza-like illness across the country, but that map only reports relative levels, from “minimal” to “very high,” rather than the numeric prevalence of the illness. And the underlying data for that map from the California Department of Health is based on flu hospitalization rates rather than the total number of people with flu-like symptoms. That being said, it is not clear how much of an effect this sample bias may have had on their estimates.
In their updated paper, the Stanford researchers have addressed some of the more important criticisms of their initial estimates of the prevalence of Covid-19 in Santa Clara. However, their finding about the total infection fatality rate remains difficult to square with the observed death rates in New York.
According to the New York City Department of Health, as of two days ago, there have been 13,938 confirmed Covid-19 deaths, with an additional 5,359 probable deaths. The city’s population is about 8.6 million, meaning that if we only include confirmed deaths then 0.16 percent of the entire city has already died from the virus (or 0.22 percent if we include probable deaths). If the infection fatality rate for the virus were really just 0.17 percent, as the Santa Clara study estimated, that would imply that virtually the entire city has already been infected. This is hardly plausible, both because herd immunity would likely stop transmission before everyone in a region is infected, and because antibody tests in the city in fact show much lower rates (on which more below).
If we include probable deaths, and assumed that every single New York City resident has indeed been infected, then the fatality rate could not be as low as 0.17 percent. (This is not to mention that Covid-19 deaths are most likely undercounted. In New York City alone, between March 11 and May 2, there have been 23,000 more deaths than would normally be expected, while only 18,706 deaths have been reported from Covid-19, suggesting that up to 4,300 additional people may have died from the virus.*)
The Stanford researchers acknowledge this wrinkle in their finding in the updated version of their paper by noting that the fatality rate may be “substantially higher in places where the hospitals are overwhelmed” or “where infections are concentrated among vulnerable individuals.” This points to the limits of ascertaining infection fatality rates from regions with a low prevalence of the disease. It also suggests that decisions about relaxing efforts to mitigate the spread of the virus should not be based on serological tests only from areas that did not see a large outbreak, since relaxation of mitigation efforts could lead to larger outbreaks with not only higher total numbers of deaths, but also higher death rates for those infected.
If we want to get a better sense of the fatality rate for the virus, we should look instead to hard hit regions like New York. The state has been conducting a large serosurvey of its own. The survey is ongoing and details have not yet been published. But according to an announcement from Governor Cuomo on April 27, at which point 7,500 people had been tested, nearly 25 percent of New York City, and nearly 15 percent statewide were found to have antibodies against the virus.
To select samples, the state set up testing sites in public areas like supermarkets, which might mitigate the self-selection bias in the Facebook-survey method used by the Stanford researchers. On the other hand, people who are walking around in public may be a higher risk group than those who are scrupulously self-isolating.
Details about the accuracy of the test are also limited: one document indicates that the specificity of the test ranges between 93 and 100 percent, which means that as many as 7 percent of the positive results could be errors. But because the actual prevalence of the disease is so much higher in New York than in Santa Clara, such false positives don’t make as much of a difference to the validity of the results. Because there are so few people with antibodies in Santa Clara (roughly between 1 and 5 percent), most of the positive findings in a test with a relatively low false positive rate might well be errors. But in New York, which clearly has had many more infections, we can expect that false positives won’t come close to outnumbering accurate positive findings.
The prevalence estimates from the survey in New York suggest that there have been about 10 times as many infections as regular diagnostic testing has found. This wouldn’t be terribly unrealistic, given the shortages of tests in the state, a third of which have come back positive. These estimates would put the infection fatality rate at 0.69 percent for the state or 0.66 percent for New York City. This would be far better than the fatality rate for confirmed cases (roughly 6 percent for the state), but would still be high enough to be very worrying: If we were to allow, say, half the country to be infected, then that fatality rate would result in over a million deaths.
While estimates of the infection fatality rate of Covid-19 obviously hold a grim fascination, they are not the most important thing we can garner from antibody studies. As the discrepancies between the fatality rate estimates in the Santa Clara and New York studies show, surveys in different regions at different stages of their outbreaks may lead to very different results. More importantly, there is not much we can do with information about the infection fatality rate.
What difference does a fraction of a percentage point in the rate make to the actions we need to take to control the pandemic today? Do we open up malls at a 0.5 percent fatality rate, and wait until a study comes along that shows a 0.3 percent rate before we open hair salons? These estimates don’t tell us anything about how to make malls and hair salons safe to visit again, they merely tell us that we might not have to worry about doing so. They are tools for managing perceptions, not for managing the crisis itself.
One potentially useful application of antibody tests could be in identifying people who have been exposed to the virus and are now immune, who can then work in higher-risk settings. Such “immunity passports” are controversial, not least because we still do not know with certainty whether, or for how long, the presence of particular antibodies against the virus will confer immunity. And of course the idea of issuing immunity passports raises a host of ethical and social issues — from discrimination against those lacking immunity to the perverse incentive they may create for people to seek out infection to acquire immunity. But as antibody testing improves we will need to decide whether as a society we want to use the tests to issue immunity passports, and how we might do so in a way that does more good than harm.
Another more immediate use of antibody testing could be finding recovered patients to serve as donors of antibody-rich blood serum, so-called “convalescent plasma” or “convalescent serum,” which some scientists have argued could be used to treat the disease or to confer short-term immunity to the virus for high-risk workers. Data from clinical trials of the treatment will be available soon, but early reports from its use in severely ill patients are promising, and the idea has a century-old track record of safety and efficacy for treating infectious diseases. However, finding recovered patients who are eligible and available to donate can be a logistical challenge, and each donor can provide only enough serum for treating 2 or 3 patients, so its usefulness is limited by the number of prospective donors we can identify.
As of this writing, we know of only about 195,000 recovered patients in the United States — about one for every six currently ill patients — so finding large numbers of recovered patients through antibody testing could allow for this treatment to become more widely available for those who need it the most. The simple tests used in these surveys may not be enough to determine if people have sufficient levels of antibody in their blood to serve as donors, but it could at least help identify prospective donors who could receive follow-up tests to determine their eligibility. As long as there are millions of unidentified people who have recovered from Covid-19, we should not allow a shortage of donors to hold back the availability of convalescent plasma therapy.
The findings from antibody studies remain uncertain and we will need more of them and more and better conventional testing to get a clearer sense of the true prevalence of the coronavirus epidemic in the United States. While we wait for such research, we should prepare to take advantage of the knowledge we might gain from it while continuing to build up our capacity to control the disease’s spread. Because whatever we learn from these studies, the cost of letting the disease spread unchecked would be unacceptably high.
Exhausted by science and tech debates that go nowhere?
