Correlation, Causation, and Confusion
Clearing up some misconceptions about statistics in science and everyday life
One weekend while on call at the hospital before the Covid-19 pandemic, three out of four of the severe stroke patients I saw developed a clot in the right middle cerebral artery of the brain. In comparison to other weekends — when, at most, one of these patients shows up in our hospital — this was shocking. In just a few days, these strokes suddenly became far more common. Was there some miasma in the air that weekend leading to clots in that particular artery? Were we witnessing an epidemic of right middle cerebral artery strokes? Highly unlikely. While it was certainly unusual, one could not make reliable assertions about why this sudden increase was occurring. In order to conclude anything definitive about a disease or a treatment, we must study a large group of patients over an extended period of time. Otherwise, the apparent trends we observe are likely due to chance alone rather than reflecting reality.
Back in March we knew little about the novel coronavirus except that it was contagious and deadly. We watched helplessly as thousands of Covid-19 patients overwhelmed hospitals in regions of China, Italy, Spain, and the United States. Entire countries shut down to mitigate the contagion. Researchers and clinicians sought guidance on how to deal with, think about, and treat this new infection. Case reports and short papers elucidating the nature of the pathogen flooded prestigious scientific journals. To give some sense of the magnitude of Covid-19 research published in 2020: As of this writing, searching for “Covid-19” on PubMed, the central repository of scientific literature, yields over 67,000 results.
The media, in its attempt to keep up with the rapidly changing science, translated findings for the public. Unfortunately, their analyses often lacked validity as they threw caution and uncertainty to the wind and exaggerated what little information scientists had. Such hyperbolic reporting obfuscated our tenuous grasp on reality — and ought to serve as a cautionary tale for science and medicine, and our interpretation of it, in the future.
One example of media exaggeration stands out in my own area of study and practice. In April, the New England Journal of Medicine (NEJM) published a short report documenting a series of five young patients (ages 33 to 49) with Covid-19 and large-vessel stroke over the course of two weeks. Frighteningly, the authors noted, “By comparison, every 2 weeks over the previous 12 months, our service has treated, on average, 0.73 patients younger than 50 years of age with large-vessel stroke.” Moreover, these five were not patients with severe Covid-19; they were only incidentally found to have the disease. Did Covid-19 cause an astronomical increase in the incidence of stroke in younger adults without coronavirus symptoms?
If so, this would affect treatment of the disease. For example, we might prescribe blood thinners to prevent such ischemic strokes, or strokes from clotting. I know some physicians who offered to put younger patients on blood thinners just in case they contracted Covid-19. But a study in August in Neurocritical Care showed that a different kind of stroke — hemorrhagic, or bleeding, stroke — in hospitalized Covid-19 patients is associated with the use of blood thinners, leading to increased mortality. Also, if the risk of stroke among the young were clear and intolerably high, we might be more hesitant to open schools and resume sporting events. So it’s important to get the possible connection between Covid-19 and stroke in young patients right.
The NEJM report caused near-panic in the press. A Washington Post headline read, “Young and middle-aged people, barely sick with covid-19, are dying of strokes.” The article quoted a physician, one of the authors of the NEJM report, stating: “This is much too powerful of a signal to be chance or happenstance.”
The reporter also referenced another study, then underway, of patients treated for “large blood blockages in their brains.” At the time, forty percent of those patients were under the age of 50. Forty percent sounds like a lot, but it was forty percent of only twelve patients. Although the Post article mentions the tiny sample size of the study, it offers no caveat about how this weakens any conclusions one might draw.
Similarly, CNN published an article on its website under the headline “Covid-19 causes sudden strokes in young adults, doctors say.” This article also quoted one of the authors of the NEJM report explaining that their work shows “a seven-fold increase in incidence of sudden stroke in young patients during the past two weeks,” all of whom tested positive for the virus, and that “the virus seems to be causing increased clotting in the large arteries, leading to severe stroke.” In addition to ignoring the importance of “seems,” the reporter failed to quote any skeptics, or even any physicians or scientists not involved with the paper, and did not caution the reader about the vanishingly small time period and sample size. Instead, sensationalist headline writers jumped to the conclusion that the virus causes strokes in young adults, and attributed this claim to the doctors themselves.
Even Medscape, a go-to source for many physicians, gravitated toward unwarranted certainty. Its brief article, entitled “COVID-19 Linked to Large Vessel Stroke in Young Adults,” describes the NEJM report, quoting one of the authors. However, here, too, the journalist offered no caution about the preliminary nature of the data.
By contrast, a U.S. News and World Report article on the NEJM report — to the credit of its author, Dennis Thompson — quoted Dr. Fadi Nahab, an associate professor of neurology and pediatrics at Emory University School of Medicine in Atlanta: “It’s unclear to me at this point if COVID-19 is definitely increasing the rate of stroke, based on these small numbers.” This type of reporting was rare; in most news stories, the findings felt definitive.
New data emerging since April has not resolved the question but only increased our uncertainty. Recall that the Washington Post article mentioned research indicating that forty percent of large-vessel stroke patients with Covid-19 were below the age of 50. But when the researcher quoted in the article later published a paper on the same subject, only one out of twelve stroke patients with a Covid diagnosis was below the age of 50. Although it was a seemingly minor correction, with the tiny study size, the share dropped dramatically, from forty percent to eight percent.
Here is one possible explanation for the discrepancy: There were actually fourteen patients overall in the study: twelve with stroke, another two with similar pathologies. Of those fourteen patients, six — or about forty percent — were age 55 or younger. So it may well be that some small miscommunications between the researcher and the reporter — about the age cutoff and whether one was considering only stroke patients or the entire group — led to the discrepancy between the Post article and the published study. If so, this shows how much of a difference seemingly arbitrary decisions about how to present data can make for studies with small sample sizes.
Larger studies have cast further doubt about a strong link between a diagnosis of Covid in the young and stroke. In a study from the Northwell Health hospital system in New York, researchers compared over eighty stroke patients with Covid to a group of stroke patients without Covid from one year prior, and found “no significant differences” with respect to age. In a large study at NYU, researchers looked at 3,556 hospitalized patients with a diagnosis of Covid. Thirty-two of them suffered stroke. The number of patients below the age of 50 was seven — or 22 percent of stroke patients. At the University of Pennsylvania, our group studied 844 hospitalized patients with Covid, finding that twenty suffered stroke. Of those patients, the number below the age of 50 was just one — or 5 percent of stroke patients.
We ought to keep a couple of caveats in mind when considering these numbers. First, due to the nature of data collection in hospital electronic records, these studies account only for hospitalized patients with Covid, not everyone who is diagnosed with Covid, to say nothing of everyone who is infected. Second, to ensure that datasets are large enough to allow for statistically significant results, researchers look at the entire population of Covid patients, and then at the subset of that population with stroke, rather than dividing the population of Covid patients by age and then analyzing the age groups by whether or not they suffered stroke. This is why the studies report how many Covid patients with stroke are under 50, rather than how many Covid patients who are under 50 have stroke. Consequently, they cannot directly answer the question that young people will find concerning: If I am hospitalized with Covid, what are the chances that I will have a stroke?
In all likelihood, of people under 50 who have Covid, the number that suffer stroke is extremely small. This can be gleaned from a few data points. Hospitalization rates for younger people with Covid are much lower than for older people, and somewhere in the area of forty percent of infected people are asymptomatic. People who are not hospitalized with severe Covid, or who are not even symptomatic — that is, most young people who are infected — will, with very few exceptions, not suffer strokes.
While many of the early reports of stroke in young patients were based on very limited evidence, this does not mean that there is in fact no elevated risk of stroke for young Covid patients. A Lancet review in September found evidence to support a possible association between Covid and increased stroke risk in younger populations, though the authors hesitate to conclude this definitively. In sum, there is probably an increased risk of stroke in Covid patients, especially in older patients with underlying risk factors, but it remains unclear how much higher the risk may be for younger people. Regardless, even if we end up finding an increased risk for the young, the media were wrong to draw such conclusions in April. The numbers should have given everyone pause from the get-go.
Blowing up flimsy data affects decisions about medical care and the public health response to the pandemic — not to mention our trust in the media itself. In a democratic republic, the media acts as a megaphone for our experts. When it rushes to publicize limited scientific data without foregrounding the uncertainty, it does the public a disservice. And we cannot afford further erosion of confidence in our press.
Good science and good epidemiology take time. They begin with observations of small groups of patients, like those in the NEJM correspondence, and lead to larger studies that allow for more robust conclusions. Outside of a pandemic, it takes years to elucidate and test a study question appropriately, to enroll a sizable patient population and develop a sound study methodology.
Rushing this process during a pandemic is all the more challenging. And so the science journals were not wrong to publish the preliminary findings. But we are all responsible for exercising caution about the threadbare data contained in them — for not making leaps of faith to compensate for these hardships. Physicians need to do a better job of recognizing when the numbers are lacking, and reporters need to do the same.
One cannot avoid invoking the boy who cried wolf: The more often public health communicators offer bombastic but misleading claims, the less likely they will be trusted in the future. Any news report with bold headlines about what “doctors say” and “science says” is likely one that withholds a good deal of the scientific story, or buries it. If we truly want to understand what scientists say, we need to be thoughtful, deliberate, and patient, lest we risk more than just a false impression.
Aaron Rothstein, M.D., is a graduate student in epidemiology and an attending neurovascular physician at the University of Pennsylvania, and the author of the New Atlantis blog Practicing Medicine.
Exhausted by science and tech debates that go nowhere?