Who Will Fact-Check the Fact-Checkers?
The New Yorker used to be considered the paragon for fact-checking among all American periodicals, and the bible of such checking was written by a former fact-checker for that magazine1, but Louise Perry discovered a howler in a recent article:
Reading the latest copy of the New Yorker magazine, published exactly a week ago, I came across this sentence in a piece by Jill Lepore: 'One study suggests that two-thirds of Americans between the ages of fifteen and thirty-four who were treated in emergency rooms suffered from injuries inflicted by police and security guards….'
This sentence jumped out to me. How could it possibly be true that 'two-thirds' of all Americans aged 15-34 visiting emergency rooms had been injured by police or security guards, given the very many other reasons why people might present for emergency treatment?2
This is a good example of critical reading, that is, reading with your brain and not just your eyes. The critical reader asks questions as she reads. The remainder of the article is an excellent example of how to do your own fact check of a dubious claim you come across, and I recommend reading the whole thing, which isn't long.
The first job of a fact-checker is to spot those alleged "facts" that need checking. Moreover, you don't need to know a lot of statistics to spot a dubious one, though of course that would help. What you need is common sense and the will to apply it. You don't need to know the actual statistic to realize that this alleged fact must be wrong; all you need is to realize that it's highly unlikely that the majority of young American adults going to the emergency room are there because of an altercation with the police, since there are so many other reasons why they would be injured. For instance, what about automobile accidents? Furthermore, if two-thirds are injured by the police, then only one-third is left for all injuries due to criminals, which would mean that the police cause at least twice as many injuries as all criminals.
Perry says that the sentence "jumped out" at her. This shows both that she was reading critically, that her mind was engaged rather than idling as she read, and that she has a well-developed sense of the plausible.
Having read critically and realized that this statistic is implausible, the next step is to check it:
I sought out the study she was referring to, and found it…. And it turns out I was right―the 'two-thirds' claim is not true. Not even close. … But it's not clear where Lepore got the 'two-thirds' figure from. Possibly she misunderstood a line from the paper itself, which includes the finding that 61.1% of people injured by police fell into the 15-34 age bracket.
If so then Lepore reversed the direction of the relationship between the age bracket and police injuries: from two-thirds of those injured by police are 15-34―which is plausible―to two-thirds of those with injuries in that age bracket were injured by police―which is implausible. However, there's another possible source of the phony stat:
…[T]he Harvard press release…reports that: "Sixty-four percent of the estimated 683,033 injuries logged between 2001-2014 among persons age 15-34 resulted from an officer hitting a civilian." Which is to say, they were injured by hitting, rather than some other use of force.
The way the press release is worded, Lepore might have thought that the over 680,000 figure was the total of all injuries requiring an emergency room visit, but it's the total of all police-related injuries3. Here's Perry's back-of-the-envelope calculation of just what proportion of emergency room injuries are caused by the police:
I did my best to work out a rough estimate of the true proportion of 15-34 year olds visiting the ER who had suffered legal intervention injuries, and arrived at a figure of 0.2%…. So I believe Lepore's claim to be off by a factor of several hundred.
I don't know exactly how Perry arrived at this result, but my own attempt to estimate it produced a figure of 0.46%4, which is in the same ballpark. She concludes:
Why does this one sentence matter? Well, firstly, it misinforms readers, several of whom…also alighted on this claim, but unlike me took it on trust.
In other words, they either were not reading critically or their implausibility detectors failed to go off. Perry draws a lesson from the magazine's failure to adequately fact-check Lepore's article:
Secondly, and perhaps more importantly, it tells us something about the political climate in a publication like the New Yorker, which was once famous for its rigorous fact checking.
We know that political bias warps cognition, sometimes catastrophically, and this is, I think, an example of that in action. Lepore read Feldman's research and she misunderstood part of it, despite being an exceptionally intelligent person. Like many other Left-leaning Democrats, she is convinced that police brutality is a huge, under-acknowledged problem in the United States, and she therefore jumped to the conclusion that this wildly inflated 'two-thirds' figure was plausible.
The staff at the New Yorker who read her piece also, we must assume, considered it to be plausible. The sentence was printed and, as of the time of writing, has not been corrected5. … A small, troubling example of the effect of political bias on journalism.
I don't know whether Perry is right about this, though my implausibility detector did not sound the alarm as I read it. However, an additional lesson is that you can't count on the news media to fact-check everything. In your own intellectual self-defense, you need to be able to read critically and know how to check the facts for yourself. Perry's article is a case study in how to do that.
Given the failure of even the august The New Yorker at basic fact-checking, the answer to the question who will fact-check the fact-checkers must be: you.
- Sarah Harrison Smith, The Fact Checker's Bible: A Guide to Getting it Right (2004), p. i.
- Louise Perry, "An untrue claim in the New Yorker speaks volumes", Unherd, 7/21/2020. All block-quotes in this entry are from this article.
- Justin M. Feldman, et al., "Temporal Trends and Racial/Ethnic Inequalities for Legal Intervention Injuries Treated in Emergency Departments: US Men and Women Age 15–34, 2001–2014", J Urban Health, 2016 Oct; 93(5): 797–807. See the "Results" section.
- If you'd like to check my work, here's how I did it: The authors of the paper linked in the previous note used the following query tool to search statistics on nonfatal injuries compiled by the Centers for Disease Control and Prevention: "Nonfatal Injury Reports, 2001-2014", Centers for Disease Control and Prevention, accessed: 7/30/2020. If you limit the intent of the injury to "Legal Intervention" (LI), the years of report from 2001 to 2014, and the age range from 15-19 to 30-34, and submit the request, the result should be: 683,033, the estimated number of LI injuries in the relevant age range used in the paper. What we want to know is what proportion that is of total injuries for the same years and age range. To get the total injuries, all you need to do is rerun the query with "All Intents", which should result in: 148,241,544.
- It has now been corrected, see: Jill Lepore, "The Invention of the Police", The New Yorker, 7/13/2020. This is the corrected article. For the uncorrected version, see: "The Invention of the Police", Internet Archive, 7/17/2020. This is the Wayback Machine's archived copy of the original article. Warning: Both versions contain the four-letter f-word. No, not "fact".
Spike it, Again
A few weeks ago1, we saw the emergence of a fad among American reporters for the word "spike" as the socio-economic shutdown began to gradually lift in some states. Predictably, the number of new coronavirus cases began to increase―or "spike" as the news media would have it―as states began to open up for business again. In addition, the number of tests for coronavirus infection has steadily increased: more tests, more cases. Luckily, however, there has been only a small concomitant rise in the number of deaths due to COVID-192.
At least some of the recent "spikes" were artifacts of the system of reporting cases: as the result of backlogs and delays on the part of cities or counties, a large number of cases were reported to the state department of health on a single day. This would then be reported breathlessly by the news media as if all of those deaths had taken place on that very day. The particular case that I discussed was in the state of Missouri, though I speculated that other states may have similar issues with their reporting systems.
On Sunday, Florida reported 15,300 coronavirus cases, the most cases reported in a single day by any state3. Predictably, the news media touted the "daily record", as in a New York Times headline:
Florida reports more than 15,000 new cases, a daily record for the U.S.4
Also predictable were the subsequent calls for the state to delay or even backtrack on its reopening. However, this "record-setting tally" is partly due to backlogs in reporting cases:
Over the weekend, Florida made international headlines when it reported a shocking number of positive COVID-19 test results: More than 15,000 in a single day. But it turns out that report contained numbers gathered over several days by a single laboratory. More than 7,000 of the 15,000 positive cases reported have been traced to GENETWORx in Richmond, Virginia. The company, which is Florida's fourth-largest processor of tests, said in a statement it looks like the Florida Department of Health reported in a single day, lab results that had been collected over the course of four to five days. That made Florida's single-day caseload appear greater than it was.5
So, almost half of the "record" number were the result of a backlog in cases from a single lab. In other words, as in Missouri, the spike was as much a spike in the reporting of cases as a genuine rise in cases. Now, it's almost certain that there has been a rise in cases since Florida began reopening, but how much of the recent rise is due to new cases and how much due to reporting delays?
Although the record-setting tally on Sunday may be partially due to how test results are reported, that is no reason to discount the sheer number of infections, one expert told the Times. In order to adjust for the number of tests, experts look at the percent of tests that come back positive. On Sunday about 16 percent of tests returned positive. And that number is relatively low compared to what the state has seen in recent weeks, when the percent of positive test results has reached nearly 20 percent on average. Although the percent positive rate is lower than normal, "16 percent is still very troubling," said José Szapocznik, a professor of public health at the University of Miami. "To me it shows that the prevalence of infections in the population is still going up." … And when infections keep creeping up "there is reason to be extremely concerned."3
Of course the prevalence of infections in the population is going up―that's obvious―but there's another reason for the high level of positive test results:
Another issue which could challenge people's trust in the test figures is the Florida Department of Health website section, which appears to show several labs only passing along positive results. The negative column is blank, making it appear that nearly 100 percent of the tests those labs performed came back positive. Orlando Health, for example, reported 512 positive cases and just 10 negatives giving the appearance of a 98 percent positive test rate. On Tuesday, they sent 10 Tampa Bay a chart showing there are several more test locations in the same healthcare group with varying positivity rates. "We're looking into this," they said in an emailed statement, "But that 98 percent positivity rate is incorrect. Our positivity rate is 9.4 percent as of July 12."5
That doesn't explain why the positivity rate was reported at over ten times what it should be. According to the Florida Department of Health, some labs were simply not reporting negative test results6. Of course, the effect of this is to raise the alarming positivity rate, but how much?
Accurate numbers are important because alarm about the "spikes", "surges", and "records" is driving politicians into reversing or delaying reopening the economy. We're even hearing again the canard about overwhelmed hospitals and an insufficient number of ventilators2. We were frightened into adopting dubious measures in the first place based on inaccurate numbers, and now we're being frightened with inaccurate numbers into continuing them.
- Spike It, 6/22/2020
- Kashmira Gander, "U.S. COVID-19 Cases Are Skyrocketing, so Why Are Deaths Down?", Newsweek, 7/8/2020
- Ian Hodgson, "Behind the Florida spike: What testing tells us about recent coronavirus cases", Tampa Bay Times, 7/14/2020
- "Florida reports more than 15,000 new cases, a daily record for the U.S.", The New York Times, 7/13/2020
- Eric Glasser, "Florida's recent record day for COVID-19 might not have been quite that high", 10 Tampa Bay, 7/14/2020
- Robert Guaderrama, "FOX 35 INVESTIGATES: Florida Department of Health says some labs have not reported negative COVID-19 results", Fox 35 Orlando, 7/13/2020
An Independence Day Patriotic Shoestring Puzzle
Mack the Finger was carrying forty shoestrings inside a brown paper bag. His shoes made a flapping sound with each step because they had no strings holding them on. Some of the strings in the bag were red, some were white, and some were blue. Mack came upon Louie the King sitting on his throne.
Well, Mack said to Louie: "I got forty red, white, and blue shoestrings in this here bag. How many strings do I need to pull out of the bag to be sure I get a pair of strings of the same color for my shoes?"
"Without peeking in the bag?" Louie asked.
"Without peeking," Mack answered.
And Louie said: "Let me think for a minute, son." And he said: "Yes, I think it can be easily done."
What did Louie the King tell Mack the Finger to do?
"Say, Mack," Louie the King asked, looking down at Mack's shoes, "do you care what color the strings are?"
"No, Louie," Mack answered, "just so long as they're the same color."
Louie the King asked Mack: "What would happen if the first three strings you pulled out of the bag were each a different color?"
Louie the King said: "Just reach in the bag and pull out four strings."
"Thanks, Louie," Mack said, "Now, do you know where I can get rid of the rest of these strings?"
And Louie said: "Just take them down to Highway 61."
Acknowledgment and Disclaimer: This puzzle is based on Bob Dylan's song "Highway 61 Revisited". The thousand non-ringing telephones is another puzzle.
The Good News is that the Bad News was Wrong
The mass hysteria over COVID-19 seems to be finally dying down, but that's partly because it's been replaced by another one involving rioting, looting, and arson, which makes me nostalgic for the previous hysteria. Can we please go back to standing six feet away from each other while wearing face masks and obsessing over toilet paper, hand sanitizer, and ventilators, instead of looting stores, lobbing Molotov cocktails, and pulling down statues? I hope that this will be the last month when the recommended reading is all about the epidemic.
- John P. A. Ioannidis, Sally Cripps & Martin A. Tanner, "Forecasting for COVID-19 has failed", International Institute of Forecasters, 6/11/2020.
Many scientists have struggled to make forecasts about [COVID-19's] impact. However, despite involving many excellent modelers, best intentions, and highly sophisticated tools, forecasting efforts have largely failed.
Experienced modelers drew early on parallels between COVID-19 and the Spanish flu that caused >50 million deaths with mean age at death being 28. … However, as of June 8, total fatalities are ~410,000 with median age ~80 and typically multiple comorbidities.
Predictions for hospital and ICU bed requirements were also entirely misinforming. Public leaders trusted models (sometimes even black boxes without disclosed methodology) inferring massively overwhelmed health care capacity. However, eventually very few hospitals were stressed, for a couple of weeks. Most hospitals maintained largely empty wards, waiting for tsunamis that never came. The general population was locked and placed in horror-alert to save the health system from collapsing. Tragically, many health systems faced major adverse consequences, not by COVID-19 cases overload, but for very different reasons. Patients with heart attacks avoided visiting hospitals for care, important treatments (e.g. for cancer) were unjustifiably delayed, mental health suffered. With damaged operations, many hospitals started losing personnel, reducing capacity to face future crises (e.g. a second wave). With massive new unemployment, more people may lose health insurance. …
Modeling resurgence after reopening also failed. E.g. a Massachusetts General Hospital model predicted over 23,000 deaths within a month of Georgia reopening. The actual number was only 896. …
See, especially, Table 1 for a compare and contrast of some of the scary forecasts with reality. In particular, notice the implicit exponential extrapolation in Governor Cuomo's remarks.
According to this article, epidemiological modelling, which is a very young science, had a poor track record even before the recent debacle. It needs to prove itself before any future large-scale, disruptive government measures are taken based on it.
- Jon Miltimore, "NPR: 'Mounting Evidence' Suggests COVID Not As Deadly as Thought. Did the Experts Fail Again?", Foundation for Economic Education, 6/12/2020.
…[O]n March 5 vaccine expert Paul A. Offit, who holds the Maurice R. Hilleman Chair of Vaccinology at the University of Pennsylvania, told Factcheck.org that he believed that the World Health Organization's 3.4 percent fatality rate figure was too high, suggesting it was well below 1 percent. "We’re more the victim of fear than the virus," Offit said, adding that the world was witnessing a "wild overreaction" to the disease. Voices like those of…Offit were quickly drowned out, however. The 24-hour news cycle fanned collective fear and outrage that more was not being done. Runs on toilet paper and masks ensued. Neil Ferguson, professor of mathematical biology at Imperial College London, predicted millions would die in the "best-case scenario."
If only we had listened to Offit, who was right, instead of Ferguson, who was disastrously wrong.
…[T]he Washington Post this week cited studies claiming the lockdown orders prevented hundreds of millions of COVID-19 infections and saved millions of lives. These findings come with caveats, however. First, one of the studies was submitted on March 22―well before the vast majority of COVID cases had even occurred. The other study was conducted by researchers at the Imperial College of London, the same school from which Ferguson hailed. (He has since resigned after it was discovered that he broke the lockdown protocol he helped design by allowing his married lover to come to his home.) Ferguson, who in 2005 said up to 200 million might die from bird flu (about 100 did), was asked by The New York Times in March what the best-case scenario was for the US during the COVID pandemic. "About 1.1 million deaths," he responded. As of June 10, Ferguson is off by about a factor of ten. Why we should continue to listen to schools that have already proven to be so disastrously wrong is anyone's guess. The "chicken little" story comes to mind.
Also, the little boy who cried "wolf". Remember that the lesson of the story of the shepherd falsely crying "wolf" is that when there are false alarms, people stop paying attention to them and may ignore a true alarm. Ferguson should be sent back to school for remedial education in epidemiology, statistics, logic, and medical ethics, instead of being forced out over an affair or even for being a hypocrite.
I haven't written anything about COVID-19 in Canada because it's hard enough trying to understand what's happening in the U.S. According to the following article, the situation in our neighbor to the north is almost exactly the same as here:
- Gwyn Morgan, "The Lockdown Contrarians Were Right", C2C Journal, 6/10/2020.
After being all-but locked down since mid-March, Canadians are emerging to face the incomprehensible damage that has occurred in just 12 weeks. Besides the patients and valiant front-line workers who succumbed to the Covid-19 virus, shutting down large sectors of the economy caused thousands of lost businesses, millions of lost jobs, and billions (perhaps trillions) in lost savings. The Prime Minister’s spending announcements…added more than $20 billion to our national debt―per week.
…There have been zero deaths of children under age 16 in Canada. And there's a growing consensus that their light viral load makes children unlikely spreaders of the virus. Rather than being the most vulnerable, as they were to the Spanish Flu a century ago, they appear to be the least vulnerable to Covid-19. Reopening schools therefore poses low risks. When summer ends, many parents need to get back to work, rather being kept home supervising their children.
This is why adults need to stop over-protecting children, which is both unnecessary and counter-productive. Let the children play together and go to school.
Don't leave pandemic response measures solely in the hands of Chief Medical Officers. They did their job of "flattening the curve" well. But the measures taken should have also considered the impact of hospital bed closures on treatment of other diseases, small business owners who face losing everything, stress-induced mental health deterioration, suicides, family violence, long-term unemployment and massive public debt.
The strategy to cope with any new outbreak should include our best financial, business, education and mental health experts working as a team. These teams should be assembled immediately as we navigate the unknown course of the corona virus. There's little point to sweeping measures that over-protect the entire population against a single virus if the resulting damage is so severe that the nation in its weakened state can’t cope with future crises, whether those are health-related or otherwise. For such crises will surely occur.
According to this article, Canada's healthcare system may have been nearly overwhelmed even before the coronavirus struck, but the curve in the U.S. was never in danger of overwhelming our system, and therefore was in no need of "flattening". Nonetheless, this is good advice about how to deal with future outbreaks.
I've neglected England for the same reasons as Canada, though part of our problem here originated in a British computer model. It's obvious from the following article that the authorities and experts there have done as poor a job as those here:
- LORD JONATHAN SUMPTION, "These people have no idea what they're doing: Ex-Supreme Court judge LORD JONATHAN SUMPTION gives a devastating verdict on our political leaders' handling of the crisis", The Daily Mail, 6/21/2020. Apparently, in England, if you're a Lord, you're name is written in all capitals.
…The Government has repeatedly claimed to be 'guided by the science'. This has in practice been a shameless attempt to evade responsibility by passing the buck to scientists for what are ultimately political, and not scientific, decisions. Scientists can advise what measures are likely to reduce infections and deaths. Only politicians can decide whether those measures make sense in economic and social terms too.
Sage, the committee of scientists advising the Government, has been very clear about this, as the minutes of its meetings show. They are not willing to become the Government's human shield, or the fall-guys for its policy misjudgments.
Ministers press them for the kind of unequivocal answers that will protect them from criticism. Scientists cover themselves by giving equivocal answers, which reflect the uncertainty of the science. The Government responds by avoiding any decision for which it would have to take political responsibility, until the pressure of events becomes irresistible, when it lurches off in a new direction. …
Judging by its minutes, Sage was unenthusiastic about closing down the hospitality industry, forbidding large gatherings or closing schools. From an early stage, it had pointed out that the real threat was to people over 70 and those with serious underlying medical conditions. Since March 5 they had been advising the Government to 'cocoon' those people, and others who either had the disease or lived in the same household.
Sage appears to have envisaged guidance rather than compulsion. 'Citizens', the behavioural scientists advised, 'should be treated as rational actors, capable of taking decisions for themselves and managing personal risk.' If this advice had been followed, it would have left almost all the economically active members of the population free to earn their livings and sustain the economy.
Indiscriminate lockdown was a panic response to the now-notorious statistical model produced on March 16 by Professor Neil Ferguson's team at Imperial College. Panic responses leave little room for reflection. No serious consideration appears to have been given to the potentially catastrophic side effects. In fact, the Imperial team did identify the main problem about a lockdown. In an earlier report to Sage, they had pointed out that once a disease had taken hold in a population, 'measures which are too effective merely push all transmission to the period after they are lifted, giving a delay but no substantial reduction in either peak incidence or overall attack rate'.
They repeated this view when they recommended a lockdown on March 16 and said that to be effective, it would need to be maintained until a vaccine was available, 'potentially 18 months or more'. They pointed out that this would involve 'enormous' social and economic costs which might themselves have a significant impact on health and wellbeing.
The Government justified its Plan B as a temporary measure designed only to delay the peak until the NHS's intensive care capacity had caught up.
This is essentially the same rationale used here in the states for the socio-economic shutdown, under the rubric "flattening the curve". What happened when it became obvious that this was unnecessary?
…[I]nstead of lifting the lockdown, it merely nibbled at its edges, announcing that its essential features would remain in place for weeks or months. No rational explanation was ever offered. But the logic of its position was that the lockdown would have to continue indefinitely. …No one in government was grown-up enough to confront the real issue: does a low risk justify a huge economic cost? …Like so many of the Government's measures, it is being maintained simply in order to avoid admitting that it was a mistake.
I don't know if this is correct, but it does explain what is otherwise baffling behavior. Why, when it became obvious in April that the early predictions were wrong, did we not change course? Why are we only now, two months later, beginning to lift the socio-economic restrictions that were supposed to "flatten a curve" that didn't need flattening? Is it just that the health and political authorities are afraid to admit error? If so, we need to work on making it easier for such authorities to admit mistakes. Otherwise, during the next crisis, we'll once again get locked into an early position taken in ignorance and be unable to adapt our response to improving knowledge.
Why were the predictions about COVID-19 so wide of the mark?