Previous Month | RSS/XML | Current | Next Month

WEBLOG

July 30th, 2020 (Permalink)

Who Will Fact-Check the Fact-Checkers?

The New Yorker used to be considered the paragon for fact-checking among all American periodicals, and the bible of such checking was written by a former fact-checker for that magazine1, but Louise Perry discovered a howler in a recent article:

Reading the latest copy of the New Yorker magazine, published exactly a week ago, I came across this sentence in a piece by Jill Lepore: 'One study suggests that two-thirds of Americans between the ages of fifteen and thirty-four who were treated in emergency rooms suffered from injuries inflicted by police and security guards….'

This sentence jumped out to me. How could it possibly be true that 'two-thirds' of all Americans aged 15-34 visiting emergency rooms had been injured by police or security guards, given the very many other reasons why people might present for emergency treatment?2

This is good example of critical reading, that is, reading with your brain and not just your eyes. The critical reader asks questions as she reads. The remainder of the article is an excellent example of how to do your own fact check of a dubious claim you come across, and I recommend reading the whole thing, which isn't long.

The first job of a fact-checker is to spot those alleged "facts" that need checking. Moreover, you don't need to know a lot of statistics to spot a dubious one, though of course that would help. What you need is common sense and the will to apply it. You don't need to know the actual statistic to realize that this alleged fact must be wrong; all you need is to realize that it's highly unlikely that the majority of young American adults going to the emergency room are there because of an altercation with the police, since there are so many other reasons why they would be injured. For instance, what about automobile accidents? Furthermore, if two-thirds are injured by the police, then only one-third is left for all injuries due to criminals, which would mean that the police cause at least twice as many injuries as all criminals.

Perry says that the sentence "jumped out" at her. This shows both that she was reading critically, that her mind was engaged rather than idling as she read, and that she has a well-developed sense of the plausible.

Having read critically and realized that this statistic is implausible, the next step is to check it:

I sought out the study she was referring to, and found it…. And it turns out I was right―the 'two-thirds' claim is not true. Not even close. … But it's not clear where Lepore got the 'two-thirds' figure from. Possibly she misunderstood a line from the paper itself, which includes the finding that 61.1% of people injured by police fell into the 15-34 age bracket.

If so then Lepore reversed the direction of the relationship between the age bracket and police injuries: from two-thirds of those injured by police are 15-34―which is plausible―to two-thirds of those with injuries in that age bracket were injured by police―which is implausible. However, there's another possible source of the phony stat:

…[T]he Harvard press release…reports that: "Sixty-four percent of the estimated 683,033 injuries logged between 2001-2014 among persons age 15-34 resulted from an officer hitting a civilian." Which is to say, they were injured by hitting, rather than some other use of force.

The way the press release is worded, Lepore might have thought that the over 680,000 figure was the total of all injuries requiring an emergency room visit, but it's the total of all police-related injuries3. Here's Perry's back-of-the-envelope calculation of just what proportion of emergency room injuries are caused by the police:

I did my best to work out a rough estimate of the true proportion of 15-34 year olds visiting the ER who had suffered legal intervention injuries, and arrived at a figure of 0.2%…. So I believe Lepore's claim to be off by a factor of several hundred.

I don't know exactly how Perry arrived at this result, but my own attempt to estimate it produced a figure of 0.46%4, which is in the same ballpark. She concludes:

Why does this one sentence matter? Well, firstly, it misinforms readers, several of whom…also alighted on this claim, but unlike me took it on trust.

In other words, they either were not reading critically or their implausibility detectors failed to go off. Perry draws a lesson from the magazine's failure to adequately fact-check Lepore's article:

Secondly, and perhaps more importantly, it tells us something about the political climate in a publication like the New Yorker, which was once famous for its rigorous fact checking.

We know that political bias warps cognition, sometimes catastrophically, and this is, I think, an example of that in action. Lepore read Feldman's research and she misunderstood part of it, despite being an exceptionally intelligent person. Like many other Left-leaning Democrats, she is convinced that police brutality is a huge, under-acknowledged problem in the United States, and she therefore jumped to the conclusion that this wildly inflated 'two-thirds' figure was plausible.

The staff at the New Yorker who read her piece also, we must assume, considered it to be plausible. The sentence was printed and, as of the time of writing, has not been corrected5. … A small, troubling example of the effect of political bias on journalism.

I don't know whether Perry is right about this, though my implausibility detector did not sound the alarm as I read it. However, an additional lesson is that you can't count on the news media to fact-check everything. In your own intellectual self-defense, you need to be able to read critically and know how to check the facts for yourself. Perry's article is a case study in how to do that.

Given the failure of even the august The New Yorker at basic fact-checking, the answer to the question who will fact-check the fact-checkers must be: you.


Notes:

  1. Sarah Harrison Smith, The Fact Checker's Bible: A Guide to Getting it Right (2004), p. i.
  2. Louise Perry, "An untrue claim in the New Yorker speaks volumes", Unherd, 7/21/2020. All block-quotes in this entry are from this article.
  3. Justin M. Feldman, et al., "Temporal Trends and Racial/Ethnic Inequalities for Legal Intervention Injuries Treated in Emergency Departments: US Men and Women Age 15–34, 2001–2014", J Urban Health, 2016 Oct; 93(5): 797–807. See the "Results" section.
  4. If you'd like to check my work, here's how I did it: The authors of the paper linked in the previous note used the following query tool to search statistics on nonfatal injuries compiled by the Centers for Disease Control and Prevention: "Nonfatal Injury Reports, 2001-2014", Centers for Disease Control and Prevention, accessed: 7/30/2020. If you limit the intent of the injury to "Legal Intervention" (LI), the years of report from 2001 to 2014, and the age range from 15-19 to 30-34, and submit the request, the result should be: 683,033, the estimated number of LI injuries in the relevant age range used in the paper. What we want to know is what proportion that is of total injuries for the same years and age range. To get the total injuries, all you need to do is rerun the query with "All Intents", which should result in: 148,241,544.
  5. It has now been corrected, see: Jill Lepore, "The Invention of the Police", The New Yorker, 7/13/2020. This is the corrected article. For the uncorrected version, see: "The Invention of the Police", Internet Archive, 7/17/2020. This is the Wayback Machine's archived copy of the original article. Warning: Both versions contain the four-letter f-word. No, not "fact".

July 15th, 2020 (Permalink)

Spike it, Again

A few weeks ago1, we saw the emergence of a fad among American reporters for the word "spike" as the socio-economic shutdown began to gradually lift in some states. Predictably, the number of new coronavirus cases began to increase―or "spike" as the news media would have it―as states began to open up for business again. In addition, the number of tests for coronavirus infection has steadily increased: more tests, more cases. Luckily, however, there has been only a small concomitant rise in the number of deaths due to COVID-192.

At least some of the recent "spikes" were artifacts of the system of reporting cases: as the result of backlogs and delays on the part of cities or counties, a large number of cases were reported to the state department of health on a single day. This would then be reported breathlessly by the news media as if all of those deaths had taken place on that very day. The particular case that I discussed was in the state of Missouri, though I speculated that other states may have similar issues with their reporting systems.

On Sunday, Florida reported 15,300 coronavirus cases, the most cases reported in a single day by any state3. Predictably, the news media touted the "daily record", as in a New York Times headline:

Florida reports more than 15,000 new cases, a daily record for the U.S.4

Also predictable were the subsequent calls for the state to delay or even backtrack on its reopening. However, this "record-setting tally" is partly due to backlogs in reporting cases:

Over the weekend, Florida made international headlines when it reported a shocking number of positive COVID-19 test results: More than 15,000 in a single day. But it turns out that report contained numbers gathered over several days by a single laboratory. More than 7,000 of the 15,000 positive cases reported have been traced to GENETWORx in Richmond, Virginia. The company, which is Florida's fourth-largest processor of tests, said in a statement it looks like the Florida Department of Health reported in a single day, lab results that had been collected over the course of four to five days. That made Florida's single-day caseload appear greater than it was.5

So, almost half of the "record" number were the result of a backlog in cases from a single lab. In other words, as in Missouri, the spike was as much a spike in the reporting of cases as a genuine rise in cases. Now, it's almost certain that there has been a rise in cases since Florida began reopening, but how much of the recent rise is due to new cases and how much due to reporting delays?

Although the record-setting tally on Sunday may be partially due to how test results are reported, that is no reason to discount the sheer number of infections, one expert told the Times. In order to adjust for the number of tests, experts look at the percent of tests that come back positive. On Sunday about 16 percent of tests returned positive. And that number is relatively low compared to what the state has seen in recent weeks, when the percent of positive test results has reached nearly 20 percent on average. Although the percent positive rate is lower than normal, "16 percent is still very troubling," said José Szapocznik, a professor of public health at the University of Miami. "To me it shows that the prevalence of infections in the population is still going up." … And when infections keep creeping up "there is reason to be extremely concerned."3

Of course the prevalence of infections in the population is going up―that's obvious―but there's another reason for the high level of positive test results:

Another issue which could challenge people's trust in the test figures is the Florida Department of Health website section, which appears to show several labs only passing along positive results. The negative column is blank, making it appear that nearly 100 percent of the tests those labs performed came back positive. Orlando Health, for example, reported 512 positive cases and just 10 negatives giving the appearance of a 98 percent positive test rate. On Tuesday, they sent 10 Tampa Bay a chart showing there are several more test locations in the same healthcare group with varying positivity rates. "We're looking into this," they said in an emailed statement, "But that 98 percent positivity rate is incorrect. Our positivity rate is 9.4 percent as of July 12."5

That doesn't explain why the positivity rate was reported at over ten times what it should be. According to the Florida Department of Health, some labs were simply not reporting negative test results6. Of course, the effect of this is to raise the alarming positivity rate, but how much?

Accurate numbers are important because alarm about the "spikes", "surges", and "records" is driving politicians into reversing or delaying reopening the economy. We're even hearing again the canard about overwhelmed hospitals and an insufficient number of ventilators2. We were frightened into adopting dubious measures in the first place based on inaccurate numbers, and now we're being frightened with inaccurate numbers into continuing them.


Notes:

  1. Spike It, 6/22/2020
  2. Kashmira Gander, "U.S. COVID-19 Cases Are Skyrocketing, so Why Are Deaths Down?", Newsweek, 7/8/2020
  3. Ian Hodgson, "Behind the Florida spike: What testing tells us about recent coronavirus cases", Tampa Bay Times, 7/14/2020
  4. "Florida reports more than 15,000 new cases, a daily record for the U.S.", The New York Times, 7/13/2020
  5. Eric Glasser, "Florida's recent record day for COVID-19 might not have been quite that high", 10 Tampa Bay, 7/14/2020
  6. Robert Guaderrama, "FOX 35 INVESTIGATES: Florida Department of Health says some labs have not reported negative COVID-19 results", Fox 35 Orlando, 7/13/2020

Puzzle
July 4th, 2020 (Permalink)

An Independence Day Patriotic Shoestring Puzzle

Mack the Finger was carrying forty shoestrings inside a brown paper bag. His shoes made a flapping sound with each step because they had no strings holding them on. Some of the strings in the bag were red, some were white, and some were blue. Mack came upon Louie the King sitting on his throne.

Well, Mack said to Louie: "I got forty red, white, and blue shoestrings in this here bag. How many strings do I need to pull out of the bag to be sure I get a pair of strings of the same color for my shoes?"

"Without peeking in the bag?" Louie asked.

"Without peeking," Mack answered.

And Louie said: "Let me think for a minute, son." And he said: "Yes, I think it can be easily done."

What did Louie the King tell Mack the Finger to do?


Recommended Reading
July 1st, 2020 (Permalink)

The Good News is that the Bad News was Wrong

The mass hysteria over COVID-19 seems to be finally dying down, but that's partly because it's been replaced by another one involving rioting, looting, and arson, which makes me nostalgic for the previous hysteria. Can we please go back to standing six feet away from each other while wearing face masks and obsessing over toilet paper, hand sanitizer, and ventilators, instead of looting stores, lobbing Molotov cocktails, and pulling down statues? I hope that this will be the last month when the recommended reading is all about the epidemic.

Previous Month | RSS/XML | Current | Next Month | Top of Page