Previous Month | RSS/XML | Current | Next Month

WEBLOG

September 28th, 2015 (Permalink)

What is cherry-picking?

"Cherry-picking" is a term that you've probably heard. Here are some examples from recent headlines:

Taxpayers group 'cherry-picks' numbers in report on high salaries…

Donald Trump is free to cherry-pick his polls.
But his decline is clear.

Don’t cherry-pick the papal message

As in the first two examples, cherry-picking usually refers to numbers, data, or statistical evidence, such as polls. However, it's sometimes used in a broader sense, as in the third headline, that applies to other types of evidence. "Cherry-picking" is a term of reproach, that is, it's not a good thing for numbers or other evidence to be cherry-picked. If it's not a good thing to pick cherries, is it a logical fallacy and, if so, why is there no entry for it in The Fallacy Files?

The term is, of course, based on an analogy to the process of harvesting the ripe fruit of the cherry tree. Though I've never picked cherries, I presume that the individual fruits on a tree ripen at different times, and the goal is to pluck from the tree only those that are sufficiently ripe, leaving the unripe cherries on the tree to ripen further. Thus, cherry-picking is a careful process of selection, so that cherry-picked evidence is, by analogy, carefully selected evidence.

But what's wrong with carefully selecting evidence? After all, the amount of evidence in the world may in fact be infinite, but even if it is finite there is still a whole lot of it. Every scientist must carefully select the data that supports a theory from among a haystack of irrelevant data. So, selectivity by itself is not the problem.

Rather, cherry-picking involves selecting only the evidence that supports a hypothesis, ignoring all that goes against it. This procedure violates a fundamental principle of inductive reasoning, namely, the requirement of total evidence, that is, when reasoning inductively use all of the relevant evidence. Or, in other words, don't cherry-pick your evidence!

The requirement of total evidence in induction is based on an important difference between deductive and inductive reasoning. In deduction, once you've got a valid argument, you can count on it remaining valid, because additional premisses will never make it invalid. In other words, a valid argument will never turn into an invalid one by the addition of new information. Inductive arguments, however, are different: new information may cause a strong inductive argument to become weak. For instance, suppose that every ripe cherry you have ever seen has been red; from this evidence you conclude that all ripe cherries are red. If one day you should encounter a ripe cherry that is not red, this new information would completely undermine your former reasoning. A single non-red ripe cherry is sufficient to show that not all ripe cherries are red. Therefore, it is vital that all of the available evidence be used in the premisses of an inductive argument.

So, it appears that cherry-picking is either the same thing as one-sidedness―see the Fallacy, below―or perhaps a type of one-sidedness that applies to statistical reasoning. However, the third headline above shows a broader use of the concept of cherry-picking, since it concerns people paying attention only to some of the Pope's recent statements and ignoring others. One-sidedness is a fallacy that seems to be rediscovered and renamed regularly, so it has acquired several aliases. For instance, in the context of journalism, it is often called "slanting"; relating to propaganda, it was dubbed "card stacking" by the Institute for Propaganda Analysis in the 1930s. "Cherry picking" appears to be yet another alias of one-sidedness, one that most often applies to numerical evidence.

Sources:

Resources: Some other discussions of cherry-picking worth checking out:

Fallacy: One-sidedness

Acknowledgment: Thanks to Gary Herstein for drawing this issue to my attention.

September 17th, 2015 (Permalink)

Debate Watch

Yesterday a so-called "debate" was held between most of the candidates for the Republican nomination for president. Calling these forums "debates" is somewhat misleading, because there is usually little debating between the candidates. Instead, they would be more accurately called joint candidate news conferences, with questions from journalists and sometimes from the audience. Moreover, the candidates often answer questions by reciting campaign speech boilerplate on the issue. As a consequence, many of the answers are not directly responsive to the questions, though they may at least be on the same general topic.

Despite these deficiencies, arguments are still sometimes made by the candidates. For instance, in yesterday's debate, moderator Jake Tapper asked the following question of candidate Ben Carson:

Jake Tapper: A backlash against vaccines was blamed for a measles outbreak here in California. Dr. Carson, Donald Trump has publicly and repeatedly linked vaccines, childhood vaccines, to autism, which, as you know, the medical community adamantly disputes. You're a pediatric neurosurgeon. Should Mr. Trump stop saying this?

Ben Carson: Well, let me put it this way, there have been numerous studies, and they have not demonstrated that there is any correlation between vaccinations and autism. This was something that was spread widely 15 or 20 years ago, and it has not been adequately…revealed to the public what's actually going on. Vaccines are very important. Certain ones. The ones that would prevent death or crippling. There are others, there are a multitude of vaccines which probably don't fit in that category, and there should be some discretion in those cases. …

Tapper: Should he stop saying it? Should he stop saying that vaccines cause autism?

Carson: Well, you know, I've just explained it to him. He can read about it if he wants to. I think he's an intelligent man and will make the correct decision after getting the real facts. …

Tapper: Mr. Trump, as president, you would be in charge of the Centers for Disease Control and the National Institutes of Health, both of which say you are wrong. How would you handle this as president?

Donald Trump: Autism has become an epidemic. Twenty-five years ago, 35 years ago, you look at the statistics, not even close. It has gotten totally out of control. I am totally in favor of vaccines. But I want smaller doses over a longer period of time. Because you take a baby in―and I've seen it―and I've seen it, and I had my children taken care of over a long period of time, over a two or three year period of time. Same exact amount, but you take this little beautiful baby, and you pump―I mean, it looks just like it's meant for a horse, not for a child, and we've had so many instances, people that work for me. Just the other day, two years old, two and a half years old, a child, a beautiful child went to have the vaccine, and came back, and a week later got a tremendous fever, got very, very sick, now is autistic. …I'm in favor of vaccines, do them over a longer period of time, same amount. … But just in little sections. …I think you're going to see a big impact on autism.
Source: "CNN Reagan Library Debate: Later Debate Full Transcript", CNN, 9/16/2015

I want to focus attention on the highlighted sections at the end, the earlier portion being stage-setting for Trump's answer to Tapper's question. At the beginning of Trump's answer, he claims that autism has become an epidemic. In the final few sentences, Trump tells a sketchy anecdote about a child who was vaccinated and is now autistic. Subsequently, he claims that spacing out the administering of vaccines over a longer period of time than currently will result in a "big impact on autism"―presumably, a large decline in the rates of autism. Trump insists that he favors vaccination, and that it's only the rate of vaccination and size of doses that he wants to change. Thus, he appears to be claiming that the amount of vaccinations, rather than the vaccines themselves, are causing autism.

This is a familiar fallback position that has been taken by anti-vaccine activists, such as Jenny McCarthy, so it's likely that Trump did not come up with the idea on his own, but heard or read about it. It allows him to avoid directly contradicting Carson's claims about the scientific evidence against the hypothesis that vaccines cause autism.

What evidence does Trump give that the number, size, or frequency of vaccinations cause autism? I see only two arguments here:

  1. The first highlighted sentences represent only half the premisses of an argument. The half that's missing is a premiss claiming that the amount of vaccinations has increased in the past decades during which the incidence of autism has increased. This is, again, a familiar claim made by anti-vaccine activists, which is why I suggest that Trump is assuming it. Otherwise, Trump's assertions about the supposed autism epidemic would simply be irrelevant to Tapper's question. The conclusion at which Trump is aiming is that these simultaneous increases show a causal relation between the amount of vaccinations and the rate of autism.

    This first argument commits the cum hoc version of a causal fallacy―see the Fallacy, below, for a full account. The fact that the amount of vaccinations has increased during the same time period that the rate of autism diagnoses has also increased is a reasonable basis for forming the hypothesis of a causal link between the two, but it isn't by itself sufficient to establish such a link.

    Fallacy: Cum Hoc Ergo Propter Hoc

  2. The second argument is an anecdote about a child being vaccinated and subsequently becoming autistic. The story has almost no details to it―for instance, we're not even told whether the child was a boy or girl―but since most children in the United States are still vaccinated―thankfully―it's a story that could be told about most children who are diagnosed with autism. As with the first argument, this may have been at one time sufficient evidence to form a hypothesis of a causal relation between vaccination and autism, but subsequent testing has not shown such a relation to exist.

    Fallacy: Post Hoc Ergo Propter Hoc

Since temporal coincidence is never sufficient by itself to establish causation, Trump's reasoning in this exchange is fallacious. In addition to being illogical, Trump is just plain factually wrong in some of the claims he makes about vaccines and autism, but I will leave it to Steven Novella to fact-check those―see the Source, below.

Source: Steven Novella, "Trump on Vaccines", Neurologica, 9/17/2015

September 15th, 2015 (Permalink)

Poll shows over two-thirds of Americans lack imagination

Why Headlines Matter

Check out these headlines:

Poll finds almost a third of Americans would support a military coup

One-Third of Americans Would Be Fine with the Military Taking Over

I've used headlines taken from newspapers and magazines as a running feature here for several years, though mostly ones that are humorously ambiguous. However, some that we've seen have misleadingly portrayed the information contained in the article.

Headlines have traditionally been supplied by editors, rather than by the authors of the stories. That's true for The Fallacy Files, too, but that's because I'm my own editor! This is one possible source of a misleading headline, since the author is more likely to understand the story than the editor―unless, of course, they're the same person.

Tabloid newspapers are infamous for their lurid headlines, such as:

Headless Body in Topless Bar

In addition, many tabloid headlines are a bait-and-switch tactic designed to get you to buy the newspaper―or, in today's online world, to click on that headline―only to find out that the story under the headline isn't nearly as interesting as the headline suggests. Increasing numbers of online publications appear to be adopting this tabloid approach to their headlines, with Slate as a prominent case―see Resource 2, below, for instance.

According to a study of the effects of headlines on people's understanding of articles, incorrect or slanted headlines can affect the reader's understanding even when the reader reads the whole story and can see that the headline is misleading―see Sources 3 & 5, below. However, a point that is not dealt with by this study is that people often read headlines but don't go on to read the headlined story, or at least don't read it all. As a result, their only source of information may be a misleading headline, and they lack even the comparison with the full article that might show the headline to be wrong or exaggerated. Part of the purpose of a headline is to give us a quick idea of what the article is about, so that we can decide whether we're interested enough to invest the time and effort in reading the whole thing. This, of course, is what leads tabloid editors to create the bait-and-switch headline that misleads you into reading an article that turns out to be different than the headline suggests.

Cases in point: the two headlines shown at the top of this entry, each of which gives a false impression of the following story. Here's a quote from the story with the second headline:

The numbers come from a YouGov survey, which polled 1,000 people online. The exact question asked was, "Is there any situation in which you could imagine yourself supporting the U.S. military taking over the powers of federal government?"

If I were asked that exact question, I would answer: "Yes, of course." This does not mean that I would support such a thing now, nor that I think the circumstances that would lead me to support such a move are at all likely or imminent―I do not. All of which makes me wonder about two-thirds of the people who answered this survey: do they just have weak imaginations?

Another thing in the above quote that would get a seasoned poll watcher's attention is the fact that this was an online poll. The story goes on to address this issue, raising a reason to be skeptical about its own alarmist headline:

Although the numbers look pretty grim, Abraham Wyner, director of the undergraduate program in statistics at the University of Pennsylvania, said that online polls are suspect when it comes to getting accurate results. "People who are participating in an online poll are generally attracted to that poll because of some variable," he said. In other words, people who respond to the poll tend to feel strongly about the poll's subject matter, so they're inherently biased.

Wyner's point is generally true for online polls, but I don't think it applies to this particular one. Apparently YouGov signs up people online who agree to participate in its polls, then it randomly selects a sample from this subpopulation for a particular survey. However, it's still possible that those who agree to participate in YouGov's polls are different from the target population―in this case, Americans. Ben Goldacre examined a similar YouGov poll and was able to compare the results with government statistics, which seem to show that YouGov's online panel is significantly different than the general population―in this case, of England. However, that was six years ago, and perhaps things have changed for the better since then. Coincidentally enough, Goldacre's article is also from The Grauniad, which was the source for the first headline above.

So, the two headlines are wrong on two counts:

  1. They misrepresent the question that was asked in the poll. There's no reason to be alarmed that a third of Americans can imagine circumstances in which they would support a military coup. That doesn't mean that they would support such a coup if it occurred tomorrow, or that they think one would be a good idea. More worrisome are the imaginatively-challenged two-thirds who answered no.
  2. Even if the question had been more in line with the misleading headlines, it's doubtful whether the sample on which this poll was based is representative of the American public. For this reason, I'm not too worried about an American imagination deficit.

Sources:

  1. "Greatest Tabloid Headlines", New York Magazine
  2. "Our Panel", YouGov US
  3. Ullrich K. H. Ecker, Stephan Lewandowsky, Ee Pin Chang & Rekha Pillai, "The Effects of Subtle Misinformation in News Headlines" (PDF). This is labelled as a prepublication copy.
  4. Ben Goldacre, "Perils on the road to PR-reviewed data", The Guardian, 2/6/2009
  5. Maria Konnikova, "How Headlines Change the Way We Think", The New Yorker, 12/17/2014
  6. Peter Moore, "Could a coup really happen in the United States?", YouGov, 9/9/2015

Via: Glenn Reynolds, "This Headline is Deceptive", Instapundit, 9/13/2015

Resources:

  1. How to Read a Poll: Scientific Versus Self-Selected
  2. Strike Three…You're Out!, 7/3/2014

Acknowledgment: The image of a newspaper clipping shown above was generated with The Newspaper Clipping Generator.

September 5th, 2015 (Permalink)

Check Out What's New

I have another review in Notre Dame Philosophical Reviews, this time of philosopher Christopher Tindale's new book The Philosophy of Argument and Audience Reception.

Source: Gary Curtis, "The Philosophy of Argument and Audience Reception", Notre Dame Philosophical Reviews

NOW PLAYING FALLACY FILES BLURB WATCH
September 4th, 2015 (Permalink)

Blurb Watch: The Black Panthers

An ad for the new documentary The Black Panthers is practically a master class in the art of blurbing in four easy lessons:

  1. The ad correctly cites a four-star rating given the movie by critic Joshua Rothkopf of Time Out New York. However, as I've pointed out in previous watches, Time Out uses a five-star rating system, so that rather than a 100% review, this is an 80%. Did the ad indicate in some way that it was four out of five stars? Of course not. Yet other movie ads will indicate when a star rating is the highest rating possible, and Time Out itself includes a gray fifth star after four red ones in its rating. So, it's not impossible to figure out a way to let the potential moviegoer in on this information.
  2. The ad puts an obligatory exclamation point after the four star blurb, though Time Out, of course, doesn't put exclamation points after its ratings. However, in blurbing, exclamation points are free! Put them anywhere!
  3. The ad has the following blurb:

    "The Black Panthers roar again in a VITAL new documentary."
    ―Alan Scherstuhl, VILLAGE VOICE

    However, this is the headline of the review, rather than a quote from the review written by Alan Scherstuhl. As I've pointed out previously, headlines are usually written by editors, who may not have seen the movie.

  4. While the The Village Voice headline blurbed does capitalize the word "vital" that's because the entire headline is in caps! However, our last blurbing lesson is that the caps lock key on your keyboard is there for a reason! USE IT WHENEVER AND WHEREVER YOU LIKE!

Sources:

Legend movie poster

Second Feature (9/10/2015): Here's something I've never seen before: Movie critic Benjamin Lee of The Guardian gave the new British gangster picture Legend a two-star review―out of five possible. The marketers for the movie then included the two stars on a poster for the film in a diabolically clever way: look just between the heads of the twin characters. Those two stars are labelled "THE GUARDIAN", but in the context of the poster, surrounded by four- and five-star reviews, the casual viewer is given the impression that there are probably at least two more stars hidden behind the two stars' heads―actually, that's only one star, Tom Hardy, playing two characters.

Something I have seen before, but not very often, is that Benjamin Lee wrote an article calling attention to this misleading use of his two-star review―see Source 1, below. If more critics would do this when they are quoted out of context in ads and posters, there would probably be less of such shenanigans. Unfortunately, there are a lot of "critics", such as Earl Dittman, whose job seems to be to get quoted in blurbs. Lee also includes a useful account of some of the classic movie marketing tricks―ones such as David Manning and Dittman that I chronicled years ago―and some recent examples of contextomies in blurbs that I missed. Check it out.

Sources: Benjamin Lee, The Guardian,

  1. "How my negative review of Legend was spun into movie marketing gold", 9/9/2015
  2. "Legend review―Tom Hardy on double duty in cartoonish Krays biopic", 9/3/2015

Via: Kaiser Fung, "Light entertainment: what is truth?", Junk Charts, 9/10/2015

September 2nd, 2015 (Permalink)

The Case of the Three Puzzling Statements

A crime has been committed by a single criminal, and the police have brought in a three-man gang of crooks for questioning. Before their lawyer arrived, each of the suspects gave a brief statement but, afterward, each refused to answer any further questions or make any additional statements.

The following are the three statements made by the suspects; the names have been changed to protect the innocent:

Tom: I didn't do it! One of the other two did it.

Dick: Tom didn't do it! Listen to him, he's innocent!

Harry: I did it! I confess, I'm guilty!

Based on Harry's confession, the district attorney plans to indict him for the crime. The police are convinced that one of the three crooks is guilty of the crime, and that each of them knows who did it. They are also sure that at least one of the trio told the truth, and that at least one lied, but they don't know who was who. Assuming that the police are correct in their conclusions, would the D. A. be making a mistake to indict Harry? Can you tell from the information given who the culprit is?

Previous Month | RSS/XML | Current | Next Month