Suspect primes

So profound is the appeal of overly potent effects that it can penetrate the cognitive defenses of people who should know better, including the researcher most famous for documenting our cognitive foibles, Nobel Prize winner Daniel Kahneman. In Chapter 4 of his bestselling book Thinking, Fast and Slow, Kahneman described a series of studies showing how subtle, almost unnoticed influences can substantially alter our thoughts and behaviors.

One study, for example, reported the "Lady Macbeth effect": After you watched a short film clip meant to elicit disgust, physically cleaning your hands was said to "wash away" that experience and lead you to judge moral transgressions to be less severe. Another study found that hanging a picture of eyes in a breakroom increased voluntary contributions to the use of the communal coffee machine. In the most famous and influential of these experiments, college students were asked to create sentences from sets of words. For some participants, half of the sets included words about being elderly, such as "wrinkle," "forgetful," and "Florida." After completing this task, the students left the laboratory and walked to the elevator. However, the experiment had not ended; a researcher surreptitiously timed their walk to test the hypothesis that the people who had been thinking about words related to old age would be unconsciously "primed" to walk more slowly, as an elderly person would. Sure enough, the students who were primed took one additional second on average to walk the approximately 10 meters to the elevator. This finding made a big splash. If subtle, unnoticed features of the world around us can have such potent influences, then we must have much less control of our actions and decisions than we intuitively believe.

Kahneman knew that these findings would be implausible to skeptical readers, so he made sure to hammer home how strong he thought the evidence was: "The idea you should focus on, however, is that disbelief is not an option. The results are not made up, nor are they statistical flukes. You have no choice but to accept that the major conclusions of these studies are true. More important, you must accept that they are true about you." He might be right that none of these findings were "made up." But in the years since Kahneman insisted that these metaphor-driven priming results were beyond question, many have been subjected to independent attempts at replication, and none has emerged unscathed.

Soon after publishing his book in 2011 – and not long after an independent laboratory failed to replicate the elderly priming study with more rigorous methods than the original – Kahneman penned a letter to priming researchers imploring them to shore up the foundations of their science by replicating each other's work. He wrote, "Your field is now the poster child for doubts about the integrity of psychological research. . . . I believe that you should collectively do something about this mess."

The letter was met mostly with silence but occasionally with denial and resistance. For example, in an interview with science journalist Ed Yong, the social psychologist Norbert Schwarz argued, "You can think of this as psychology's version of the climate-change debate . . . the consensus of the vast majority of psychologists closely familiar with work in this area gets drowned out by claims of a few persistent priming skeptics." In science, comparing critics to climate-change deniers is an extremely low blow.

Six years later, John Bargh, the senior author of the influential elderly-walking study, published a book arguing that subtle factors have pervasive influences on our actions and thoughts in daily life, even proposing that these priming effects could be harnessed for a new form of psychotherapy. His book contained no evidence of grappling with the failures to replicate his own work and that of others. Instead, it ignored them. He omitted studies that other researchers had failed to replicate – including his elderly priming effect—but included similar, not-yet-replicated studies, many of them from the same scientific papers as the ones he left out of the book. If you read Bargh's book without any foreknowledge, you would have no idea that the main scientific field it covers was a "poster child for doubts about the integrity of psychological research."

In the same year that Bargh published his book, Kahneman reflected on the effects of his letter: "I hoped that the authors of this research would rally to bolster their case by stronger evidence, but this did not happen." If these social priming effects are so potent that they determine our daily thoughts, actions, and behaviors, their adherents should have had little trouble reproducing them in well-controlled laboratory studies. Instead, they spent more effort arguing that inde pendent, direct replication – a bedrock principle of science found in elementary school textbooks – was irrelevant to their field.

Holding the warmth effect to the fire

Although few researchers who had published findings of social priming took up Kahneman's challenge, outsiders did. Like many psychologists, we were intrigued by the remarkable findings of the elderly priming study. In our own discipline, cognitive psychology, priming is an established phenomenon, but there it refers to the notion that seeing one word or image will slightly enhance our ability to see or process an identical or related word or image moments later. A core principle in cognitive psychology is that priming becomes weaker as the difference in meaning between the prime and the target increases; the weaker the association and the more conceptual jumps between them, the weaker the effect. The notion that unscrambling sentences for a few minutes unconsciously spread to the general idea of aging and thence to the association between aging and walking speed, thereby causing someone to walk more slowly in a different place some time later, is implausible in light of what we know from decades of rigorous priming research.

Nonetheless, there was a chance that Bargh had discovered one of those extraordinarily rare butterfly effects. Rather than accept the potency of these metaphorical priming results at face value or dismiss them out of hand, we decided to check for ourselves. We worked with our students to replicate a more recent finding from the Bargh group that followed the same priming logic. That study, published in Science in 2008, tested the idea that experiencing physical warmth would activate the concept of warmth, thereby priming other meanings of warmth, including interpersonal warmth, and leading people to judge other people to be "warmer." The paper reported a big effect in each of two experiments. People who held a warm cup of coffee gave "personality warmth" ratings about half a point higher on a 1–7 scale than did people who held a cold cup. And people who briefly held a warm therapeutic pack acted more prosocially than did those who held a cold therapeutic pack. We followed the procedures of both of these experiments as closely as we could, except that we tested more than three times as many participants. We found virtually no effect of holding something warm on how people thought or behaved immediately afterward.

As a scientific journal editor, Dan has overseen replication efforts by independent teams of researchers that scrutinized similar claims of powerful effects from small interventions. Most have not held up. One of these, a 1979 study by Thomas Srull and Robert Wyer, helped to launch the entire literature on such priming effects; it provided a recipe for later researchers to follow and has been cited over 2,400 times. College student participants first rearranged sets of words to form sentences (the same task Bargh used years later to prime elderliness). They then read a brief story in which the main character acted in ways that potentially could be interpreted as hostile. For some participants, most of the word sets could only form sentences that described hos tile actions. For others, most formed neutral sentences. Those primed by descrambling hostile sentences rated the story character as three points more hostile on a 0-to-10 scale. When converted to a common statistical scale, that difference in hostility ratings was more than twice as big as such obvious differences as the heights of men and women or the number of years older and younger people expect to work before retiring. Yet in the project Dan edited, across twenty-two replication attempts of that study design, all using a standardized protocol, with more than seven thousand participants in total, the average increase in hostility ratings was a mere 0.08 points.

In 2017, Ulrich Schimmack of the University of Toronto analyzed each of the priming studies that Daniel Kahneman's book had cited as incontrovertible evidence and found that most of those original studies provided little statistical evidence for their claims. Accordingly, replication studies conducted by independent laboratories since 2011 have found that people don't actually walk more slowly after unscrambling sentences related to aging, washing one's hands does not make moral judgments less severe, getting people to recall the Ten Commandments doesn't make them more honest, and flashing images of money does not make people more selfish.

Kahneman later admitted that he had been wrong to place so much trust in "the results of underpowered studies with unreasonably small samples" and that he had blinded himself to their implausible potency: "I knew all I needed to know to moderate my enthusiasm for the surprising and elegant findings that I cited, but I did not think it through." The robustness Kahneman initially saw in social priming research was that of a castle made of sand. But after witnessing six years of failed replications, Kahneman revised his views. He noted that the size of behavioral priming effects "cannot be as large and as robust as my chapter suggested," and he warned that authors like him "should be wary of using memorable results of underpowered studies as evidence for their claims." A Nobel laureate– who had written an influential paper decades earlier about the dangers of putting faith in results from small studies – admitted that he himself had been hooked by the supposed potency of priming.

Had Kahneman approached the priming phenomenon with more skepticism, he might have realized how implausibly big the effects were. One study he cited, for example, claimed that priming people with photographs of classrooms and school lockers made them more likely to vote for additional school funding. The primes were so potent that their effect was larger than the difference in support for school funding between people with and without children! The same logic would have cast doubt on the warmth priming studies that we could not replicate. The original paper reported an impact of holding a warm therapeutic pack on generosity that would be almost 50 percent larger than the difference in charitable contributions made by high- and low-income people. If that were true, then nonprofit organizations would have learned long ago to do all their fundraising on warm summer days – which they don't.

Nobody's Fool: Why We Get Taken In and What We Can Do About It by Daniel Simons & Christopher Chabris (£25, Basic Books) is available now.