It's fine if your studies don't work outside the lab

In praise of theoretical psychology

My friend Mickey Inzlicht recently wrote Ten Years a Skeptic, where he talked about his “coming-out moment” ten years ago as a full-blown critic of social psychology. Back then, he had published a viral blog post saying that his colleagues were doing their science wrong, and many of their published findings were bogus.

I have a lot of respect for this. Everyone agrees about the replication crisis now, but this was a while ago, and what Mickey wrote had a professional and personal cost, including the loss of a close friendship. And the post itself was persuasive and eloquent— I quote it at length in my book Psych.

Now Mickey is worrying about something else.

Replicability is a low bar for a science; it simply means we can reproduce our results when we try. But what if the results we’re reproducing say nothing about the real world? This is what I worry about now—that all these ingenious experiments we run say more about our cleverness than they do about the world we’re trying to understand … Lab experiments, I came to realize, are merely proofs of concept, demonstrations that something could happen under specific conditions. But that's a far cry from showing that these effects matter in the real world.

He gives an example.

Say we run an experiment showing that when people exercise self-control in a lab, they eat fewer cookies or chips or whatever tempting snack we put in front of them. Hooray! Hypothesis confirmed. Time to write that self-help book about how self-control is the key to weight loss, right? Not so fast. That lab study, pristine and controlled as it might be, tells us remarkably little about whether self-control helps people lose weight in the messy reality where we all live. In the real world, there are thousands of competing priorities and stimuli demanding our attention: work stress, family obligations, that neighbour who keeps playing Kenny Rogers at 3 in the morning. Maybe self-control does work outside the lab, but its effect is so tiny it gets drowned out by all these other, more powerful forces.

Marina Milyavskaya and I actually tested the power of momentary self-control in the real world. We followed people over weeks and months, tracking their moment-to-moment self-control and their progress toward various goals, including weight loss. What we found—much to our surprise and perhaps chagrin—was that self-control didn't predict success at all. The lab effects, so clean and promising, simply evaporated in the noise of real life.

So, what was the point of that original lab study? All it really did was prove that something could happen under perfectly controlled conditions.

Mickey goes on to say a lot of other things in his post, but I want to zoom in on this argument, which I’ve heard many times.

Agreed—if you want your research to have immediate practical implications, your effect has to show up in “the noise of real life.” A weight loss intervention that works only in carefully controlled conditions is useless, and would be a poor basis for a best-seller on how to shed your love handles.

The thing is, psychologists are not mainly in the business of writing self-help books.

Psychology is many things, but at its core, it’s the study of the mind. It’s an attempt to use the methods of science to answer a host of questions that are central to our being. Early in my book Psych, I have a partial list of such questions:

• Where does knowledge come from?

• How does the mind of a child differ from that of an adult?

• How do we perceive the world?

• What is the relationship between language and thought?

• How do our biases affect how we see and remember the world?

• Are we rational beings?

• What motivates us—and what is the purpose of feelings such as fear, disgust, and compassion?

• How do we think of other people—including those from other social and ethnic groups

• How (and why) do we differ in personality, intelligence, and other traits?

• What is the cause of different mental illnesses?

These are fascinating questions. My own view is that knowing their answers would be of great value even if it didn’t lead to any tangible improvement in the world. In this regard, psychology is akin to other sciences. When evolutionary biologists come to a new understanding of dinosaur evolution, say, this is a worthy accomplishment. It’s good to know things.

Laboratory studies are one way (not the only way) that science comes to know things. By establishing “perfectly controlled conditions” for our experiments, we’re doing what every other scientist does—testing our theories by focusing on specific contrasts. To take a simple example, researchers interested in how the brain figures out the location of sounds in the environment spend a lot of money and effort creating soundproof booths so that they can run their experiments without any extraneous variables (in this case, other noises) messing things up. They do this for the same reason that chemists keep their test tubes clean.

A cool laboratory study is a thing of beauty. Here’s one I read about in an excellent new paper by Chaz Firestone and Dorsa Amir. The paper is about the Müller-Lyer illusion, shown below. The red line on top looks longer—but it isn’t!

There is a debate in the field over the origin of the illusion. Many scholars think that it arises because we are raised in environments that have certain visual features, such as houses with straight lines and right angles. It’s learned. Others, like Firestone and Amir, think that it’s built into the visual system. It’s hard-wired.

If it’s hard-wired in humans, you might expect it to be hard-wired in other animals as well. And this brings me to the study I really like, by Maria Santacà and Christian Agrillo. It’s called “Perception of the Müller-Lyer illusion in guppies.” Here’s how Firestone and Amir describe the study, along with an illustration that they provide.

During a training phase, each fish was given a food reward for swimming to the longer of two horizontal lines. After demonstrating proficiency in training, a test phase displayed the Müller-Lyer stimulus, whose two central lines are equal in length.

What did the guppies do? If they don’t get the illusion, they should be random, since the lines are actually the same length. But no!

Remarkably, the fish who had learned that approaching a longer line yields a food reward chose to approach the line with inward-facing arrows attached to it, as if it appeared longer to them.

So cool.

Does it matter if the finding replicates? Of course. If other guppy labs redo the study and don’t get the same finding, we should worry that it isn’t real (a statistical fluke, perhaps) and lose our confidence that the Müller-Lyer illusion reflects neural structures that are widespread across the animal kingdom. Replication is critical.

Does it matter if the finding shows up in the real world? Not in the slightest. Maybe it’s a subtle effect, and if you plopped these guppies into a dirty fish tank, with some bratty kid tapping on the side, and some flashy neon tetras darting around and giving them the side-eye, the guppies would behave randomly. The point of the study wasn’t to demonstrate some practical technique to get your guppy to go where you want it to. The point was to test the hypothesis that guppies see a certain illusion. To test such a hypothesis it’s totally kosher to establish an unrealistically distraction-free environment.

To recap: Suppose you have an intervention, and you want it to work in the real world. Then you have to go outside the lab. This is the point that Mickey was making and he’s entirely right. (And he’s right as well that many of his colleagues are too quick to draw strong conclusions about real-world interventions based on sterile laboratory studies.)

But suppose you are interested in how the mind works—about, say, the Müller-Lyer illusion. Then such laboratory studies are the cat’s pajamas.

Actually, even if you only care about practical implications, you should respect laboratory research because the knowledge you get from them often provides the foundation for actionable interventions. Laboratory studies of the factors that lead to memory distortion can help us develop better practices for interviewing witnesses; carefully controlled research in visual perception can help us build robots that can see; studies of mental processing, carried out in perfectly pristine conditions, might one day lead to a better understanding, and ultimately treatments and cures, of diseases such as Alzheimer’s.

I’ll end by going back to Mickey’s dismissal of his own finding:

All it really did was prove that something could happen under perfectly controlled conditions.

Too harsh! We don’t just care about interventions; we also care about how the mind works. Finding out that something can happen under perfectly controlled conditions can be psychological science at its best.