Psychology is ok

A response to Adam Mastroianni

Adam Mastroianni re-posted his very interesting post, so I figured I’d re-post my moderately interesting response.

---

[Thanks to Adam for his thoughtful and gracious comments on an earlier draft.]

A couple of weeks ago, Adam Mastroianni put up a post complaining about the state of psychology. I was (and am) impressed with it, and said so:

As sometimes happens, people took my endorsement of the post as saying that I agree with Mastroianni. But I don’t, not entirely. Keep in mind that: (smart + depressing + by an interesting thinker) does not necessarily = right.

The issues that Mastroianni raises are important ones, and worth responding to. Also, he mentions me in the post and gets me wrong in a small way, and I want to clear this up as well.

---

Mastroianni’s main argument turns on the consequences (or really, the lack, of consequences) of certain recent cases of alleged fraud

This whole debacle matters a lot socially: careers ruined, reputations in tatters, lawsuits flying. But strangely, it doesn't seem to matter much scientifically. That is, our understanding of psychology remains unchanged. If you think of psychology as a forest, we haven't felled a tree or even broken a branch. We've lost a few apples. 

Turning to an earlier case of confessed fraud, that of Diederik Stapel, he goes on.

So what was the scientific fallout of Stapel's demise? What theories had to be rewritten? What revisions did we have to make to our understanding of the human mind?

Basically none, as far as I can tell. The universities where Stapel worked released a long report cataloging all of his misdeeds, and the part called “Impact of the fraud” (section 3.7 if you're following along at home) details all sorts of reputational harm: students, schools, co-authors, journals, and even psychology itself all suffer from their association with Stapel. It says nothing about the scientific impact—the theories that have to be rolled back, the models that have to be retired, the subfields that are at square one again. And looking over Stapel's retracted work, it's because there are no theories, models, or subfields that changed much at all. The 10,000+ citations of his work now point nowhere, and it makes no difference.

As a young psychologist, this chills me to my bones. Apparently is possible to reach the stratosphere of scientific achievement, to publish over and over again in “high impact” journals, to rack up tens of thousands of citations, and for none of it to matter. Every marker of success, the things that are supposed to tell you that you're on the right track, that you're making a real contribution to science—they might mean nothing at all. So, uh, what exactly am I doing?

---

As I interpret the logic of Mastroianni’s argument so far, it’s this:

If our science is going well, discovering that specific findings by single investigators are mistaken (due to error, fraud, poor design, whatever) should have major consequences.

This doesn’t happen, so our science is not going well.

I think the logic is wrong, though. This isn’t how science actually works. For the theories and the phenomenon that really count, our confidence in them is built from hundreds or thousands of papers, so dropping a few out of the pile shouldn’t make much of a difference.

Take a basic finding from psycholinguistics—the word frequency effect. This is the discovery that we are quicker to read and recognize words that are more frequent, even controlling for factors such as length. If you have to tap a button when you see an English word, you’ll tap quicker for “apple” than for “anode”.

This is a robust finding, and what this means if you deleted a researcher from the timeline, we would still believe in it. Maybe our confidence would drop a bit—now there are 280 papers showing it, instead of 285 papers—but it won’t make any real difference.

The same is true for findings such as age differences in the false belief task, male-female differences in the age of one’s ideal sexual partner, the fact that babies understand words before they begin to speak, the high prevalence of spider and snake phobias even in urban populations, the many similarities between depression and anxiety, and the fact that white people tend to show a pro-white bias in implicit attitude tasks.

It reflects well on these findings that they are robust enough that you can remove an experiment—or even a researcher—and it doesn’t make a difference. To turn it around, if a single case of fraud (or even a series of frauds by a single investigator) causes you to give up on a psychological claim, then maybe you shouldn’t have taken it seriously in the first place.

So I would propose the opposite of what Mastroianni says:

If our science is going well, discovering that specific findings by single investigators are mistaken (due to error, fraud, poor design, whatever) should NOT have major consequences.

---

Mastroianni anticipates this point, writing

Maybe the impact of any single scientist is simply too small to be seen from a distance. If you deleted a whole bunch of papers from across the literature, though, that would really make a difference, and we’d have to rebuild big parts of the field from the ground up. Right?

No, not really. We did delete those papers, and nothing much happened. In 2015, a big team of researchers tried to redo 100 psychology studies, and about 60% failed to replicate. This finding made big waves and headlines, and it's already been cited nearly 8,000 times.

But the next time someone brings it up, ask them to name as many of the 100 studies as they can. My bet is they top out at zero. I'm basically at zero myself, and I've written about that study at length. (I asked a few of my colleagues in case I'm just uniquely stupid, and their answers were: 0, 0, 0, 0, 1, and 3.)

He concludes:

What a damning indictment of our field!

There’s a lot to be said about this 2015 paper and the replication crisis in general, and my own overall take is a bit different. But regardless, I again don’t quite get the logic of Mastroianni’s argument. The replication failures were based on papers on wildly different topics. Psychology is a very broad field, and almost none of the papers relate to topics I study or know anything about. This sort of specialization is true for every psychologist, so what’s so damming about the fact that nobody can list the papers that got dinged?

And isn’t this true for sciences in general? Should someone who studies the musculature of horses lose sleep because their colleague down the hall failed to replicate her experiments on mRNA transport? I don’t think so.

What would be a damming indictment of our field is if there were multiple failed replications in a single domain of psychology and it didn’t make a difference. Let’s put it like this:

If our science is going well, discovering that many studies that bear on a single finding or phenomenon are mistaken (due to error, fraud, poor design, whatever) should have major consequences.

This happens! Mastroianni himself gives an example, talking about

The now-much-ridiculed “social priming” studies

and how we are so skeptical about them now because they keep failing to replicate! Psychology is, here at least, doing things right.

---

Mastroianni makes a further argument, based on a tweet of mine.

… earlier this year, the psychologist Paul Bloom asked exactly this question on Twitter:

A bunch of psychologists weighed in, and their responses bring me a deep sense of despair:

“psychopathology symptoms have small world network properties”

“people's bodies and brains synchronize when they are interacting”

“monkeys can use money (and pay for sex)”

Look, this isn't a systematic study; it's just a person asking for opinions on the internet. … Plenty of these findings are interesting and some are useful (especially if you are a rich, lonely monkey). I think there's some terrific psychology that doesn't get mentioned here; I highlight some in Underrated ideas in psychology.

But there's no world-changing insight like relativity, evolution, or DNA, nor any smaller-but-still-very-cool discoveries like polymerase chain reaction, CRISPR, or Higgs bosons. Only a few psychological discoveries are mentioned by more than one commenter, except for “most psychology studies are bunk.” If Bloom can't think of any major recent discoveries, and if none of his friends can agree on any major recent discoveries, then maybe there aren't any major recent discoveries.

I agree with a lot of this. The responses really were pretty goofy, and I think Mastroianni is right that we haven’t come up with an insight like relativity and evolution. (More on this later). But I was unhappy about this:

“If Bloom can't think of any major recent discoveries”

But I had been thinking of several major recent discoveries when I wrote the tweet! I was writing a popular article summarizing Psychology’s Greatest Hits and was just checking to see if there were any I had missed.

You might be thinking I’m bluffing at this point, so here’s a quick list of ten (Check out Psych (soon out in paperback!) for more details on each.) All of these are robust findings, discovered (or at least substantially built upon) by psychologists in the last few decades, and I think that most of them are interesting to non-specialists.

1. Babies have a surprisingly sophisticated understanding of the physical and social world before their first birthday.

2. Our conscious experience of the world is sharply limited; if our attention is elsewhere, we often fail to see what’s right in front of our noses. Check out this classic demo.

3. Memory is not an accurate recording, and despite what many of us think, our recollection of the past is highly distorted. False memories are not difficult to implant, and if you think you have an accurate memory of something that happened a year ago, you are almost certainly wrong.

4. Perception is a complex inferential process; what you see is influenced by your unconscious expectations of how the world works. As a vivid example, developed by Edward Adelson, B looks lighter than A because it’s in shadow, we have an unconscious assumption that shadows darken objects, and so the mind “corrects” for the extra darkness. If you cover up the rest of the scene, isolating the two squares, you’ll see that A and B are actually the same color.

   

5. All sorts of psychological traits are heritable to a strong degree—not just the obvious ones such as intelligence, but also surprising ones like how religious you are.

6. Many sex differences are culture-specific, but others, such as differences in desire for sexual variety, are universal, showing up everywhere in the world.

7. We overestimate the likelihood of infrequent but conspicuous events, such as plane crashes and shark attacks.

8. There is a universal list of things and experiences that frighten people regardless of where they have been raised, such as snakes, spiders, darkness, and heights. This suggests that our fears have been shaped by natural selection.

9. There are universal features of physical attractiveness—such as facial symmetry—and even babies prefer to look at faces that possess these features.

10. Studies of people from dozens of different countries find that, as we pass middle age, we get more agreeable, more conscientious, and less neurotic.

---

I’ll end with three points of agreement with Mastroianni. First, although he never comes out and says it, I get the sense that when he talks about psychology he means “social psychology”. And I agree that it’s harder to find a list of robust findings from this subfield. I have some ideas as to why but will save them for another day.

Second, he has a great discussion of “proto-paradigms” in psychology that I didn’t discuss here, and I strongly recommend reading. (I agree with most of it.)

Third, to repeat from above, he writes

But there's no world-changing insight like relativity, evolution, or DNA, nor any smaller-but-still-very-cool discoveries like polymerase chain reaction, CRISPR, or Higgs bosons.

I agree. I think the ten findings above (and I could have easily added so many more) are interesting and important, but, no, they don’t reach that level of “world-changing insights”. I hope that psychology might get there, but perhaps not in my lifetime.