I agree with you on this! The replication crisis may not have changed the landscape of the field in the way that Mastroianni suggests it should (that is, it didn't make us all crumble into nothingness), but it absolutely transformed the way research is being done and the statistics that people are using. The younger generation of psych trainees & junior faculty are using better + more replicable methods and are way more aware of the dangers of p-hacking. The funding + hiring + publication incentives still need to change, but the kids are alright.
Do you think 2, 3 and 4 *should* actually be more impactful than they are?
I would guess that the vast majority of people operate on the basis of naive realism and accurate memory contrary to these findings — like they’re wearing “augmented” reality goggles (augmented in evolutionary historical terms) and don’t know it.
Great points throughout, and I think I agree with your argument.
The word frequency effect is a nice example—psycholinguistics does have a quite robust set of replicable empirical findings (to say nothing of psychophysics): it’s true that deleting a given paper shouldn’t necessarily cause us to change our minds much, and that is more a sign of the robustness of the effect across papers than a theoretical weakness.
I get the sense that Mastroianni’s frustration is more with the underlying theoretical scaffolding of the field (which I think connects to the claim that there are no “big ideas”), but I agree with you that the evidence marshaled in his original piece doesn’t entail theoretical inadequacies per se.
In my view a fundamental challenge is that psychology deals with slippery constructs that are hard to operationalize, and I don’t know whether that challenge is going away anytime soon.
I think Mastroianni is also making one other point that you didn't address (perhaps because it's sort of orthogonal to the other points about psychology more generally), which is this:
- Stapel was considered highly successful in his field (many papers, 10,000+ citations etc.)
- Stapel turns out not to be very important in his field (deleting his many papers and orphaning 10,000+ citations doesn't seem to overturn anything particularly significant)
- So are we defining "highly successful" completely wrongly?
Or as Mastroianni put it, "Every marker of success, the things that are supposed to tell you that you're on the right track, that you're making a real contribution to science—they might mean nothing at all."
And I think that this is a very significant question to ask. This is also a theme that Mastroianni has raised elsewhere, as have many others: is the pursuit of publication counts and citation numbers undermining the pursuit of good science? And this applies not only to individual researchers who want to publish and be cited, but also to publications who also measure themselves, at least in part, on how heavily they are cited.
I would suggest that psychology - and by the way, much of the rest of science - has succumbed to Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure". As academia has increasingly defined who is an important scientist by publication and citation numbers, those numbers became a goal in their own right, and the numbers became decreasingly useful as a measure of importance.
I really like this exercise of trying to identify the most significant findings, and I'd agree that 1-6 are worthy candidates. 7-10 are interesting/useful but not field-changing.
BUT among the first six, all of those findings emerged prior to the 1980's, definitely not "in the last few decades". I'm sure you know that #4 "perceptual inference" goes back to Helmholz, before psychology was even called psychology.
I don't think change blindness is the most substantial illustration of the limitations of consciousness, but even that finding goes back to the 1970's, before ubiquitous peer review and other homogenizing practices.
In "recent decades" we've given more confidence to these six findings but mostly we've just been filling in the details---for example: understanding average developmental differences between 10 months and 14 months, but not overturning the fundamental theories/concepts that organized the paradigm in the first place.
So it feels like your examples actually underscore Adam's point.
Your article does not only serve as a response to Mastroianni's suggestions, but also an indicator to how psychology-related experiments should be scrutinized and appraised by researchers in general. If you originally have trust in a certain psychological principle, even if you find a single case of fraud, you would likely take into account countless other studies and researches that still back up the principle. Therefore, unless it's a case where the most representative research is attacked, or where the total sum of researches are unconvincing or shallow in the first place, Mastroianni's concerns should be inconsequential.
I think this shows how researchers can also fall vulnerable to confirmation bias. If you find a principle valid, the faulty case would simply be one outlier among countless other studies that say otherwise. However, if you have doubts in a principle in the first place, that would be when you would solely spotlight the single faulty research against the rest.
5,6,8 and 9 are findings from behavioral genetics that most psychologists consider debunked pseudoscience. Even w(o)ikipedia says the damns the whole field and labels it controversial.
Very interesting. Your top-10 findings list was especially interesting. I have always wanted to ask a psychologist if anyone in your field has made an attempt to falsify the "mimetic desire" theory discussed by René Girard. I recently read about his theory in his analysis of Shakespeare called Theatre of Envy, which I heard about in a Tyler Cowen podcast and I have no idea what to make of it because I'm not smart enough. But if it were true it would seem to be a top-10 type discovery and apparently Shakespeare discovered it. And if it were false then that would be a pretty big deal too since Girard has recently been gaining steam.
Many basic learning phenomena (e.g. conditional reflexes, operant schedule effects and stimulus control) are so easily replicated that they are often used as basic class demonstrations. And, perhaps more important, they are easily conducted at the level of the individual, thus avoiding the many drawbacks of large sample NHST research. Such inductive research is easily subjected to interparticipant replications.
I agree with you on this! The replication crisis may not have changed the landscape of the field in the way that Mastroianni suggests it should (that is, it didn't make us all crumble into nothingness), but it absolutely transformed the way research is being done and the statistics that people are using. The younger generation of psych trainees & junior faculty are using better + more replicable methods and are way more aware of the dangers of p-hacking. The funding + hiring + publication incentives still need to change, but the kids are alright.
Do you think 2, 3 and 4 *should* actually be more impactful than they are?
I would guess that the vast majority of people operate on the basis of naive realism and accurate memory contrary to these findings — like they’re wearing “augmented” reality goggles (augmented in evolutionary historical terms) and don’t know it.
Great points throughout, and I think I agree with your argument.
The word frequency effect is a nice example—psycholinguistics does have a quite robust set of replicable empirical findings (to say nothing of psychophysics): it’s true that deleting a given paper shouldn’t necessarily cause us to change our minds much, and that is more a sign of the robustness of the effect across papers than a theoretical weakness.
I get the sense that Mastroianni’s frustration is more with the underlying theoretical scaffolding of the field (which I think connects to the claim that there are no “big ideas”), but I agree with you that the evidence marshaled in his original piece doesn’t entail theoretical inadequacies per se.
In my view a fundamental challenge is that psychology deals with slippery constructs that are hard to operationalize, and I don’t know whether that challenge is going away anytime soon.
I think Mastroianni is also making one other point that you didn't address (perhaps because it's sort of orthogonal to the other points about psychology more generally), which is this:
- Stapel was considered highly successful in his field (many papers, 10,000+ citations etc.)
- Stapel turns out not to be very important in his field (deleting his many papers and orphaning 10,000+ citations doesn't seem to overturn anything particularly significant)
- So are we defining "highly successful" completely wrongly?
Or as Mastroianni put it, "Every marker of success, the things that are supposed to tell you that you're on the right track, that you're making a real contribution to science—they might mean nothing at all."
And I think that this is a very significant question to ask. This is also a theme that Mastroianni has raised elsewhere, as have many others: is the pursuit of publication counts and citation numbers undermining the pursuit of good science? And this applies not only to individual researchers who want to publish and be cited, but also to publications who also measure themselves, at least in part, on how heavily they are cited.
I would suggest that psychology - and by the way, much of the rest of science - has succumbed to Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure". As academia has increasingly defined who is an important scientist by publication and citation numbers, those numbers became a goal in their own right, and the numbers became decreasingly useful as a measure of importance.
Your list of 10 is really impressive (I noted this at the time, but I think it bears repeating - those are great insights).
I really like this exercise of trying to identify the most significant findings, and I'd agree that 1-6 are worthy candidates. 7-10 are interesting/useful but not field-changing.
BUT among the first six, all of those findings emerged prior to the 1980's, definitely not "in the last few decades". I'm sure you know that #4 "perceptual inference" goes back to Helmholz, before psychology was even called psychology.
I don't think change blindness is the most substantial illustration of the limitations of consciousness, but even that finding goes back to the 1970's, before ubiquitous peer review and other homogenizing practices.
In "recent decades" we've given more confidence to these six findings but mostly we've just been filling in the details---for example: understanding average developmental differences between 10 months and 14 months, but not overturning the fundamental theories/concepts that organized the paradigm in the first place.
So it feels like your examples actually underscore Adam's point.
Your article does not only serve as a response to Mastroianni's suggestions, but also an indicator to how psychology-related experiments should be scrutinized and appraised by researchers in general. If you originally have trust in a certain psychological principle, even if you find a single case of fraud, you would likely take into account countless other studies and researches that still back up the principle. Therefore, unless it's a case where the most representative research is attacked, or where the total sum of researches are unconvincing or shallow in the first place, Mastroianni's concerns should be inconsequential.
I think this shows how researchers can also fall vulnerable to confirmation bias. If you find a principle valid, the faulty case would simply be one outlier among countless other studies that say otherwise. However, if you have doubts in a principle in the first place, that would be when you would solely spotlight the single faulty research against the rest.
5,6,8 and 9 are findings from behavioral genetics that most psychologists consider debunked pseudoscience. Even w(o)ikipedia says the damns the whole field and labels it controversial.
This rocks!
Very interesting. Your top-10 findings list was especially interesting. I have always wanted to ask a psychologist if anyone in your field has made an attempt to falsify the "mimetic desire" theory discussed by René Girard. I recently read about his theory in his analysis of Shakespeare called Theatre of Envy, which I heard about in a Tyler Cowen podcast and I have no idea what to make of it because I'm not smart enough. But if it were true it would seem to be a top-10 type discovery and apparently Shakespeare discovered it. And if it were false then that would be a pretty big deal too since Girard has recently been gaining steam.
Many basic learning phenomena (e.g. conditional reflexes, operant schedule effects and stimulus control) are so easily replicated that they are often used as basic class demonstrations. And, perhaps more important, they are easily conducted at the level of the individual, thus avoiding the many drawbacks of large sample NHST research. Such inductive research is easily subjected to interparticipant replications.
The plane crashed and nobody checked the bodies because it’s a not a 747, it’s a light plane and the pilot survived the crash