I agree with you on this! The replication crisis may not have changed the landscape of the field in the way that Mastroianni suggests it should (that is, it didn't make us all crumble into nothingness), but it absolutely transformed the way research is being done and the statistics that people are using. The younger generation of psych trainees & junior faculty are using better + more replicable methods and are way more aware of the dangers of p-hacking. The funding + hiring + publication incentives still need to change, but the kids are alright.
Do you think 2, 3 and 4 *should* actually be more impactful than they are?
I would guess that the vast majority of people operate on the basis of naive realism and accurate memory contrary to these findings — like they’re wearing “augmented” reality goggles (augmented in evolutionary historical terms) and don’t know it.
Great points throughout, and I think I agree with your argument.
The word frequency effect is a nice example—psycholinguistics does have a quite robust set of replicable empirical findings (to say nothing of psychophysics): it’s true that deleting a given paper shouldn’t necessarily cause us to change our minds much, and that is more a sign of the robustness of the effect across papers than a theoretical weakness.
I get the sense that Mastroianni’s frustration is more with the underlying theoretical scaffolding of the field (which I think connects to the claim that there are no “big ideas”), but I agree with you that the evidence marshaled in his original piece doesn’t entail theoretical inadequacies per se.
In my view a fundamental challenge is that psychology deals with slippery constructs that are hard to operationalize, and I don’t know whether that challenge is going away anytime soon.
In the moment of reading, I had EXACTLY the same reaction as Jason S. #3, especially, would be absolutely world-changing if we could only bring ourselves to TRULY believe it - as in actually internalizing its insight and acting on that instead of our more naive assumption of its opposite. For one thing, the ways in which so much of the practice of law is conducted would HAVE to be changed. Especially criminal law and the conduct of trials. #2 and #4 both "color" #3, when one thinks about what would be needed in the conduct of society writ large, were we actually to try to make the world better by acting on a belief in these results. So, I'm thinking that social psychology IS implicated, and quite heavily, in this list. Thinking harder about the list after the fact, it seems #7, too, COULD impact how we do regulatory law (but, so far, mostly doesn't). Finally, #5, and to a lesser extent, #6 and #9, could have much greater impact on social activity were we not so predisposed to ignore them as being in violation of popular contemporary political ideologies. So it's not really that these are not "world-changing" - it's that we won't LET them be.
In my youth, I thought of being a psychologist. I had 56 course hours in psychology and needed 4 more in the field (+ a few required courses in other fields) to qualify for my undergraduate degree in psych and I was feeling quite like Adam Mastroianni. It seemed like every issue of Psych. Today has a new wunderkind psych theory to erase all mankind's psychological problems/
Knots was poetry and Being & Nothingness was philosophy, but they seemed to describe man's psychological better than any...basically a communication deficit that I didn't see any others addressing.
But Sarte went nuts (more or less)trying to justify his new found Marxism in the 30's, and the same year Knots appeared Kingsley Hall was shut down.
These guys (I thought) understand our prime mental turmoil better than any of the "cures" being offered but then when they turned to their own cures they had no more to offer. Well I dropped out of psychology to pursue philosophy.
But this doesn't seem to be Adam's problem quite as much. He seems to be less troubled by the theories of psychology and more about its efficacy and whether it matters if psychologists crib.
I agree it should matter, but I disagree that it doesn't. I think psychology has entered into becoming more scientific and failed, unsustainable ideas are becoming recognized when they can't be verified successfully and I think (perhaps wrongly) that only in the last twenty years have we had any way to verify psychological perspectives. So its in its infancy, only now, as a science, in my estimation and there will be more and more theories, papers, tests, procedures, and ideas of how the brain functions that will be able to be rejected if they cant be sustainably verified.
In short, what may be rejected isn't necessarily bad and that what can't be verified is being exposed more frequently is very beneficial towards psychiatry becoming a legitimate science.
P.S. Sartre is not generally recognized today as a psychiatrist, but he was a prominent psychology student in the 20's and early 30's who studied under most of the prominent post-Freudians in France, Berlin & Vienna. He also worked in the asylum in Paris for a few years. Being & Nothingness emerged out of his thesis to earn his Ph.D. in psychiatry, and it success led him to abandon completing the thesis/
I think Mastroianni is also making one other point that you didn't address (perhaps because it's sort of orthogonal to the other points about psychology more generally), which is this:
- Stapel was considered highly successful in his field (many papers, 10,000+ citations etc.)
- Stapel turns out not to be very important in his field (deleting his many papers and orphaning 10,000+ citations doesn't seem to overturn anything particularly significant)
- So are we defining "highly successful" completely wrongly?
Or as Mastroianni put it, "Every marker of success, the things that are supposed to tell you that you're on the right track, that you're making a real contribution to science—they might mean nothing at all."
And I think that this is a very significant question to ask. This is also a theme that Mastroianni has raised elsewhere, as have many others: is the pursuit of publication counts and citation numbers undermining the pursuit of good science? And this applies not only to individual researchers who want to publish and be cited, but also to publications who also measure themselves, at least in part, on how heavily they are cited.
I would suggest that psychology - and by the way, much of the rest of science - has succumbed to Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure". As academia has increasingly defined who is an important scientist by publication and citation numbers, those numbers became a goal in their own right, and the numbers became decreasingly useful as a measure of importance.
I don't mean to suggest an unusality that only indicates psychology, but from its earliest post-Freudian generation both behavioral studies and case histories were often questioned on whether they were fraudulent.
And yet, perhaps, Mastroianni may have a point to be said in his favor and markers of what is successful should be reevaluated continuously, not just after scandals like Watson's or Burt's fabricated twin studies in the 60's. So both bad methodology like Watson's and fabricated data such as Cyril Burt presented (both highly respected in their time) still persist to influence.
Therin is the real problem and what leads to both, psychiatrists sometimes want their results to be different than it is and they will alter or fudge data to make it so.
Not unique to psychiatry and much more of a political issue. What disturbs me more than anything, perhaps because it was one of the leading reasons I dropped out of psychology, was Burt's studies because they continue to be propagated by supremacists and there is no evidence that any follow-up" studies have been better.
Perhaps the greatest crisis in science as a whole, but certainly in those outside of chemistry and physics, is the expectation of certainty. The proliferation of statistical tools and professional publishing has led to an expectation that we can divine simple truths from complex systems like sociology, economics, human biology, nutrition, etc.
The issue is not that we have a replicability crisis, it is that we as a society have an epistemic crisis - we expect more "truth" than the world, our senses, and our instruments can provide.
Psychology is not dead because we cannot treat it like a Hubermann episode, and philosophy is not dead because we can't prove it through double-blind experiments. Complex systems rarely yield to simple truths, so our modes of contemplation have to calibrate to a realistic level of resolution.
I really like this exercise of trying to identify the most significant findings, and I'd agree that 1-6 are worthy candidates. 7-10 are interesting/useful but not field-changing.
BUT among the first six, all of those findings emerged prior to the 1980's, definitely not "in the last few decades". I'm sure you know that #4 "perceptual inference" goes back to Helmholz, before psychology was even called psychology.
I don't think change blindness is the most substantial illustration of the limitations of consciousness, but even that finding goes back to the 1970's, before ubiquitous peer review and other homogenizing practices.
In "recent decades" we've given more confidence to these six findings but mostly we've just been filling in the details---for example: understanding average developmental differences between 10 months and 14 months, but not overturning the fundamental theories/concepts that organized the paradigm in the first place.
So it feels like your examples actually underscore Adam's point.
Your article does not only serve as a response to Mastroianni's suggestions, but also an indicator to how psychology-related experiments should be scrutinized and appraised by researchers in general. If you originally have trust in a certain psychological principle, even if you find a single case of fraud, you would likely take into account countless other studies and researches that still back up the principle. Therefore, unless it's a case where the most representative research is attacked, or where the total sum of researches are unconvincing or shallow in the first place, Mastroianni's concerns should be inconsequential.
I think this shows how researchers can also fall vulnerable to confirmation bias. If you find a principle valid, the faulty case would simply be one outlier among countless other studies that say otherwise. However, if you have doubts in a principle in the first place, that would be when you would solely spotlight the single faulty research against the rest.
5,6,8 and 9 are findings from behavioral genetics that most psychologists consider debunked pseudoscience. Even w(o)ikipedia says the damns the whole field and labels it controversial.
Very interesting. Your top-10 findings list was especially interesting. I have always wanted to ask a psychologist if anyone in your field has made an attempt to falsify the "mimetic desire" theory discussed by René Girard. I recently read about his theory in his analysis of Shakespeare called Theatre of Envy, which I heard about in a Tyler Cowen podcast and I have no idea what to make of it because I'm not smart enough. But if it were true it would seem to be a top-10 type discovery and apparently Shakespeare discovered it. And if it were false then that would be a pretty big deal too since Girard has recently been gaining steam.
Many basic learning phenomena (e.g. conditional reflexes, operant schedule effects and stimulus control) are so easily replicated that they are often used as basic class demonstrations. And, perhaps more important, they are easily conducted at the level of the individual, thus avoiding the many drawbacks of large sample NHST research. Such inductive research is easily subjected to interparticipant replications.
I agree with you on this! The replication crisis may not have changed the landscape of the field in the way that Mastroianni suggests it should (that is, it didn't make us all crumble into nothingness), but it absolutely transformed the way research is being done and the statistics that people are using. The younger generation of psych trainees & junior faculty are using better + more replicable methods and are way more aware of the dangers of p-hacking. The funding + hiring + publication incentives still need to change, but the kids are alright.
Do you think 2, 3 and 4 *should* actually be more impactful than they are?
I would guess that the vast majority of people operate on the basis of naive realism and accurate memory contrary to these findings — like they’re wearing “augmented” reality goggles (augmented in evolutionary historical terms) and don’t know it.
Great points throughout, and I think I agree with your argument.
The word frequency effect is a nice example—psycholinguistics does have a quite robust set of replicable empirical findings (to say nothing of psychophysics): it’s true that deleting a given paper shouldn’t necessarily cause us to change our minds much, and that is more a sign of the robustness of the effect across papers than a theoretical weakness.
I get the sense that Mastroianni’s frustration is more with the underlying theoretical scaffolding of the field (which I think connects to the claim that there are no “big ideas”), but I agree with you that the evidence marshaled in his original piece doesn’t entail theoretical inadequacies per se.
In my view a fundamental challenge is that psychology deals with slippery constructs that are hard to operationalize, and I don’t know whether that challenge is going away anytime soon.
In the moment of reading, I had EXACTLY the same reaction as Jason S. #3, especially, would be absolutely world-changing if we could only bring ourselves to TRULY believe it - as in actually internalizing its insight and acting on that instead of our more naive assumption of its opposite. For one thing, the ways in which so much of the practice of law is conducted would HAVE to be changed. Especially criminal law and the conduct of trials. #2 and #4 both "color" #3, when one thinks about what would be needed in the conduct of society writ large, were we actually to try to make the world better by acting on a belief in these results. So, I'm thinking that social psychology IS implicated, and quite heavily, in this list. Thinking harder about the list after the fact, it seems #7, too, COULD impact how we do regulatory law (but, so far, mostly doesn't). Finally, #5, and to a lesser extent, #6 and #9, could have much greater impact on social activity were we not so predisposed to ignore them as being in violation of popular contemporary political ideologies. So it's not really that these are not "world-changing" - it's that we won't LET them be.
I ended up writing about the replication crisis in psych this week after being (partially) inspired by this post! https://darbysaxbe.substack.com/p/cogsplained-by-my-ai-doppelganger
In my youth, I thought of being a psychologist. I had 56 course hours in psychology and needed 4 more in the field (+ a few required courses in other fields) to qualify for my undergraduate degree in psych and I was feeling quite like Adam Mastroianni. It seemed like every issue of Psych. Today has a new wunderkind psych theory to erase all mankind's psychological problems/
Knots was poetry and Being & Nothingness was philosophy, but they seemed to describe man's psychological better than any...basically a communication deficit that I didn't see any others addressing.
But Sarte went nuts (more or less)trying to justify his new found Marxism in the 30's, and the same year Knots appeared Kingsley Hall was shut down.
These guys (I thought) understand our prime mental turmoil better than any of the "cures" being offered but then when they turned to their own cures they had no more to offer. Well I dropped out of psychology to pursue philosophy.
But this doesn't seem to be Adam's problem quite as much. He seems to be less troubled by the theories of psychology and more about its efficacy and whether it matters if psychologists crib.
I agree it should matter, but I disagree that it doesn't. I think psychology has entered into becoming more scientific and failed, unsustainable ideas are becoming recognized when they can't be verified successfully and I think (perhaps wrongly) that only in the last twenty years have we had any way to verify psychological perspectives. So its in its infancy, only now, as a science, in my estimation and there will be more and more theories, papers, tests, procedures, and ideas of how the brain functions that will be able to be rejected if they cant be sustainably verified.
In short, what may be rejected isn't necessarily bad and that what can't be verified is being exposed more frequently is very beneficial towards psychiatry becoming a legitimate science.
P.S. Sartre is not generally recognized today as a psychiatrist, but he was a prominent psychology student in the 20's and early 30's who studied under most of the prominent post-Freudians in France, Berlin & Vienna. He also worked in the asylum in Paris for a few years. Being & Nothingness emerged out of his thesis to earn his Ph.D. in psychiatry, and it success led him to abandon completing the thesis/
I think Mastroianni is also making one other point that you didn't address (perhaps because it's sort of orthogonal to the other points about psychology more generally), which is this:
- Stapel was considered highly successful in his field (many papers, 10,000+ citations etc.)
- Stapel turns out not to be very important in his field (deleting his many papers and orphaning 10,000+ citations doesn't seem to overturn anything particularly significant)
- So are we defining "highly successful" completely wrongly?
Or as Mastroianni put it, "Every marker of success, the things that are supposed to tell you that you're on the right track, that you're making a real contribution to science—they might mean nothing at all."
And I think that this is a very significant question to ask. This is also a theme that Mastroianni has raised elsewhere, as have many others: is the pursuit of publication counts and citation numbers undermining the pursuit of good science? And this applies not only to individual researchers who want to publish and be cited, but also to publications who also measure themselves, at least in part, on how heavily they are cited.
I would suggest that psychology - and by the way, much of the rest of science - has succumbed to Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure". As academia has increasingly defined who is an important scientist by publication and citation numbers, those numbers became a goal in their own right, and the numbers became decreasingly useful as a measure of importance.
I don't mean to suggest an unusality that only indicates psychology, but from its earliest post-Freudian generation both behavioral studies and case histories were often questioned on whether they were fraudulent.
And yet, perhaps, Mastroianni may have a point to be said in his favor and markers of what is successful should be reevaluated continuously, not just after scandals like Watson's or Burt's fabricated twin studies in the 60's. So both bad methodology like Watson's and fabricated data such as Cyril Burt presented (both highly respected in their time) still persist to influence.
Therin is the real problem and what leads to both, psychiatrists sometimes want their results to be different than it is and they will alter or fudge data to make it so.
Not unique to psychiatry and much more of a political issue. What disturbs me more than anything, perhaps because it was one of the leading reasons I dropped out of psychology, was Burt's studies because they continue to be propagated by supremacists and there is no evidence that any follow-up" studies have been better.
Perhaps the greatest crisis in science as a whole, but certainly in those outside of chemistry and physics, is the expectation of certainty. The proliferation of statistical tools and professional publishing has led to an expectation that we can divine simple truths from complex systems like sociology, economics, human biology, nutrition, etc.
The issue is not that we have a replicability crisis, it is that we as a society have an epistemic crisis - we expect more "truth" than the world, our senses, and our instruments can provide.
Psychology is not dead because we cannot treat it like a Hubermann episode, and philosophy is not dead because we can't prove it through double-blind experiments. Complex systems rarely yield to simple truths, so our modes of contemplation have to calibrate to a realistic level of resolution.
I really like this exercise of trying to identify the most significant findings, and I'd agree that 1-6 are worthy candidates. 7-10 are interesting/useful but not field-changing.
BUT among the first six, all of those findings emerged prior to the 1980's, definitely not "in the last few decades". I'm sure you know that #4 "perceptual inference" goes back to Helmholz, before psychology was even called psychology.
I don't think change blindness is the most substantial illustration of the limitations of consciousness, but even that finding goes back to the 1970's, before ubiquitous peer review and other homogenizing practices.
In "recent decades" we've given more confidence to these six findings but mostly we've just been filling in the details---for example: understanding average developmental differences between 10 months and 14 months, but not overturning the fundamental theories/concepts that organized the paradigm in the first place.
So it feels like your examples actually underscore Adam's point.
Your list of 10 is really impressive (I noted this at the time, but I think it bears repeating - those are great insights).
The plane crashed and nobody checked the bodies because it’s a not a 747, it’s a light plane and the pilot survived the crash
Your article does not only serve as a response to Mastroianni's suggestions, but also an indicator to how psychology-related experiments should be scrutinized and appraised by researchers in general. If you originally have trust in a certain psychological principle, even if you find a single case of fraud, you would likely take into account countless other studies and researches that still back up the principle. Therefore, unless it's a case where the most representative research is attacked, or where the total sum of researches are unconvincing or shallow in the first place, Mastroianni's concerns should be inconsequential.
I think this shows how researchers can also fall vulnerable to confirmation bias. If you find a principle valid, the faulty case would simply be one outlier among countless other studies that say otherwise. However, if you have doubts in a principle in the first place, that would be when you would solely spotlight the single faulty research against the rest.
5,6,8 and 9 are findings from behavioral genetics that most psychologists consider debunked pseudoscience. Even w(o)ikipedia says the damns the whole field and labels it controversial.
This rocks!
Very interesting. Your top-10 findings list was especially interesting. I have always wanted to ask a psychologist if anyone in your field has made an attempt to falsify the "mimetic desire" theory discussed by René Girard. I recently read about his theory in his analysis of Shakespeare called Theatre of Envy, which I heard about in a Tyler Cowen podcast and I have no idea what to make of it because I'm not smart enough. But if it were true it would seem to be a top-10 type discovery and apparently Shakespeare discovered it. And if it were false then that would be a pretty big deal too since Girard has recently been gaining steam.
Many basic learning phenomena (e.g. conditional reflexes, operant schedule effects and stimulus control) are so easily replicated that they are often used as basic class demonstrations. And, perhaps more important, they are easily conducted at the level of the individual, thus avoiding the many drawbacks of large sample NHST research. Such inductive research is easily subjected to interparticipant replications.