An excellent article. I would add that much of my work in experimental physics, in a lab, was to figure out what factors could be safely ignored.
For example, thermal scattering of electrons going through a crystal produces a diffuse background in a diffraction pattern. If you are trying to work out the symmetry of the crystal, the thermal scattering blur usually isn't "blurry" enough to stop you from working out the symmetry. But a crystal that is highly contaminated (random stuff has accumulated on its surface due to a poor vacuum) and you definitely won't be able to work the symmetry out.
So as well as finding "effects" in a lab, you can also rule out real-world factors when the experiment is slowly but surely introduced to the real world.
I would also add that the depressing thing about being an experimental physicist is that it often takes decades for your lab research to become relevant in the real world (lifetimes if you are an astrophysicist), if it ever does. That is just how slow the process is. We don't jump straight from a single lab experiment into the real world. We build the foundations first.
It does seem like there are some disanalogies with lab conditions for other aspects of nature. How a feather falls in a vacuum can tell you something about a basic principle of nature. It is not so easy to tell when you are isolating a basic principle of how psychology works in the lab. The complexity of the brain, and the fact that psychologies evolved to work in "messy", i.e., non-laboratory controlled environments, is obviously relevant.
I think this is a good argument and in general I’m very supportive of controlled studies that falsify theories—but that said, the extent to which a controlled lab study is useful or informative basically corresponds to whether it is, in fact, engaged in theory falsification. If a lab study is merely providing evidence consistent with a theory (or if the theory is not articulated with sufficient precision), then it’s mostly a kind of existence proof: there’s at least one setting under which the theory’s predictions hold. Not useless, but not as theoretically (or practically) impactful. I find that psychophysics and much of cognitive psychology does do deductive, falsification-based work—but that’s not always the case.
I just finished the late Bob Altemeyer's penultimate book, The Authoritarians, and this question was on my mind throughout. Altemeyer developed various tests that he claimed measured predisposition to be an authoritarian leader or follower, then deployed the tests on his psych students in Manitoba. His findings were replicated by colleagues who ran tests on their psych students in the USA. [Does testing psych undergrads for course credit count as controlled lab conditions?] His "highly sophisticated tests" seem to be opinion surveys on hot button socio-political issues of the day. He generalizes his results to cover the rest of humanity and says he has discovered a deep truth of social psych. The whole time I'm thinking that what he's done is discovered a deep truth about the socio-political opinions of psych students in Manitoba. How actionable is this insight, really?
To his credit. Altemeyer published the ebook for free and told us to do our own experiments to replicate or falsify his findings: https://theauthoritarians.org/
Interesting! As a new EdD student studying to be a scholar-practitioner in human learning, this post gives me a useful lens for understanding research — thank you! It does seem that the practitioner side is sometimes too quick to try to translate lab research into actionable (self-helpish) advice.
On the surface it might seem like this reflects the reductionism vs holism debate in philosophical jargon. That the causal chain in real world conditions is considerably more complicated?
Respect for Michael for admitting ones own field of study does not replicate.
Respect also for recognising the issue with empathy. More major thinkers seem to understand this.
But what about Lee Jussim and Stereotype Accuracy vs Stereotype threat. Are people really externalising and internalising? Or is it telling us something about human nature.
How much of the replication crisis is caused by empathy? Not a correlation with empathy?
The anti-MAGA professor who hits a MAGA person who is just trying to communicate with a bike lock?
Paul, Thanks for this interesting piece. You would enjoy our U of Chicago standing quip:
It makes sense it practice, but does it work in theory?
Best wishes,
Randall Paul
typo, sorry. 'in' practice
An excellent article. I would add that much of my work in experimental physics, in a lab, was to figure out what factors could be safely ignored.
For example, thermal scattering of electrons going through a crystal produces a diffuse background in a diffraction pattern. If you are trying to work out the symmetry of the crystal, the thermal scattering blur usually isn't "blurry" enough to stop you from working out the symmetry. But a crystal that is highly contaminated (random stuff has accumulated on its surface due to a poor vacuum) and you definitely won't be able to work the symmetry out.
So as well as finding "effects" in a lab, you can also rule out real-world factors when the experiment is slowly but surely introduced to the real world.
I would also add that the depressing thing about being an experimental physicist is that it often takes decades for your lab research to become relevant in the real world (lifetimes if you are an astrophysicist), if it ever does. That is just how slow the process is. We don't jump straight from a single lab experiment into the real world. We build the foundations first.
It does seem like there are some disanalogies with lab conditions for other aspects of nature. How a feather falls in a vacuum can tell you something about a basic principle of nature. It is not so easy to tell when you are isolating a basic principle of how psychology works in the lab. The complexity of the brain, and the fact that psychologies evolved to work in "messy", i.e., non-laboratory controlled environments, is obviously relevant.
I think this is a good argument and in general I’m very supportive of controlled studies that falsify theories—but that said, the extent to which a controlled lab study is useful or informative basically corresponds to whether it is, in fact, engaged in theory falsification. If a lab study is merely providing evidence consistent with a theory (or if the theory is not articulated with sufficient precision), then it’s mostly a kind of existence proof: there’s at least one setting under which the theory’s predictions hold. Not useless, but not as theoretically (or practically) impactful. I find that psychophysics and much of cognitive psychology does do deductive, falsification-based work—but that’s not always the case.
I just finished the late Bob Altemeyer's penultimate book, The Authoritarians, and this question was on my mind throughout. Altemeyer developed various tests that he claimed measured predisposition to be an authoritarian leader or follower, then deployed the tests on his psych students in Manitoba. His findings were replicated by colleagues who ran tests on their psych students in the USA. [Does testing psych undergrads for course credit count as controlled lab conditions?] His "highly sophisticated tests" seem to be opinion surveys on hot button socio-political issues of the day. He generalizes his results to cover the rest of humanity and says he has discovered a deep truth of social psych. The whole time I'm thinking that what he's done is discovered a deep truth about the socio-political opinions of psych students in Manitoba. How actionable is this insight, really?
To his credit. Altemeyer published the ebook for free and told us to do our own experiments to replicate or falsify his findings: https://theauthoritarians.org/
Interesting! As a new EdD student studying to be a scholar-practitioner in human learning, this post gives me a useful lens for understanding research — thank you! It does seem that the practitioner side is sometimes too quick to try to translate lab research into actionable (self-helpish) advice.
On the surface it might seem like this reflects the reductionism vs holism debate in philosophical jargon. That the causal chain in real world conditions is considerably more complicated?
Respect for Michael for admitting ones own field of study does not replicate.
Respect also for recognising the issue with empathy. More major thinkers seem to understand this.
But what about Lee Jussim and Stereotype Accuracy vs Stereotype threat. Are people really externalising and internalising? Or is it telling us something about human nature.
How much of the replication crisis is caused by empathy? Not a correlation with empathy?
The anti-MAGA professor who hits a MAGA person who is just trying to communicate with a bike lock?
https://www.youtube.com/watch?v=ZGaIFfH7bT0
A better example of Against Empathy you could not find nowadays.
What one is searching for, Messieurs, is COMPASSION.......
Granted, I didn’t read the whole article. But, “hypothesis confirmed” doesn’t sound like good understanding of how science works…