This is similar to Biblical scholars’ “Criterion of Embarrassment.” If an account is embarrassing to its authors’ worldview, it’s regarded as more likely to be true.
Another suggestion I have is that we should require newely minted PhDs to take an oath like doctors do -- an oath that recognizes the special position of trust they have as an expert and requires them to swear to -- when speaking as an expert -- always attempt to convey the full state of the evidence not cherry pick positions for personal comfort or partisan benefit, to endeavor to publish in ways that add to our overall understanding and never p-hack or hide unwanted outcomes and to speak up to correct the record about their area of expertise even when they fear the consequences of how people would react.
While I think part of the benefit would just be getting academics to think more seriously about the impact of their work the most important aspect is that it creates an excuse for why you are speaking up.
If you're an epidemiologist who is considering standing up to say BLM rallies are dangerous because of the potential for COVID spreading you will reasonably worry people will infer the reason you spoke up is because you are against BLM. An oath like this gives you another reason you can point to.
This is why I believe all scientific journals should reserve a slot (or a number of them) for the best rebuttal piece submitted within some period of time after some publication.
The exact structure can be tweaked but the key observation is that, more than politics, publications matter to academics and if you can get a publication more easily by challenging a finding they'll do that. And you can give the original authors their own response as well and the net effect will be to create a more accurate academic record.
Sure, but I find the political partisanship framing of the piece reductive and to some extent counterproductive because discussions will again fall into the tedious and unproductive us-versus-them discussion that led to the intellectual dark web fiasco.
I find that there are two broader, more fundamental issues, worth exploring:
1- Scientific Rigor and Popular Appeal: Research that aligns with popular hopes or beliefs - regardless of political orientation - often receives less rigorous scrutiny. This extends far beyond any single ideological domain. Recent examples include claims about social media's impact on youth, smartphone effects on cognition, mindfulness benefits, and psychedelics' therapeutic potential. When mainstream media eagerly amplifies these findings, journal editors may face pressure to expedite publication of studies that warrant more thorough examination.
2-The Misuse of Empirical Research for Moral Arguments: We should ground moral positions in ethical reasoning rather than contingent empirical findings. Take workplace diversity: If we believe in its intrinsic value, we should advocate for it based on moral principles of human dignity and representation. Supporting diversity solely through productivity studies is both intellectually dishonest and strategically flawed - few would accept fully homogeneous teams even if data suggested they performed better. Similar issues arise with environmental corporate responsibility and other cases where empirical arguments are used to buttress fundamentally moral positions.
The issues you raise are good ones (though I disagree that moral positions have to be disengaged from empirical facts.) I'm not sure, though, why you believe that my own topic shouldn't be discussed.
I don't think that's what I said. I just thought it would have been interesting to make the more general point about hype, bias and rigor. Maybe it's election fatigue, but I seem to have a knee-jerk aversion to any progressives vs conservatives discussions. Sorry if I seemed dismissive. I've been following you on podcasts and in writing for years, and you always seem to have a better, more insightful, detached take on these issues – I hope to read more of them here.
If you use empirical data to argue why moral positions are good (like in the PNAS paper you mention: "[Our results] serve as an important call to continue the diversification of the medical workforce"), then you place the debate on empirical grounds, where, as an honest scientist, you should be open to changing your position in light of new evidence. Most wouldn't – and shouldn't – because the core argument for diversity rests on human dignity and equality, not performance metrics.
"because the core argument for diversity rests on human dignity and equality, not performance metrics."
And the core argument against diversity rests on the same things.
When it comes to basketball, no-one gives a fig for diversity. They seem to be happy for blacks to be "over represented" there. And theere is a reson why we are happy with that, because it can be proven scientifically that the mass of black men are better are basketball than the mass of peoploe from other races. But of course, we don't really resort to science in determining that diversity is not important in basketball. We hust know that the game wouldn't be as good if the players had to look "more like America".
What is important for human dignity is not enforced diversity, which by its very nature will exclude people of talent, but openess to diversity. Equality means equality of opportunity, not equality of outcome.
Ginger Rogers didn't do anywhere near all the moves that Astaire did, as anyone who's watched more than a few minutes of them dancing knows. The old cliche about her doing "all of it but backwards in heels" is, ironically, another example of exactly the same unskeptical wishcasting that you rightly call out regarding scientific-research findings.
For what it's worth, I first heard the Ginger assertion made by Ann Richards, then Texas Governor (during her failed reelection against George W. Bush), as part of her feminist mythology. I didn't believe it then, as I had read how much work Fred and Hermes Pan, his choreographer, put into the dances long before Ginger came on to the set. The point of the meme (sorry to be anachronistic) is not whether it is true but as a useful point of reference for this piece. In that context, I think the meme works. Also, in other fields, such as the natural sciences (I'm a geophysicist), the work that supports one's preferred paradigm (which underscores your funded research project) is going to get your recommendation for publication, while you may jump through hoops to find objections to that manuscript which could sink your project. At the receiving end of such gymnastics my response has been to reexamine my research, usually find something I had missed before while completely rewriting the work, publishing it elsewhere, and, if it sinks somebody's project, be blissfully ignorant (although I'm still bugged about the initial treatment). Science, whether natural or social, is very human.
On point, Paul. I’m of the opinion that publishing research results tends to be beneficial even when the results are uncomfortable or when there is a potential for harm. But I can’t help wondering if my opinion is situational.
Your example of the retraction of the Nature Communications article is a good one. There is clearly a potential for harm created by publishing the results, but what about the harm from suppressing the results? Suppose we continue to assign female mentors to female scientists in the unwitting belief that it is better for them? Or, more importantly, because we refuse to acknowledge a difference between assigning a male versus female mentor, we’re not studying why the difference exists and how to correct it?
On the other hand, I could envision a scenario where a group, in trying to show that there are no statistically significant differences in intelligence between races, inadvertently shows that there are. Do they publish? Unlike the previous example, there is arguably no benefits to society or science in their results (at least, none that I can think of), and the potential for harm is extraordinary. I’m afraid that if I’m a reviewer of their paper, I might suggest not publishing.
Your last example is a lot like the question of whether a doctor should kill one organ donor patient in surgery so he can save 5 with his organs. Yes, if you imagine the doctor could do so with no risk of ever being discovered it's a hard question -- in the real world it's easy because the harms that would result in people distrusting the medical system in the case the doctor gets caught far outweigh the benefits.
Ultimately, the value of most scientific publication derives from it's ability to shift what people believe (no point publishing into the void) and if you don't believe troubling findings would be published if true then you don't get much information from the scientific literature in the first place.
Reading the literature has convinced me (eg the very nice Stanford study on traffic stops) that blacks are generally treated worse by the police (of all races) and a number of other important things. But the fact that journals published results challenging the claims about differences in rates of fatal shootings was equally important to me being persuaded so I knew it wasn't true that they'd have refused to publish a devastating counter argument. However on some issues I am starting to distrust the scientific record for exactly that reason.
In other words the harmful stuff needs to be published because if it's not we'll figure it out and stop trusting it all.
Counterpoint- if there is a statistically significant intelligence gap between races, isn't that important to know? And isn't knowing that it exists the first step in understanding why it exists? What if, hypothetically, one race was being systematically exposed to some intelligence-reducing factor (say, lead in water pipes)? That would mean the intelligence gap was something we could actually ameliorate- unless of course we collectively plug our ears and hum by suppressing those findings for fear of harm.
One thing that I would add to buttress your point in the conclusion as to the desirability of truth seeking institutions is that we have so few out there in the world! Besides science, how many institutions do we really have which strive (ideally) to incentivize truth-maximization? There aren’t that many.
That’s why it’s so important I feel to keep ideology out of science. This is true even if you think ideology is super important. There already are so many institutions out there dedicated to maximizing ideological aims, let’s not make science one of them, otherwise we run into the danger of running out of any institution which incentives the search for the truth, and that would be disastrous. Surely even the most ideologically committed person would agree that we should have at least one institution out there dedicated to discovering truth, so let’s keep truth-warping incentives out of science.
If we lived in a world where most institutions besides scientific ones were boring “just the facts please” types of social organizations dedicated to truth-maximization, and ideology was a dirty word, then maybe it might make sense to promote ideology at the expense of truth. But we don’t live in that world.
This is great, and receives all the more legitimacy (as per the article!) coming from someone who (I think) is generally fairly liberal. It's a shame that the necessity of this sort of rational evaluation of evidence (which would always be necessary, not just in response to a particular political climate) isn't more widely understood. I imagine we do it intuitively in many contexts.
As an outsider, I wish there was more talk about these sorts of issues from principled people. It seems like whenever someone does attempt this, it becomes their whole "thing", they become less principled/lose themselves to fighting a larger culture war and their criticism loses its legitimacy (I think Jordan Peterson is a decent, though perhaps especially odd, example of this). If the "minority of professors" you mentioned who are more ideologically driven is indeed a minority, then the majority should be doing a better job of being explicit about how they think these issues should be handled. Otherwise, presumably, the minority dictates the attitudes and standards of the field as a whole.
I'm having trouble articulating this in detail, but I think you get idea.
This recent article about climate scientist Patrick Brown provides a good account of his experience of the dynamic you’re describing and his attempt to stay oriented towards the full picture…
I follow him and Roger Pielke Jr. for exactly the purpose of checking the overall tendency of climate journalism towards more alarmist accounts of the science.
So... in conclusion, you hate freedom ? ;-) Seriously, great article. I know you are primarily an academic, but you write in such an accessible writing style that I wish more people would adopt from that world.
Good article. The reforms I would like to see involve a clear acknowledgement that truth and activism are not the same thing, and are often in conflict. Perhaps universities could do more to establish a wall of separation between truth/knowledge and activism by establishing dedicated schools of Activism, and moving the various 'studies' programs into that.
Well, if you're into 19th century Norwegian drama, it's certainly a go-to. I bring it up because Henrik Ibsen was, for most of his career, a "truth-at-all-costs" sort of guy. His play An Enemy of the People concerns a doctor who discovers that the water at his town's new spa is contaminated. He wants, reasonably, the truth to come out, but the political pressure against doing so mounts, and he quickly becomes - well, the title.
Ibsen softened his view in later life. The Wild Duck, a late play, deals with a young man who is determined that the truth come out about his family, and damn the consequences. The consequences are predictably awful.
I recoil against the "truth....but managed truth" attitude that prevails in a lot of the academy (if you think it's bad in the sciences, I recommend not looking at the humanities). But I understand its origins and rationale. The development of academic disciplines - and the modern university - coincided with hegemonic cultural and scientific racism. That "truth" was/is used as the blunt cudgel of repression can't invalidate "Truth." But it can give us a moment or two to think if what we're about to publish might be adding a few more footpounds of pressure to the boot on somebody's neck.
I very much like the idea of a "Ginger Rogers" test, but if next week Verso published "The Bell Curve: The Sequel" I'm afraid I would find little edifying in the ensuing conflagration.
1) Love "Ginger Rogers findings" (althoughy, how sure are you about this analysis? ;) ).
2) Much like the king being ridiculed (or whatever) for one day of the year, should we institute a rule that for one day a year, Lee Jussim gets to add red pen wherever he likes?
3) Particularly egregious (myself included I'm sure) is when folks argue with a well researched finding that , if true, would significantly benfit humanity. No-one likes good news. Let it bleed and lead.
The Astaire/Rogers comment is cute, and now iconic, but inaccurate. More than half the time the couple dance beside each other, facing forward, moving sideways, forwards and backwards, and another significant movement is rotation, when each person is forwards and backwards by turn. Beyond that, if Astaire leads her backwards, he has to return, which often entails reversing backwards while Rogers goes forward. But most striking, and possibly their signature, is dancing beside each other facing forward. Their precise unison is one of their appeals. It's the equality of the virtuosity that is so powerful.
This is similar to Biblical scholars’ “Criterion of Embarrassment.” If an account is embarrassing to its authors’ worldview, it’s regarded as more likely to be true.
https://en.m.wikipedia.org/wiki/Criterion_of_embarrassment
thanks -- that's excellent
Another suggestion I have is that we should require newely minted PhDs to take an oath like doctors do -- an oath that recognizes the special position of trust they have as an expert and requires them to swear to -- when speaking as an expert -- always attempt to convey the full state of the evidence not cherry pick positions for personal comfort or partisan benefit, to endeavor to publish in ways that add to our overall understanding and never p-hack or hide unwanted outcomes and to speak up to correct the record about their area of expertise even when they fear the consequences of how people would react.
While I think part of the benefit would just be getting academics to think more seriously about the impact of their work the most important aspect is that it creates an excuse for why you are speaking up.
If you're an epidemiologist who is considering standing up to say BLM rallies are dangerous because of the potential for COVID spreading you will reasonably worry people will infer the reason you spoke up is because you are against BLM. An oath like this gives you another reason you can point to.
This is why I believe all scientific journals should reserve a slot (or a number of them) for the best rebuttal piece submitted within some period of time after some publication.
The exact structure can be tweaked but the key observation is that, more than politics, publications matter to academics and if you can get a publication more easily by challenging a finding they'll do that. And you can give the original authors their own response as well and the net effect will be to create a more accurate academic record.
Sure, but I find the political partisanship framing of the piece reductive and to some extent counterproductive because discussions will again fall into the tedious and unproductive us-versus-them discussion that led to the intellectual dark web fiasco.
I find that there are two broader, more fundamental issues, worth exploring:
1- Scientific Rigor and Popular Appeal: Research that aligns with popular hopes or beliefs - regardless of political orientation - often receives less rigorous scrutiny. This extends far beyond any single ideological domain. Recent examples include claims about social media's impact on youth, smartphone effects on cognition, mindfulness benefits, and psychedelics' therapeutic potential. When mainstream media eagerly amplifies these findings, journal editors may face pressure to expedite publication of studies that warrant more thorough examination.
2-The Misuse of Empirical Research for Moral Arguments: We should ground moral positions in ethical reasoning rather than contingent empirical findings. Take workplace diversity: If we believe in its intrinsic value, we should advocate for it based on moral principles of human dignity and representation. Supporting diversity solely through productivity studies is both intellectually dishonest and strategically flawed - few would accept fully homogeneous teams even if data suggested they performed better. Similar issues arise with environmental corporate responsibility and other cases where empirical arguments are used to buttress fundamentally moral positions.
The issues you raise are good ones (though I disagree that moral positions have to be disengaged from empirical facts.) I'm not sure, though, why you believe that my own topic shouldn't be discussed.
I don't think that's what I said. I just thought it would have been interesting to make the more general point about hype, bias and rigor. Maybe it's election fatigue, but I seem to have a knee-jerk aversion to any progressives vs conservatives discussions. Sorry if I seemed dismissive. I've been following you on podcasts and in writing for years, and you always seem to have a better, more insightful, detached take on these issues – I hope to read more of them here.
If you use empirical data to argue why moral positions are good (like in the PNAS paper you mention: "[Our results] serve as an important call to continue the diversification of the medical workforce"), then you place the debate on empirical grounds, where, as an honest scientist, you should be open to changing your position in light of new evidence. Most wouldn't – and shouldn't – because the core argument for diversity rests on human dignity and equality, not performance metrics.
"because the core argument for diversity rests on human dignity and equality, not performance metrics."
And the core argument against diversity rests on the same things.
When it comes to basketball, no-one gives a fig for diversity. They seem to be happy for blacks to be "over represented" there. And theere is a reson why we are happy with that, because it can be proven scientifically that the mass of black men are better are basketball than the mass of peoploe from other races. But of course, we don't really resort to science in determining that diversity is not important in basketball. We hust know that the game wouldn't be as good if the players had to look "more like America".
What is important for human dignity is not enforced diversity, which by its very nature will exclude people of talent, but openess to diversity. Equality means equality of opportunity, not equality of outcome.
This will depend a lot on your morality. Consequentialists / Utilitarians will look to outcomes to determine morality and hence empirical evidence.
Ginger Rogers didn't do anywhere near all the moves that Astaire did, as anyone who's watched more than a few minutes of them dancing knows. The old cliche about her doing "all of it but backwards in heels" is, ironically, another example of exactly the same unskeptical wishcasting that you rightly call out regarding scientific-research findings.
Interesting, thanks. I'll admit that I've never seen their movies; it's just a nice metaphor.
For what it's worth, I first heard the Ginger assertion made by Ann Richards, then Texas Governor (during her failed reelection against George W. Bush), as part of her feminist mythology. I didn't believe it then, as I had read how much work Fred and Hermes Pan, his choreographer, put into the dances long before Ginger came on to the set. The point of the meme (sorry to be anachronistic) is not whether it is true but as a useful point of reference for this piece. In that context, I think the meme works. Also, in other fields, such as the natural sciences (I'm a geophysicist), the work that supports one's preferred paradigm (which underscores your funded research project) is going to get your recommendation for publication, while you may jump through hoops to find objections to that manuscript which could sink your project. At the receiving end of such gymnastics my response has been to reexamine my research, usually find something I had missed before while completely rewriting the work, publishing it elsewhere, and, if it sinks somebody's project, be blissfully ignorant (although I'm still bugged about the initial treatment). Science, whether natural or social, is very human.
On point, Paul. I’m of the opinion that publishing research results tends to be beneficial even when the results are uncomfortable or when there is a potential for harm. But I can’t help wondering if my opinion is situational.
Your example of the retraction of the Nature Communications article is a good one. There is clearly a potential for harm created by publishing the results, but what about the harm from suppressing the results? Suppose we continue to assign female mentors to female scientists in the unwitting belief that it is better for them? Or, more importantly, because we refuse to acknowledge a difference between assigning a male versus female mentor, we’re not studying why the difference exists and how to correct it?
On the other hand, I could envision a scenario where a group, in trying to show that there are no statistically significant differences in intelligence between races, inadvertently shows that there are. Do they publish? Unlike the previous example, there is arguably no benefits to society or science in their results (at least, none that I can think of), and the potential for harm is extraordinary. I’m afraid that if I’m a reviewer of their paper, I might suggest not publishing.
Your last example is a lot like the question of whether a doctor should kill one organ donor patient in surgery so he can save 5 with his organs. Yes, if you imagine the doctor could do so with no risk of ever being discovered it's a hard question -- in the real world it's easy because the harms that would result in people distrusting the medical system in the case the doctor gets caught far outweigh the benefits.
Ultimately, the value of most scientific publication derives from it's ability to shift what people believe (no point publishing into the void) and if you don't believe troubling findings would be published if true then you don't get much information from the scientific literature in the first place.
Reading the literature has convinced me (eg the very nice Stanford study on traffic stops) that blacks are generally treated worse by the police (of all races) and a number of other important things. But the fact that journals published results challenging the claims about differences in rates of fatal shootings was equally important to me being persuaded so I knew it wasn't true that they'd have refused to publish a devastating counter argument. However on some issues I am starting to distrust the scientific record for exactly that reason.
In other words the harmful stuff needs to be published because if it's not we'll figure it out and stop trusting it all.
Counterpoint- if there is a statistically significant intelligence gap between races, isn't that important to know? And isn't knowing that it exists the first step in understanding why it exists? What if, hypothetically, one race was being systematically exposed to some intelligence-reducing factor (say, lead in water pipes)? That would mean the intelligence gap was something we could actually ameliorate- unless of course we collectively plug our ears and hum by suppressing those findings for fear of harm.
One thing that I would add to buttress your point in the conclusion as to the desirability of truth seeking institutions is that we have so few out there in the world! Besides science, how many institutions do we really have which strive (ideally) to incentivize truth-maximization? There aren’t that many.
That’s why it’s so important I feel to keep ideology out of science. This is true even if you think ideology is super important. There already are so many institutions out there dedicated to maximizing ideological aims, let’s not make science one of them, otherwise we run into the danger of running out of any institution which incentives the search for the truth, and that would be disastrous. Surely even the most ideologically committed person would agree that we should have at least one institution out there dedicated to discovering truth, so let’s keep truth-warping incentives out of science.
If we lived in a world where most institutions besides scientific ones were boring “just the facts please” types of social organizations dedicated to truth-maximization, and ideology was a dirty word, then maybe it might make sense to promote ideology at the expense of truth. But we don’t live in that world.
This is great, and receives all the more legitimacy (as per the article!) coming from someone who (I think) is generally fairly liberal. It's a shame that the necessity of this sort of rational evaluation of evidence (which would always be necessary, not just in response to a particular political climate) isn't more widely understood. I imagine we do it intuitively in many contexts.
As an outsider, I wish there was more talk about these sorts of issues from principled people. It seems like whenever someone does attempt this, it becomes their whole "thing", they become less principled/lose themselves to fighting a larger culture war and their criticism loses its legitimacy (I think Jordan Peterson is a decent, though perhaps especially odd, example of this). If the "minority of professors" you mentioned who are more ideologically driven is indeed a minority, then the majority should be doing a better job of being explicit about how they think these issues should be handled. Otherwise, presumably, the minority dictates the attitudes and standards of the field as a whole.
I'm having trouble articulating this in detail, but I think you get idea.
I agree with all that. Many people, including me, hate getting into culture war issues because they want it to take over their lives and identity.
Wonderful piece, Paul! I will definitely be assigning this to my students and sharing it with a lot of people.
This recent article about climate scientist Patrick Brown provides a good account of his experience of the dynamic you’re describing and his attempt to stay oriented towards the full picture…
https://grist.org/science/patrick-brown-profile-climate-scientist-criticized-study/
I follow him and Roger Pielke Jr. for exactly the purpose of checking the overall tendency of climate journalism towards more alarmist accounts of the science.
I came to post that Grist profile but Jason got there first...
So... in conclusion, you hate freedom ? ;-) Seriously, great article. I know you are primarily an academic, but you write in such an accessible writing style that I wish more people would adopt from that world.
Good article. The reforms I would like to see involve a clear acknowledgement that truth and activism are not the same thing, and are often in conflict. Perhaps universities could do more to establish a wall of separation between truth/knowledge and activism by establishing dedicated schools of Activism, and moving the various 'studies' programs into that.
Somebody hasn't read Ibsen's The Wild Duck. ;)
guilty as charged!
Well, if you're into 19th century Norwegian drama, it's certainly a go-to. I bring it up because Henrik Ibsen was, for most of his career, a "truth-at-all-costs" sort of guy. His play An Enemy of the People concerns a doctor who discovers that the water at his town's new spa is contaminated. He wants, reasonably, the truth to come out, but the political pressure against doing so mounts, and he quickly becomes - well, the title.
Ibsen softened his view in later life. The Wild Duck, a late play, deals with a young man who is determined that the truth come out about his family, and damn the consequences. The consequences are predictably awful.
I recoil against the "truth....but managed truth" attitude that prevails in a lot of the academy (if you think it's bad in the sciences, I recommend not looking at the humanities). But I understand its origins and rationale. The development of academic disciplines - and the modern university - coincided with hegemonic cultural and scientific racism. That "truth" was/is used as the blunt cudgel of repression can't invalidate "Truth." But it can give us a moment or two to think if what we're about to publish might be adding a few more footpounds of pressure to the boot on somebody's neck.
I very much like the idea of a "Ginger Rogers" test, but if next week Verso published "The Bell Curve: The Sequel" I'm afraid I would find little edifying in the ensuing conflagration.
Abolish peer review.
1) Love "Ginger Rogers findings" (althoughy, how sure are you about this analysis? ;) ).
2) Much like the king being ridiculed (or whatever) for one day of the year, should we institute a rule that for one day a year, Lee Jussim gets to add red pen wherever he likes?
3) Particularly egregious (myself included I'm sure) is when folks argue with a well researched finding that , if true, would significantly benfit humanity. No-one likes good news. Let it bleed and lead.
The Astaire/Rogers comment is cute, and now iconic, but inaccurate. More than half the time the couple dance beside each other, facing forward, moving sideways, forwards and backwards, and another significant movement is rotation, when each person is forwards and backwards by turn. Beyond that, if Astaire leads her backwards, he has to return, which often entails reversing backwards while Rogers goes forward. But most striking, and possibly their signature, is dancing beside each other facing forward. Their precise unison is one of their appeals. It's the equality of the virtuosity that is so powerful.
Thanks! As I said to the another commenter, I've never actually seen the movies. I just liked the metaphor. But I appreciate the clarification.