In its own small way, the existence of the SCORE report indicates a degree of health in the scientific enterprise, which is only as good as its error metabolism is. The kind of critical work SCORE represents identifies error in the body scientific, and the kind of publishing venue Nature represents makes these identifications visible.
This is exactly right! And it's the part that's easy to miss when the headline is '50% fail.' The fact that this project exists, got funded for seven years, and landed in Nature is itself evidence that the immune system is (at least somewhat) working. The question is whether the metabolism is fast enough. Brodeur's numbers from 2022-23 suggest it's speeding up in the fields that built the infrastructure. The fields that haven't are still running a fever...
I'm a sociologist and am obviously unhappy with how sociology fared here. My response is to discuss this with my1st year grad students and to say, let's do better. (Transparency is taking hold in sociology, though, to be sure, more slowly than elsewhere.) Thanks for highlighting this important work.
The fact that you're putting it in front of first-year students is where it actually matters. Norms don't change through journal editorials much (as you know). They change when someone in a methods seminar says "here's what the data look like, and here's what we're going to do differently."
Good to hear that sociology's transparency infrastructure is coming along, and SCORE giving you a concrete set of numbers to point at in that classroom is worth something.
The trajectory matters more than the snapshot. Good on you.
Seems to me that side of this is its own horror show, and in some ways a harder problem than what SCORE is measuring? At least in social science you can usually rerun an analysis or field a new survey.
Would actually be curious whether you're seeing the same institutional pattern (that the fields or labs with enforced transparency norms do measurably better) or whether the biology introduces enough noise that infrastructure alone can't close the gap...?
The problem is that attempts at reproducibility in biomedicine are very costly and time-consuming. One of the most serious attempts I am aware of was the Reproducibility Project: Cancer Biology. It was a large, multi-year effort with roughly $1.5 million in funding that aimed to replicate key experiments from around 50 high-impact cancer biology papers (we are talking Nature, Cell etc.).
In the end, due to cost and practical constraints, only a subset of the planned experiments could actually be replicated, and each replication took several months and tens of thousands of dollars. The results were mixed: many findings could not be reproduced, and even when they were reproduced, the effect sizes were often smaller or more uncertain than originally reported.
So basically, as far as I’m aware, it‘s difficult to find out if interventions on the level of institutional infrastructure work because you only get indirect hints and learn what‘s reproducible only years later (if at all, some concepts are just forgotten and never get enough traction).
Richard Feynman criticized sociology and psychology—as "cargo cult science," arguing they adopt the outward forms of scientific inquiry (data gathering, jargon, professional titles) without adhering to rigorous empirical testing, resulting in few, if any, foundational laws and lacked predictive power and rigor.
I love to challenge papers which use proxy metrics (social sciences inherently struggle to measure things in isolation) to see if the authors attempted to falsify the premises. Either way it is generally a struggle towards opaqueness, whereas in hard sciences falsification leads to clarity.
To be fair, Feynman's (one of my intellectual heroes, and I know I’m not unique in that) actual target was a bit narrower than that — his "Cargo Cult Science" talk (1974 Caltech commencement) went after parapsychology, sloppy education research, and poorly controlled psych experiments. He wasn't arguing that sociology and psychology can't produce laws or lack rigor as fields. He was arguing that individual researchers sometimes skip the part where you try to prove yourself wrong. It's a critique of practices, not disciplines necessarily.
That said, his underlying point — that going through the motions of science isn't the same as doing science — is basically what SCORE just demonstrated with a $30 million budget and seven years of labor. The fields that built real accountability infrastructure (data editors, code checks, sharing mandates) reproduced at nearly twice the rate of fields that didn't. Feynman would probably be delighted by that finding.
He'd also probably point out that it took us fifty years to run the experiment while playing his bongos.
This read like AI generated content, so I paid closer attention and nearly immediately noticed an error in one of the graphs: You wrote "Political science: 84% of papers with data available were approximately or precisely reproduced, 66% precisely. Economics: 77% approximately or precisely, 72% precisely. "
But the accompanying figure clearly shows economics with a smaller percentage of precise replication, something below 70%.
The fact that I found this error this quickly strongly suggests there are many other errors in this post.
The text numbers come directly from Miske et al. and the original chart was built from a visual read of the Nature & Views figures — economics was showing ~65% precisely reproduced when the paper reports 72%. That's my mistake. Fixed now. Appreciate the close read.
(I do use AI to edit, btw, but the ideas/content are mine. Still my mistake.)
In its own small way, the existence of the SCORE report indicates a degree of health in the scientific enterprise, which is only as good as its error metabolism is. The kind of critical work SCORE represents identifies error in the body scientific, and the kind of publishing venue Nature represents makes these identifications visible.
This is exactly right! And it's the part that's easy to miss when the headline is '50% fail.' The fact that this project exists, got funded for seven years, and landed in Nature is itself evidence that the immune system is (at least somewhat) working. The question is whether the metabolism is fast enough. Brodeur's numbers from 2022-23 suggest it's speeding up in the fields that built the infrastructure. The fields that haven't are still running a fever...
I'm a sociologist and am obviously unhappy with how sociology fared here. My response is to discuss this with my1st year grad students and to say, let's do better. (Transparency is taking hold in sociology, though, to be sure, more slowly than elsewhere.) Thanks for highlighting this important work.
The fact that you're putting it in front of first-year students is where it actually matters. Norms don't change through journal editorials much (as you know). They change when someone in a methods seminar says "here's what the data look like, and here's what we're going to do differently."
Good to hear that sociology's transparency infrastructure is coming along, and SCORE giving you a concrete set of numbers to point at in that classroom is worth something.
The trajectory matters more than the snapshot. Good on you.
Great that this type of work is being done.
I wrote a piece on the reproducibility crisis in preclinical biomedical research (on my substack) from some first hand experience in the field.
Seems to me that side of this is its own horror show, and in some ways a harder problem than what SCORE is measuring? At least in social science you can usually rerun an analysis or field a new survey.
Would actually be curious whether you're seeing the same institutional pattern (that the fields or labs with enforced transparency norms do measurably better) or whether the biology introduces enough noise that infrastructure alone can't close the gap...?
The problem is that attempts at reproducibility in biomedicine are very costly and time-consuming. One of the most serious attempts I am aware of was the Reproducibility Project: Cancer Biology. It was a large, multi-year effort with roughly $1.5 million in funding that aimed to replicate key experiments from around 50 high-impact cancer biology papers (we are talking Nature, Cell etc.).
In the end, due to cost and practical constraints, only a subset of the planned experiments could actually be replicated, and each replication took several months and tens of thousands of dollars. The results were mixed: many findings could not be reproduced, and even when they were reproduced, the effect sizes were often smaller or more uncertain than originally reported.
So basically, as far as I’m aware, it‘s difficult to find out if interventions on the level of institutional infrastructure work because you only get indirect hints and learn what‘s reproducible only years later (if at all, some concepts are just forgotten and never get enough traction).
Some really important research and a *very* helpful explanation plus insights here.
very kind, thank you. :) tell your friends, I guess? :)
Richard Feynman criticized sociology and psychology—as "cargo cult science," arguing they adopt the outward forms of scientific inquiry (data gathering, jargon, professional titles) without adhering to rigorous empirical testing, resulting in few, if any, foundational laws and lacked predictive power and rigor.
I love to challenge papers which use proxy metrics (social sciences inherently struggle to measure things in isolation) to see if the authors attempted to falsify the premises. Either way it is generally a struggle towards opaqueness, whereas in hard sciences falsification leads to clarity.
To be fair, Feynman's (one of my intellectual heroes, and I know I’m not unique in that) actual target was a bit narrower than that — his "Cargo Cult Science" talk (1974 Caltech commencement) went after parapsychology, sloppy education research, and poorly controlled psych experiments. He wasn't arguing that sociology and psychology can't produce laws or lack rigor as fields. He was arguing that individual researchers sometimes skip the part where you try to prove yourself wrong. It's a critique of practices, not disciplines necessarily.
That said, his underlying point — that going through the motions of science isn't the same as doing science — is basically what SCORE just demonstrated with a $30 million budget and seven years of labor. The fields that built real accountability infrastructure (data editors, code checks, sharing mandates) reproduced at nearly twice the rate of fields that didn't. Feynman would probably be delighted by that finding.
He'd also probably point out that it took us fifty years to run the experiment while playing his bongos.
This read like AI generated content, so I paid closer attention and nearly immediately noticed an error in one of the graphs: You wrote "Political science: 84% of papers with data available were approximately or precisely reproduced, 66% precisely. Economics: 77% approximately or precisely, 72% precisely. "
But the accompanying figure clearly shows economics with a smaller percentage of precise replication, something below 70%.
The fact that I found this error this quickly strongly suggests there are many other errors in this post.
The text numbers come directly from Miske et al. and the original chart was built from a visual read of the Nature & Views figures — economics was showing ~65% precisely reproduced when the paper reports 72%. That's my mistake. Fixed now. Appreciate the close read.
(I do use AI to edit, btw, but the ideas/content are mine. Still my mistake.)
There are 3 “it’s not x it’s y” formulations which make me nervous about AI use elsewhere. I’d avoid it for editing tbh.