I've defended sociology, including on my Substack, and I think most research-active sociologists are trying to draw conclusions that are interesting and sufficiently justified by the data to get past reviewers. But the replication/reproducibility study is truly troubling, sociology is definitely embattled politically, and this piece (I mean Kyle's) seems fair to me, if we don't quibble over details--such as the significance of Sociologists for Palestine or the recent ASA election.
One thing that would help is greater recognition for well-designed studies with null results. A relative of mine is a high-energy physicist who wrote his dissertation on his work at CERN, at which he failed to find a new particle -- but established the conditions under which none was produced, which counts as a contribution. In sociology, "didn't find a significant result" generally means "didn't find anything worth publishing." That explains the significance chasing.
I think that sociologists will catch up in terms of data/code sharing, if we survive long enough, and that our sluggishness there is partly due to our methodological diversity, where in a lot of legitimate sociological research there's no data that can be shared and no code (or significance testing) at all. That may be diminishing the internal pressure to do things differently.
Happy to concede those details. :) Sociologists for Palestine was texture more than evidence. The ASA election I'd hold onto a little harder, since a neutrality candidate losing 51-37 is at least a sign the inside-reform door is closed, but I take your point not to hang the argument on either.
The null-results point is the best thing in this thread. Your CERN relative is the whole case in one story: physics can reward a well-built "we looked, it isn't there" because the null is informative when the field is chained to reality. Sociology's "didn't find anything worth publishing" is the incentive gradient working upstream of anyone's honesty.
Reward a clean null and you fix a lot of the significance-chasing without moralizing at a single person.
Your methodological-diversity point is fair, and it's a real limit on my framing. SCORE only audited quantitative work (I flagged that in the linked piece but not here), and plenty of good sociology has no code or data file to share, so "install a data editor" isn't universal. But my read is the underlying pressure is the same on both sides of the methods divide: publication rewards the interesting result, researcher degrees of freedom are everywhere, and the field hasn't built much that makes being wrong expensive for either tradition. The quantitative side just has a tractable audit. The interpretive side needs one of its own, and nobody's built it yet...
Appreciate you engaging this way, and for defending the field without pretending it's fine.
I was Melissa Wilde's colleague and am currently Rory McVeigh's colleague and the first I heard about Melissa's position about neutrality was when the Chronicle piece came out! What I had read was their respective positions on Sociologists for Palestine, which didn't seem very different to me. Agreed on qualitative sociology; we had a dust-up over that in connection with Alice Goffman not very long ago.
Kyle, I don’t think you’ve got this wrong exactly, but there might be something else going on. In my area, I’m not sure my associated fields of rural sociology and forest sociology have the same kind of culture. They continue to study useful things and help people and institutions with real-world problems, including folks at CSU. So perhaps it isn’t the “field of sociology” so much as “when sociology departments are disjunct from the responsibility to work with real people and their problems. In that void perhaps they make up their own perceived problems and become a self-reinforcing clique of thinking.
what a very good point...the subfield variance matters here too. A lot of applied land-grant work stays tethered because they answer to someone with a real problem: a county, an agency, a landowner who'll tell you the model was wrong when the thing you predicted doesn't happen. Fast, cheap correction. Take the client and the consequence away and you get exactly the void you describe, where a subfield starts inventing its own problems and grading itself on them.
So maybe the better frame isn't "the field is captured," it's "the field is captured wherever the reality clock went slow or got switched off."
The applied corners kept their auditor. Really glad you raised this...
I have an MS in Demography from Wisconsin 30 years ago. Technically my degree was in the College of Agriculture, and I got to know the rural and environmental sociologists. I was an RA for Bill Freudenberg, who was an outstanding person and scholar. I agree with you that much of this area is not politicized. But this does not invalidate the points made by the sociology critics cited in Kyle's article and the WSJ article. From a larger picture perspective, there are parts of sociology - crime and demography - that could easily reconstitute elsewhere. Org. Theory has pretty much moved to management even though AJS and ASR publish their stuff.
"Reality is the auditor" is an interesting phrase well beyond the precinct of sociology. The example of economics which has been extolling the virtues of globalization and free trade for a generation, to just now recognizing this is a significant problem. The other example is the whole climate science and green transition where the policy imperatives have overcame the science and technology inadequacies. These suggest the issue is fields where policy influence is THE primary purpose.
Publicly funded university programs shouldn't be involved with policy advocacy. Their role could be more usefully policy auditing.
Medical science and physics both have more than a few examples of invalid theories/products persisting despite being wrong/useless. See parity conservation in the weak force and non-drowsy Sudafed, for starters. It seems to me that this piece (and others) makes an unfair/inaccurate assumption that these conditions plague only the social sciences.
Additionally, social reality is constructed through intersubjective interpretations of the world. Full stop. These are far more mutable than the laws of physics, and in fact, can radical alter when human perception also changes. One of humanity’s fascinating capabilities is to create world-altering ideas that have no basis in physical reality. There may not be a single molecule of mercy in the universe nor an atom of democracy nor even a libation of liberal. But these concepts, however ill-defined, change the world.
Therefore, the social sciences must be kept open enough to allow for the emergence and reinforcement of such ideas as well as the improved understanding of how they work. If we move to a binary pass-fail model for research and we stifle the magnificent power of these disciplines, and kneecap our own potential for true progress.
That is not to say that the current model of gatekeeping and resource allocation is not flawed. In fact, we know it has been, at least in part, captured by both corporate and national security interests, and subverted to reinforce dogma. But like democracy itself, it may be the worst form…except for all the others.
I think both your examples actually make my case. Parity conservation was bedrock physics right up until somebody tested it, and then it was gone in a few months. Non-drowsy Sudafed is on its way out too; the FDA finally moved to pull oral phenylephrine in 2024, decades after the evidence turned. That's reality batting last on two different clocks, which is the whole argument. And I'm not letting the fast fields off the hook. The piece says outright that physics and medicine get things wrong too. They just get caught sooner and cheaper. The rot is everywhere.
On the construction point, I'm with you more than you'd guess. Democracy, mercy, liberal, not one of them has a molecule, and they've all reshaped the world. I recently spent a whole long essay on "liberal" meaning almost nothing and mattering enormously. But that "full stop" of yours is really a comma. A concept being constructed doesn't mean claims about it can't be checked. "Democracy" is contested all the way down; "this reform bumped turnout six points" still either holds up or it doesn't. That second kind of sentence is where the whole replication problem lives. The map's up for grabs. The territory still bites.
And no one is proposing a pass/fail test for ideas. That's the thing that scares me, and it's exactly what I call the bulldozer: some outside power deciding which fields get to exist. The reformer's version is way more boring. Show your data and your code so somebody can check the claim. That doesn't lay a finger on the imaginative, world-making side of social science; it just makes being wrong about the measurable part cost you something. You're right that the current setup is captured and broken. I'm not saying burn the auditor down. I'm saying rebuild the one the field already fired.
If the two examples provided confirm your thesis then I'm confused about the argument. I don't think the parity conservation of the weak force is so straight forward a case. For 30 years, physics operated on a false assumption because of over-confidence. In fact, when rival hypotheses were proposed, the researchers were ridiculed and laughed out of the room. And while the test did overturn established convention, Chien-Shiumg Wu was excluded from the Nobel, for a variety of reasons, one being that she was an experimentalist. Physics favored theorists.
That does not seem to me to be a resounding endorsement of the honesty of physics. As for the honesty of medicine, you said in your article that medicine stays honest because drugs either work in trial or they do not. And yet... numerous tests suggested for years that non-drowsy Sudafed was basically a placebo, and yet it still took decades to remove it from the shelves. And that's just one example among many.
In both the above cases, I doubt the cost of being wrong cost much. In fact, being wrong probably lucratively advanced careers.
Your argument suggested to me that you seem to think that the social sciences have some kind of distinct ideological/structural wound at their center which is not present in physical sciences. That complexity combined with longer time-scales combined with establishment egos produces institutional rot unique to our fields. But I see that happening everywhere.
If that is not what you are arguing, then I misunderstood.
From my viewpoint, the social sciences are straddling physical and non-physical reality, and the nature of research is structurally different than in the physical sciences.
If you are, in fact, arguing that the integrity of the social sciences fall short of the physical sciences, could you provide specific examples?
But don't let "degree" turn into "therefore the same": the moment someone actually tested parity, physics folded in months, while social science can shrug off a failed replication and keep citing the original, no penalty, because nobody's keeping score.
Ego's everywhere; correction isn't. A verdict that takes thirty years and can be blamed on anything is exactly what lets a field drift where physics can't.
What are the specific examples of theories persisting beyond replicability that demonstrates the social sciences have a different kind of problem? The research I’ve seen on this pertaining to PS and econ doesn’t draw alarming conclusions. Meanwhile, medicine seems to have its own problems with this issue.
Given the need for flexible constructs in the social sciences (liberal being one such example), is there a problem here that is substantially different than in physical sciences?
Also, in regards to the politically neutral part of this post. Are you arguing that the social sciences should be politically neutral? In theory, that sounds great. But in the context we live in now, when even fundamental values in our society, like the Bill of Rights, or even human rights and opposition to genocide, are being politicized, I’d caution against a strict adherence to anti-politiciation. We, so-called experts in governance, should be free to advance the tools of a decent society, even if those tools become politicized. In that case, it is not the discipline that is doing the politicization, it is an external force. And scholars ought to have the liberty to respond as scholars.
Medical science has a replication problem, too. So do other fields mainly because researchers lack a sufficient understanding of statistics. Getting reviewers who know the field and statistics is not easy.
From what little I know, agreed on medicine, the preclinical replication numbers are as grim as anything in social science.
Where I'd push a little: I don't think it's mainly a stats-literacy problem. Some of it is, sure, but plenty of people who understand power and multiple comparisons perfectly well still chase significance, because the reward is for the interesting result, not the one that holds up. Better training helps at the margin; it doesn't change what gets published or tenured. That's why the fix that actually moved the numbers wasn't teaching everyone more statistics, it was structural: journals hiring a data editor to check the code before publication, so nobody has to hope the three reviewers happened to know both the subfield and the methods.
But then again, I think everyone should take at least two research methods and epistemology classes at a minimum. :)
You're also right that finding that reviewer is hard. So stop relying on finding them, and build the audit into the pipeline instead.
Yes, though an audit is a form of review. Yet I've been around statisticians enough to know they sometimes disagree. That leaves the generally accepted practices that auditors follow.
Last week an academic came on Breaking Points to describe his research on how incorporation of computer screens into schools lowers educational outcomes. I don't recall if he was officially a "sociologist", but that's clearly the sort of thing a sociologist would study. He gave a measured and convincing reason why he got the result he did—you could even see him struggling on the spot how not to get too in the weeds with the stats he did to handle various confounding factors. His is not a decisive answer. It's not ideology-free. (Of course Breaking Points with its strong anti-AI bent would feature him.) But it's a meaningful data point, and as a member of the general public I came away feeling better informed. This is the baby that is getting thrown out with the bathwater.
Your argument assumes that saving the field is worthwhile presumably because the field can do good and useful things. But if a particular field generates 80% or more BS, then is it actually doing any good, particularly if there is no effective process to sort the wheat from the chaff? Perhaps there is a case for Darwinian evolution by removing the defective institution?
I've defended sociology, including on my Substack, and I think most research-active sociologists are trying to draw conclusions that are interesting and sufficiently justified by the data to get past reviewers. But the replication/reproducibility study is truly troubling, sociology is definitely embattled politically, and this piece (I mean Kyle's) seems fair to me, if we don't quibble over details--such as the significance of Sociologists for Palestine or the recent ASA election.
One thing that would help is greater recognition for well-designed studies with null results. A relative of mine is a high-energy physicist who wrote his dissertation on his work at CERN, at which he failed to find a new particle -- but established the conditions under which none was produced, which counts as a contribution. In sociology, "didn't find a significant result" generally means "didn't find anything worth publishing." That explains the significance chasing.
I think that sociologists will catch up in terms of data/code sharing, if we survive long enough, and that our sluggishness there is partly due to our methodological diversity, where in a lot of legitimate sociological research there's no data that can be shared and no code (or significance testing) at all. That may be diminishing the internal pressure to do things differently.
Happy to concede those details. :) Sociologists for Palestine was texture more than evidence. The ASA election I'd hold onto a little harder, since a neutrality candidate losing 51-37 is at least a sign the inside-reform door is closed, but I take your point not to hang the argument on either.
The null-results point is the best thing in this thread. Your CERN relative is the whole case in one story: physics can reward a well-built "we looked, it isn't there" because the null is informative when the field is chained to reality. Sociology's "didn't find anything worth publishing" is the incentive gradient working upstream of anyone's honesty.
Reward a clean null and you fix a lot of the significance-chasing without moralizing at a single person.
Your methodological-diversity point is fair, and it's a real limit on my framing. SCORE only audited quantitative work (I flagged that in the linked piece but not here), and plenty of good sociology has no code or data file to share, so "install a data editor" isn't universal. But my read is the underlying pressure is the same on both sides of the methods divide: publication rewards the interesting result, researcher degrees of freedom are everywhere, and the field hasn't built much that makes being wrong expensive for either tradition. The quantitative side just has a tractable audit. The interpretive side needs one of its own, and nobody's built it yet...
Appreciate you engaging this way, and for defending the field without pretending it's fine.
I was Melissa Wilde's colleague and am currently Rory McVeigh's colleague and the first I heard about Melissa's position about neutrality was when the Chronicle piece came out! What I had read was their respective positions on Sociologists for Palestine, which didn't seem very different to me. Agreed on qualitative sociology; we had a dust-up over that in connection with Alice Goffman not very long ago.
Kyle, I don’t think you’ve got this wrong exactly, but there might be something else going on. In my area, I’m not sure my associated fields of rural sociology and forest sociology have the same kind of culture. They continue to study useful things and help people and institutions with real-world problems, including folks at CSU. So perhaps it isn’t the “field of sociology” so much as “when sociology departments are disjunct from the responsibility to work with real people and their problems. In that void perhaps they make up their own perceived problems and become a self-reinforcing clique of thinking.
what a very good point...the subfield variance matters here too. A lot of applied land-grant work stays tethered because they answer to someone with a real problem: a county, an agency, a landowner who'll tell you the model was wrong when the thing you predicted doesn't happen. Fast, cheap correction. Take the client and the consequence away and you get exactly the void you describe, where a subfield starts inventing its own problems and grading itself on them.
So maybe the better frame isn't "the field is captured," it's "the field is captured wherever the reality clock went slow or got switched off."
The applied corners kept their auditor. Really glad you raised this...
I have an MS in Demography from Wisconsin 30 years ago. Technically my degree was in the College of Agriculture, and I got to know the rural and environmental sociologists. I was an RA for Bill Freudenberg, who was an outstanding person and scholar. I agree with you that much of this area is not politicized. But this does not invalidate the points made by the sociology critics cited in Kyle's article and the WSJ article. From a larger picture perspective, there are parts of sociology - crime and demography - that could easily reconstitute elsewhere. Org. Theory has pretty much moved to management even though AJS and ASR publish their stuff.
"Reality is the auditor" is an interesting phrase well beyond the precinct of sociology. The example of economics which has been extolling the virtues of globalization and free trade for a generation, to just now recognizing this is a significant problem. The other example is the whole climate science and green transition where the policy imperatives have overcame the science and technology inadequacies. These suggest the issue is fields where policy influence is THE primary purpose.
Publicly funded university programs shouldn't be involved with policy advocacy. Their role could be more usefully policy auditing.
Medical science and physics both have more than a few examples of invalid theories/products persisting despite being wrong/useless. See parity conservation in the weak force and non-drowsy Sudafed, for starters. It seems to me that this piece (and others) makes an unfair/inaccurate assumption that these conditions plague only the social sciences.
Additionally, social reality is constructed through intersubjective interpretations of the world. Full stop. These are far more mutable than the laws of physics, and in fact, can radical alter when human perception also changes. One of humanity’s fascinating capabilities is to create world-altering ideas that have no basis in physical reality. There may not be a single molecule of mercy in the universe nor an atom of democracy nor even a libation of liberal. But these concepts, however ill-defined, change the world.
Therefore, the social sciences must be kept open enough to allow for the emergence and reinforcement of such ideas as well as the improved understanding of how they work. If we move to a binary pass-fail model for research and we stifle the magnificent power of these disciplines, and kneecap our own potential for true progress.
That is not to say that the current model of gatekeeping and resource allocation is not flawed. In fact, we know it has been, at least in part, captured by both corporate and national security interests, and subverted to reinforce dogma. But like democracy itself, it may be the worst form…except for all the others.
I think both your examples actually make my case. Parity conservation was bedrock physics right up until somebody tested it, and then it was gone in a few months. Non-drowsy Sudafed is on its way out too; the FDA finally moved to pull oral phenylephrine in 2024, decades after the evidence turned. That's reality batting last on two different clocks, which is the whole argument. And I'm not letting the fast fields off the hook. The piece says outright that physics and medicine get things wrong too. They just get caught sooner and cheaper. The rot is everywhere.
On the construction point, I'm with you more than you'd guess. Democracy, mercy, liberal, not one of them has a molecule, and they've all reshaped the world. I recently spent a whole long essay on "liberal" meaning almost nothing and mattering enormously. But that "full stop" of yours is really a comma. A concept being constructed doesn't mean claims about it can't be checked. "Democracy" is contested all the way down; "this reform bumped turnout six points" still either holds up or it doesn't. That second kind of sentence is where the whole replication problem lives. The map's up for grabs. The territory still bites.
And no one is proposing a pass/fail test for ideas. That's the thing that scares me, and it's exactly what I call the bulldozer: some outside power deciding which fields get to exist. The reformer's version is way more boring. Show your data and your code so somebody can check the claim. That doesn't lay a finger on the imaginative, world-making side of social science; it just makes being wrong about the measurable part cost you something. You're right that the current setup is captured and broken. I'm not saying burn the auditor down. I'm saying rebuild the one the field already fired.
If the two examples provided confirm your thesis then I'm confused about the argument. I don't think the parity conservation of the weak force is so straight forward a case. For 30 years, physics operated on a false assumption because of over-confidence. In fact, when rival hypotheses were proposed, the researchers were ridiculed and laughed out of the room. And while the test did overturn established convention, Chien-Shiumg Wu was excluded from the Nobel, for a variety of reasons, one being that she was an experimentalist. Physics favored theorists.
That does not seem to me to be a resounding endorsement of the honesty of physics. As for the honesty of medicine, you said in your article that medicine stays honest because drugs either work in trial or they do not. And yet... numerous tests suggested for years that non-drowsy Sudafed was basically a placebo, and yet it still took decades to remove it from the shelves. And that's just one example among many.
In both the above cases, I doubt the cost of being wrong cost much. In fact, being wrong probably lucratively advanced careers.
Your argument suggested to me that you seem to think that the social sciences have some kind of distinct ideological/structural wound at their center which is not present in physical sciences. That complexity combined with longer time-scales combined with establishment egos produces institutional rot unique to our fields. But I see that happening everywhere.
If that is not what you are arguing, then I misunderstood.
From my viewpoint, the social sciences are straddling physical and non-physical reality, and the nature of research is structurally different than in the physical sciences.
If you are, in fact, arguing that the integrity of the social sciences fall short of the physical sciences, could you provide specific examples?
Degree, not kind, I'll grant you, A.
But don't let "degree" turn into "therefore the same": the moment someone actually tested parity, physics folded in months, while social science can shrug off a failed replication and keep citing the original, no penalty, because nobody's keeping score.
Ego's everywhere; correction isn't. A verdict that takes thirty years and can be blamed on anything is exactly what lets a field drift where physics can't.
What are the specific examples of theories persisting beyond replicability that demonstrates the social sciences have a different kind of problem? The research I’ve seen on this pertaining to PS and econ doesn’t draw alarming conclusions. Meanwhile, medicine seems to have its own problems with this issue.
Given the need for flexible constructs in the social sciences (liberal being one such example), is there a problem here that is substantially different than in physical sciences?
PS and Econ reproducibility:
https://www.nature.com/articles/s41586-026-10251-x?utm_source=chatgpt.com
Replicability preclinical cancer:
https://elifesciences.org/articles/71601v2?utm_source=chatgpt.com
Persistence of amyloid hypothesis:
https://www.nejm.org/doi/full/10.1056/NEJMoa2304430?utm_source=chatgpt.com
Also, in regards to the politically neutral part of this post. Are you arguing that the social sciences should be politically neutral? In theory, that sounds great. But in the context we live in now, when even fundamental values in our society, like the Bill of Rights, or even human rights and opposition to genocide, are being politicized, I’d caution against a strict adherence to anti-politiciation. We, so-called experts in governance, should be free to advance the tools of a decent society, even if those tools become politicized. In that case, it is not the discipline that is doing the politicization, it is an external force. And scholars ought to have the liberty to respond as scholars.
Medical science has a replication problem, too. So do other fields mainly because researchers lack a sufficient understanding of statistics. Getting reviewers who know the field and statistics is not easy.
From what little I know, agreed on medicine, the preclinical replication numbers are as grim as anything in social science.
Where I'd push a little: I don't think it's mainly a stats-literacy problem. Some of it is, sure, but plenty of people who understand power and multiple comparisons perfectly well still chase significance, because the reward is for the interesting result, not the one that holds up. Better training helps at the margin; it doesn't change what gets published or tenured. That's why the fix that actually moved the numbers wasn't teaching everyone more statistics, it was structural: journals hiring a data editor to check the code before publication, so nobody has to hope the three reviewers happened to know both the subfield and the methods.
But then again, I think everyone should take at least two research methods and epistemology classes at a minimum. :)
You're also right that finding that reviewer is hard. So stop relying on finding them, and build the audit into the pipeline instead.
Yes, though an audit is a form of review. Yet I've been around statisticians enough to know they sometimes disagree. That leaves the generally accepted practices that auditors follow.
Last week an academic came on Breaking Points to describe his research on how incorporation of computer screens into schools lowers educational outcomes. I don't recall if he was officially a "sociologist", but that's clearly the sort of thing a sociologist would study. He gave a measured and convincing reason why he got the result he did—you could even see him struggling on the spot how not to get too in the weeds with the stats he did to handle various confounding factors. His is not a decisive answer. It's not ideology-free. (Of course Breaking Points with its strong anti-AI bent would feature him.) But it's a meaningful data point, and as a member of the general public I came away feeling better informed. This is the baby that is getting thrown out with the bathwater.
Your argument assumes that saving the field is worthwhile presumably because the field can do good and useful things. But if a particular field generates 80% or more BS, then is it actually doing any good, particularly if there is no effective process to sort the wheat from the chaff? Perhaps there is a case for Darwinian evolution by removing the defective institution?