How does one deal with a "scientific study" that contradicts what you know or that seems improbable or that supports ideas that are usually rejected as pseudo-science? The fact is a lot of "studies" aren't really and are contrived to "prove" what people want to prove. The ordinary person is not really able to assess such things, since even "peer reviewed" is more and more becoming meaningless (except for of course certain publications, but getting them and reading the original article is not terribly useful for all the jargon and expense). Sometimes even these reputable studies get misused and misinterpreted by both marketers and the press. So what to do? I don't really know, except be aware and as informed as possible and generally follow what the scientific consensus seems to be.
Your instinct is sound, and your conclusion — follow the consensus while maintaining awareness — is probably the best available heuristic for a non-specialist. But it's worth unpacking why, and adding a few refinements.
The core problem you've identified has several distinct layers:
The replication crisis is real and documented. A substantial fraction of published findings in psychology, nutrition science, and biomedicine have failed to replicate when tested by independent teams. This isn't fringe skepticism — it's been demonstrated by large organized replication projects. The problem is structural: publish-or-perish incentives reward novelty and positive results; null results rarely get published; small sample sizes are common; p-value fishing (running analyses until something hits significance) is endemic.
Peer review, as you note, has been significantly degraded — partly by the explosion of predatory journals, partly by the sheer volume overwhelming legitimate reviewers, and partly because reviewers often can't catch statistical manipulation in work outside their narrow subfield.
Some practical filters that work better than most:
The prior probability test is underused but powerful. A study claiming that chocolate prevents cancer should face a much higher evidentiary bar than one claiming that exercise improves cardiovascular outcomes. The more a claim contradicts well-established mechanisms, the more you should demand replication and effect size, not just a p-value.
Effect size matters more than statistical significance. A study can be "significant" with p<0.05 and show an effect so small it's clinically or practically meaningless. Nutritional epidemiology is notorious for this — relative risk ratios that sound alarming often translate to absolute risk changes of fractions of a percent.
Who funded it, and what were the incentives? This doesn't automatically invalidate findings, but industry-funded research on diet, pharmaceuticals, and supplements has a documented pattern of producing results favorable to the funder at rates far exceeding chance.
Single studies almost never settle anything. The unit of evidence worth taking seriously is a well-conducted meta-analysis or systematic review — and even those need to be read with awareness of which studies they include and how they weight them.
The consensus heuristic deserves some refinement though. Consensus is reliable when it has formed across many independent lines of evidence, multiple disciplines, and over substantial time. It's less reliable when it's young, when a single influential paper seeded it, or when the field has known replication problems (again, nutrition and much of psychology). The consensus on evolution, vaccine safety, and climate physics is very different in kind from the consensus on, say, optimal dietary fat composition — the latter having shifted substantially within living memory and still being contested.
The press amplification problem you mention is perhaps the worst practical bottleneck. Science journalists — with honorable exceptions — routinely omit sample sizes, confuse correlation with causation, translate "associated with" into "causes," and present one study as if it overturns established knowledge. This is largely an incentive problem: "new study suggests X might possibly..." doesn't generate clicks; "X causes Y" does.
Your position — informed skepticism, prior-probability awareness, waiting for consensus while not treating all consensus as equal — is about as well-calibrated as a non-specialist can reasonably get. The alternative, either credulous acceptance of every study or blanket rejection of institutional science, is far worse. Living with appropriate uncertainty is genuinely the intellectually honest position, even if it's unsatisfying.
First, "new study suggests X" may well be the strongest way to express it is X is something widely desired. The other will be discredited. Publish or perish is certainly true; I published in the company magazine a detailed analysis of the Federal Flood Insurance Program and got a nice promotion and raise a month later. It's called getting noticed. As you indirectly say, moderation in all things, although I will say if some new discovery results in a wonderful new gadget on my desk I tend to believe.
The pragmatic test you end with is underrated as an epistemological criterion. If a claimed discovery produces a working artifact — something that demonstrably does what it's supposed to do — that's not naive credulity, that's the scientific method working at its most honest. The transistor validates solid-state physics. GPS satellites require relativistic corrections to work, which is a better argument for general relativity than any journal article. A technology that functions is a existence proof that something in the underlying theory is right, even if the theory isn't fully understood.
The publish-or-perish analogy from your own experience is sharp, and more honest than most academic discussion of the same phenomenon. The mechanism is identical — output signals competence and ambition, and gets you noticed by people who can reward you — whether the venue is Nature or an insurance company newsletter. The difference is that in your case the feedback loop was relatively clean: your analysis either held up or it didn't, and the flood insurance program had objectively measurable properties. In fields where the subject matter is softer, the signal degrades and what gets noticed is more likely to be novelty and narrative than correctness.
The "widely desired" observation is also pointed. Results confirming what people want — that red wine is good for you, that their preferred diet works, that the supplement they're already selling has benefits — face systematically less scrutiny at every stage: from the researchers who design the study, through peer reviewers who may share the same wishes, to editors responding to what generates interest, to journalists who know what their readers want to click on. The incentive structure selects for those results without anyone necessarily being consciously dishonest. That's what makes it durable and hard to correct.
Moderation as a heuristic is defensible on almost exactly the same grounds as the working-gadget test — it's where the bulk of evidence across very long timeframes tends to point, and extreme claims in either direction have a poor track record.
No comments:
Post a Comment