Friday , June 25 2021
Home / Lars P. Syll / What kind of evidence do RCTs provide?

What kind of evidence do RCTs provide?

Summary:
What kind of evidence do RCTs provide? Perhaps it is supposed that the assumptions for an RCT are generally more often met (or meetable) than those for other methods. What justifies that? Especially given that the easiest assumption to feel secure about for RCTs—that the assignment is done “randomly”—is far from enough to support orthogonality, which is itself only one among the assumptions that need support. I sometimes hear, “Only the RCT can control for unknown unknowns.” But nothing can control for unknowns that we know nothing about. There is no reason to suppose that, for a given conclusion, the causal knowledge that it takes to stop post‐randomization correlations in an RCT is always, or generally, more available or more reliable than the

Topics:
Lars Pålsson Syll considers the following as important:

This could be interesting, too:

Lars Pålsson Syll writes The epistemic fallacy

Lars Pålsson Syll writes Challenging causal models — the low birth weight paradox

Lars Pålsson Syll writes Is causality only in the mind?

Lars Pålsson Syll writes Science manufacturing ignorance

What kind of evidence do RCTs provide?

Randomized Control Trials Feed Our Fetish for Single-Focus Interventions -  ICTworksPerhaps it is supposed that the assumptions for an RCT are generally more often met (or meetable) than those for other methods. What justifies that? Especially given that the easiest assumption to feel secure about for RCTs—that the assignment is done “randomly”—is far from enough to support orthogonality, which is itself only one among the assumptions that need support. I sometimes hear, “Only the RCT can control for unknown unknowns.” But nothing can control for unknowns that we know nothing about. There is no reason to suppose that, for a given conclusion, the causal knowledge that it takes to stop post‐randomization correlations in an RCT is always, or generally, more available or more reliable than the knowledge required for one or another of the other methods to be reliable.

It is also essential to be clear what the conclusion is. As with any study method, RCTS can only draw conclusions about the objects studied—for the RCT, the population enrolled in the trial, which is seldom the one we are interested in. The RCT method can be expanded of course to include among its assumptions that the trial population is a representative sample of the target. Then it follows deductively that the difference in mean outcomes between treatment and control groups is an unbiased estimate of the ATE of the target population. How often are we warranted in assuming that, though, and on what grounds? Without this assumption, an RCT is just a voucher for claims about any except the trial population. What then justifies placing it above methods that are clinchers for claims we are really interested in—about target populations?

Nancy Cartwright

Lars Pålsson Syll
Professor at Malmö University. Primary research interest - the philosophy, history and methodology of economics.

Leave a Reply

Your email address will not be published. Required fields are marked *