by RCTs risk distorting our knowledge base The claimed hierarchy of methods, with randomized assignment being deemed inherently superior to observational studies, does not survive close scrutiny. Despite frequent claims to the contrary, an RCT does not equate counterfactual outcomes between treated and control units. The fact that systematic bias in estimating the mean impact vanishes in expectation (under ideal conditions) does not imply that the (unknown) experimental error in a one-off RCT is less than the (unknown) error in some alternative observational study. We obviously cannot know that. A biased observational study with a reasonably large sample size may well be closer to the truth in specific trials than an underpowered RCT … The questionable
Topics:
Lars Pålsson Syll considers the following as important: Theory of Science & Methodology
This could be interesting, too:
Lars Pålsson Syll writes Randomization and causal claims
Lars Pålsson Syll writes Race and sex as causes
Lars Pålsson Syll writes Randomization — a philosophical device gone astray
Lars Pålsson Syll writes Keynes on the importance of ‘causal spread’
RCTs risk distorting our knowledge base
The claimed hierarchy of methods, with randomized assignment being deemed inherently superior to observational studies, does not survive close scrutiny. Despite frequent claims to the contrary, an RCT does not equate counterfactual outcomes between treated and control units. The fact that systematic bias in estimating the mean impact vanishes in expectation (under ideal conditions) does not imply that the (unknown) experimental error in a one-off RCT is less than the (unknown) error in some alternative observational study. We obviously cannot know that. A biased observational study with a reasonably large sample size may well be closer to the truth in specific trials than an underpowered RCT …
The questionable claims made about the superiority of RCTs as the “gold standard” have had a distorting influence on the use of impact evaluations to inform development policymaking, given that randomization is only feasible for a non-random subset of policies. When a program is community- or economy-wide or there are pervasive spillover effects from those treated to those not, an RCT will be of little help, and may well be deceptive. The tool is only well suited to a rather narrow range of development policies, and even then it will not address many of the questions that policymakers ask. Advocating RCTs as the best, or even only, scientific method for impact evaluation risks distorting our knowledge base for fighting poverty.