by The pitfalls of randomization The core purpose of RCTs is to use random assignment in order to ensure that the unconfoundedness assumption essential to identifying an average treatment effect holds. In the abstract, this is a strong argument for the method. Problems arise, however, when pristine asymptotic properties confront the muddy realities of field applications and strict control over fully exogenous assignment almost inevitably breaks down, for any of a variety of reasons discussed below or in the preceding section on ethical dilemmas. The end result is that the attractive asymptotic properties of RCTs often disappear in practice, much like the asymptotic properties of other IV estimators. We term this the “faux exogeneity” problem because, like
Topics:
Lars Pålsson Syll considers the following as important: Theory of Science & Methodology
This could be interesting, too:
Lars Pålsson Syll writes Randomization — a philosophical device gone astray
Lars Pålsson Syll writes Keynes on the importance of ‘causal spread’
Lars Pålsson Syll writes Use your brain!
Lars Pålsson Syll writes What is a good model?
The pitfalls of randomization
The core purpose of RCTs is to use random assignment in order to ensure that the unconfoundedness assumption essential to identifying an average treatment effect holds. In the abstract, this is a strong argument for the method. Problems arise, however, when pristine asymptotic properties confront the muddy realities of field applications and strict control over fully exogenous assignment almost inevitably breaks down, for any of a variety of reasons discussed below or in the preceding section on ethical dilemmas. The end result is that the attractive asymptotic properties of RCTs often disappear in practice, much like the asymptotic properties of other IV estimators. We term this the “faux exogeneity” problem because, like faux pearls, the product commonly looks of high value and one must scrutinize carefully in order to detect the fundamental flaws in construction so that many naïve consumers are regrettably duped …
Unobservably heterogeneous treatments, encouragement bias and sampling bias in economic studies undercut the ‘gold standard’ claim that RCTs reliably identify the (local) average treatment effect for the target population (i.e., that RCT estimates have internal validity). Just as the original gold standard depended on a range of strong assumptions – that ultimately proved untenable, leading to the collapse of the gold standard – so does the claim of internal validity depend on multiple, strong, often-contestable assumptions. As with studies based on conventional, observational data, the development economics community needs to interrogate underlying identifying assumptions before accepting RCT results as internally valid …
A core pitfall is that experiments typically treat human beings as subjects, not as agents. When measurable outcomes are the core variables of interest, as is typically true in evaluation research, the behavioral mechanisms that yield these outcomes in the non-experimental economy are almost inevitably subordinated in research design. This problem is compounded when the phenomena of interest – such as market equilibria, outcomes that fundamentally depend on collective action, etc. – arise from complex multi-agent interactions not readily reproducible in experiments, as we discuss below, with reference to capital access. Furthermore, it is by no means clear that purging agents’ endogenous behavioral response is always desirable given that the core question of interest is what will happen in response to real people’s non-random responses to the introduction of a policy or project or technology. Precise answers to the wrong question are not always helpful.
The present RCT idolatry is dangerous. Believing randomization is the only way to achieve scientific validity blinds people to searching for and using other methods that in many contexts are better. Insisting on using only one tool often means using the wrong tool.
Randomization is not a panacea. It is not the best method for all questions and circumstances. Proponents of randomization make claims about its ability to deliver causal knowledge that is simply wrong. There are good reasons to share Deeaton’s scepticism on the now popular — and ill-informed — view that randomization is the only valid and the best method on the market. It is not.
Researchers who use randomization-based research strategies often set up problem formulations that are not at all the ones we really want answers to, to achieve ‘exact’ and ‘rigorous’ results. Design becomes the main thing, and as long as one can get more or less clever experiments in place, they believe they can draw far-reaching conclusions about both causality and the ability to generalize experimental outcomes to larger populations. Unfortunately, this often means that this type of research has a negative bias away from interesting and important problems towards prioritizing method selection. As Barrett and Carter rightly notice:
Precise answers to the wrong question are not always helpful.