by Causal interaction and external validity As yours truly has repeatedly argued on this blog, randomized control trials (RCTs) usually do not provide evidence that their results are exportable to other target systems. The almost religious belief with which its propagators portray it, cannot hide the fact that RCTs cannot be taken for granted to give generalizable results. Randomized evaluations have become widespread in development economics in recent decades, largely due to the promise of identifying policy-relevant causal effects. A number of concerns have been raised in response … [One] concern, which is the subject of the present contribution, is that current research based on experimental methods does not adequately address the problem of
Topics:
Lars Pålsson Syll considers the following as important: Statistics & Econometrics
This could be interesting, too:
Lars Pålsson Syll writes What statistics teachers get wrong!
Lars Pålsson Syll writes Statistical uncertainty
Lars Pålsson Syll writes The dangers of using pernicious fictions in statistics
Lars Pålsson Syll writes Interpreting confidence intervals
Causal interaction and external validity
As yours truly has repeatedly argued on this blog, randomized control trials (RCTs) usually do not provide evidence that their results are exportable to other target systems. The almost religious belief with which its propagators portray it, cannot hide the fact that RCTs cannot be taken for granted to give generalizable results.
Randomized evaluations have become widespread in development economics in recent decades, largely due to the promise of identifying policy-relevant causal effects. A number of concerns have been raised in response … [One] concern, which is the subject of the present contribution, is that current research based on experimental methods does not adequately address the problem of extrapolating from empirical findings to policy claims relating to other populations (“external validity”) …
Combining insights from prior literature on experimental methods in social science and econometric formulations of external validity yields three important insights. First, that plausibly attaining external validity requires ex ante knowledge of covariates that influence the treatment effect along with empirical information on these variables in the experimental and policy populations. This, in turn, implies that “atheoretical” replication-based resolutions to the external validity problem are unlikely to be successful except for extremely simple causal relations, or very homogeneous populations, of a kind that appears unlikely in social science. Finally, the formal requirements for external validity are conceptually analogous to the assumptions needed for causal identification using observational data. Together these imply a much more modest interpretation of the policy relevance of past work that has not addressed these issues. Furthermore, the resultant challenges for making policy claims premised on randomized evaluations are substantial, if not insurmountable, in many cases of interest.
Muller’s article underlines the problem many ‘randomistas’ end up with when underestimating heterogeneity and interaction. It does not just turn up as an external validity problem when trying to ‘export’ regression results to different times or different target populations. It is also often an internal problem to the millions of regression estimates that economists produce every year.
‘Ideally controlled experiments’ tell us with certainty what causes what effects — but only given the right ‘closures.’ Making appropriate extrapolations from (ideal, accidental, natural or quasi) experiments to different settings, populations or target systems, is not easy. ‘It works there’ is no evidence for ‘it will work here.’ Causes deduced in an experimental setting still have to show that they come with an export-warrant to the target population/system. The causal background assumptions made have to be justified, and without licenses to export, the value of ‘rigorous’ and ‘precise’ methods — and ‘on-average-knowledge’ — is despairingly small.
RCTs have very little reach beyond giving descriptions of what has happened in the past. From the perspective of the future and for policy purposes they are as a rule of limited value since they cannot tell us what background factors were held constant when the trial intervention was being made.
RCTs usually do not provide evidence that the results are exportable to other target systems. RCTs cannot be taken for granted to give generalizable results. That something works somewhere for someone is no warranty for us to believe it to work for us here or even that it works generally.