by What kind of ‘rigour’ do RCTs provide? The bad news is, first, that there is no reason in general to suppose that an ATE [Average Treatment Effect] observed in one population will hold in others. That is what the slogan widespread now in education and elsewhere registers: “Context matters”. The issue in this paper is not though about when we can expect a study result to hold elsewhere but rather when we can have EBPP-style [Evidence-Based Policy and Practice] “rigorous” evidence about any of the kinds of claims needed in practice. Here too, the news is bad: There are no good explicit methods with detailed content for inferring either general causal claims or causal predictions about what will happen in a specific case that look anything like rigorous
Topics:
Lars Pålsson Syll considers the following as important: Statistics & Econometrics
This could be interesting, too:
Lars Pålsson Syll writes Is the p-value dead?
Lars Pålsson Syll writes Alternatives to RCTs
Lars Pålsson Syll writes My critique of RCTs
Lars Pålsson Syll writes The history of econometrics
What kind of ‘rigour’ do RCTs provide?
The bad news is, first, that there is no reason in general to suppose that an ATE [Average Treatment Effect] observed in one population will hold in others. That is what the slogan widespread now in education and elsewhere registers: “Context matters”. The issue in this paper is not though about when we can expect a study result to hold elsewhere but rather when we can have EBPP-style [Evidence-Based Policy and Practice] “rigorous” evidence about any of the kinds of claims needed in practice. Here too, the news is bad: There are no good explicit methods with detailed content for inferring either general causal claims or causal predictions about what will happen in a specific case that look anything like rigorous in the sense for which RCTs are extolled and which we might hope for given the talk of rigour throughout the EBPP literature …
Although RCTs and other study designs can provide rigorous evidence about the effects of educational programmes in study populations, you need a lot more for figuring out whether and how to use those same programmes in your school …
The take-home lesson for EBPP institutions is that there is work to be done, new work of a new kind. It is time to rethink EBPP philosophy and in train to refocus where our efforts are put. The primary focus currently is on piling up gold nuggets (or good stiff twigs) – study results one can be sure of. But no heap of gold nuggets will add up to a general truth nor tell us what will happen next. That requires a panoply of information different in kind from what we are now vetting and disseminating. How shall we categorise this additional information, how organise it, how relate it to different plans of action? The big job now is to take on this far more amorphous, far more ambitious – and far more helpful – project.
Science and policy proposals should never be based on making heroically unreal tractability assumptions in the pursuit of ‘rigour’. Models and research designs building on such assumptions should make us naturally suspicious about their relevance and definitely weaken our degree of confidence in the proposals they produce.
