Tuesday , November 5 2024
Home / Lars P. Syll / Evidence-based policy

Evidence-based policy

Summary:
‘Ideally controlled experiments’ tell us with certainty what causes what effects — but only given the right closures. Making appropriate extrapolations from (ideal, accidental, natural or quasi) experiments to different settings, populations or target systems, is not easy. ‘It works there’ is no evidence for ‘it will work here.’ Causes deduced in an experimental setting still have to show that they come with a transportability warrant to the target population. The causal background assumptions made have to be justified, and without licenses to transport, the value of ‘rigorous’ and ‘precise’ methods — and ‘on-average-knowledge’ — is often despairingly small. Like us, you want evidence that a policy will work here, where you are. Randomized controlled trials (RCTs) do not

Topics:
Lars Pålsson Syll considers the following as important:

This could be interesting, too:

Lars Pålsson Syll writes What statistics teachers get wrong!

Lars Pålsson Syll writes Statistical uncertainty

Lars Pålsson Syll writes The dangers of using pernicious fictions in statistics

Lars Pålsson Syll writes Interpreting confidence intervals

‘Ideally controlled experiments’ tell us with certainty what causes what effects — but only given the right closures. Making appropriate extrapolations from (ideal, accidental, natural or quasi) experiments to different settings, populations or target systems, is not easy. ‘It works there’ is no evidence for ‘it will work here.’ Causes deduced in an experimental setting still have to show that they come with a transportability warrant to the target population. The causal background assumptions made have to be justified, and without licenses to transport, the value of ‘rigorous’ and ‘precise’ methods — and ‘on-average-knowledge’ — is often despairingly small.

Evidence-based policyLike us, you want evidence that a policy will work here, where you are. Randomized controlled trials (RCTs) do not tell you that. They do not even tell you that a policy works. What they tell you is that a policy worked there, where the trial was carried out, in that population. Our argument is that the changes in tense – from “worked” to “work” – are not just a matter of grammatical detail. To move from one to the other requires hard intellectual and practical effort. The fact that it worked there is indeed fact. But for that fact to be evidence that it will work here, it needs to be relevant to that conclusion. To make RCTs relevant you need a lot more information and of a very different kind.

So, no, yours truly finds it hard to share the enthusiasm and optimism on the value of (quasi)natural experiments and all the statistical-econometric machinery that comes with it. Guess I’m still waiting for the transportability warrant …

Lars Pålsson Syll
Professor at Malmö University. Primary research interest - the philosophy, history and methodology of economics.

Leave a Reply

Your email address will not be published. Required fields are marked *