Randomised controlled trials — a retreat from the bigger questions

by
Lars Pålsson Syll
My articles My siteAbout meMy booksMy videos
Follow on:Twitter

from Lars Syll

Nobel prizes are usually given in recognition of ideas that are already more or less guaranteed a legacy. But occasionally they prompt as much debate as admiration. This year’s economics award, given to Abhijit Banerjee, Esther Duflo and Michael Kremer … recognised the laureates’ efforts to use randomised controlled trials (RCTs) to answer social-science questions … RCT evangelists sometimes argue that their technique is the “gold standard”, better able than other analytical approaches to establish what causes what. Not so, say some other economists … Results are contextually dependent in ways that are hard to discern; a finding from a study in Kenya might not reveal much about policy in Guatemala …

Advanced economies grew rich as a result of a broad transformation that affected everything from the aspirations of working people to the functioning of the state, not by making a series of small, technocratic changes, no matter how well-supported by evidence …

Indeed, some economists have a sneaking suspicion that the rise of RCTs represents a pivot not just to smaller questions but also to smaller ambitions … Researchers are still guided by theory, which shapes the empirical questions that get asked and whether results are interpreted as capturing some deeper aspect of an economy’s nature. But a world in which economists are mostly policy-tweakers—or “plumbers”, in Ms Duflo’s phrase—is very different from the one to which many economists once aspired.

The Economist

It is nowadays widely believed among mainstream economists that the scientific value of randomization — contrary to other methods — is totally uncontroversial and that randomized experiments are free from bias. When looked at carefully, however, there are in fact few real reasons to share this optimism on the alleged ’experimental turn’ in economics. Strictly seen, randomization does not guarantee anything.

‘Ideally controlled experiments’ tell us with certainty what causes what effects — but only given the right ‘closures.’ Making appropriate extrapolations from (ideal, accidental, natural or quasi) experiments to different settings, populations or target systems, is not easy. Causes deduced in an experimental setting still have to show that they come with an export-warrant to their target populations.

The almost religious belief with which its propagators — like Duflo, Banerjee and Kremer — portray it, cannot hide the fact that RCTs cannot be taken for granted to give generalizable results. That something works somewhere is no warranty for us to believe it to work for us here or even that it works generally.

The present RCT idolatry is dangerous. Believing there is only one really good evidence-based method on the market — and that randomization is the only way to achieve scientific validity — blinds people to searching for and using other methods that in many contexts are better. RCTs are simply not the best method for all questions and in all circumstances. Insisting on using only one tool often means using the wrong tool.

This year’s ‘Nobel prize’ winners think that economics should be based on evidence from randomised experiments and field studies. Duflo et consortes want to give up on ‘big ideas’ like political economy and institutional reform and instead go for solving more manageable problems the way plumbers do. But that modern time ‘marginalist’ approach sure can’t be the right way to move economics forward and make it a relevant and realist science. A plumber can fix minor leaks in your system, but if the whole system is rotten, something more than good old fashion plumbing is needed. The big social and economic problems we face today is not going to be solved by plumbers performing RCTs.