Monday , December 16 2019
Home / Real-World Economics Review / The experimental dilemma

The experimental dilemma

Summary:
From Lars Syll We can either let theory guide us in our attempt to estimate causal relationships from data … or we don’t let theory guide us. If we let theory guide us, our causal inferences will be ‘incredible’ because our theoretical knowledge is itself not certain … If we do not let theory guide us, we have no good reason to believe that our causal conclusions are true either of the experimental population or of other populations because we have no understanding of the mechanisms that are responsible for a causal relationship to hold in the first place, and it is difficult to see how we could generalize an experimental result to other settings if this understanding doesn’t exist. Either way, then, causal inference seems to be a cul-de-sac. Nowadays many mainstream economists

Topics:
Lars Syll considers the following as important:

This could be interesting, too:

Editor writes The social environment as a cause in economics

John Quiggin writes No takers for a nuclear grand bargain

Lars Syll writes 2019 ‘Nobel prize’ reveals the poverty of economics

V. Ramanan writes UK Elections 2019 And Brexit

from Lars Syll

The experimental dilemmaWe can either let theory guide us in our attempt to estimate causal relationships from data … or we don’t let theory guide us. If we let theory guide us, our causal inferences will be ‘incredible’ because our theoretical knowledge is itself not certain … If we do not let theory guide us, we have no good reason to believe that our causal conclusions are true either of the experimental population or of other populations because we have no understanding of the mechanisms that are responsible for a causal relationship to hold in the first place, and it is difficult to see how we could generalize an experimental result to other settings if this understanding doesn’t exist. Either way, then, causal inference seems to be a cul-de-sac.

Nowadays many mainstream economists maintain that ‘imaginative empirical methods’ — especially randomized experiments (RCTs) — can help us to answer questions concerning the external validity of economic models. In their view, they are, more or less, tests of ‘an underlying economic model’ and enable economists to make the right selection from the ever-expanding ‘collection of potentially applicable models.’

It is widely believed among economists that the scientific value of randomization — contrary to other methods — is totally uncontroversial and that randomized experiments are free from bias. When looked at carefully, however, there are in fact few real reasons to share this optimism on the alleged ’experimental turn’ in economics. Strictly seen, randomization does not guarantee anything.

‘Ideally controlled experiments’ tell us with certainty what causes what effects — but only given the right ‘closures.’ Making appropriate extrapolations from (ideal, accidental, natural or quasi) experiments to different settings, populations or target systems, is not easy. ‘It works there’ is no evidence for ‘it will work here’. Causes deduced in an experimental setting still have to show that they come with an export-warrant to the target population. The causal background assumptions made have to be justified, and without licenses to export, the value of ‘rigorous’ and ‘precise’ methods — and ‘on-average-knowledge’ — is despairingly small.

The experimental dilemmaThe almost religious belief with which its propagators — this year’s ‘Nobel prize’ winners Duflo, Banerjee and Kremer included — portray it, cannot hide the fact that RCTs cannot be taken for granted to give generalizable results. That something works somewhere is no warranty for us to believe it to work for us here or even that it works generally.

The present RCT idolatry is dangerous. Believing there is only one really good evidence-based method on the market — and that randomization is the only way to achieve scientific validity — blinds people to searching for and using other methods that in many contexts are better. RCTs are simply not the best method for all questions and in all circumstances. Insisting on using only one tool often means using the wrong tool.

About Lars Syll
Lars Syll
Lars Jörgen Pålsson Syll (born November 5, 1957) is a Swedish economist who is a Professor of Social Studies and Associate professor of Economic History at Malmö University College. Pålsson Syll has been a prominent contributor to the economic debate in Sweden over the global financial crisis that began in 2008.

Leave a Reply

Your email address will not be published. Required fields are marked *