by From Lars Syll Without question, the experimental approach has produced genuine insights … All the same, there are serious limitations to a strategy centered on experimental design: 1. Good experimental design results in internal validity, where measurements actually measure the things they’re supposed to and confounding influences are suppressed. External validity, the extent to which results can be generalized to a wider array of situations beyond the confines of the experiment is a different matter. There are two specific aspects of experimentalism that raise questions on this front, the tendency for experiments to be small, local and time-bound … 2. The strategy of experimental design virtually requires a reductionist, small-bore approach to social change. A more sweeping,
Topics:
Lars Pålsson Syll considers the following as important: Uncategorized
This could be interesting, too:
Editor writes Water Flowing Upwards: Net financial flows from developing countries
John Quiggin writes Machines and tools
Eric Kramer writes An economic analysis of presidential immunity
Angry Bear writes Protesting Now and in the Sixties and Seventies
from Lars Syll
Without question, the experimental approach has produced genuine insights … All the same, there are serious limitations to a strategy centered on experimental design:
1. Good experimental design results in internal validity, where measurements actually measure the things they’re supposed to and confounding influences are suppressed. External validity, the extent to which results can be generalized to a wider array of situations beyond the confines of the experiment is a different matter. There are two specific aspects of experimentalism that raise questions on this front, the tendency for experiments to be small, local and time-bound …
2. The strategy of experimental design virtually requires a reductionist, small-bore approach to social change. A more sweeping, structural approach to poverty and inequality introduces too many variables and defeats experimental control. Thus, without any explicit ideological justification, we end up with incremental reformism when the entire social configuration may be the true culprit …
Using experimental methods to incorporate more learning in program administration should be standard practice; perhaps some day it will be. But the big problems in poverty and oppression are too complex and encompassing to be reduced to experimental bits, and there is no substitute for theoretical analysis and a willingness to take chances with large-scale collective action.
‘Ideally controlled experiments’ tell us with certainty what causes what effects — but only given the right ‘closures.’ Making appropriate extrapolations from (ideal, accidental, natural or quasi) experiments to different settings, populations or target systems, is not easy. “It works there” is no evidence for “it will work here”. Causes deduced in an experimental setting still have to show that they come with an export-warrant to the target population/system. The causal background assumptions made have to be justified, and without licenses to export, the value of ‘rigorous’ and ‘precise’ methods — and ‘on-average-knowledge’ — is despairingly small.
Apart from these methodological problems, I do think there is also a rather disturbing kind of scientific naïveté in the kind of experimentalist approach to combatting poverty that this years ‘Nobel prize’ winners Duflo/Banerjee/Kramer represent. The way they present their whole endeavour smacks of not so little ‘scientism’ where fighting poverty becomes a question of applying ‘objective’ quantitative ‘techniques.’ But that can’t be the right way to fight poverty! Fighting poverty and inequality is basically a question of changing the structure and institutions of our economies and societies.