by Decision making — trustworthiness vs relevance The random assignment plus masking are supposed to make it likely that the two groups have the same distribution of causal factors. It is controversial how confident these measures should make us that they do this. This issue bears on the trustworthiness of causal claims backed by RCTs. As we noted, trustworthiness is the central topic of many other guides. But we aim to move beyond that; we concentrate on relevance … Randomization is often defended by the claim that it is the only way to deal with unknown causal factors. If so, then an ideal RCT can be the superior choice if you are not very secure that you know much about what the significant causal factors are. Supposing that you are in this situation,
Topics:
Lars Pålsson Syll considers the following as important: Statistics & Econometrics
This could be interesting, too:
Lars Pålsson Syll writes The importance of ‘causal spread’
Lars Pålsson Syll writes Applied econometrics — a messy business
Lars Pålsson Syll writes Feynman’s trick (student stuff)
Lars Pålsson Syll writes Difference in Differences (student stuff)
Decision making — trustworthiness vs relevance
The random assignment plus masking are supposed to make it likely that the two groups have the same distribution of causal factors. It is controversial how confident these measures should make us that they do this. This issue bears on the trustworthiness of causal claims backed by RCTs. As we noted, trustworthiness is the central topic of many other guides. But we aim to move beyond that; we concentrate on relevance …
Randomization is often defended by the claim that it is the only way to deal with unknown causal factors. If so, then an ideal RCT can be the superior choice if you are not very secure that you know much about what the significant causal factors are. Supposing that you are in this situation, then ranking good RCT studies above otherwise good studies that do not mask and randomize seems correct — so long as it is remembered as well that what is at stake is trustworthiness, not relevance or cost effectiveness or moral acceptability …
We have dealt here with effectiveness, but we do not say that effectiveness is all you have to consider. We are not introducing yet another apparently comprehensive technique for cutting through the complexities of decision making. We have looked at no more than one, important, corner of the decision-making process, where we think that contemporary emphasis on trustworthiness over relevance has led us astray, and we hope to have shown how you can set about seeing what evidence you will need if you are to choose effective policies. Whatever else may be needed, that must be worth having.
Yours truly’s view is that nowadays many social scientists maintain that ‘imaginative empirical methods’ — such as natural experiments, field experiments, lab experiments, and RCTs — can help us answer questions concerning the external validity of models used in social sciences. In their view, they are more or less tests of ‘an underlying model’ that enable them to make the right selection from the ever-expanding ‘collection of potentially applicable models.’ When looked at carefully, however, there are not many convincing reasons to share this optimism.
Many ‘experimentalists’ claim that it is easy to replicate experiments under different conditions and therefore a fortiori easy to test the robustness of experimental results. But is it really that easy? Population selection is rarely simple. Most social scientists — including economists — that use natural experiments, do as a rule not work with random samples taken from well-defined populations. Had the problem of external validity only been about inference from sample to population, this would be no critical problem. But the really interesting inferences are those we try to make from specific labs/experiments/fields to specific real-world situations/institutions/ structures that we are interested in understanding or (causally) explaining. And then the population problem is more difficult to tackle.
Achieving ‘as-if’ randomization settings is not enough. At the end of the day, what counts when we evaluate natural experiments is substantive and policy relevance — as in John Snow’s path-breaking ‘shoe-leather’ cholera study in 1855 — and not if we come up with more and more contrived instrumental-variables designs or not.
‘Ideally controlled experiments’ tell us with certainty what causes what effects — but only given the right ‘closures.’ Making appropriate extrapolations from (ideal, accidental, natural or quasi) experiments to different settings, populations or target systems is difficult. “It works there” is no evidence for “it will work here”. Causes deduced in an experimental setting still have to show that they come with an export warrant to the target population/system. The causal background assumptions made have to be justified, and without licenses to export, the value of ‘rigorous’ and ‘precise’ methods is rather small. The contemporary emphasis on ‘trustworthiness’ and ‘rigour’ over relevance certainly often leads us astray.