by From Lars Syll Statistician Andrew Gelman has an interesting blogpost up on what inference in science really means: I like Don Rubin’s take on this, which is that if you want to go from association to causation, state very clearly what the assumptions are for this step to work. The clear statement of these assumptions can be helpful in moving forward … Another way to say this is that all inference is about generalizing from sample to population, to predicting the outcomes of hypothetical interventions on new cases. You can’t escape the leap of generalization. Even a perfectly clean randomized experiment is typically of interest only to the extent that it generalizes to new people not included in the original study. I agree — but that’s also why we so often fail (even when having the
Topics:
Lars Pålsson Syll considers the following as important: Uncategorized
This could be interesting, too:
Merijn T. Knibbe writes Christmas thoughts about counting the dead in zones of armed conflict.
Lars Pålsson Syll writes Mainstream distribution myths
Dean Baker writes Health insurance killing: Economics does have something to say
Lars Pålsson Syll writes Debunking mathematical economics
from Lars Syll
Statistician Andrew Gelman has an interesting blogpost up on what inference in science really means:
I like Don Rubin’s take on this, which is that if you want to go from association to causation, state very clearly what the assumptions are for this step to work. The clear statement of these assumptions can be helpful in moving forward …
Another way to say this is that all inference is about generalizing from sample to population, to predicting the outcomes of hypothetical interventions on new cases. You can’t escape the leap of generalization. Even a perfectly clean randomized experiment is typically of interest only to the extent that it generalizes to new people not included in the original study.
I agree — but that’s also why we so often fail (even when having the best intentions) when it comes to making generalizations in social sciences.
What strikes me again and again when taking part of the results of randomized experiments is that they really are very similar to theoretical models. They all have the same basic problem — they are built on rather artificial conditions and have difficulties with the ‘trade-off’ between internal and external validity. The more artificial conditions, the more internal validity, but also less external validity. The more we rig experiments/models to avoid the ‘confounding factors,’ the less the conditions are reminiscent of the real ‘target system.’ The nodal issue is basically about how scientists using different isolation strategies in different ‘nomological machines’ attempt to learn about causal relationships. I doubt the generalizability of the (randomized or not) experiment strategy because the probability is high that causal mechanisms are different in different contexts and that lack of homogeneity/stability/invariance don’t give us warranted export licenses to the ‘real’ societies.
Evidence-based theories and policies are highly valued nowadays. Randomization is supposed to best control for bias from unknown confounders. The received opinion — including Rubin and Gelman — is that evidence based on randomized experiments therefore is the best.
More and more economists have also lately come to advocate randomization as the principal method for ensuring being able to make valid causal inferences. Especially when it comes to questions of causality, randomization is nowadays considered some kind of “gold standard”. Everything has to be evidence-based, and the evidence has to come from randomized experiments.
But just as econometrics, randomization is basically a deductive method. Given the assumptions (such as manipulability, transitivity, Reichenbach probability principles, separability, additivity, linearity, etc., etc.) these methods deliver deductive inferences. The problem, of course, is that we will never completely know when the assumptions are right. [And although randomization may contribute to controlling for confounding, it does not guarantee it, since genuine ramdomness presupposes infinite experimentation and we know all real experimentation is finite. And even if randomization may help to establish average causal effects, it says nothing of individual effects unless homogeneity is added to the list of assumptions.] Real target systems are seldom epistemically isomorphic to our axiomatic-deductive models/systems, and even if they were, we still have to argue for the external validity of the conclusions reached from within these epistemically convenient models/systems. Causal evidence generated by randomization procedures may be valid in ‘closed’ models, but what we usually are interested in, is causal evidence in the real target system we happen to live in.
Many advocates of randomization want to have deductively automated answers to fundamental causal questions. But to apply ‘thin’ methods we have to have ‘thick’ background knowledge of what’s going on in the real world, and not in (ideally controlled) experiments. Conclusions can only be as certain as their premises — and that also goes for methods based on randomized experiments.
So yours truly agrees with Gelman that “all inference is about generalizing from sample to population.” But I don’t think randomized experiments — ideal or not — take us very far on that road. Randomized experiments in social sciences are far from being the ‘gold standard’ they so often are depicted as.