by Randomization and causal claims A couple of weeks ago yours truly had a post up here where Julia Rohrer discussed possible alternatives to RCTs for making causal claims: It is instructive to consider cases in which most people readily accept causal claims in the absence of randomized experiments. Nowadays, few people doubt the effects of tobacco smoking on lung cancer. But in the 1950s, tobacco lobbyists embraced the idea that a genetic predisposition caused both a tendency to smoke and lung cancer … In other words, they claimed that there was an unblocked backdoor path. This idea was dispelled not by randomized, controlled experiments in humans, but by highly consistent results of observational studies using various controls and different sampling
Topics:
Lars Pålsson Syll considers the following as important: Theory of Science & Methodology
This could be interesting, too:
Lars Pålsson Syll writes Kausalitet — en crash course
Lars Pålsson Syll writes Race and sex as causes
Lars Pålsson Syll writes Randomization — a philosophical device gone astray
Lars Pålsson Syll writes Keynes on the importance of ‘causal spread’
Randomization and causal claims
A couple of weeks ago yours truly had a post up here where Julia Rohrer discussed possible alternatives to RCTs for making causal claims:
It is instructive to consider cases in which most people readily accept causal claims in the absence of randomized experiments. Nowadays, few people doubt the effects of tobacco smoking on lung cancer. But in the 1950s, tobacco lobbyists embraced the idea that a genetic predisposition caused both a tendency to smoke and lung cancer … In other words, they claimed that there was an unblocked backdoor path. This idea was dispelled not by randomized, controlled experiments in humans, but by highly consistent results of observational studies using various controls and different sampling designs, experimental evidence from rodent studies, and demonstration of a plausible mechanism …
A plausible mechanism is also what greatly increases scientists’ confidence in the causal effect of human activity on the climate: Human activity, such as industrial processes, increases the atmospheric concentrations of greenhouse gases. Atmospheric greenhouse gases, in turn, warm the Earth’s surface through an uncontroversial mechanism, the greenhouse effect … And a plausible mechanism is also the reason why one does not need randomized controlled trials to conclude that parachute use during free fall reduces mortality …
In a comment on Rohrer’s article, Sander Greenland begs to differ:
The cited article is for the most part concerned with review of uncontroversial methods and advice. Unfortunately, taken out of context, the quoted passage from the conclusion seems to me to invoke a weakly reasoned analogy, coming uncomfortably close to political rhetoric rather than methodology. To start, plausibility is a finely graded subjective continuum that can mislead when discussed as if only present or absent. Thus I don’t think it helpful to say “a plausible mechanism is … the reason why one does not need randomized controlled trials to conclude that parachute use during free fall reduces mortality”. In this case as in many others (like firearm hazards) the mechanism of the hazard is obvious, empirically in our face from start to finish (jump to impact); also obvious here what to do about it (wear a well-prepared parachute). I suspect invoking this kind of example only undermines the important argument that more subtle cases require “highly consistent results of observational studies using various controls and different sampling designs, experimental evidence from rodent studies, and demonstration of a plausible mechanism” to persuade all but those invested in denying an effect.
Further, the smoking-lung cancer example is itself remote from environmental concerns on the scale of the global-warming issue: Lung damage from smoking was obvious to some observers as far back as the early 1600s, when smoking became common enough in England to notice the effects. And from then on it was obvious what to do about it: Discourage individuals from smoking. But for global warming the current legitimate controversy is ever less about whether it is from human activity and ever more about what should be done about it. The causal questions in debate are now what will be the actual benefits and harms of the many policies proposed to slow it down. There is far less known about the consequences of those policies than their promoters are prepared to admit, consequences that may involve a range of concerns from the spread of infectious-disease vectors, to flooding, to a myriad of economic consequences.
No randomized trial can contribute much to climate policy formation because the consequences are extremely intertwined with each other and embedded within a complex global system with no overarching administration. This is far removed from the experimental ideal of simple independent units with treatments controlled by one team following a pre-specified protocol to the letter. This removal just as thoroughly undermines nonexperimental policy evaluations as experimental ones. It renders fallacious analogies with controversies such as what was to be done about hazards for which experiments were infeasible but policy targets were obvious and achievable with simple, limited-cost policies, like banning smoking advertisements, elimination of lead in petrol and paint, and so on.
I cannot see how methodology or policy driven by faulty rhetorical analogies will aid in avoiding the adverse consequences of global warming or any similarly complex global problem. It may sadly be that the best we can do is blunt adverse consequences by using interventions of proven effectiveness, such as methods for control of disease vectors, for prevention of flood damage, and for coping with agricultural impacts on the scale of what is to come.
I guess the discussion will continue …
Within economics nowadays there is an almost religious belief with which its propagators — including ‘Nobel prize’ winners like Esther Duflo, Abhijit Banerjee and Michael Kremer — portray it, but it cannot hide the fact that RCTs cannot be taken for granted to give generalizable results. That something works somewhere is no warranty for us to believe it works for us here or that it generally works. Whether an RCT is externally valid or not, is never a question of study design. What ‘works’ is always a question of context.
Leaning on an interventionist approach often means that instead of posing interesting questions on a social level, the focus is on individuals. Instead of asking about structural socio-economic factors behind, e.g., gender or racial discrimination, the focus is on the choices individuals make. Duflo et consortes want to give up on ‘big ideas’ like political economy and institutional reform and instead solve more manageable problems ‘the way plumbers do.’
Yours truly is far from sure that is the right way to move economics forward and make it a relevant and realist science. The ‘identification problem’ is certainly more manageable in plumbing, but we should never forget that clinical trials and medical studies have another dimensionality and heterogeneity than what we encounter in most social and economic contexts.
A plumber can fix minor leaks in your system, but if the whole system is rotten, something more than good old-fashioned plumbing is needed. The big social and economic problems we face today will not be solved by plumbers performing interventions or manipulations in the form of RCTs.