by The lack of theory in social experiments Jason Collins discusses a paper by Milkman et al. that presented “a megastudy testing 54 interventions to increase the gym visits of 61,000 experimental participants” … Collins’s discussion seems reasonable to me. In particular, I agree with his big problem about the design of this “mega-study,” which is that there’s all sorts of rigor in the randomization and analysis plan, but no rigor at all when it comes to deciding what interventions to test. Unfortunately, this is standard practice in policy analysis! Indeed, if you look at a statistics book, including mine, you’ll see lots and lots on causal inference and estimation, but nothing on how to come up with the interventions to study in the first place … What are
Topics:
Lars Pålsson Syll considers the following as important: Economics
This could be interesting, too:
Lars Pålsson Syll writes Klas Eklunds ‘Vår ekonomi’ — lärobok med stora brister
Lars Pålsson Syll writes Ekonomisk politik och finanspolitiska ramverk
Lars Pålsson Syll writes NAIRU — a harmful fairy tale
Lars Pålsson Syll writes Isabella Weber on sellers inflation
The lack of theory in social experiments
Jason Collins discusses a paper by Milkman et al. that presented “a megastudy testing 54 interventions to increase the gym visits of 61,000 experimental participants” …
Collins’s discussion seems reasonable to me. In particular, I agree with his big problem about the design of this “mega-study,” which is that there’s all sorts of rigor in the randomization and analysis plan, but no rigor at all when it comes to deciding what interventions to test.
Unfortunately, this is standard practice in policy analysis! Indeed, if you look at a statistics book, including mine, you’ll see lots and lots on causal inference and estimation, but nothing on how to come up with the interventions to study in the first place …
What are those 54 interventions, anyway? Just some things that a bunch of well-connected economists wanted to try out. Well-connected economists know lots of things, but maybe not so much about motivating people to go to the gym.
A related problem is variation: These treatments, even when effective, are not simply push-button-X-and-then-you-get-outcome-Y. Effects will be zero for most people and will be highly variable among the people for whom effects are nonzero. The result is that the average treatment effect will be much smaller than you expect. This is not just a problem of “statistical power”; it’s also a conceptual problem with this whole “reduced-form” way of looking at the world. To put it another way: Lack of good theory has practical consequences.