Tyler Cowen today flags a paper by Currie, Kleven, and Zwiers on changing practices in economics, and highlights the following: Panel A illustrates a virtually linear rise in the fraction of papers, in both the NBER and top-five series, which make explicit reference to identification. This fraction has risen from around 4 percent to 50 percent of papers. This caught my eye, because Matt Yglesias at Vox recently highlighted a study claiming to show large educational benefits from air filtering in schools: This paper identifies the achievement impact of installing air filters in classrooms for the first time. To do so, I leverage a unique setting arising from the largest gas leak in United States history, whereby the offending gas company installed air
Topics:
Eric Kramer considers the following as important: Featured Stories, Taxes/regulation, US/Global Economics
This could be interesting, too:
Angry Bear writes Japanese Prime Minister Fumio Kishida addresses Congress . . .
Bill Haskell writes Manipulating Supply Chains and Manufacturing, for Corporate Influence and Profit
Eric Kramer writes Milei and dollarization
Joel Eissenberg writes How much should a life-saving drug cost?
Tyler Cowen today flags a paper by Currie, Kleven, and Zwiers on changing practices in economics, and highlights the following:
Panel A illustrates a virtually linear rise in the fraction of papers, in both the NBER and top-five series, which make explicit reference to identification. This fraction has risen from around 4 percent to 50 percent of papers.
This caught my eye, because Matt Yglesias at Vox recently highlighted a study claiming to show large educational benefits from air filtering in schools:
This paper identifies the achievement impact of installing air filters in classrooms for the first time. To do so, I leverage a unique setting arising from the largest gas leak in United States history, whereby the offending gas company installed air filters in every classroom, office and common area for all schools within five miles of the leak (but not beyond). This variation allows me to compare student achievement in schools receiving air filters relative to those that did not using a spatial regression discontinuity design. I find substantial improvements in student achievement: air filter exposure led to a 0.20 standard deviation increase in mathematics and English scores, with test score improvements persisting into the following year. Air testing conducted inside schools during the leak (but before air filters were installed) showed no presence of natural gas pollutants, implying that the effectiveness of air filters came from removing common air pollutants and so these results should extend to other settings. The results indicate that air filter installation is a highly cost-effective policy to raise student achievement and, given that underprivileged students attend schools in highly polluted areas, one that can reduce the pervasive test score gaps that plague public education.
Kevin Drumm was quick to spot the problem in the paper (click through to see the data).
Then Andrew Gelman weighed in:
I don’t want to pick on the author of the above paper, who’s studying an important problem with a unique dataset using generally accepted methods. And it’s just a preprint. It’s good practice for people to release preliminary findings in order to get feedback. Indeed, that’s what’s happening here. That’s the way to go: be open about your design and analysis, share your results, and engage the hivemind. It was good luck to get the publicity right away so now there’s the opportunity to start over on the analysis and accept that the conclusions probably won’t be so clear, once all is said and done.
If there’s a problem, it’s with the general attitude in much of economics, in which it is assumed that identification strategy + statistical significance = discovery. That’s a mistake, and it’s something we have to keep talking about. But, again, it’s not about this particular researcher. Indeed, now the data are out there—I assume that at least the average test scores for each school and grade can be released publicly?—other people can do their own analyses. Somebody had to get the ball rolling.
I am not sure what to make of this. Generally, I think the empirical turn and credibility revolution in economics represents huge intellectual progress. The ability of economists to understand markets and make useful policy prescriptions has increased, and at least in some fields this has increased the influence of economists with policymakers. Consider this:
Two prior studies, conducted in 1966 and in 1979, examined the role of economic research in health policy development. Both concluded that health economics had not been an important contributor to policy. Passage of the Affordable Care Act offers an opportunity to reassess this question. We find that the evolution of health economics research has given it an increasingly important role in policy. Research in the field has followed three related paths over the past century-institutionalist research that described problems; theoretical research, which proposed relationships that might extend beyond existing institutions; and empirical assessments of structural parameters identified in the theoretical research. These three strands operating in concert allowed economic research to be used to predict the fiscal and coverage consequences of alternative policy paths. This ability made economic research a powerful policy force. Key conclusions of health economics research are clearly evident in the Affordable Care Act.
Of course there are problems, but my sense is that people are really concerned with getting things right, that there is pretty robust debate within the profession (and Gelman should add his very knowledgeable outside perspective), and I’m pretty optimistic overall. But maybe I’m too complacent? Is someone going to write an article in 10 years claiming that most published economics research is false?