by In a blog post the other day, Noah Smith returned again to the discussion about the ’empirical revolution’ in economics and how to — if it really does exist — evaluate it. Counter those who think quasi-experiments and RCTs are the true solutions to finding causal parameters, Noah argues that without structural models empirical results are only locally valid. And you don’t really know how local “local” is. If you find that raising the minimum wage from to doesn’t reduce employment much in Seattle, what does that really tell you about what would happen if you raised it from to in Baltimore? That’s a good reason to want a good structural model. With a good structural model, you can predict the effects of policies far away from the current state of the world. If
Topics:
Lars Pålsson Syll considers the following as important: Statistics & Econometrics
This could be interesting, too:
Lars Pålsson Syll writes What statistics teachers get wrong!
Lars Pålsson Syll writes Statistical uncertainty
Lars Pålsson Syll writes The dangers of using pernicious fictions in statistics
Lars Pålsson Syll writes Interpreting confidence intervals
In a blog post the other day, Noah Smith returned again to the discussion about the ’empirical revolution’ in economics and how to — if it really does exist — evaluate it. Counter those who think quasi-experiments and RCTs are the true solutions to finding causal parameters, Noah argues that without structural models
empirical results are only locally valid. And you don’t really know how local “local” is. If you find that raising the minimum wage from $10 to $12 doesn’t reduce employment much in Seattle, what does that really tell you about what would happen if you raised it from $10 to $15 in Baltimore?
That’s a good reason to want a good structural model. With a good structural model, you can predict the effects of policies far away from the current state of the world.
If only that were true! But it’s not.
Structural econometrics — essentially going back to the Cowles programme — more or less takes for granted the possibility of a priori postulating relations that describe economic behaviours as invariant within a Walrasian general equilibrium system. In practice that means the structural model is based on a straightjacket delivered by economic theory. Causal inferences in those models are — by assumption — made possible since the econometrician is supposed to know the true structure of the economy. And, of course, those exact assumptions are the crux of the matter. If the assumptions don’t hold, there is no reason whatsoever to have any faith in the conclusions drawn, since they do not follow from the statistical machinery used!
Structural econometrics aims to infer causes from probabilities, inferred from sample data generated in non-experimental settings. Arguably, it is the most ambitious part of econometrics. It aims to identify economic structures, robust parts of the economy to which interventions can be made to bring about desirable events. This part of econometrics is distinguished from forecasting econometrics in its attempt to capture something of the ‘real’ economy in the hope of allowing policy makers to act on and control events …
By making many strong background assumptions, the deductivist [the conventional logic of structural econometrics] reading of the regression model allows one — in principle — to support a structural reading of the equations and to support many rich causal claims as a result. Here, however, the difficulty is that of finding good evidence for many of the assumptions on which the approach rests. It seems difficult to believe, even in cases where we have good background economic knowledge, that the background information will be sufficiently to do the job that the deductivist asks of it. As a result, the deductivist approach may be difficult to sustain, at least in economics.
The difficulties in providing an evidence base for the deductive approach show just how difficult it is to warrant such strong causal claims. In short, as might be expected there is a trade-off between the strength of causal claims we would like to make from non-experimental data and the possibility of grounding these in evidence. If this conclusion is correct — and an appropriate elaboration were done to take into account the greater sophistication of actual structural econometric methods — then it suggests that if we want to do evidence-based structural econometrics, then we may need to be more modest in the causal knowledge we aim for. Or failing this, we should not act as if our causal claims — those that result from structural econometrics — are fully warranted by the evidence and we should acknowledge that they rest on contingent, conditional assumptions about the economy and the nature of causality.
Maintaining that economics is a science in the ‘true knowledge’ business, yours truly remains a skeptic of the pretences and aspirations of — both structural and non-structural — econometrics. So far, I cannot see that it has yielded much in terms of relevant, interesting economic knowledge. Over all the results have been bleak indeed.
Firmly stuck in an empiricist tradition, econometrics is only concerned with the measurable aspects of reality. But there is always the possibility that there are other variables — of vital importance and although perhaps unobservable and non-additive, not necessarily epistemologically inaccessible — that were not considered for the econometric modeling.
Most econometricians still concentrate on fixed parameter models and the structuralist belief/hope that parameter-values estimated in specific spatio-temporal contexts are exportable to totally different contexts. To warrant this assumption one, however, has to convincingly establish that the targeted acting causes are stable and invariant so that they maintain their parametric status after the bridging. The endemic lack of predictive success of the econometric project indicates that this hope of finding fixed parameters is a hope for which there really is no other ground than hope itself.
Most of the assumptions that econometric modeling presupposes are not only unrealistic — they are plainly wrong.
If economic regularities obtain they do it (as a rule) only because we engineered them for that purpose. Outside man-made ‘nomological machines’ they are rare, or even non-existant. Unfortunately that also makes most of the achievements of both structural and non-structural econometric forecasting and ‘causal explanation’ rather useless.
Invariance assumptions need to be made in order to draw causal conclusions from non-experimental data: parameters are invariant to interventions, and so are errors or their distributions. Exogeneity is another concern. In a real example, as opposed to a hypothetical, real questions would have to be asked about these assumptions. Why are the equations “structural,” in the sense that the required invariance assumptions hold true? Applied papers seldom address such assumptions, or the narrower statistical assumptions: for instance, why are errors IID?
The tension here is worth considering. We want to use regression to draw causal inferences from non-experimental data. To do that, we need to know that certain parameters and certain distributions would remain invariant if we were to intervene. Invariance can seldom be demonstrated experimentally. If it could, we probably wouldn’t be discussing invariance assumptions. What then is the source of the knowledge?
“Economic theory” seems like a natural answer, but an incomplete one. Theory has to be anchored in reality. Sooner or later, invariance needs empirical demonstration, which is easier said than done.
