by A meta-analysis identifies a set of studies, each of which provides one or more estimates of the effect of some intervention. For example, one might be interested in the impact of job training programs on prisoner behavior after release. For some studies, the outcome of interest might be earnings; do inmates who participate in job training programs have higher earnings after release than those who do not? For other studies, the outcome might be the number of weeks employed during the first year after release. For a third set of studies, the outcome might be the time between release and getting a job. For each outcome, there would likely be several research reports with varying estimates of the treatment effect. The meta-analysis seeks to provide a summary estimate over all of the studies … The outcome is both pleasing and illusory. The subjects in treatment and control (even in a randomized controlled experiment … ) are not drawn at random from populations with a common variance; with an observational study, there is no randomization at all. It is gratuitous to assume that standardized effects are constant across studies: it could be, for instance, that the average effects themselves are approximately constant but standard deviations vary widely.
Topics:
Lars Pålsson Syll considers the following as important: Statistics & Econometrics
This could be interesting, too:
Lars Pålsson Syll writes The history of econometrics
Lars Pålsson Syll writes What statistics teachers get wrong!
Lars Pålsson Syll writes Statistical uncertainty
Lars Pålsson Syll writes The dangers of using pernicious fictions in statistics
A meta-analysis identifies a set of studies, each of which provides one or more estimates of the effect of some intervention. For example, one might be interested in the impact of job training programs on prisoner behavior after release. For some studies, the outcome of interest might be earnings; do inmates who participate in job training programs have higher earnings after release than those who do not? For other studies, the outcome might be the number of weeks employed during the first year after release. For a third set of studies, the outcome might be the time between release and getting a job. For each outcome, there would likely be several research reports with varying estimates of the treatment effect. The meta-analysis seeks to provide a summary estimate over all of the studies …
The outcome is both pleasing and illusory. The subjects in treatment and control (even in a randomized controlled experiment … ) are not drawn at random from populations with a common variance; with an observational study, there is no randomization at all. It is gratuitous to assume that standardized effects are constant across studies: it could be, for instance, that the average effects themselves are approximately constant but standard deviations vary widely. If we seek to combine studies with different kinds of outcome measures (earnings, weeks worked, time to first job), standardization seems helpful. And yet, why are standardized effects constant across these different measures? Is there really one underlying construct being measured, constant across studies, except for scale? We find no satisfactory answers to these critical questions …
The interesting question is why the technique is so widely used. One possible answer is this. Meta-analysis would be a wonderful method if the assumptions held. However, the assumptions are so esoteric as to be unfathomable and hence immune from rational consideration: the rest is history.