Simpson’s Paradox

by
Asad Zaman
My articles My booksMy videos
Follow on:Twitter

from Asad Zaman

Statistics and Econometrics today are done without any essential reference to causality – this is much like try to figure out how birds fly without taking into account their wings. Judea Pearl “The Book of Why” Chapter 2 tells the bizarre story of how the discipline of statistics inflicted causal blindness on itself, with far-reaching effects for all sciences that depend on data. These notes are planned as an accompaniment and detailed explanation of the Pearl, Glymour, & Jewell textbook on Causality: A Primer. The first steps to understand causality involve a detailed analysis of the Simpson’s Paradox. This has been done in the sequence of six posts, which are listed, linked, and summarized below

1-Simpson’s Paradox: Suppose that there are only two departments at Berkeley, and that they have different admit ratios for women. In Humanities 40% of female applicants are admitted, while in Engineering 80% are admitted. What will be the overall admit ratio of women to Berkeley? The overall admit ratio is a weighted average of 40% and 80% where the weights are the proportions of females who apply to the two departments.  Similarly, if 20% of male applicants are admitted to Humanities while 60% are admitted to Engineering, then the overall admit ratio is a weighted average of 20% and 60%, with weights depending on the proportion of males which apply to the two departments. This is what lead to the possibility of Simpson’s Paradox. As the numbers have been set up, both Engineering and Humanities favor females, who have much higher admit ratios than male. If males apply mostly to Engineering, then the overall admit ratio for men will be closer to 60%. If females apply mostly to humanities, their overall admit ration will be closer to 40%. So, looking at the overall ratios, it will appear that admissions favor males, who have higher admit ratios. The key question is: which of these comparisons is correct? Does Berkeley discriminate against males, the story told be departmental admit ratios? Or does it discriminate against females, as the overall admit ratios indicate? The main lesson from the analysis in this sequence of posts is that the answer cannot be determined by the numbers. Either answer can be correct, depending on the hidden and unobservable causal structures of the real world which generate the data.

2-Simpson’s Paradox: This post elaborates on Berk’s explanation of the paradox for Berkeley admissions. His explanation can be understood as a causal path diagram where gender affects choice of department. Both gender and choice of department affect the admissions rate. With this causal structure, gender is a confounding variable when it comes to departmental admission ratios. These must be calculated conditionally on gender – that is, separately for men and women. However, departments are NOT a confounding factor when it comes to the effect of gender on admissions rate. Gender affects admissions through two channels – one is a direct affect on admissions ratios, and the second is an indirect effect via choice of department. Female gender affects admission positively via the direct affect which is favorable. However the indirect affect is negative since females choose the more difficult department in larger numbers. The numbers can be set up so that the negative indirect effect overwhelms the positive direct affect, creating the Simpson’s Paradox. But this entire analysis is dependent on a particular causality structure, and different causal structures can lead to entirely different analyses for exactly the same set of numbers. This is the main point of this sequence of posts – to show the hidden and unobservable real world causal structures MUST be considered for meaningful data analysis. Current econometrics and statistics does not pay attention to causality and hence often leads to meaningless analysis.

3-Simpson’s Paradox: This post considers alternative causal structures for Berkeley admissions which lead to conclusions radically different from Berk’s original analysis. We first consider a case where gender affects department choice, while admit ratio depends only on department, and is completely gender neutral. If females choose more difficult departments, there will be a spurious correlation between admit ratios and gender, creating a misleading impression of discrimination against females. A second example is considered where admissions depend purely on SAT scores, and has no relationship to gender or to department. Nonetheless, if gender affects SAT Scores and choice of department, we can replicate the exact same numbers of the original data, which would create the misleading impressions that departments discriminate by gender, and some departments are more difficult to get into than others. In fact, admissions policy is same across departments, and depends only on SAT scores. The point of these analyses is that exactly the same observed data can correspond to radically different causal structures, and lead to radically different conclusions about discrimination with respect to gender.

4-Simpson’s Paradox: Contrary to the perspective taken by conventional statistics texts, and some forms of econometric analysis (VAR models), we cannot do data analysis without knowing where the numbers come from. The jobs of the field expert and the statistical consultant cannot be separated. To illustrate this point, we consider the same data generated for the Berkeley admissions, and consider it as batting averages of two different batters against left and right-handed pitchers. Then the Simpson’s Paradox takes the following form. Frank has higher batting average than Tom against left-handed pitchers and he also has higher batting average than Tom against left-hand pitchers. However, the overall batting average of Tom is higher than that of Frank. As the manager of the team, which one of the two should you send out when it is critical to get an extra hit or two? If we consider left and right handed pitchers separately, Frank is better than Tom for both, and hence we should send Frank. However, overall batting average of Tom is better, suggesting that we should send out Tom. The answer depends on the causal structure. If the choice of pitchers is EXOGENOUS – independent of the batter choice – then Frank is the better choice. If adversary coach looks at the batter to decide on the pitchers, then the choice of pitchers is endogenous, and in this case Tom may be the better choice.

5-Simpson’s Paradox: To further drive home the fact that data analysis cannot be confined to numbers, and divorced from the real world environment which generated the data, we consider a third interpretation of the same data set used for Berkeley admissions. In this interpretation, we look at the effect of a drug on recovery rates from a disease. The Simpson Paradox takes the form that the drug decreases recovery rates in females, and also decreases recovery rates in males. So, it is bad for males and it is bad for females. But when we look at the population as a whole, we find that the drug improves recovery rate. So, the drug is good for the general population. A causal path diagram shows that gender must be exogenous – it cannot be affected by the drug. Thus gender is a confounding variable, we must condition on this variable to get the right measure of the effect of drug on recovery. Thus we conclude that the drug is bad for everyone, and lowers the recovery rate for everyone, even though the overall data tells us otherwise. But now consider the same data set with gender replaced by blood pressure, and suppose that the drug affects blood pressure. Suppose low blood pressure is a positive factor in recovery, while the drug has a toxic-effect so that the direct impact is negative. However, the drug also lowers the blood pressure, which creates a positive factor for recovery. The combined effect can be favorable, and this is what should be considered when administering the drug.