by Or not enough. Nate Silver has long performed fairly complicated calculations with (lots of) raw data. Even back when he was Poblano at DailyKos this was highly controversial. Now that he leads a huge team at fivethirtyeight.com, it is almost necessary just to decide whether to trust them, because it is very time consuming to read their explanations of their algorithms. In particular, obsessive poll watchers such as your humble correspondent, have noted that Senate ratings by fivethirtyeight.com and realclearpolitics.com don’t always move together. Very often the difference between the complicated calculations at fivethirtyeight.com and the simple averages at realclearpolitics.com is greater than the change of either from week to week. One very important
Topics:
Robert Waldmann considers the following as important: Journalism, politics
This could be interesting, too:
Bill Haskell writes Monthly payments could get thousands of homeless people off the streets
Angry Bear writes A Doctor at Cigna Said Her Bosses Pressured Her to Review Patients’ Cases Too Quickly
Steve Roth writes How Did Under-40s Get So Much Richer During Covid?
Eric Kramer writes Eric Segall tells us what he really thinks about the Roberts court
Or not enough. Nate Silver has long performed fairly complicated calculations with (lots of) raw data. Even back when he was Poblano at DailyKos this was highly controversial. Now that he leads a huge team at fivethirtyeight.com, it is almost necessary just to decide whether to trust them, because it is very time consuming to read their explanations of their algorithms.
In particular, obsessive poll watchers such as your humble correspondent, have noted that Senate ratings by fivethirtyeight.com and realclearpolitics.com don’t always move together. Very often the difference between the complicated calculations at fivethirtyeight.com and the simple averages at realclearpolitics.com is greater than the change of either from week to week.
One very important (and controversial) aspect of the fivethirtyeight.com approach is correcting raw polls for house effects. The corrected number which goes into the smoother used to make forecasts is a prediction of what a random pollster would report (more likely to be selected if they have performed well in the past). This means that the calculation is first an estimate of a (quality rating weighted) central tendency of polls based on the observation. Then this is averaged (with slightly more weight on more recnt polls) to give the forecast.
The justification is *not* based on the claim that the (weighted) average poll is unbiased, but rather that such bias can’t be estimated. The report probabilities relatively close to 50-50 (compared at least to bug eating non welcher Sam Wang) because they allow a random term for the unknown common bias. Rather the claim is that removing house effects can be done and reduces the noise in the smoothed average.
I think the way to test such a claim is to try to predict not the outcome (which is occurs only every other year) but the next poll (which is biased only because of the unmeasurable common bias).
My impression is that they are too cautious when removing house effects — that the corrected numbers deviate from the others in the same direction as the uncorrected numbers. So Rasmussen reports polls which (compared to the others and not the unknown truth) are biased Republican. The corrected Rasmussen numbers fivethirtyeight uses also tend to look unusually good for Republicans. This would happen if reluctance to process the numbers too much lead to smaller corrections than would be optimal (that would minimize forecast error).
To test this impression, I took the uncorrected and corrected Democratic leads from generic ballot polls with data from July 1 2018 on (I only went back that far because I was doing this by hand and got bored).
There is one very interesting result
Number of obs = 134
R-squared = 0.0589
Root MSE = 2.725
corrected | Coef. Std. Err. t
correction | -.4733886 .1647134 -2.87
_cons | 8.448304 .2714733 31.12
Corrected is the corrected number used by fivethirtyeight to forecast outcomes. correction is that number minus the raw number reported by the pollster.
There is a statistically significant correlation between the corrected number and the correction. A larger correction (1.4733886) times as large would minimize the variance of the more_corrected numbers.
This correction to the correction remains statistically significant also when a lowess smoothed (with parameter 0.8) series (pcorrected) is included. Rather strikingly (but not to poll obsessives) pcorrected has varied very little — there is no noticeable trend in the polls).
corrected | Coef. Std. Err. t
correction | -.4482108 .1651148 -2.71
pcorrected | 1.287806 .9229588 1.40
_cons | -2.000649 7.493556 -0.27
The main expainable part of the variance in the corrected polls is the under-correction for house effects.
OK so it’s time to admit that I talked about forecasting the next poll. I look at the next (raw) poll in the sample ldraw.
Number of obs = 133
R-squared = 0.0414
Root MSE = 3.4038
–
ldraw | Coef. Std. Err. t
correction | .4870599 .2121298 2.30
corrected | -.0009333 .1096151 -0.01
_cons | 6.833945 .9893328 6.91
Now that is weird. The idea was that in addition to a coefficient close to one on the lagged corrected poll, there would be a postiive coefficient on the correction. This would happen if the polls were corrected too little (to little of a good thing). Instead there is a significant coefficient on the correction (corrected too little) and an almost exactly zero coefficient on the lagged poll (as would happen if there were no true change in voting intentions).
In any case, there is more statistically evidence that the polls aren’t corrected enough by fivethirtyeight.com