Tag Archives: statistical significance

The knives are out for the p-value. This statistical quantity is the Holy Grail for empirical researchers across the world -- if your study finds the right p-value, you can get published in a credible journal, and possibly get a good university tenure-track job and research funding. Now a growing chorus of voices wants to de-emphasize or even ban this magic number. But the crusade against p-values is likely to be a distraction from the real problems afflicting scientific inquiry.... The...

From my 2009 paper with Weakliem: Throughout, we use the term statistically significant in the conventional way, to mean that an estimate is at least two standard errors away from some “null hypothesis” or prespecified value that would indicate no effect present. An estimate is statistically insignificant if the observed value could reasonably be explained by simple chance variation, much in the way that a sequence of 20 coin tosses might happen to come up 8 heads and 12 tails; we would say...

Summary: The end-in-view is doing good science and avoiding junk science, which is proliferating. Adjusting standards, etc. are only means to an end. There are no silver bullets or magic wands. Doing good science depends on good design, accurate measurement, and replication. Statistical Modeling, Causal Inference, and Social ScienceWhen considering proposals for redefining or abandoning statistical significance, remember that their effects on science will only be indirect! Andrew Gelman |...

If you are keeping up with this. Some finer points.From the epistemological point of view, criticism of statistical significance here is based on questioning a criterion that is stipulated, i.e., defined arbitrarily.Doing so gives formalization and modeling a greater importance than advancing understanding. That is unscientific.A pragmatic approach is more appropriate than a strictly formal one, especially as an institutional norm.Formal rigor is a necessary condition but not a sufficient...

As shown over and over again when significance tests are applied, people have a tendency to read ‘not disconfirmed’ as ‘probably confirmed.’ Standard scientific methodology tells us that when there is only say a 10 % probability that pure sampling error could account for the observed difference between the data and the null hypothesis, it would be more ‘reasonable’ to conclude that we have a case of disconfirmation. Especially if we perform many independent tests of our hypothesis and they...

AbstractIn science publishing and many areas of research, the status quo is a lexicographic decision rule in which any result is first required to have a p-value that surpasses the 0.05 threshold and only then is consideration—often scant—given to such factors as prior and related evidence, plausibility of mechanism, study design and data quality, real world costs and benefits, novelty of finding, and other factors that vary by research domain. There have been recent proposals to change the...