by Summary:
I think there are two separate issues here that depend on intent. "P-hacking" likely implies intent, and that is not necessarily a factor in all cases, and it may well not be in many if not most cases. In some cases there may be intent to persuade by playing loose, or even to deceive. I recall that How to Lie with Statistics was required reading in the Stat 101 course I took over fifty years ago. But this is not the only issue. As Richard Feynman famously observed, science is about not fooling ourselves. This applies to each of us individually owing to cognitive bias. Humans are smart, but we mare still primates. Nobody is entirely free of cognitive-affective bias. So we have to take steps to counter this tendency. Science was developed as an instrument to address this.
Topics:
Mike Norman considers the following as important: cognitive bias, data, data selection, formal rigor, scientific modeling, statistical inference, statistical reasoning
This could be interesting, too:
I think there are two separate issues here that depend on intent. "P-hacking" likely implies intent, and that is not necessarily a factor in all cases, and it may well not be in many if not most cases. In some cases there may be intent to persuade by playing loose, or even to deceive. I recall that How to Lie with Statistics was required reading in the Stat 101 course I took over fifty years ago. But this is not the only issue. As Richard Feynman famously observed, science is about not fooling ourselves. This applies to each of us individually owing to cognitive bias. Humans are smart, but we mare still primates. Nobody is entirely free of cognitive-affective bias. So we have to take steps to counter this tendency. Science was developed as an instrument to address this.
Topics:
Mike Norman considers the following as important: cognitive bias, data, data selection, formal rigor, scientific modeling, statistical inference, statistical reasoning
This could be interesting, too:
Mike Norman writes econintersect — 50 Cognitive Biases In The Modern World
Mike Norman writes Why we are wrong — Chris Dillow
Mike Norman writes Lars P. Syll — On the applicability of statistics in social sciences
I think there are two separate issues here that depend on intent. "P-hacking" likely implies intent, and that is not necessarily a factor in all cases, and it may well not be in many if not most cases.
In some cases there may be intent to persuade by playing loose, or even to deceive. I recall that How to Lie with Statistics was required reading in the Stat 101 course I took over fifty years ago. But this is not the only issue.
As Richard Feynman famously observed, science is about not fooling ourselves. This applies to each of us individually owing to cognitive bias. Humans are smart, but we mare still primates.
Nobody is entirely free of cognitive-affective bias. So we have to take steps to counter this tendency. Science was developed as an instrument to address this.
The reason we use rigorous method is to avoid, or at least minimize, our tendency toward being shaped by cognitive biases such as confirmation bias and anchoring.
Methodology is about using instruments on good data in a rigorous fashion that reduces not only error in application but also bias.
There is no method that completely eliminates error and bias. On one hand, GIGO, and being human, on the other.
A lot of the problems in doing science as well as applying other rigorous instruments lies in measurement. The highest level of formality does nothing to affect errors in measurement. I happened to be thinking about the issues around measurement just prior to reading this post.
And lot of the most interesting things are difficult to measure when humans are involved and psychology enters into the data significantly. History also present issues regarding not only data quality and availability but also changing context that affects the data.
It's good we are having a debate about p-values, since there are issues there than do seem to be influential in a negative way. And it is not only the stat, but also the data that the method is being applied to.
There are essentially three areas of interest. The first is the method, in this case probability and statistics as formal method. The second is data and its reliability and precision, along with data collection. The third is data processing and selection. All of these are subject to error and manipulation. This is especially a problem when data sets are proprietary and are not transparent.
But the debate should not stop there. The methodological debate is not over, as some would have it. Science is always tentative on discovery and it is a work in progress. Science is often viewed as a fixed body of true knowledge. That is not a good approach to doing science. The fundamental principle of science is questioning authority, especially that of received belief, intuition and common sense.
Humans are fallible, and it is doubtful that we can ever finally work out all the kinks epistemologically and methodologically. We are a work in progress, too, just as is science.
As a discipline becomes more formalized, there is a greater tendency to emphasize formal rigor at the expense of data and evidence, especially when there are issues around data and evidence. Such tendencies are fertile ground for cognitive-affective bias.
Epistemology, logic, and methodology are foundational to gaining reliable knowledge. We need to keep this in mind.
On one hand, the search for absolute knowledge is a chimera since no criteria can be established as absolute. Criteria are stipulated. This realization should make us humble — and careful.
On the other hand, humans are not lost in a sea of relativity either. History has shown that it is possible to arrive at knowledge that is reliable and practical if intelligence is applied and bias reduced.
P-hacking and data dredging
Lars P. Syll | Professor, Malmo University