by Data clusters are everywhere, even in random data. Someone who looks for an explanation will inevitably find one, but a theory that fits a data cluster is not persuasive evidence. The found explanation needs to make sense, and it needs to be tested with uncontaminated data. Similarly, someone who fires enough bullets at enough targets is bound to hit one. A researcher who tests hundreds of theories is bound to find evidence supporting at least one. Such evidence is unconvincing unless the theory is sensible and confirmed with fresh data. When you hear that the data support a theory, don’t be persuaded until you’ve answered two questions. First, does the theory make sense? If it doesn’t, don’t be easily persuaded that nonsense is sensible. Second, is there a Texas
Topics:
Lars Pålsson Syll considers the following as important: Statistics & Econometrics
This could be interesting, too:
Lars Pålsson Syll writes What statistics teachers get wrong!
Lars Pålsson Syll writes Statistical uncertainty
Lars Pålsson Syll writes The dangers of using pernicious fictions in statistics
Lars Pålsson Syll writes Interpreting confidence intervals
Data clusters are everywhere, even in random data. Someone who looks for an explanation will inevitably find one, but a theory that fits a data cluster is not persuasive evidence. The found explanation needs to make sense, and it needs to be tested with uncontaminated data.
Similarly, someone who fires enough bullets at enough targets is bound to hit one. A researcher who tests hundreds of theories is bound to find evidence supporting at least one. Such evidence is unconvincing unless the theory is sensible and confirmed with fresh data.
When you hear that the data support a theory, don’t be persuaded until you’ve answered two questions. First, does the theory make sense? If it doesn’t, don’t be easily persuaded that nonsense is sensible. Second, is there a Texas sharpshooter in the house? Did the person promoting the theory look at the data before coming up with the theory? Were hundreds of theories tested before settling on the theory being promoted? If there is a smoking gun, withhold judgment on the theory until it has been tested with different data.