Multiple Measures: P-Hacking Your Way To Fame | Part 3 of 6

This video is part of a series of videos on p-hacking, what it is, and why it’s so dangerous for science. I give a pretty detailed high-level explanation of p-hacking in the first video in this series, so if you’re not familiar with the idea, please have a look there first. I’ll put a link to that video below. But in a few seconds, researchers are motivated to get what’s called a p-value to be below a threshold of .05. If they do that, their findings are considered meaningful and they can typically publish their results. If they don’t, well all their work is largely wasted. And to get those p-values below .05, there are some very dubious and unethical approaches they can take. In this video, we’ll dig in to one of those unethical approaches that researchers can use to p-hack their data...by collecting multiple measures and only reporting those that “work.”
Welcome to Data Demystified. I’m Jeff Galak and today we’ll dig deeper into p-hacking so that you can understand how to spot it when you see research results and avoid it when you do the research yourself. The goal here is to build intuition, so we’ll avoid heavy duty math and statistics, and focus on what you really need to know.
Calculation for multiple comparisons: 1 - ((1 - .05)^5)
P-Hacking Series Playlist: • P-Hacking
Video 1: P-Hacking Explained (Released on 6/24/2021): • P-Hacking Your Way To ...
Video 2: Dropping Conditions that "Work" (Released on 7/1/2021): • Dropping Conditions th...
Video 3: Multiple Measures Misuse (Released on 7/8/2021): • Multiple Measures: P-H...
Video 4: Covariate Misuse (Released on 7/15/2021): • Covariate Misuse: P-Ha...
Video 5: Selective Stopping Rules (Released on 7/22/2021): • Selective Stopping Rul...
Video 6: P-Curve (Released on 7/29/2021): • P-Curve : P-Hacking Yo...
Link to video about statistical significance testing: • Statistical Significan...
Link to video about randomized experiments and causation: • Correlation Doesn't Me...
Link to video about False Positives: • False Positives vs. Fa...
Link to academic paper by Simmons, Nelson and Simonhson: journals.sagepub.com/doi/full...
Link to academic paper by John, Lowenstein, and Prelec: www.cmu.edu/dietrich/sds/docs...
Follow me at:
LinkedIn: / jeff-galak-768a193a
Patreon: / datademystified
Equipment Used for Filming:
Nikon D7100: amzn.to/320N1FZ
Softlight: amzn.to/2ZaXz3o
Yeti Microphone: amzn.to/2ZTXznB
iPad for Teleprompter: amzn.to/2ZSUkNh
Camtasia for Video Editing: amzn.to/2ZRPeAV

[@bon12121]

5:21 Amazing analogy. Never thought of it like that.

[@ajaykulkarni576]

Jeff, great video, need a little more explanation on 1 - ((1 - .05)^5)

[@DataDemystified]

Okay,...I clearly didn't put enough information there. That's a binomial probability looking to see how often a specific event occurs. Here we are saying, what is the probability that a 5% event (1 in 20) occurs AT LEAST once across 5 tries. The way you do that is say that the odds of the event NOT occurring each time is 95% (that's the 1 - .05). Then we say that this event NOT occurring 5 times is 95% x 95% x 95% x 95% x 95% (that's the ^.5). Then we say...well if that's all the odds of the event NOT occurring 5 times...then the odds of it occurring AT LEAST once is 1 - that. Here's a nice calculator that helps figure all this out: stattrek.com/online-calculator/binomial.aspx

[@yisun1437]

In the first example the problematic part is not to identify shipping error, but lack of validation on independent data. The first part is normal considering nobody knows beforehand what factor matters

[@DataDemystified]

Thank you for the comment! To be clear, if the researchers in my example replicated their results focusing on shipping errors alone, you are correct. Replication solves most p-hacking issues. However, that there isn't an a priori specific prediction grossly inflates type-1 error (false positives). The replication is meant to minimize that risk, but, sadly, most researchers (in academia and otherwise) don't even consider how much type-1 error is inflated by not pre-specifying a single, specific prediction.

[@OneOfTheLostOnes]

What would be the real p value of the study done by the consultants on management? 0.226?

[@DataDemystified]

The most typical correction is called a bonferroni correction. With that, you take your p-value and multiply it by the number of statistical tests that you ran.In this case, if the p-value were .05 and they ran five tests, that would yield a corrected p-value of .05 x 5 = .25.

[@OneOfTheLostOnes]

@@DataDemystified awesome! Thanks!!!