Selective Stopping Rules: P-Hacking Your Way To Fame | Part 5 of 6

Selective stopping rules. This video is part of a series of videos on p-hacking, what it is, and why it’s so dangerous for science. I give a pretty detailed high-level explanation of p-hacking in the first video in this series, so if you’re not familiar with the idea, please have a look there first. I’ll put a link to that video below. But in a few seconds, researchers are motivated to get what’s called a p-value to be below a threshold of .05. If they do that, their findings are considered meaningful and they can typically publish their results. If they don’t, well all their work is largely wasted. And to get those p-values below .05, there are some very dubious and unethical approaches they can take. In this video, we’ll dig in to one of those unethical approaches that researchers can use to p-hack their data...by stopping data collection when your results are “good.”
Welcome to Data Demystified. I’m Jeff Galak and today we’ll dig deeper into p-hacking so that you can understand how to spot it when you see research results and avoid it when you do the research yourself. The goal here is to build intuition, so we’ll avoid heavy duty math and statistics, and focus on what you really need to know.
P-Hacking Series Playlist: • P-Hacking
Video 1: P-Hacking Explained (Released on 6/24/2021): • P-Hacking Your Way To ...
Video 2: Dropping Conditions that "Work" (Released on 7/1/2021): • Dropping Conditions th...
Video 3: Multiple Measures Misuse (Released on 7/8/2021): • Multiple Measures: P-H...
Video 4: Covariate Misuse (Released on 7/15/2021): • Covariate Misuse: P-Ha...
Video 5: Selective Stopping Rules (Released on 7/22/2021): • Selective Stopping Rul...
Video 6: P-Curve (Released on 7/29/2021): • P-Curve : P-Hacking Yo...
Link to video about statistical significance testing: • Statistical Significan...
Link to video about randomized experiments and causation: • Correlation Doesn't Me...
Link to video about False Positives: • False Positives vs. Fa...
Link to academic paper by Simmons, Nelson and Simonhson: journals.sagepub.com/doi/full...
Link to academic paper by John, Lowenstein, and Prelec: www.cmu.edu/dietrich/sds/docs...
Follow me at:
LinkedIn: / jeff-galak-768a193a
Patreon: / datademystified
Equipment Used for Filming:
Nikon D7100: amzn.to/320N1FZ
Softlight: amzn.to/2ZaXz3o
Yeti Microphone: amzn.to/2ZTXznB
iPad for Teleprompter: amzn.to/2ZSUkNh
Camtasia for Video Editing: amzn.to/2ZRPeAV

[@howexistential]

With a large enough sample size, does the p-value converge to its "true" value due to CLT? Also, thanks for putting these videos out!

[@DataDemystified]

Pretty much yes! As sample size increases, random error goes down.

[@unbridled_exciton]

@@DataDemystified I think your statement is incorrect in a practical sense.
From a theoretical POV, if we have a small population, then it might be possible to sample the whole population. In such setting it is true that all the p-values will converge to one "true" p-value, calculated from the whole population.
However, in practice our sample sizes are small when compared to the population. So, the samples will differ and produce different p-values. For example, it is known that when H0 is true and H0 is a simple hypothesis the p-values will have a uniform(0,1) distribution. In such case there is no "true" p-value: all the p-values from 0 to 1 are equally possible.
At the same time, I think that it is true that as we accumulate more data, our p-value becomes more robust to change caused by addition of one more data point. But that is not to say that the p-value converges to some true value.