Dropping Conditions that "Work": P-Hacking Your Way To Fame | Part 2 of 6
HTML-код
- Опубликовано: 30 июн 2021
- This video is part of a series of videos on p-hacking, what it is, and why it’s so dangerous for science. I give a pretty detailed high-level explanation of p-hacking in the first video in this series, so if you’re not familiar with the idea, please have a look there first. I’ll put a link to that video below. But in a few seconds, researchers are motivated to get what’s called a p-value to be below a threshold of .05. If they do that, their findings are considered meaningful and they can typically publish their results. If they don’t, well all their work is largely wasted. And to get those p-values below .05, there are some very dubious and unethical approaches they can take. In this video, we’ll dig into one of those unethical approaches that researchers can use to p-hack their data...by dropping experimental conditions that “don’t work.”
Welcome to Data Demystified. I’m Jeff Galak, and in this video we’re going to dig into one of the key p-hacking approaches: dropping conditions that “don’t work.” So let’s jump right in.
P-Hacking Series Playlist: • P-Hacking
Video 1: P-Hacking Explained (Released on 6/24/2021): • P-Hacking Your Way To ...
Video 2: Dropping Conditions that "Work" (Released on 7/1/2021): • Dropping Conditions th...
Video 3: Multiple Measures Misuse (Released on 7/8/2021): • Multiple Measures: P-H...
Video 4: Covariate Misuse (Released on 7/15/2021): • Covariate Misuse: P-Ha...
Video 5: Selective Stopping Rules (Released on 7/22/2021): • Selective Stopping Rul...
Video 6: P-Curve (Released on 7/29/2021): • P-Curve : P-Hacking Yo...
Link to video about statistical significance testing: • Statistical Significan...
Link to video about randomized experiments and causation: • Correlation Doesn't Me...
Link to video about False Positives: • False Positives vs. Fa...
Link to academic paper by Simmons, Nelson and Simonhson: journals.sagepub.com/doi/full...
Link to academic paper by John, Lowenstein, and Prelec: www.cmu.edu/dietrich/sds/docs...
Follow me at:
LinkedIn: / jeff-galak-768a193a
Patreon: / datademystified
Equipment Used for Filming:
Nikon D7100: amzn.to/320N1FZ
Softlight: amzn.to/2ZaXz3o
Yeti Microphone: amzn.to/2ZTXznB
iPad for Teleprompter: amzn.to/2ZSUkNh
Camtasia for Video Editing: amzn.to/2ZRPeAV
6:42 earning your way to a sub.
It seems to me that you are suggesting that researchers change their general hypotheses from the more incentive to work out the more people will work out to : Numerous conditions exist to show that paying people more to work out results in more people working out. Now, you can show all your data and identify the pattern which satisfies the hypothesis. This would also give you the chance to identify other factors that explain the failures to meet the hypothesis. Undoubtedly, given at least 10 comparisons, there is more of a financial incentive to work out that works given your scenario. That is statistically valuable, isn't it? And in every comparison where p
Hi Jason. Thanks for the comment! Lots here so let me take it one at a time.
First, the intuition. If I failed to make it clear, that's on me. Sorry! The idea is that unless you expressly pre-specify a SINGLE prediction, you could, if you were unethical, selectively pick conditions that "work" to support your hypothesis. If you fail to correctly adjust for this flexibility (which is easy enough if you want to...see "Bonferroni correction"), you are seriously over-stating the veracity of your results.
Science more generally: yes, scientists are humans and are driven by incentives. Some will use less than ideal research practices, some will bend the truth, and some will outright fabricate their results. HOWEVER, science isn't about a single finding. No one will take a single finding seriously enough to do anything about it. IF you showed me tomorrow a published paper that suggests that people can read minds, I wouldn't just automatically assume it to be true. I would try and replicate the result and so would many other scientists. If we all, collectively, find more-or-less the same thing, then science advances and we know something new. If that original mind-reading scientist cheated in some way, we will all mostly fail to replicate their results and will just move on. There are literally hundreds of thousands of academic papers published every single year...most of which die in obscurity. The ones that are both interesting AND true, get replicated and change the world.
At the end of the day, you SHOULD be skeptical of any single scientific finding...just like you should be skeptical of anything that is surprising. But if that skepticism is honest, you can let go of it with enough evidence. A single finding is not enough evidence in most cases.
The point of this series on p-hacking is to equip everyone with the knowledge to look for bad science and, quite frankly, just ignore it. Good science will replicate and will update our understand of the world...this just helps us detect the bad science so we don't waste resources trying to verify something that isn't true to begin with.
All the best,...
@@DataDemystified You are probably the most generous person I have ever seen on RUclips. Your willingness to engage is testament to the quality of your work. Thanks so much for a brilliant explanation on both my comments and everyone else's that I have read. Great job. Thank you.
Thank you for your very kind words!
I really hate the disencentive to publish work that didn't fit the original narrative and, especially, work that ends up proving that something didn't really have an effect, or was insignificant. I think it would be terribly important to publish work stating that paying people different amounts of money to go to they gym didn't necessarily improve gym time in a manner that would affect the real world. If we can trust the results of that trial then we can move on to maybe finding something that does work, and anyone else who reads that study won't try to repeat it, but if it gets left out then it leaves the door open for other researchers to design the same experiment, fail, and not get published, which is really inefficient. I would much rather see the study get published showing the reality of the situation, allowing the rest of the researchers to collectively devise other tests.
I think your sentiment is right in spirit, but, like everything else, the devil is in the details. I 100% agree that null-effects are important. The challenge, though (and the typical pushback) is that a null effect in an experiment can result from two places: 1) the hypothesis is wrong and 2) the experiment didn't actually test what it claimed it tested. If all experiments were absolutely perfectly designed and all measures were perfect at capturing whatever it is they wanted to capture, we'd only have null results due to (1). However, the reality is that we have no way of knowing if a null result is due to the hypothesis being wrong or, to be super reductive, the experimenter just not being good at their job. For instance, in the gym-going experiment from the video, what if the results all came back with a null (everyone was equally willing to go to the gym, regardless of payment), but this was all tested during the height of the Covid-19 pandemic when most gyms were closed and/or people weren't willing to go to gyms. You would find no difference in compensation's influence on gym attendance, but that null result would most likely have to do with Covid, and not with incentives. In other words, the reason for the null isn't that they hypothesis is wrong (it might be, but we don't know), but because the experimenter didn't consider that observing a positive finding, under those circumstances is nearly impossible. This is an extreme case, but the same logic applies with inattentive participants, poor measures, insufficiently strong manipulations, etc... I'm not saying all nulls are worthless, but we have to accept that nulls are just that...a lack of evidence to support a hypothesis. They DO NOT refute the hypothesis...they just don't expressly support. Thanks for the comment!!
@@DataDemystified interesting take, however do you think it is possible with the current methodology to reverse the hypothesis, so the research question would be to validate whether there's no _practical_ difference with different incentives?