P-Hacking: Crash Course Statistics #30
HTML-код
- Опубликовано: 21 авг 2024
- Today we're going to talk about p-hacking (also called data dredging or data fishing). P-hacking is when data is analyzed to find patterns that produce statistically significant results, even if there really isn't an underlying effect, and it has become a huge problem in science since many scientific theories rely on p-values as proof of their existence! Today, we're going to talk about a few ways researchers have "hacked" their data, and give you some tips for identifying and avoiding these types of problems when you encounter stats in your own lives.
XKCD's comic on p-hacking: xkcd.com/882/
Crash Course is on Patreon! You can support us directly by signing up at / crashcourse
Thanks to the following Patrons for their generous monthly contributions that help keep Crash Course free for everyone forever:
Mark Brouwer, Trevin Beattie, Satya Ridhima Parvathaneni, Erika & Alexa Saur, Glenn Elliott, Justin Zingsheim, Jessica Wode, Eric Prestemon, Kathrin Benoit, Tom Trval, Jason Saslow, Nathan Taylor, Brian Thomas Gossett, Khaled El Shalakany, Indika Siriwardena, SR Foxley, Sam Ferguson, Yasenia Cruz, Eric Koslow, Caleb Weeks, Tim Curwick, D.A. Noe, Shawn Arnold, Ruth Perez, Malcolm Callis, Advait Shinde, William McGraw, Andrei Krishkevich, Rachel Bright, Mayumi Maeda, Kathy & Tim Philip, Eric Kitchen, Ian Dundore, Chris Peters
--
Want to find Crash Course elsewhere on the internet?
Facebook - / youtubecrashcourse
Twitter - / thecrashcourse
Tumblr - / thecrashcourse
Support Crash Course on Patreon: / crashcourse
CC Kids: / crashcoursekids
It has been shown that p-values barely under 0.05 are statistically over represented. This was shown with a p-value < 0.05.
I'm dead 😂
Haha. That's funny.
Actually, there is a small misconception in this video (the table in the beginning). When we reject the alternative hypothesis that does not mean that the the null hypothesis is true. It simply means that we do not have statistical evidence to reject the null hypothesis - we cannot say with 100 % statistical certainty that Ho is true. Andy Fields writes in 'Discovering Statistics in SPSS': "If the p-value is greater than 0,05 you can decide to reject the alternative hypothesis but that is not the same as the null hypothesis being true" (Fields, 2018: 76).
This crash course is great. The level is fairly high (I believe that most people with a PhD could even learn a thing or two), and it's still comprehensible.
Great job!
During my Ba and MA I certainly did feel the strong push to "just try this or that" so the p-values would be a little bit more acceptable. I never really got a straight and clear answer how that wasn't similar to cheating. But everyone seemed to think it was quite okay - That was a weird experience.
Gaslighting, that's called right?
If you aren't publishing (and as a student, you probably weren't expected to publish) then p-hacking isn't a concern. If you're learning the "art" of statistics, you need to experiment with the way different analyses work.
**sighs** You explained this so much better than my biometry prof. Thank you!
I think I loved the peas on the laptop. I'm confident with a p=0.04 that did not NOT love them.
@Rowan Brown Luk je kemi roa? Juk no ke lemapu ke hoa?
Poor kid... He fell asleep studying and gets called an honest mistake :D
Plot twist: those peas are actually green jellybeans.
This is amazing! I dont understand anything you talked about . Still love it👌🏽
All of that and you didn't mention the common terms "fishing" and "exploratory analyses", nor how such approaches can be used ethically to generate new hypotheses, or be methodologically accommodated, such as with screening and hold-out samples (cross-validation).
The bowl of peas sitting at that angle and on a laptop is very distressing!
I just want to say, when you calculate that with twenty tests all looking for p < 0.05, there is a 65% chance that we will get a false positive, you said that this "might be higher than you would expect". My statistically challenged brain sees it the other way, though: if each test has a 5% chance of giving a false positive and we run 20 tests, then we get a 100% chance of a false positive because 20 * 5% = 100%!!
(Of course I understand why this is wrong, and you've explained it very well over the course of this series--I just wanted to point out what seemed to me to be the obvious mistake to make!)
You can think of it like "what's the chances that I'll get NO false positives in 20 tests of p = 0.05", which is ( 1 - 0.05 ) ^ 20 = 0.358 or 36%, so on the other hand, 1 - 0.358 = 0.642, or 64.2% chance of getting more than 1 false positive.
Someone needs to start a scientific journal that only publishes papers in which the null hypothesis is not rejected.
XauriEL Zwaan already done. Journal of negative results or something like that. But it’s not much cited...
Hi! I really like your videos, I'll eventually go through all of them. Please make a video on multi-variate analysis. I' beginning to understand them but would really like for you to explain them (PCA, CVA)
i had to pause the damn video to laugh like a goat when i saw the peas on the laptop XD
I was a bit confused about the Bonferroni correction, because surely, for a total p-value of 0.05 and 5 tests, the p-value per test should be
p < 1 - 5th.root(0.95)
But apparently, the simple division always leads to a smaller value than this calculation, making it basically close enough, and an even stricter limit. Although it surprises me that you didn't point this out, as that quotient comes kind of out of nowhere.
Yes it's a conservative approximation (that's close enough if you don't have a huge number of tests) that almost everyone uses without knowing that it is always (a little) too conservative (I guess that's why she doesn't mention it). The accurate correction (for independent tests) which you pointed out is called the Sidak correction :)
Yes, she also didn't talk about other solutions ie cross validation. Bringing up only Bonferroni (and also not talking about how conservative it is) and not talking about the issue more fully can be a bit misleading /:
I use the Bonferroni correction quite often. There are more than 20 thousand human genes in our DNA. Sometimes I need to look for genes which **significantly** change under a treatment. Without the Bonferroni correction, I would be doing over 20 thousand different p-values, which of course I would find something that appears "significant" with a p < 0.05.
I can see how one could easily come to the wrong conclusion if one is not careful under that circumstance.
WIth such a number of tests though maybe you could use another correction. Bonferonni is always a little too conservative, you could use the more acurrate Sidak correction. Or even Holm's or Hochberg's procedure...
@@nadjal Yeah totally. I use some R packages that let me choose from a variety of multiple testing procedures. Bonferroni is just one
P-Hacking was my nickname in highschool.
Phaking hell
how to get p-hacking out of "for profit" research?
1-make only "no profit" & massively perreviewed research significant enough for laws.
2-implement UBI and/or have a "national scientific research fund" that allows for scientists to follow through theier research within ethical conditions for themselfs and their results.
Peas and laptop for P-hacking... XD
This is such an important video
Awesome video! Thank you!
I always love your props! DFTBAQ!
Chad and Stacy studio 💪💪💪💪
So, in order to believe any statistic we see, and even then the statistic might itself be a chance result, we need Ph.D.'s in statistical analysis, an army of scientist employees, and sophisticated lab and computer equipment to verify claims. Oh, and access to the journals said statistic was published in, each of which will cost a pretty penny.
Not just CC, but in general, the media and experts do a fine job of outlining the problem but rarely give solutions and even when they do, the solutions are impractical or theoretical.
The same media and experts, due to shenanigans like p-hacking, have broken the public's trust in what they have to offer.
My solution: mandatory statistics classes beginning in Kindergarten. This will take forty years for any critical mass of trust to return to the societal influencers because in forty years, today's kindergarteners will have the power to influence and the old guard of prevaricating and unethical doyens will have died or be too old to dictate the direction of society in any meaningful way.
Or, wait for the Matrix plugin seats. They're coming!
Michael Crichton was talking about this 20 years ago. He suggested that the government fund a return to basic reseach and that researchers make their data available so that others can replicate their studies. His senate testimony is available online.
Pee hacking: pissing in a bottle while stuck in traffic
Or, using someone else's urine to pass a drug screening test.
Excellent
How to get a tiny p-value every time? It's very easy conceptually. Increase your sample size. In fact, in areas of science where very large sample sizes are the norm, the p-values are almost always less than 0.05. Hmm.....so if with very large studies p-values are kind of pointless....does this mean that p-values are kinda dumb in general. Yes, and many, many people think that and write about it.
So how did p-value type analyses become so popular? Easy. It provides a veneer of objectivity when scientific decision making is not totally objective. You can't have a machine spit out a yes or no, usually. Also, with moderate sample sizes, a low p-value indicates a relatively strong strength of effect. Hmm...so why not just look at strength of effect directly and lay everything out so that individuals can decide upon it? Well, that's a better approach.
Cue first year math students, people that have taken one stats course, or a semi-competent stats teacher/researcher to call me crazy.
Research this question about null hypothesis and significance testing and the controversy around it. There are hundreds of resources.
Yeah, the p-value is just a number you can use to argue your results. You should still be able to adapt your argument to your experiment/methods. Picking .05 as a cutoff (as it was said on other CC:Statistics videos) is just arbitrary and means nothing without context. The problem is that statistics isn't the most prestigous field of mathematics, but it is one of the most important one in ALL fields of science. So there are few really good Statisticians, that are good at teaching new generations.
You still need to have an effect (however small) to get a tiny p-value no matter how large the sample size. The way p-hacking is done is with lots of tests on lots of variables, not lots of samples.
Thank you for doing this!
Oh
Thank you for the advice
contrast two results out of every 24. (3/3+1/3) : (3/4+1/4)
I liked this video, but it was basically a drawn out way of saying that if you don't take everything into account or intentionally leave some things out you can get bad results.
can you guys do a crash course forensics??
Awesome!
why is it called family wise error??
Wait you mean buzzfeed aint legit?
"You feel bad about taking their million dollars."
You should never feel bad about taking Kevin O'Leary's money. p
Oramchi poki chutki he? Mera nor ke gor cun chu! Lakchi goy he?
That broken laptop annoys me
I will never eat a green jelly bean again.
Watch out people: the green jellybeans are really peas.
I only found out this statistic video 2 days before my exam dammit
Regards Adrian, I am planning to carry out an empirical study with p hacking being the subject at hand, could you please a case study that could be worked upon.
Thanks in advance !!!!!!
This puts the sigh in science and the lie in believe the scientists, they all agree.
Is there a relationship between p-hacking and Simpson paradox?
Vicente Rivera Not really, they are different things.
20
So what does hacked :p mean?
Is this the last episode? There hasn't been an upload of this series in 2 weeks...
Justyna Izabela
She says “I’ll see you next time”, which implies that there is either another episode or that she will present another show.
0:21 Groan. 🙄
She reminds me a bit of Terri Garr.
So much power with math. Hehehehe
phacking hell
Seems like p-values are obsolete
"Obsolete" implies there is an alternative that is better; if you know of a better mathematical tool, you might be eligible for a lot of money. lol
But that information useless in that it does not offer any details about which variables (jelly beans) are significant or even how many! By adjusting your p-values you've CHANGED your null hypothesis. There are better ways of doing this!
Haha a bowl of P's
oh no, is it over? =o
Jaii Raph
She says “I’ll see you next time”, which implies that there is either another episode or that she will present another show.
i heart your faith. Let us pray. But, in like a mathy way & that. w/graphs.
*thumbs up*
Are the peas placed on the laptop because the peas are hacking? (P-Hacking?)
> significant
lel
Statistics are always a good tool to lie.
Yello
Math T_T
What about it?
Aspiring Cloud Expert succks
i lovecars Search "vihart" in the RUclips search bar. It might change your mind.
Flaming Basketball Club why?
22nd comment!
Im early :)
Is calling it "hacking" supposed to make it sound cool? Did we need another neologism for this old concept?
First :D
Congrats