MINI LESSON 7: P-Values and P-Value Hacking: a simplified lecture.
HTML-код
- Опубликовано: 8 фев 2025
- We saw that 1) many metrics are stochastic, 2) what is stochastic can be hacked. This is the simplification of my work showing that "p-values are not p-values", i.e. highly sample dependent, with a skewed distribution. For instance for a "true" P value of .11, 53% of observations will show less than .05. This allows for hacking: in a few trials a researcher can get a fake p-value of .01.
Paper is here and in Chapter 19 of SCOFT (Statistical Conseq of Fat Tails):
arxiv.org/pdf/...
This might be my favourite probability mooc because it starts with "if you don't know p-value means, forget about it"
Was not aware of electronic version provided free. Well done. I feel better now about listening to Fooled By Randomness on RUclips. Since then, I have purchased all 5 of Incerto. Listen daily. Thanks for teaching!
The content we do not deserve... thank you for helping many of us stand on the shoulders of giants. I'm certain you bring much relief to those few who scar themselves for having their skin in the game.
In medicine, repeat the same clinical trial a sufficient number of times, then only publish the positive studies (p
At around 9.00 Taleb says 'in medicine N tends to be large'. Is he implying that in medicine or biostatistics it is ok to use p-value because they use a large sample (N) ?
@@RostamiMehdi Thanks for your response. I spoke to few of my friends in the biopharma sector. They still heavily use p-values. While they kind of understand the drawbacks of it, they state that they give emphasis on 'effect size' and thus justify the usage of p-values.
@@venkatraman42 Due to the issues taleb mentions, you need really low p-values to get meaningful results. If you have very large n, it is possible to get these results. In this case use of p-value is ok, but not recommended.
@@venkatraman42They should!!
Another perfectly executed debunk of an overly used statistical tool. Very well done.
Maestro, your series of videos have changed the way I approach learning basic statistics. 🙏
I'm loving these mini lectures, such powerful statements made from the basics. I started them not too long ago and the piqued my interested to grab a couple of your books. Making my way through my first one Fooled by Randomness. It's been a while since I've enjoyed a book this much.
Been waiting for this. Always love it when Prof Taleb uploads
Hi Taleb! Awesome exposition! 😁👍 My intuitive take away is any sort of inference, even from a large data set, if expressed as a parameter must necessarily be itself stochastic and needs to be interpreted with caution!!! 😁👍
For anyone interested to see a simulation, the following setting in R gives you the P-hacking dilemma when the real p-value is 0.11.
bias
Thank you, exactly what I was looking for!
The great teacher of our time. Thank you!
The vast majority of biosciences like them too, it's a huge problem. Thanks for the lectures and the books.
Merci beaucoup pour les explications Monsieur Taleb, et pour le lien.
Thankful as always for calling out BS!
Sir, it is not just psychologists who abuse this notion. A former employer (with a PhD in statistics) would choose when to use bonferoni Holm adjustment based on the likelihood our client would stop our funding based on the results of our spurious regressions. Data scientists in the wild don't like to think about randomness
"Probability... is the acceptance of the lack of certainty in our knowledge and the development of methods for dealing with our ignorance. Outside of textbooks and casinos, probability almost never presents itself as a mathematical problem" (Taleb, 2004, p. x). That is one of my favorite part of his book, Fooled by Randomness.
il miglior professore del mondo. Grazie Signore
Friend ! I finished fooled by randomness last night, good writing style and valuable lessons, I will read the other ones. I'm playing around with a small equity options portfolio for hedging and speculation purposes, long volatility, for fun (and hopefully profits), it's really nice to get your insight.
thank you so much. because of you I became a professional stock trader
Please can you tell us about an alternative metric that can allow us to make confident decisions on whether or not to adopt a hypothesis?
Thanks so much for this series and for making the book digitally free!
Ευχαριστούμε Δάσκαλε!
Every lecture you put out makes me more an more interested in probability/statistics.
(An appreciation I unfortunately did not develop as much when in school / university)
Anyone looking it up in the book, it is actually page 349.
Felt good recognising the Jorge L Borges collection in the background
Based and redpilled, Mr. Taleb.
Nassim, would be huge if you could run us through some of your favorite books from that shelf behind you in a short video
These videos are great! Thank you sir!
When we are making inference, we are interested in estimating a population characteristic (the so-called "estimand"), e.g. the mean height in a population of humans. For this population there is no "true p-value", just the true mean height, which we could measure if we just measured all heights of all individuals and averaged. In this video NNT considers a single experiment (measure height of a sample of n individuals of this population), and calculates a p-value against a null hypothesis (in the video the null was mean of x equal 0). Then, he creates an imaginary infinite population of identically performed experiments, which would be created by drawing samples of equal size n from the population of humans. For each of these infinite experiments he calculates the p-value, to obtain the distribution of p-values and "mean p-value". Thus, this concept of "mean p-value" corresponds to a population of iid studies, and NOT to the population of humans we started with. Thus, I found the whole concept of this video confusing, because the estimand that NNT seems to be focused on (mean p-value in the infinite imaginary experiments of equal size) does not refer to the population of interest (the indiviual humans). Having said that, I agree that p-values are a very weird and unintuitive tool for making inference, and in most cases we already know that the null hypothesis does not exactly hold (i.e. mean x cannot be EXACTLY zero). Thus, dichotomizing p with 0.05 or 0.005 or 0.00005 in many cases is answering a question we already know the answer to (i.e. the null does not hold).
Hi, you forget that by definition statistics is making *general* claims from individual observations, not descriptive facts about samples. And all stochastic metrics (correlation, mean, variance) are supposed to come with a distribution. It is just that p-val has a distribution *but* statisticians did not know it.
I just wanted say thanks! I was quite confused by the video and this comment has made it clear. Not the Taleb's point (still very relevant) is much clearer :).
Ευχαριστούμε Μαέστρο!
I love this before watching it
Thank you very much! I learn so much and you put into words things about psychology that I intuitively know. Do you think there's any hope for the field of Psychology?
good illustration of p-hacking, thanx!
Thank you kindly ✍️
So it sounds like the main gripe here that Nassim has is that the sample sizes typically used are too small to gain any meaningful insight. So P-values can be used but look for much smaller values to confirm statistical significance?
Abraços do Brasil !
Thanks Nassim
Good, Lord. After listening to this, I feel as if someone punched me in the stomach...
I wish I understood this, but since you give me permission to move on right at the beginning of the video I'm just going to do that. Thanks!
Thanks.
Holy moly, you are amazing. Sorry for that one comment I made on Twitter 3-years ago in your reply box. (It was my only comment to you, but it was also very uninformed, not insulting to anyone, but just very uninformed, LOL.)
What would be a large n?
can anyone explain what he means by the "n being taken out"?
number of observations in the sample
What are the prerequisites for someone right out of high school to undertake your technical book on fat tails?
A few stat courses. Or just STAT 101 plus self-education.
@@nntalebproba Thanks, will start today! Any recommendations on good Moocs?
@@nntalebproba so my student is doing her undergraduate project on the idea of fat tails, based on your book. And we're loving it!
When will you give lecture on twitter spaces or even youtube live? I think it would be really great
Dr. Taleb how can I get you to sign your book during for me during these times ? I can’t travel to NY.
Molto interessante grazie assai
So is the moral of the story that p-value is garbage? Am I understanding this correctly?
So may be we can propose empirically what n should be for each confidence interval. People understand rules.
What people actually need is good judgement. There is no one size fits all set of rules unfortunately.
@@DanielJanzon it s way tougher changing (yet improving) people judgment faculty than just setting a minimum ratio that should start being acceptable/sound :)
You should start a tiktok
The p value itself is stochastic!
Uncle Nas black board please!
Do you think the Bayesian approach to P values is of better use?
p values suck, confidence intervals for the win!
They myth of statistical significance
Fair point, I'm missing the alternatives though..
Nothing is better than BS.
perhaps there isn't. If there isn't, it does not give us an excuse to continue using it incorrectly.
If I understand one of his frustrations correctly, it is that people look for alternatives. If there is no alternative people will continue to use something that they should not be using because they don't have an alternative. When really, they should not be using it at all (even with the absence of an alternative).
What does BS stand for? (Foreign student)
@@gerardosandoval5370 or as proff Taleb noted, use it with a very strict criterion like 0.001
@@scrrrdamn8210 BULLSHIT