What is a p-value? (Updated and extended version)
HTML-код
- Опубликовано: 29 сен 2024
- An introduction to the concept of the p-value, in the context of one-sample Z tests for the population mean. Much of the underlying logic holds for other tests as well. I discuss what a p-value is, then using simulation I illustrate its distribution when the null hypothesis is true and when the null hypothesis is false. I then give a rough guideline of what different p-values mean in terms of evidence against the null hypothesis.
Thanks, I guess. Although I believe I bring some real value to the class in a lecture setting, I have no problem with students learning the material on their own. Which is one of the reasons why I supply all of these materials. They don't magically appear out of the ether though, and I hope that my creating them is one of the things that makes me a "good prof".
Interesting to watch as every episode gives some new perspectives from my textbook
You are undoubtedly the best teacher for statistics. I have not seen anyone else explain these concepts with the accuracy and simplicity with which you do.
I agree
It's not the easiest concept in the world. If you remember that it's a probability, and the smaller the p-value, the greater the evidence against the null hypothesis, that often gets you pretty far.
Thanks!
One of these days I'll put all my R scripts online, but I'm not going to post it here. (I'd have to dig it up, and make sure it's the right one, etc.) Cheers.
This is way faster than going to class.
You're a good prof, however.
Thank you very much! Best of luck on the test.
Very nice lectures, thank you! One question though: It is not clear why the p-value distribution for the case when H_0 is true is uniform. Why is it not skewed to the right? Higher p-values should be more likely in that case, no?
I agree that it's not obvious, but it's true notwithstanding. And I've never thought of a reasonable intuitive explanation. It's one of those (rare) things that I state without proof or justification and expect students to remember. It follows from the notion that if X is a continuous random variable then F(X)~U(0,1). The proof of *that* is pretty simple, but requires a little more background knowledge that I expect of my intro stats students or the general audience for this video. That proof is typically covered in a first course in mathematical statistics.
The p-value having a uniform distribution under H_0 isn't obvious, and it's not what most people would expect. But I think it's an important notion, if we are to make appropriate conclusions based on the p-value.
Not from me. (It's on the agenda, but it might take a little while before I get to it.) I'm sure there are some good ones out there, but I don't have any recommendations.
Hi!
Question! Getting a small p-value is great. It gives strong evidence against null hypothesis. But what about p-values that are close to 0.50 ( 50% ) or 0.975 ( 97.5 % ). Do they suggest only this:
- Poor evidence against null hypothesis
Or something else interesting can be inferred from these p-values too?
I have read thousands of books and articles on p-values, but you are the first person ever to give a novel and easy to understand definition of the p-value. This is really great, congratulations. Your definition should be used by all statistics teachers! Usualy they define p-value using the word "extreme" which causes huge confusion to almost all students!
Awesome videos !!!!! It's so easy to understand Sokal & Rohlf through your classes, please make some videos about regression and correlation, you sir are helping biologists from Bazil, THANKS !!
You're very welcome Gustavo! And thanks for the compliment! I'm glad to hear you find my videos helpful. I'm very glad I have the chance to offer a little help to people around the world. Cheers.
cant say how appreciation I have for this channel! Thank you JB!
You're welcome! I'm glad to be of help.
Came here from the John Hopkins Data Science specialization :)
your videos are highly understandable !!
thanks a lot !!
you, sir, are my hero.
+D'vIsis Creations I'm glad I could help!
+jbstatistics phew! now I understood. Previous video with same name was fast and furious!
Isnt the distribution of p values for mu=0 normaly distributed instead of uniformly distributed?
No. If: 1) We have a continuous test statistic, 2) the assumptions of the test are met, 3) the null hypothesis is true, then the p-value has a U(0,1) distribution. Note that the p-value is a probability, and thus could never truly have a normal distribution.
Excellent. I know for the purity of the concept, it is necessary to concentration on the "null hypothesis". But for consistency and balance, it is of value to state the "research hypothesis", that indicates the value of disproving the null hypothesis.
Thank you very much for your video, but I have two doubts. As per the tutotial distribution of that Z test statistic follows a normal distribution "IF THE NULL HYPOTHESIS IS TRUE"
Q.1.First of all how can we say that Z test statistic will follow a normal distribution?
Q.2. If we say Z test statistic follows normal distribution if null Hypothesis is true, In case we reject null hypothesis then our assumption to take Z tests statistic follows normal distribution itself is not correct(as it was only correct if null hypothesis is true). And we rejected null hypothesis taking a incorrect assumption. Isn't it?
Some how I am not able to swallow that, can you clear these doubts?
Thanks
Could u through an example explain the meaning the probability of getting the test statistic with even greater evidence against the Ho if the Null hypothesis is true?
Could u use two tail hypothesis testing example and another example tossing a coin multiple trials in series & various combinations? These examples would clarify beyond what u have already clarified so well. Would like to thank you.
Hello from SMU!
Hi from Guelph! Southern Methodist, St. Mary's, or somewhere else?
@@jbstatistics Haha am from Singapore Management University, thanks for your video its really insightful and helpful for my stats this semester. Anyways cheers to you, take care and hope u get back in releasing more sweet videos!
Hi professor, on the illustration of the 3 scenarios where you demonstrated the histogram of p-values with respect to true null-hypothesis being u=0, and false null-hypothesis being u=1 and u=2, since the scenarios with u=1 and u=2 would generate more samples with sample means > 0 (on the right hand side of std normal distribution of u=0), would it not result in more Z-scores of positive end of the distribution tail, hence p-values closer to 1 instead of 0 (as shown in the histograms of u=1 and 2)? How did you get the p-values that you used to plot the histograms ?
As far as "how did you get the p-values" goes, I: 1) Drew a sample from the distribution as given in the video, 2) Calculated the Z stat and p-value in the usual ways, 3) Did that a million times.
I used a *two-sided* alternative hypothesis, so the further the true value of mu is from the hypothesized mean, all else being equal the distribution of the p-value will be bunched closer to 0.
We would only get the situation you described (the p-value bunched more toward 1), if we are using a one-sided alternative ***and the true difference is in the other direction***.
@@jbstatistics Hello professor, I went back to your other video that explains p-value, indeed, I didn't grasp the idea until you explained again above. Thank you so much.
@@weisanpang755 I'm glad it worked out, and I'm glad to be of help!
Does the meaning of the p-value change depending on if your z-value turns out to be negative?
I had a hard time to get this in lecture...but this made it much easier...I appreciate it 😇🙌
Yes, you do provide value to the classroom itself.
Yes, providing the extra resources and putting in the extra time goes a LONG way towards becoming a good prof in my opinion. If you haven't received awards for teaching already, you certainly deserve to.
I also plan to become a professor (chem) and I will be following examples such as yours.
hello. I have downloaded this vid. thanks for your help.
is there any video related to joint prob distribution and marginal prob distributions?
If sampling distribution is known and so much normal as shown, then it means the sample means or sample proportions have accumulated around the population mean, then why we reject the null hypothesis? Caz of one sample evidence and probability of that happening? In that case, assuming a sampling distribution around mean of sample means/sample proportions nearing population mean itself would be wrong?
I'm not following what you're getting at here. Care to rephrase?
4:25 it is totally difficult to understand the justification behind assumptions on face value. instead please show how they were arrived in first place. your approach will only force viewers to memorize the conditions. please explain why uniform distribution. your videos are so popular next to khan academy and so many times I come here searching to understand such topics.
I'm not looking for people to memorize things, and I do my absolute best to illustrate the rationale and motivation behind the topics I discuss. However, showing that the distribution of the p-value under H_0 for continuous test statistics is U(0,1) is *far* beyond the scope of this video. There is simply no way to explain that neatly without getting into mathematical statistics. It's based on the notion that if X is a continuous random variable, and F(x) is its cumulative distribution function, then F(X)~U(0,1). Showing this mathematically is beyond the scope of the current discussion, and if I trying to show it would takes us very, very, very far away from the point of this video.
I was so confused about the concept of the p-value and this video was simply perfect and exactly what I needed to put everything together! Thank you so much for this video!
You are very welcome Veronika. And thanks very much for the compliment!
So if the p value is less than the alpha level 5%, that means "the probability of alternative hypothesis happening under the conditions that null is true "is pretty small, so we reject the null in favor of alternative, am I correct?
Not quite. In some ways you're thinking along the right lines, but your statement would need to be rephrased. It doesn't make sense to state "the probability of alternative hypothesis happening under the conditions that null is true" , since the alternative hypothesis is either true or false, and it's always false if the null is true.
A small p-value means that, if the null hypothesis were true, it would be very unlikely to get the observed value of the test statistic or something more extreme. If we have chosen to use an alpha level of 5%, then a p-value less than 0.05 would lead to rejection of the null in favour of the alternative.
This video was really helpful, thank you so much!
I often come to this channel due to the clear and concise explanations. Really appreciate you created this content.
Thanks for the compliment! I'm glad to be of help.
+jbstatistics what software do you use to generate 1mil samples?
To this point, all of the simulations in these videos have been done in R.
at 8:52 or even prior, it's claimed that if H_o true, p-value must be uniformly distributed. Not quite sure why is that and what's a simple proof of this? thx
It's possible that others have thought of a nice, intuitive explanation for that, but I haven't, and I've never seen one. As far as the proof goes, it's quite simple, *but* only if one already knows that, for any continuous random variable X, the distribution of the cumulative distribution function F(X) is U(0,1). The proof of *that* is very short and straightforward, but only if one knows the change of variable method in calculus. The fact that F(X) ~ U(0,1) tells us that, for example, if we were to repeatedly randomly sample from a normal distribution, and calculate the area to the left of those randomly sampled values, the areas would have the continuous uniform distribution on the 0-1 interval. That's true for any continuous random variable X. Proving that F(X) ~ U(0,1) in these spots is usually done in a first course in mathematical statistics. As I said above, it's a quick proof, but only if one has the proper tools available. Once we know that F(X) ~U(0,1), then showing the p-value has the same uniform distribution (under H_0) is quite simple. For example, first recognize that for a left tailed test, the random variable that is the p-value *is* the cumulative distribution function of the random variable that is the test statistic. I hope this helps a little!
@@jbstatistics ah right right. It comes straight from the property of continuous random variable. Thanks for pointing me to it, and your explanation helps. Thanks a lot!
I couldn't understand distribution plot for p-values. As per my understanding, p- values are given, then one can calculate the "strength" of evidence(statistics) and compare against the given p-value to reject or keep the hypothesis. I am not able to wrap my head about plotting a p-value distribution. p-value is a continuous random variable, having point probability of 0. How can we plot it using million iterations.
The p-value is based on the value of the test statistic, which is based on sample data. The test statistic is a random variable that takes on a value once the sample data is obtained. In repeated sampling, the test statistic would vary from sample to sample, and it thus has a probability distribution. Since the p-value is based on the value of the test statistic, the p-value is also a random variable that has a probability distribution. Once the sample is drawn, the test statistic can be calculated, and that value leads to the p-value. Small p-values give strong evidence against the null hypothesis.
Thanks for such a prompt reply. I searched for "calculating p-value from a test statistic" and found another great video by you. You are awesome. I have been watching your videos from past 5 days and the statistics talks at variuos pycons now appear oversimplified or silly. Thank you sir. I had no background in stats, but I feel like a stats graduate in just 5 days, having good understanding of confidence intervals, hypothesis testing, central limit theorem, type 1,2 errors and p-values. Though I know there is much more to learn .
I'm having trouble understanding why the p-value has a uniform distribution.
A p-value close to 1 means that our sample mean was close to the population mean.
A p-value close to 0 means that our sample mean was very far from the population mean.
The sampling distribution of sample means is normal. The standardized version of the sampling distribution is also normal. Then, how is the distribution of p-values not normal?
First, the p-value is a probability, and so its possible values are bounded by 0 and 1 (and thus it cannot be truly normally distributed). Also, "population mean" should be replaced by "hypothesized mean" in lines 2 and 3 of your comment. The fact that the p-value has a uniform distribution under H_0 is not trivial to show. It is related to the fact that the distribution of the CDF of any continuous probability distribution is Unif(0,1). That notion is typically covered in a first course in mathematical statistics.
Thank you, sir! :)
Fascinating results in the second half of the video. Thanks so much!
You are very welcome!
Great video! Thank you so much!
You are very welcome!
Salute to your channel and video!!
Thanks!
Very useful, thanks!
neither do I
Thanks for plotting the histogram of p-values : )
Maybe it's helpful if you prove that:
Suppose `X` is an RV and `F` is CDF of `X` and `Y` is defined as
Y = F(X),
then `Y` ~ U(0, 1).
I don't think a proof of that is appropriate for this video. While it's a fairly simple proof if one comfortable with the change of variable technique in calculus, some still find it conceptually a little tricky. And it's far outside of what I was trying to achieve here -- an explanation that is appropriate for an applied introductory statistics course. I think proving that the CDF for any continuous r.v. is U(0,1) would be far more distracting than it's worth, given the nature of the video.
@@jbstatistics You're right : )
@@jbstatistics i look for internet of the proof , i did not find it , i find only simplist one , i will be appreciated if you send it to me please .The hardest one ☺️
@@haithamabdelrahman6276 There are some different approaches here: math.stackexchange.com/questions/868400/showing-that-y-has-a-uniform-distribution-if-y-fx-where-f-is-the-cdf-of-contin
@@jbstatistics
Thanks , prof , i already see this proof on stack exchange before you recommend it , my reason to look for this proof because i was so desperately looking for the values for p value min an max on the uniform distribution , p is always [0,1] , and as i remember the p should be greater than alpha in non rejection region, or we fail to reject null hypothesis when p greater than our significant level , so p should be between (alpha,1]
From other side of we are sampling p in the rejection region p shoud be [0,alpha)
Is that logic or there is something i missed !
It is not true that the smaller the p-value the greater the evidence against the null-hypothesis. See for example the wikipedia page on p-values.
In the absence of other information, and for smallish p-values, it it true. Sure, we can dream up scenarios such as:
Two studies, A and B. Exact same info going in, exact same study design, exact same analysis. For study A we find a p-value of 0.71 and for B we find a p-value of 0.55. For both of those there is absolutely no evidence against the null hypothesis whatsoever.
Two studies, A and B. Suppose we *know* going in that the null hypothesis in A is false, whereas in B we know it's true. For study A we find a p-value of 0.12, whereas in B we find a p-value of 0.02. One could argue that neither one provide any evidence whatsoever regarding the veracity of the null hypothesis, since we know the underlying reality to begin with.
But for any given study, all other information being equal, and smallish p-values (such that there is *some* evidence against the null), the statement holds. A p-value of 0.0007 is greater evidence against the null hypothesis than a p-value of 0.03. I think that's the important bit and it's an important point for learners of introductory statistics to grasp. Statements such as yours cause much more confusion than they are worth.
@@jbstatistics But it has some educational value to point out one of the biggest misconceptions about p-values right? "For typical analysis, using the standard α = 0.05 cutoff, the null hypothesis is rejected when p ≤ .05 and not rejected when p > .05. The p-value does not, in itself, support reasoning about the probabilities of hypotheses but is only a tool for deciding whether to reject the null hypothesis."
In other words it is just a tool to calculate whether the test statistic fell in the critical region given the significance level. By the way your videos are great! Just wanted to point this out for some discussion.
@@TheOskro I vehemently disagree that it's one of the biggest misconceptions about p-values, and I think pointing it out has net negative pedagogical value for the vast majority. But I'm happy to have the discussion. And thanks for the compliment.
Your quote involves a hypothesis test at a fixed alpha level, where we're simply "rejecting" the null hypothesis if the p-value is less than or equal to the chosen significance level. Sure, there we reach the exact same conclusion for a p-value of 0.0499 as for 10^(-80), but that speaks more to the silliness of carrying out tests at a fixed alpha level (especially blindly choosing 0.05) than to misconceptions about p-values. In the *vast* majority of practical applications, we're not actually making a decision between the null and the alternative hypothesis, we're simply assessing the evidence. If no decision is being made based on this single test, and we're simply assessing the evidence, then I strongly feel that reporting the p-value and not forcing a significance level on the test is the way to go. That's far from a universal belief, but I'm definitely not the only one.
It reminds me a little of those who make a big deal of stating something like "We can't say that the *probability* the parameter lies in the 95% confidence interval is 0.95, since the parameter either lies in the interval or it does not. So the probability the parameter lies in the interval is either 0 or 1, but we can be 95% confident that it's 1." I personally feel that complicates the issue in a very silly way, and that it has net negative pedagogical value, but they are technically correct from a certain viewpoint.
And I'd argue that even people who say they agree with your quote think that a p-value of 10^(-80) provides a hell of a lot more evidence against the null hypothesis than a p-value of 0.02, even if they stated up front that they were going to carry out the test at a significance level of 0.05.
@@jbstatistics "In the vast majority of practical applications, we're not actually making a decision between the null and the alternative hypothesis, we're simply assessing the evidence." Ah I think we both have a different definition for the p-value in mind then. I am used to the definition $p = \inf\{\alpha \in [0,1]: t \in C_{\alpha}\}$ where $C_{\alpha}$ is the critical region and in that case the p-value is defined by the hypothesis test. What will be the definition in terms of "assessing the evidence".
@@TheOskro Smallest significance level for which we could still reject the null. Sure. What has more evidence against the null, a situation where the smallest significance level at which we could reject the null is one in a quintillion, or one where the smallest significance level at which we could reject the null is one in twenty?
Thanks...
You are very welcome!