Corrections: 3:23 I should have said "To understand why dividing by n underestimates the variation around the population mean". 3:40 The estimated mean was switched with the population mean. Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/
At around 8:35, you should've used asterisk '*' character instead of 'x' character for multiplication. I was a bit confused and thought you wrote 2*(x-v)*x-1 instead of 2*(x-v)*(-1). Great video by the way!
I think we're encouraged to purchase a double dam t-shirt or sweatshirt, which is more of a financial incentive than an award but who doesn't like getting paid to be awesome? I'll probably pick one up this weekend
Amazing Josh!! I can't imagine how much hard work goes into simplifying the complex statistics concepts and coming up with these amazing videos. And on top of that your ingenious ideas of adding humor and musical creativity, taking the content to another level. If there was an Oscar for tutoring you'd be the undisputed winner. BAMMM !!- simply the best educator on RUclips....
Came to this video for "Why Dividing By N Underestimates the Variance" but got to know why absolute values are not used in Variance calculation. Literally cried, Prof. Josh.. Kudos to you. You are supporting me to understand the topics in statistics. I will support you regularly after I get a job soon. And I'm sure your teachings are required for many of the upcoming students in the coming decades. In India we have a concept called "Guru Kulam", and I see you as my guru (Not the term commonly known in the western world, this is more about respect)
I searched for this on a number of online resources, some mentioned "n" while others "n-1", leaving me confused. This is the best possible explanation to the problem you made it really easy for us to understand. Thanks a lot !!! Bammmm subscribed and shared with friends.
Josh, This is a total hypnotism you did with BAMs, echos, and other sounds. You mastered the art of making us stick there. I been searching for statistics and machine learning videos where they have kind of a roadmap, and simple explanations for complex topics, and this is it. You saved my life for sure, my donation is on its way, I know anything is small for what goes into making these. Hats off to you, you are a LEGEND. We owe you...
I have nothing but admiration; this is the clearest explanation that I've seen so far that does not shy away from the underlying math, yet still keeping it understandable for those with minimal math background. I feel like a bit of a fool when I see the contrast between my own attempts to explain this correction factor and your explanation.
the best accessible explanation I can find in the whole internet for this mystery. then just as I was about to say "aha! you missed out something!" towards the end of the video, you seemed to have read my mind and "p.s. if you are wondering why n-1 and not 0.5 or 2 .... " you are so so spot-on!
@@statquest I agree - best explanation i have found and i'm sharing this video with all my students. THANK YOU! So.... any chance that next video is coming out soon? (or has come out already?)
I rather get a clear and understanding explanation with "BAMS" like i'm five, than a 50 pages long explanation with words like "trivial" and abbreviations (q.e.d) and just feel depressed and left clueless. And an other very important thing: Only if you *really* understood the topic, you can explain it with easy words. Very well done, Josh! Thank you very much!
Yet another video from this channel that leaves me speechless. I've never really understood this concept until I've watched your video. Thank you very much, again.
came from calculating the mean, variance and SD video. Did not expect a proof for why variance = x-bar. This is a really good in depth video i've ever watched for statistics. Thank you very much.
Bam!!! I've watched lots of your videos after I discovered the one explaining the standard error. You make me understand stats concepts more clearly. Please continue making these awesome videos (machine learning too)! 5 dollars donated!
Big thanks from Taiwan. I have been asking why not dividing by n since high school...but all I get from my teacher was only "a rule of thumb". Now I know the reason behind and thanks to statquest. BAM!!
Unfortunately, it will be a while before I get to it. I've got covariance and correlation coming up next, then a few machine learning videos, but then I'll loop back to expected values. It's a topic that I've wanted to work on for quite some time.
My uneducated guess about n-x is that the bigger the magnitude diff. between the population and your sample size, the larger x would be. Because as this magnitude get smaller and smaller, the need for x to have any significant value, disappears. My biggest question is why would x lead to this unitary value when your sample size is little. But we'll see what Josh explains about that.
Intuitively: the number you're dividing stands for the degrees of freedom you have. In other words: how many data points are allowed to vary freely. The reason that this is 1 less here is, as the video hinted at, because of the sample mean. If someone shared with you n-1 data points of their sample distribution of n points, and you know what the sample mean is, then you can easily calculate what the last data point is. I.e. that last data point doesn't have any freedom to vary, just because it was crucial in defining the sample mean. This doesn't matter if you know what the population mean is, precisely because the sample distribution didn't decide its value. Therefore all n values in a sample distribution with known population mean can be used to make an unbiased estimator, while only n-1 degrees of freedom can be used to have an unbiased estimator when all you known is the sample mean. Mathematically: en.m.wikipedia.org/wiki/Bias_of_an_estimator The first example (in the examples tab) shows why it should be n-1, and not n, or n-whatever.
I click the like button before I watch it because I'm always sure I'll love it! Thanks so much for making this series. You'll never know how helpful it has been in my life
Thank you for this explanation. When I was learning stat in university, I did not understand well, why we divide by (n-1) instead of n to estimate sample variance. You explained it so clearly in a way that I will never forget what I learnt. Thank you Josh !!!
Amazing explanation of why we use the square of the errors instead of the absolute value! I always asked myself that and all the teachers said it was just to give a bigger weight to the errors! We need the statquest on expected value!
I just read wiki and found that even divided by n-1, we still underestimate the standard deviation (although we don't underestimate the variance anymore). I feel that's somewhat mind-blowing, since calculating sample std is such an ordinary job for statisticians, and it is surprisingly BIASED (and I am sure the standard error formula is also biased)...
I don't even remember what I was confused about in particular, but I remember feeling very happy to see this video. Will revisit this in the following days. Psst, you're a gem ;)
Man, I always thought that statistics doesn’t make any sense at all and that people should just blindly chug into weird formulas without questioning, but this was absolutely mind opening. Not even khan academy could explain the proof!
Josh, thank you so so so much for this amazing video!!!! It's really really helpful and is certainly a moment of enlightenment for me. But please please pleaseeeeeee, make the video about why specifically divide by one 😢😢😢, pleaseeeee
BAM! I have not seen this concept explained better anywhere else ever. Have you gotten around to making the follow-up video on 'expected values' ? Can't thank you enough for your channel
I've got the video on expected values ruclips.net/video/KLs_7b7SKi4/видео.html and ruclips.net/video/OSPr6G6Ka-U/видео.html , but there are still a few steps to go after that... :(
im currently learning data analytics and trying to figure out ab testing and bam! here i am! thank you so much for making statistics fun and easy to understand! double bam!
Thanks for explanation! I understand that differences between the SAMPLE data and the sample mean are smaller than the differences between the SAMPLE data and the population mean. BUT! We are not interested in the difference between the SAMPLE data and the population mean, rather we are looking for the difference between the TRUE POPULATION data and the population mean (the population variance). And it's not clear why this value would be larger. I mean sample data is centered around sample mean the same way population data is centered around population mean. Comparing sample data with population mean feels to be misleading.
The best estimate we can do is the estimate of the variance around the sample mean, which is probably an underestimate, but not always. So this is the best we can do.
I have been SO stressed out about a project I'm working on, and 3:15 made me laugh so hard!!! I didn't even realize how stressed out I was until I caught myself laughing for the first time in weeks. Thank you Josh!!! **sob**
I really want to understand why we use n-1 instead of substituting any other number instead of 1. I'm guessing it has something to do with the way we approximate the mean and the variance. I think it's related to properties the normal distribution has and such. I think that to truly understand that analytically I'd have to integrate over all possible outcomes while taking into account all the probabilities and then calculating the average. It really excites me, but I don't know where I can find the information needed to understand the subject in more depth. Can you give me some advice on what textbooks I should read, please? I'd really really appreciate that!
Waited the whole video to know why it was n-1 and not n-2 etc... "that mistery will be resolved in the next episode" Felt like watching an overstretched TV series in some way haha. I understand it's capital to show first that sample variance underestimate true variance, but could mention earlier that you ll not focus on "why" it is n-1. :p Thank you though wonderful content!
I tried to be careful with the title of this video with "Why dividing by N underestimates the variance" instead of "Why n-1 gives us an unbiased estimate". That being said, I really wanted to explain exactly why n-1 works, but the proof is relatively advanced.
Great video! We clearly see that estimated variation is smaller than desired so we have to make it bigger. We can make it by dividing by n-1, but also by n-2 or n-1.5 or n-100. Why n-1?
Loved the video. But didn't understand something clearly. The variance is the least around the calculated mean. But that is only when the data x remains the same right? How can you compare it with the population variance which has a lot more data points and the summation is therefore different?
We are not comparing it to the population variance. We are simply comparing the variance of the data calculated around the sample mean compared to the variance of the data calculated around the population mean.
I do understand the way yo calculate min variance But @ 14:54 How did you conclude in that "Thus valus around sample mean is always less than population mean" ?
Although you make everything look so simple, your teaching pedagogy requires a lot of hardwork( to make the slides particulalry). I hope that every teacher puts in the same kind of hardwork and assume there students to be in 5th grade that way every class will be a pleasureable experience of life.
it is because some teachers don't know how to teach. They learn from textbook's concept. Memorize them, then give those back to students. I am not being rude but it is the reality. In order to be able to explain well to new learners, teachers must be able to understand the concepts well. Teaching is a hard skill to master. Nowadays, lot of taught concepts are assumed true or left blank during teaching . That's why if those students become teachers, they won't be able to explain.
@@statquest Thanks a lot for responding! ... and sorry, as I noticed after reading more comments, that you had already answered this question many times. Quest on!
I thought that the sample variance [ (sigma(Xi - X bar)^2)/n ] is trying to estimate population variance [ (sigma(X - mew)^2)/N ], so at 16:04, why do we care about how the sample standard deviation estimates the variation in the *Data* around the population mean, as opposed to how it estimates the actual population variance (which is what it’s set out to estimate)?
Hi Josh, where did you study about it, is it from Bessel's correction or Karl Pearson. I am interested to ready a bit about the history behind it. Can you please suggest a book or paper where the original discovery was made. Thanks in advance.
@@shubhamtalks9718 The concepts seem complicated because people that do not really understand them try to teach them. How did I learn them? Years of really hard work. I read everything I can about a subject, then I re-read it. Then I re-read it again. Then I make a program based on my ideas and see what happens. Then I re-read everything over again. And sooner or later I figure it out. But it takes a lot of time and a lot of work. Sometimes I worry I will not succeed, and sometimes I fail, but I keep trying anyway.
This intuitively makes more sense to me now. If I take a sample, the sample mean may end up being larger or smaller than the population mean. But the sample variance can never be larger than the population variance, it might be equal to it, but most probably it will be smaller.
I think you have to be a little bit careful with what you mean by "sample variance" and "population variance". As long as you're comparing an estimated population variance using the sample data and actual population mean vs an estimated population variance using the sample data and sample mean. But, comparing the estimated population variance using the sample data and sample mean vs the actual population variance (all data and actual population mean) doesn't have the guarantee that sample variance will be lower than population variance.
Sorry I am still getting confused. At 5:29, in the inequality, both left hand side and right hand side are using the same n, which is the number of samples. You argued that the right hand side is greater so that we need to make the left hand side larger by dividing n-1. However, the right hand side is not the actual population variance. The actual population variance should be using a much larger n to calculate. What we are doing here is to estimate the population variance but not the right hand side. Thinking to this point, all the linkage seems broken. How can I relate the right hand side to the population variance? It is true that the inequality holds. But it does not mean also the population variance is always greater than the left hand side. Thanks for your videos. They inspire me and teach me a lot.
Sometimes we know the population mean, but don't know the variance, so we sill have to estimate it. That is what is going on on the right side of the equation.
The end of this video suggests that the video on Expected Values will elaborate on why subtracting 1 is not arbitrary. The Expected Values video says that it is the first step towards getting to this video. Does anyone have the appropriate chronological order of the videos? Or if I'm stuck in a time loop, let me know!
The expected value is a step in the direction of understanding why subtracting 1 is not arbitrary, but it doesn't get you all the way. Unfortunately I haven't made videos for the rest of the steps yet. If you're in a hurry, check out: online.stat.psu.edu/stat415/lesson/1/1.3
I think there is little problem here @8:47 where we use chain rule .......where that x-1 come I know -1 is derivative of -v but why there is x and where it goes later. By the way nice explanation .
Super content in the video. But i have a doubt. Its been mentioned that the sample mean will always be less than the Population mean (14:58). But i think that this is a completely relative term. What if the sampled data has more quantity of samples which have higher value than the population mean? I think the sample mean will be greater than the population mean. Kindly correct me if wrong.
I believe you are misunderstanding what the video is saying. The equation shows the sample variance is always less than the population variance and the text says that that relation will always be true unless the sample mean is exactly the same as the population mean.
Corrections:
3:23 I should have said "To understand why dividing by n underestimates the variation around the population mean".
3:40 The estimated mean was switched with the population mean.
Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/
BAM BUM hahah
Please Give Video on degrees of freedom please🙇
At around 8:35, you should've used asterisk '*' character instead of 'x' character for multiplication. I was a bit confused and thought you wrote 2*(x-v)*x-1 instead of 2*(x-v)*(-1). Great video by the way!
@@m3c4nyku43 noted
is there some sort of award we can give this guy? please?!
:)
I think we're encouraged to purchase a double dam t-shirt or sweatshirt, which is more of a financial incentive than an award but who doesn't like getting paid to be awesome? I'll probably pick one up this weekend
@@jacobmoore8734 Thanks! :)
@@jacobmoore8734 award + t-shirt = double bam! just ordered my own shirt. gonna wear it to my statistics test in 2 weeks
Really, your way is too unique. one of the best
Amazing Josh!!
I can't imagine how much hard work goes into simplifying the complex statistics concepts and coming up with these amazing videos. And on top of that your ingenious ideas of adding humor and musical creativity, taking the content to another level.
If there was an Oscar for tutoring you'd be the undisputed winner.
BAMMM !!- simply the best educator on RUclips....
BAM! :)
Came to this video for "Why Dividing By N Underestimates the Variance" but got to know why absolute values are not used in Variance calculation. Literally cried, Prof. Josh.. Kudos to you. You are supporting me to understand the topics in statistics. I will support you regularly after I get a job soon. And I'm sure your teachings are required for many of the upcoming students in the coming decades. In India we have a concept called "Guru Kulam", and I see you as my guru (Not the term commonly known in the western world, this is more about respect)
Thank you so much!!! It means a lot to me.
I searched for this on a number of online resources, some mentioned "n" while others "n-1", leaving me confused. This is the best possible explanation to the problem you made it really easy for us to understand. Thanks a lot !!! Bammmm subscribed and shared with friends.
Awesome!!! Thank you very much for subscribing and sharing my videos with your friends. :)
Josh, This is a total hypnotism you did with BAMs, echos, and other sounds. You mastered the art of making us stick there. I been searching for statistics and machine learning videos where they have kind of a roadmap, and simple explanations for complex topics, and this is it. You saved my life for sure, my donation is on its way, I know anything is small for what goes into making these. Hats off to you, you are a LEGEND. We owe you...
Wow, thank you!
I have nothing but admiration; this is the clearest explanation that I've seen so far that does not shy away from the underlying math, yet still keeping it understandable for those with minimal math background. I feel like a bit of a fool when I see the contrast between my own attempts to explain this correction factor and your explanation.
I'm glad you like the video so much. Thanks! :)
I've been 2 years asking how to plot variance, why sample variance (also sd) divided by n-1. And this is best explanation i ever had
Awesome! :)
the best accessible explanation I can find in the whole internet for this mystery. then just as I was about to say "aha! you missed out something!" towards the end of the video, you seemed to have read my mind and "p.s. if you are wondering why n-1 and not 0.5 or 2 .... " you are so so spot-on!
Thank you very much! :)
@@statquest I agree - best explanation i have found and i'm sharing this video with all my students. THANK YOU! So.... any chance that next video is coming out soon? (or has come out already?)
@@naysannaderi5135 I hope the next video will come out soon. Possibly in the next 4 months or so. I hope!
I rather get a clear and understanding explanation with "BAMS" like i'm five, than a 50 pages long explanation with words like "trivial" and abbreviations (q.e.d) and just feel depressed and left clueless. And an other very important thing: Only if you *really* understood the topic, you can explain it with easy words. Very well done, Josh! Thank you very much!
Thank you very much!!!! :)
Yet another video from this channel that leaves me speechless. I've never really understood this concept until I've watched your video. Thank you very much, again.
Wow, thank you!
came from calculating the mean, variance and SD video. Did not expect a proof for why variance = x-bar. This is a really good in depth video i've ever watched for statistics. Thank you very much.
bam!
Bam!!! I've watched lots of your videos after I discovered the one explaining the standard error. You make me understand stats concepts more clearly. Please continue making these awesome videos (machine learning too)!
5 dollars donated!
Thank you very, very much. I really appreciate it. :)
Thank you for this!! The first time I saw the formula for the sample variance I wondered why the n-1 was there, this is a great explanation.
Thanks!
hello, keej
i hate u mate
Big thanks from Taiwan. I have been asking why not dividing by n since high school...but all I get from my teacher was only "a rule of thumb". Now I know the reason behind and thanks to statquest. BAM!!
Hooray! :)
Yes! I had already feared that the n-? question won't be explained. Glad to hear that you will explain this unsolved mystery in the next video!
Unfortunately, it will be a while before I get to it. I've got covariance and correlation coming up next, then a few machine learning videos, but then I'll loop back to expected values. It's a topic that I've wanted to work on for quite some time.
My uneducated guess about n-x is that the bigger the magnitude diff. between the population and your sample size, the larger x would be. Because as this magnitude get smaller and smaller, the need for x to have any significant value, disappears. My biggest question is why would x lead to this unitary value when your sample size is little. But we'll see what Josh explains about that.
Intuitively: the number you're dividing stands for the degrees of freedom you have. In other words: how many data points are allowed to vary freely. The reason that this is 1 less here is, as the video hinted at, because of the sample mean. If someone shared with you n-1 data points of their sample distribution of n points, and you know what the sample mean is, then you can easily calculate what the last data point is. I.e. that last data point doesn't have any freedom to vary, just because it was crucial in defining the sample mean. This doesn't matter if you know what the population mean is, precisely because the sample distribution didn't decide its value. Therefore all n values in a sample distribution with known population mean can be used to make an unbiased estimator, while only n-1 degrees of freedom can be used to have an unbiased estimator when all you known is the sample mean.
Mathematically: en.m.wikipedia.org/wiki/Bias_of_an_estimator
The first example (in the examples tab) shows why it should be n-1, and not n, or n-whatever.
Thank you, 10 years of confusion made clear by this 15 mins of video.
Hooray! I'm glad the video was helpful. :)
I click the like button before I watch it because I'm always sure I'll love it! Thanks so much for making this series. You'll never know how helpful it has been in my life
Hooray!!! Thank you very much! :)
Thank you for this explanation. When I was learning stat in university, I did not understand well, why we divide by (n-1) instead of n to estimate sample variance. You explained it so clearly in a way that I will never forget what I learnt. Thank you Josh !!!
Hooray! :)
Michael Scott: Why don't you explain this to me like I'm five?
Josh Starmer: Bammm!!
and understood ...
thank you : ) !
BAM! :)
@@statquest
Can you provide the slides for all the statistics videos you used to explain the concepts
@@Deepak-uv8du I have PDF study guides for some of my videos here: statquest.org/studyguides/
Amazing explanation of why we use the square of the errors instead of the absolute value! I always asked myself that and all the teachers said it was just to give a bigger weight to the errors! We need the statquest on expected value!
Thanks! I'm working on the expected value, but it still might be a few months before it's ready.
I think for even n there wouldn't even be a minimum point, rather a flat line between the 2 middle samples
This video is such a gem! Thanks for explaining the root of this concept which is not easy to find even in statistics books.
Glad it was helpful!
I have always hated statistics but I just today found this channel and this guy explains everything elegantly! ❤😊
Wow, thank you!
I just read wiki and found that even divided by n-1, we still underestimate the standard deviation (although we don't underestimate the variance anymore). I feel that's somewhat mind-blowing, since calculating sample std is such an ordinary job for statisticians, and it is surprisingly BIASED (and I am sure the standard error formula is also biased)...
interesting
@@statquest Yeah. This is the wiki page.
en.wikipedia.org/wiki/Unbiased_estimation_of_standard_deviation
Kids today are so lucky they can review their stats online like this with great teachers.
:)
Awesome and the best video with most simplified explaination.
Thank you! :)
Statquest, JBstatistics and Khan Academy.....You guys are just amazing !!.....Thank you for all you have done for us
Thank you! :)
Thank you, I haven't known about these channels
Thank you so much for the clear and simple explanation. This is an example for when showing the proof is better than only trying to give an intuition.
Thanks
This is epic, never got a better or clearer explanation for this particular problem. Hats off!🙌
Thanks a ton!
Aah! Finally end. What a excellent work by you!! Statquest rocks ❤.. Thank you sir. You helped a lot in my carrier ❤.
Thanks!
I don't even remember what I was confused about in particular, but I remember feeling very happy to see this video. Will revisit this in the following days. Psst, you're a gem ;)
Thank you very much! :)
This is the best explanation that I've come across for this. And I really liked that you gave a proof for general set of observations. Thanks a lot.
Awesome, thank you!
Man, I always thought that statistics doesn’t make any sense at all and that people should just blindly chug into weird formulas without questioning, but this was absolutely mind opening. Not even khan academy could explain the proof!
Thanks!
This channel is just incredible, well done!
Thank you very much! :)
Josh, thank you so so so much for this amazing video!!!! It's really really helpful and is certainly a moment of enlightenment for me. But please please pleaseeeeeee, make the video about why specifically divide by one 😢😢😢, pleaseeeee
One day I'll do it! I promise!
BAM! I have not seen this concept explained better anywhere else ever. Have you gotten around to making the follow-up video on 'expected values' ? Can't thank you enough for your channel
I've got the video on expected values ruclips.net/video/KLs_7b7SKi4/видео.html and ruclips.net/video/OSPr6G6Ka-U/видео.html , but there are still a few steps to go after that... :(
The last point about absolute value explains a lot! I was always wondering why squaring data is so much more common than taking absolute values!
bam! :)
This is excellent, I am looking forward to the next one.
im currently learning data analytics and trying to figure out ab testing and bam! here i am! thank you so much for making statistics fun and easy to understand! double bam!
Happy to help!
The most impressive explanation I've ever seen.
Thanks!
I admire this explanation... Amazing. I really look forward to the expected values video!
Thank you. I started working on the expected value video, but it will still be awhile before I finish since I have many other projects to work on.
Mind = Blown.
Thankyou from Indonesia.
Thanks!
I can only have love for these videos, thank you Josh and all the team if you have any.
Thank you! It's just me doing all this.
Great explanation!!
I'm loving every second of your videos!!! Cheers!!
Thank you! :)
I'm eagerly awaiting the expected values quest! Thank you so much for making these videos, I love watching them before sleep.
Awesome! It's on the to-do list, but it might not be done for awhile. :(
@@statquest That's cool, take your time to keep making awesome videos. I still have loads of your videos on my to-watch list!
Thank you St Josh for this illuminating explanation :)
My pleasure!
8:22 `the way he said "Whaat" is so cute.. I'm in love
:)
this is literally what I was trying to get a clear understanding on in the last few days? what are the chances? no seriously what are the chances?
That's awesome! :)
"Future is nooow, BAM " - #LOL #respect #welldone #thanks
Thank you! :)
Thanks for explanation!
I understand that differences between the SAMPLE data and the sample mean are smaller than the differences between the SAMPLE data and the population mean. BUT! We are not interested in the difference between the SAMPLE data and the population mean, rather we are looking for the difference between the TRUE POPULATION data and the population mean (the population variance). And it's not clear why this value would be larger.
I mean sample data is centered around sample mean the same way population data is centered around population mean. Comparing sample data with population mean feels to be misleading.
The best estimate we can do is the estimate of the variance around the sample mean, which is probably an underestimate, but not always. So this is the best we can do.
I love the way you explain these topics, great work!
Thanks!
Good job Josh!! Waiting for StatQuest on Expected Values! I am the one wondering why not dividing by 'n-0.5' or 'n-2'
Thanks!
I have been SO stressed out about a project I'm working on, and 3:15 made me laugh so hard!!! I didn't even realize how stressed out I was until I caught myself laughing for the first time in weeks. Thank you Josh!!! **sob**
Hooray!!! Good luck with your project. I hope it goes well. :)
16:38 , it's not resolved in the expected values video. How to know that prof.
Unfortunately I haven't had time to do the follow up video. The best I can do is give you this link for now: online.stat.psu.edu/stat415/lesson/1/1.3
@@statquest thanks for immediate response prof.😊
I really want to understand why we use n-1 instead of substituting any other number instead of 1. I'm guessing it has something to do with the way we approximate the mean and the variance. I think it's related to properties the normal distribution has and such. I think that to truly understand that analytically I'd have to integrate over all possible outcomes while taking into account all the probabilities and then calculating the average. It really excites me, but I don't know where I can find the information needed to understand the subject in more depth. Can you give me some advice on what textbooks I should read, please? I'd really really appreciate that!
See: online.stat.psu.edu/stat415/book/export/html/886
@@statquest thank you, I will definitely read that!
Thanks!
Wow!!! Thank you so much for supporting StatQuest!!! :)
Greatest explanation so far!
Thank you! :)
Your work is impeccable. BAM!
Thank you!
This is awesome explanation. Waiting for quest on 'Expected Values'....BAM!
Me too. Hopefully I can get to it soon.
You left in a cliff hanger of expected values :((
Love your videos tho, thanks for these!
I'm working on it, but everything I do takes longer than I would like. :)
ASTOUNDING EFFECTS & EXPLAINATIONS!
SUBSCRIBED TRIPLE BAM!!!
bam!
Waited the whole video to know why it was n-1 and not n-2 etc... "that mistery will be resolved in the next episode"
Felt like watching an overstretched TV series in some way haha.
I understand it's capital to show first that sample variance underestimate true variance, but could mention earlier that you ll not focus on "why" it is n-1. :p
Thank you though wonderful content!
I tried to be careful with the title of this video with "Why dividing by N underestimates the variance" instead of "Why n-1 gives us an unbiased estimate".
That being said, I really wanted to explain exactly why n-1 works, but the proof is relatively advanced.
I finally know why n-1 is used. Thank you so much!
Bam!
Oh no... I'm falling deeper and deeper into this rabbit hole
:)
You are a very great teacher, i like your coaching style, keep going on!
Thank you! 😃
Great video! We clearly see that estimated variation is smaller than desired so we have to make it bigger. We can make it by dividing by n-1, but also by n-2 or n-1.5 or n-100. Why n-1?
One day I'll make that video, for now, see: online.stat.psu.edu/stat415/lesson/1/1.3
I usually hit like after the first BAMMM. This is some super great stuff Josh.
Thank you very much! :)
wow that n-1 has something to do with E(X)? Im waiting for it!
Super interesting! Thanks for your work!
Thanks!
Loved the video. But didn't understand something clearly. The variance is the least around the calculated mean. But that is only when the data x remains the same right? How can you compare it with the population variance which has a lot more data points and the summation is therefore different?
We are not comparing it to the population variance. We are simply comparing the variance of the data calculated around the sample mean compared to the variance of the data calculated around the population mean.
1:47
Draw it on the graph how
Are you talking about that curvy line around the histogram?
The red line with arrows on each end represents the population standard deviation.
@@statquest yeah thanks alot that's what i meant
I do understand the way yo calculate min variance
But @ 14:54 How did you conclude in that "Thus valus around sample mean is always less than population mean" ?
Because the sample mean is the value that minimizes the variance, any other value will give you a larger variance.
THX!!! Looking forward for the STATQUEST on expected Values ;))))
Me too!
15:46 it’s the god moment 👏🏻👏🏻👏🏻👏🏻
bam!
I wish you were my stats teacher!! Amazing job!!!
Thank you! :)
@@statquest really waiting for the expected value video to get explanation of n-1. When can we expect it?
@@shouryanand456 Unfortunately, it might be a while. I've got a full plate until after the summer.
I wanted to know why we deduct exactly 1, but I guess that only takes 20 aditional minutes to explain. Hooraay!
Thanks for the videos :)
It's true. We have to dive into expected values and that is a whole new topic.
baaammm! subscribed.
Awesome! :)
BAAM! me too
Although you make everything look so simple, your teaching pedagogy requires a lot of hardwork( to make the slides particulalry). I hope that every teacher puts in the same kind of hardwork and assume there students to be in 5th grade that way every class will be a pleasureable experience of life.
Wow, thank you!
it is because some teachers don't know how to teach. They learn from textbook's concept. Memorize them, then give those back to students. I am not being rude but it is the reality. In order to be able to explain well to new learners, teachers must be able to understand the concepts well. Teaching is a hard skill to master. Nowadays, lot of taught concepts are assumed true or left blank during teaching . That's why if those students become teachers, they won't be able to explain.
"The future is now" I'm dying
BAM! :)
Your Voice is magical 🌹🌹🌹
Thank you!
Very nice explanation, god bless you josh!
Thank you! :)
Great video!
Where is the one about Expected Values?
I cannot wait with such a cliffhanger! GoT finale can wait...
Very funny! Yes, I have my work to do. I hope to get to expected values before too long.
@@statquest Thanks a lot for responding! ... and sorry, as I noticed after reading more comments, that you had already answered this question many times. Quest on!
What a explanation.
I don't have money, else I'd have contributed.
The least I could do is share, which I already did.
BAM !
BAM! :)
I thought that the sample variance [ (sigma(Xi - X bar)^2)/n ] is trying to estimate population variance [ (sigma(X - mew)^2)/N ], so at 16:04, why do we care about how the sample standard deviation estimates the variation in the *Data* around the population mean, as opposed to how it estimates the actual population variance (which is what it’s set out to estimate)?
Because the estimation of the variation in the data around the population is unbiased.
This the best explanation ever
Thank you!
Nice! PS. Small typo at 8:42, you say -1 but write x-1.
That 'x' is a "times" symbol. So it's "times -1", not "x - 1"
@@statquest thx I was confused too - by the way - your videos among the very best one can find!!! Thank you so much!!!
Thanks so much for the explanation, super clear as always
Glad it was helpful!
Hi Josh, where did you study about it, is it from Bessel's correction or Karl Pearson. I am interested to ready a bit about the history behind it. Can you please suggest a book or paper where the original discovery was made. Thanks in advance.
The idea for this came from Bessel's correction.
8:21 -> I will watch a thousand times and I will laugh out loud a thousand times 😂
Hooray! :)
Man, you are great. From where did you learn these concepts? Keep making videos and enlighten us. Thank you.
Thanks! :)
@@statquest When I try to learn these concepts they seem complicated to me. From where did you learn these concepts?
@@shubhamtalks9718 The concepts seem complicated because people that do not really understand them try to teach them.
How did I learn them? Years of really hard work. I read everything I can about a subject, then I re-read it. Then I re-read it again. Then I make a program based on my ideas and see what happens. Then I re-read everything over again. And sooner or later I figure it out. But it takes a lot of time and a lot of work. Sometimes I worry I will not succeed, and sometimes I fail, but I keep trying anyway.
@@statquest Thanks😁
Excellent. Thank you for a great explanation.
Glad you enjoyed it!
This intuitively makes more sense to me now. If I take a sample, the sample mean may end up being larger or smaller than the population mean. But the sample variance can never be larger than the population variance, it might be equal to it, but most probably it will be smaller.
That's exactly right. :)
I think you have to be a little bit careful with what you mean by "sample variance" and "population variance". As long as you're comparing an estimated population variance using the sample data and actual population mean vs an estimated population variance using the sample data and sample mean. But, comparing the estimated population variance using the sample data and sample mean vs the actual population variance (all data and actual population mean) doesn't have the guarantee that sample variance will be lower than population variance.
Sorry I am still getting confused. At 5:29, in the inequality, both left hand side and right hand side are using the same n, which is the number of samples.
You argued that the right hand side is greater so that we need to make the left hand side larger by dividing n-1.
However, the right hand side is not the actual population variance. The actual population variance should be using a much larger n to calculate.
What we are doing here is to estimate the population variance but not the right hand side.
Thinking to this point, all the linkage seems broken. How can I relate the right hand side to the population variance?
It is true that the inequality holds. But it does not mean also the population variance is always greater than the left hand side.
Thanks for your videos. They inspire me and teach me a lot.
Sometimes we know the population mean, but don't know the variance, so we sill have to estimate it. That is what is going on on the right side of the equation.
Amazing video!
I think that you should teach another subject. Maybe MathQuest? That would be amazing!
Maybe one day!
Clarity brings understanding
Bam! :)
Hey josh, yet another cool explanation.
Thank you! :)
The end of this video suggests that the video on Expected Values will elaborate on why subtracting 1 is not arbitrary. The Expected Values video says that it is the first step towards getting to this video. Does anyone have the appropriate chronological order of the videos? Or if I'm stuck in a time loop, let me know!
The expected value is a step in the direction of understanding why subtracting 1 is not arbitrary, but it doesn't get you all the way. Unfortunately I haven't made videos for the rest of the steps yet. If you're in a hurry, check out: online.stat.psu.edu/stat415/lesson/1/1.3
Thank you! I will look forward to that and check this out for now.
I think there is little problem here @8:47 where we use chain rule .......where that x-1 come I know -1 is derivative of -v but why there is x and where it goes later.
By the way nice explanation .
The little 'x' means "times" and the big "X" is a variable. Sorry for the confusion.
@@statquest Thank You so much for explanation .I get it now.
From 2:38 to 2:40, the symbol for estimated mean was switched to population mean. Great video anyway
Ooops. That's a typo. Thanks for pointing that out. I've updated the pinned comment with it.
Super content in the video. But i have a doubt. Its been mentioned that the sample mean will always be less than the Population mean (14:58). But i think that this is a completely relative term. What if the sampled data has more quantity of samples which have higher value than the population mean? I think the sample mean will be greater than the population mean. Kindly correct me if wrong.
I believe you are misunderstanding what the video is saying. The equation shows the sample variance is always less than the population variance and the text says that that relation will always be true unless the sample mean is exactly the same as the population mean.