The full Neural Networks playlist, from the basics to deep learning, is here: ruclips.net/video/CqOfi41LfDw/видео.html Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/
In neural network, does the gradient for parameters are calculated parallel? For example: when I start with finding gradient for all the 7 parameters, do I calculate all 7 parameters simultaneously by taking the previous iteration values or, first I calculate the bias gradient and get the new bias, then calculate predicted value by new bias and then calculate gradient for w3 ? And so on till w1 ?
I am watching all of these eventhough i already graduated with masters with focus on machine learning and deep learning. its actually amazing how much I am learning even as a intermediate student.
Despite the simple explanations, these videos continuously make me doubt my mathematical abilities for about 15 minutes. But without fail, there’ll be a DOUBLE BAM! out of left field and suddenly everything’s clear in my head. Thank you! You’re doing God’s work.
One of these days I'd love to see you do a NN to watch the process you produced on these videos get lined up with some code, maybe python or R. It's incredible work you do, hell you are helping me survive my masters program. If you put it up, I'd trust the content. Thank you for all your hard work
Thanks Josh for these videos, I passed my data mining exam by watching your videos, now preparing for the ML exam. Your explanation is brilliant, I learn topics of 3 lectures with these 18 minutes videos. Please continue to publish such valuable content, you save lives of many people like me.
I'm a student for CS at the Hebrew University of Jerusalem, study right now IML course. Your lectures so help me and my friends, and I really want to thank you. You're a great & funny teacher and your lessons are a perfect example to how to teach at the 21 century. Tnx again
BAM !!! I'm doing my PHD in this field, and it is the BEST serie of videos that I have watched since the bigenning of my study ! Thank you so much for that :D
Such hard work. Thank you Josh, you are helping generations with this + all your videos. Step by step learning with examples is the right way to learn anything !
What a great explanation to such complex topic. I can't imagine the amount of effort you put in to create such detailed videos along with spoken text. One of the best youtube channel I have ever come across ! Hats off to you .. Don't BAM me :)
Had to watch 17:09 at least ten times. But now I get the most dificult part: the orange and blue curves, represent... the orange and blue curves. Without that, I would be completely lost 😆. Thanks for the hard work. Amazing series!
wow! WoW! WOW!, I have always been scared of math, cus it took me hell lot of time to understand, but you just explain it as smooth as butter, Thanks a lot really!!
These videos are so much better than 3blue1brown, he starts with complicated analogies and examples and then delves into heavy math whereas this simplifies the problem using simpler examples and works through all the small steps
thank you so much for these videos. i hated math back in high school, but now in my mid 20's i would rather do math than play video games. all thanks to your tutorials
that was exactly what I needed. It would be great if you could 'also' do an application through one of Python libraries in order to show a real application by scripting with using this knowledge.
This is just awesome. I had started learning machine learning algorithm from multiple sources until I found your youtube channel. And now I don't have to check for any other source for understanding any ML algorithm. Looking Forward for more deep learning videos as my area of interest is deep learning. Could you help me with a good book for deep learning? And thanks for such wonderful videos.
Love your videos man, very helpful at providing detail without sacrificing clarity. However I have noticed quite a few errors across the videos, generally small errors such as saying the wrong numbers or when calling up examples such as in this video at 9:26. input_3 would be 0, not 1. Again, it is not a major error and the information provided is nonetheless exemplary however it does make following along a tad challenging when trying to listen to the video rather than watching it like a hawk. Keep up the good work man, much appreciated x
I'm glad you like my videos. It is indeed unfortunate that a few of them have small "typos". However, the example you provide is not one of them. The inputs to the neural network are the x-axis coordinates, not the y-axis coordinates. The 3rd data point has an x-axis coordinate of 1 and a y-axis coordinate of 0. Thus, for the 3rd data point, the input to the neural network is 1 and the desired output is 0. So, not only is this not a major error, it's not an error at all.
As usual, your videos are totally awesome, I like them much and easy to understand. I wonder if you will make a video about spatial transcriptomic analysis please since you uploaded the scRNA three years ago considering the spatial analysis is now more famous?
[Notes] 6:44 Notation for activation functions 2:50 Initialize weights using standard normal distribution. Q: Why N(0,1)? -- A: Just one of many ways to initialize weights. [ref. 9:50 of ruclips.net/video/GKZoOHXGcLo/видео.html&ab_channel=StatQuestwithJoshStarmer] Initialize bias with 0 since bias terms frequently start from 0. 4:334:48 plot SSR with respect to b3
Hey Josh, great video as always!! Can you also please point to some source with examples (with answers) which we can practice on our own? I know there are tons of them on internet, but you know, your selection will be really helpful as always!!
basically, taking derivatives of losses with respect to unknown terms to find how quickly the loss is changing if we change the parameters is the essence of this whole Machine learning thing.
Hi Josh, great video as always. One question, if I want to speak in epoch and batch terms for this video, is it correct to say that this video shows one epoch, which includes one batch that contains all 3 data points we have (Batch Gradient Descent process)? Thanks a lot !!!
Thanks for your great video as always! I have a question though after watching this video and the other SGD video you've made in the past. When calculating the gradients for each parameter with regular gradient descent, we are plugging in all of the samples into the derivative of the loss function w.r.t the current parameter; versus we will just randomly pick one sample in the same process with SGD being used. If that's the case, then what will be the purpose of looping through all the samples (with regular GD) in a complete epoch if we are already using all the samples when calculating the gradients? Thanks in advance!
I'm not sure I fully understand your question. The difference between "regular" and "stochastic" gradient descent in this context has to do with the summation. In "regular", the summation goes from 1 to 'n', where 'n' is the number of samples. In "stochastic", the summation goes from 1 to m, were 'm' is < 'n' and is the number of samples randomly selected for the iteration. Does that help?
@@statquest Thank you for the quick reply! Yes that’s helpful and I think I’m understanding that part. I was mixing the concept of Gradient Descent with epoch/batch numbers, but I guess whether the GD is stochastic or not has nothing to do with the general epoch/batching concept when running a neural network, as we would still need to go over all the samples in a full epoch.
Great video and explanation. But I'm missing something simple. The blue and orange lines are added to render the green line, right? It appears (I'm squinting) that, after convergence, the middle dose (the 1/2 dose; actually, just to the left of it) value is 1 but the intersection of the blue and orange lines is at about -.5. Adding those together gives -1, not 1. What am I missing??
@@statquest Thank you for answering even after 2 years! Funnily enough, while I wrote the elaboration of my question here, I stumbled upon the answer myself. Thank you again for your commitment. Let me tell you, that the work you do absolutely outclasses any learning material that I have stumbled across. I will definitely check out your book! Great work!
To get a better understanding of how we determine this derivative, check out the StatQuest on The Chain Rule: ruclips.net/video/wl1myxrtQHQ/видео.html It will explain exactly where that -1 comes from.
I think for the sake of clarity and rigour, it should be noted that all of the differentials are partial. Otherwise, some people may wonder why implicit differentiation wasn't used in such cases where W2 was differentiated with respect to W1 or vice versa.
Hey just a question! Around 14:00, why are you taking the derivative of SSR with respect to w_3 and w_4 rather than y_2,i and y,1_i? What is the logic between choosing taking the derivative with respect to the weight rather than the functions themselves?
Ah nevermind, its because you want to optimize the weights w_3 and w_4, so you just take their derivative to get step size and so forth... im so dumb haha! Im assuming that in the next part then you will optimize the weights w_1 and w_2 by also connecting them to the derivative of the loss function with respect to the weights, so itll be a huge bonkers chain rule in action
Why do you not average the derivative of the SSR (the gradient). What I mean by average is dividing the derivative of the SSR by the number of training examples. I read online that this is more common practice unless we are doing stochastic gradient descent. I was a little bit confused as this was not clarified. Thanks for the video though it really helped me understand the topic better.
As the video shows, it works just fine without averaging the SSR. However, we have a relatively small dataset and that keeps the derivative from getting out of hand. If we had tons and tons of data, the SSR alone might lead to a massive derivative that's too big to be helpful, and averaging could help with that.
We include the derivative of the SoftPlus activation function in the next video (part 2), when we optimize all of the weights and biases, including the ones to the left of the activation functions: ruclips.net/video/GKZoOHXGcLo/видео.html
Sir, one question I have. when you said we randomly select w3 and w4 from standard distrib in the first iteration, that is any values from standard distrib table or we select no's w.r.t. given dataset?
In this example I selected random value from a standard normal distribution. This is a normal distribution with mean = 0 and standard deviation = 1 and is completely independent of the data.
BAM!! I finally understand but.... Am I correct to say that if I was optimizing 3 weights and biases at the same time i would do gradient descent in a function with 3 dimensions (1 for each weight and bias)??
Hi Josh ! Love your videos, could you please explain why normal distribution is used to initialise w3 and w4 or else if you have already uploaded a video on normal distribution can you tag it
It's just a standard way to do it. However, you can use uniform distributions or other distributions if you would like. One thing people like about the normal distribution is that changing the standard deviation for each hidden layer can make it easier to train deeper models (models with lots of hidden layers).
Small question: since we fiddle with all (or part) of the parameters at once, and for example bias is dependent on weights on the graph, does that mean they fight with each other? Can something be done about it? Like we calculate the derivatives for current forward pass, ok, but then changing all parameters at once to what the think is optimal might throw off everything, since they can't communicate in any way, how does it not explode?
In my video on gradient descent, I show how to optimize two parameters at the same here: ruclips.net/video/sDv4f4s2SB8/видео.html In that video, we're trying to fit a straight line to some data points and are using gradient descent to find the best values for two parameters, the y-axis intercept and the slope. If you watch, you'll see a fancy graph, where one axis represents different values for the y-axis intercept and another axis represents different values for the slope. When we optimize both at the same time, we take a step towards a better intercept on that axis and take a step towards a better slope on that axis, which is different, and doesn't affect the one the intercept is on. So the parameters don't fight each other because each one gets its own axis to work on. That being said, we can still get stuck in a local minimum, but it's like progress in one parameter can be negated by progress in another.
@@statquest Ah, this makes a lot of sense now, I think I know why it was misleading for me - in the end all you see a numerical value, the error, but behind the scenes the partial derivatives take apart the loss function in their own domains, so it's not just one number. Thank you for very descriptive response!
In statistics, data science, machine learning and almost all programming languages, the default base for the log function is 'e', and that's what I use here.
@@statquest Basically I'm just confused about if the final curve approximating the 3 points is a "curve" as in a polynomial, or just a set of 3 points. Because when we add the two activation functions, you talked about adding them at each point as if we were adding the equations for the lines themselves, in order to get the final line. But it seems like instead we're just adding the y values at each input (the 3 given inputs) rather than a line itself..?
@@dahirou_harden The adding is done for all possible x-axis coordinates (or input values), and thus, we are adding the lines themselves, not just the 3 points. The points (or circles) on the lines are just to illustrate the concept of adding y-axis values, and do not to limit the adding to just those points.
So what is the clever part of back prop? Why does it have a special name and it isn't just called "gradient estimation"? How does it save time? It looks like it just calculates all derivatives one by one
Hi, how to understand back? not forward or other direction? I mean the video is nice, but didn't explain to clear why backward is important. Why not forward?
got it. At the back point, the derivative is much simpler than the derivatives at the front. So, as we would like to go from simple to hard, we'd choose from back to front. That's why it's backpropagation, which is discussed in the next video. BR
The full Neural Networks playlist, from the basics to deep learning, is here: ruclips.net/video/CqOfi41LfDw/видео.html
Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/
In neural network, does the gradient for parameters are calculated parallel?
For example: when I start with finding gradient for all the 7 parameters, do I calculate all 7 parameters simultaneously by taking the previous iteration values or, first I calculate the bias gradient and get the new bias, then calculate predicted value by new bias and then calculate gradient for w3 ? And so on till w1 ?
@@sahanamd707 Everything is done at the same time.
Thank you
the way you explain things,so patiently and in depth, i feel 200% more confident in the topic afterwards
Awesome! :)
I'm writing from Brazil. This channel's playlist about neural networks is a masterpiece.
Muito obrigado! :)
join a cartel
I wish I had this when I was first learning backpropagation! Can I "work my way backward" with this knowledge haha
BAM! :)
Pro tip: you can watch movies on flixzone. Me and my gf have been using it for watching lots of of movies lately.
@Roman Randall Definitely, have been using Flixzone for years myself :)
haha good one !!
@Roman Randall and @Amos Zahir are bots but nice one
I really like the medieval guitar sound you added when you said "fancy notation" , you effort can really be seen in the little details
Thanks!
I am watching all of these eventhough i already graduated with masters with focus on machine learning and deep learning. its actually amazing how much I am learning even as a intermediate student.
bam!
We all should buy his book, he deserves it given the quality of these videos!!
Thank you!!! :)
yes!
@@statquest
Hey man,
just bought your book,will be arriving in a few days via amazon.All these topics are covered right?
@@MultiSamarjit The basics of neural networks and backpropagation are covered. The other topics are listed here: statquest.org/statquest-store/
OMG THE HAPPINESS I WAS FEELING WHEN I UNDERSTOOD EVERYTHING, you seriously are a really good teacher.
Thank you!
Despite the simple explanations, these videos continuously make me doubt my mathematical abilities for about 15 minutes. But without fail, there’ll be a DOUBLE BAM! out of left field and suddenly everything’s clear in my head. Thank you! You’re doing God’s work.
Bam! :)
Triple BAM (Explanation)+Triple BAM (Animations)......
You are a very great teacher Josh Starmer :) :)
Wow, thanks!
The details are just out of this world. Amazing. Breath-taking and short of words.
Thanks!
One of these days I'd love to see you do a NN to watch the process you produced on these videos get lined up with some code, maybe python or R. It's incredible work you do, hell you are helping me survive my masters program. If you put it up, I'd trust the content. Thank you for all your hard work
Thank you! And good luck with your masters degree.
Thanks Josh for these videos, I passed my data mining exam by watching your videos, now preparing for the ML exam. Your explanation is brilliant, I learn topics of 3 lectures with these 18 minutes videos. Please continue to publish such valuable content, you save lives of many people like me.
Thank you and good luck with your exam! Let me know how it goes.
@@statquest Josh, are you planning to make a video about batch normalization?
@@vusalaalakbarova7378 Not soon. Currently I'm working on a series of videos about how to build neural networks with pytorch and pytorch_lightning.
I'm a student for CS at the Hebrew University of Jerusalem, study right now IML course. Your lectures so help me and my friends, and I really want to thank you. You're a great & funny teacher and your lessons are a perfect example to how to teach at the 21 century. Tnx again
Wow! Thank you very much! BAM! :)
How the heck do you have this much understanding in each concept...you are irreplaceable.
Thanks!
BAM !!! I'm doing my PHD in this field, and it is the BEST serie of videos that I have watched since the bigenning of my study ! Thank you so much for that :D
Thanks and good luck!
Such hard work. Thank you Josh, you are helping generations with this + all your videos. Step by step learning with examples is the right way to learn anything !
Thank you!
What a great explanation to such complex topic. I can't imagine the amount of effort you put in to create such detailed videos along with spoken text. One of the best youtube channel I have ever come across ! Hats off to you .. Don't BAM me :)
Wow, thank you!
I agree 100%
Had to watch 17:09 at least ten times. But now I get the most dificult part: the orange and blue curves, represent... the orange and blue curves. Without that, I would be completely lost 😆.
Thanks for the hard work. Amazing series!
I'm glad you figured it out! :)
hahahaha
wow! WoW! WOW!, I have always been scared of math, cus it took me hell lot of time to understand, but you just explain it as smooth as butter, Thanks a lot really!!
Thank you very much!
These videos are so much better than 3blue1brown, he starts with complicated analogies and examples and then delves into heavy math whereas this simplifies the problem using simpler examples and works through all the small steps
Thank you!
I totally agree with you 👏
BAMMMMMMM!
I like the animation in the last part and the music with Fan~cy notation lol
BAM! :)
You are One of the Best Content Creator I have ever Seen.
Wow, thanks!
woah this was some amazing teaching skills sir, you're totally gifted with that
Thanks! 😃
A Brief Indepth Explanation.Thank you Sir
Glad you liked it
thank you so much for these videos. i hated math back in high school, but now in my mid 20's i would rather do math than play video games. all thanks to your tutorials
Wow! That's awesome! Thank you!
Thank you for your clear explanations with the simple example! Great work, and very useful.
Glad it was helpful!
Really loving these videos, thank you so much for your work Josh
Thank you!
The little harpsichord really tickled me haha, love it
:)
This NN series is so underrated just 124K I mean come on
Thanks!
that was exactly what I needed. It would be great if you could 'also' do an application through one of Python libraries in order to show a real application by scripting with using this knowledge.
Thanks! I would like to do that.
You make this stuff so accessible, well done!
Thank you!
Mega BAMM!! I really love the explanation. Awesome!
Thanks!
this playlist is excellent
Thank you!
Hats off to you Josh!!
So nicely explained ❤
Glad you liked it!
That was the best explanation I had ever seen. Thank you very much.
Thank you! :)
Thank you so much for sharing your knowledge, it is really so increadibly helped me understand the basics of the NN.
Glad it was helpful!
This is simply beautiful!. You are the best.
Thank you!
I am on vacation in Hawaii but I am watching your neural network video. This video is so entertaining to watch :) Tai
BAM! Have a great vacation! :)
@@statquest thank you! you too. have a nice day
You're a legend Doctor Starmer.
Thanks!
I wish I could have taken your classes when I was back in high school.
bam! :)
Well you actually make complex things super easy.Hats off and of course BAAA...M!!!
Bam! :)
This is just awesome. I had started learning machine learning algorithm from multiple sources until I found your youtube channel. And now I don't have to check for any other source for understanding any ML algorithm. Looking Forward for more deep learning videos as my area of interest is deep learning. Could you help me with a good book for deep learning? And thanks for such wonderful videos.
This series ends (for now) with Convolutional Neural Networks, so just keep watching to learn about deep learning.
amazing video, thanks for all your hard work on this.
Glad you enjoyed it!
Great explanation as usual. Thank you very much.
Thanks again!
thank you so much sir for clearly explaining everything
Thanks!
God bless this Good Man.
Thanks!
Best channel ever
Wow! Thank you! :)
I love you Statquest
:)
Love your videos man, very helpful at providing detail without sacrificing clarity. However I have noticed quite a few errors across the videos, generally small errors such as saying the wrong numbers or when calling up examples such as in this video at 9:26. input_3 would be 0, not 1. Again, it is not a major error and the information provided is nonetheless exemplary however it does make following along a tad challenging when trying to listen to the video rather than watching it like a hawk. Keep up the good work man, much appreciated x
I'm glad you like my videos. It is indeed unfortunate that a few of them have small "typos". However, the example you provide is not one of them. The inputs to the neural network are the x-axis coordinates, not the y-axis coordinates. The 3rd data point has an x-axis coordinate of 1 and a y-axis coordinate of 0. Thus, for the 3rd data point, the input to the neural network is 1 and the desired output is 0. So, not only is this not a major error, it's not an error at all.
Hey Josh, this is dope. Hope you could do some videos about the Hessian and Jacobian matrices, Thanks.
I'll keep those topics in mind.
As usual, your videos are totally awesome, I like them much and easy to understand. I wonder if you will make a video about spatial transcriptomic analysis please since you uploaded the scRNA three years ago considering the spatial analysis is now more famous?
I'll keep it in mind.
BAM! Great work!
Thank you!
[Notes]
6:44 Notation for activation functions
2:50 Initialize weights using standard normal distribution. Q: Why N(0,1)? -- A: Just one of many ways to initialize weights. [ref. 9:50 of ruclips.net/video/GKZoOHXGcLo/видео.html&ab_channel=StatQuestwithJoshStarmer]
Initialize bias with 0 since bias terms frequently start from 0.
4:33 4:48 plot SSR with respect to b3
bam!
Thanks. Great video again and again.
Thank you very much! :)
So underrated
Glad you think so! :)
i hate stats, and i hate statquest. But i keep finding myself on this channel again and again
noted
YOU ARE THE BEST!
Thanks!
Hey Josh, great video as always!!
Can you also please point to some source with examples (with answers) which we can practice on our own?
I know there are tons of them on internet, but you know, your selection will be really helpful as always!!
I don't have anything yet, but I will create a "how to do neural networks" video soon.
Hi Josh! Love the videos. Do you have any posts for building models in R/Rstudio on neural networks? Thanks,Tina
Not yet!
Im gonna make a AI agent that create youtube bots that promotes your channel. You really deserve all kudos.
Bam!
Yeaaaa finally new episodde
:)
superb, thanks a lot.
Thanks!
I love you man! You have a sense of humor about you that is rare in deez parts lol
Thank you!
I can not thank you enough!
Thanks!
THIS MAN IS AN ANGEL! :D QUADRUPLE BAM!
Thank you! :)
very informative ty
:)
You are amazing!
Wow, thank you!
basically, taking derivatives of losses with respect to unknown terms to find how quickly the loss is changing if we change the parameters is the essence of this whole Machine learning thing.
yep
Hi Josh, great video as always. One question, if I want to speak in epoch and batch terms for this video, is it correct to say that this video shows one epoch, which includes one batch that contains all 3 data points we have (Batch Gradient Descent process)? Thanks a lot !!!
Yes, that is correct.
Thanks for your great video as always! I have a question though after watching this video and the other SGD video you've made in the past. When calculating the gradients for each parameter with regular gradient descent, we are plugging in all of the samples into the derivative of the loss function w.r.t the current parameter; versus we will just randomly pick one sample in the same process with SGD being used. If that's the case, then what will be the purpose of looping through all the samples (with regular GD) in a complete epoch if we are already using all the samples when calculating the gradients? Thanks in advance!
I'm not sure I fully understand your question. The difference between "regular" and "stochastic" gradient descent in this context has to do with the summation. In "regular", the summation goes from 1 to 'n', where 'n' is the number of samples. In "stochastic", the summation goes from 1 to m, were 'm' is < 'n' and is the number of samples randomly selected for the iteration. Does that help?
@@statquest Thank you for the quick reply! Yes that’s helpful and I think I’m understanding that part. I was mixing the concept of Gradient Descent with epoch/batch numbers, but I guess whether the GD is stochastic or not has nothing to do with the general epoch/batching concept when running a neural network, as we would still need to go over all the samples in a full epoch.
great content!
Thanks!
Great video and explanation. But I'm missing something simple. The blue and orange lines are added to render the green line, right? It appears (I'm squinting) that, after convergence, the middle dose (the 1/2 dose; actually, just to the left of it) value is 1 but the intersection of the blue and orange lines is at about -.5. Adding those together gives -1, not 1. What am I missing??
You forgot to add the bias term.
Absolutely amazing! I‘ve got a question though. How do we know if we are going the right direction when calculating the new parameter.
The derivative tells us what direction to change the parameter. To see more details, see: ruclips.net/video/sDv4f4s2SB8/видео.html
@@statquest Thank you for answering even after 2 years!
Funnily enough, while I wrote the elaboration of my question here, I stumbled upon the answer myself.
Thank you again for your commitment. Let me tell you, that the work you do absolutely outclasses any learning material that I have stumbled across. I will definitely check out your book! Great work!
13:35 Do you mean the derivative of observed - predicted? Wouldn't that be a derivative of a single number? Or does it always just come out to be -1?
To get a better understanding of how we determine this derivative, check out the StatQuest on The Chain Rule: ruclips.net/video/wl1myxrtQHQ/видео.html It will explain exactly where that -1 comes from.
@@statquest Oh the derivative of the negative intercept? ok thanks
I think for the sake of clarity and rigour, it should be noted that all of the differentials are partial. Otherwise, some people may wonder why implicit differentiation wasn't used in such cases where W2 was differentiated with respect to W1 or vice versa.
noted
BAM! that was good!
Thanks!
OMGGGGG, Is that you who sings at "big bang theory", S12, E24???!!!!!
I wish! :)
Hey just a question! Around 14:00, why are you taking the derivative of SSR with respect to w_3 and w_4 rather than y_2,i and y,1_i? What is the logic between choosing taking the derivative with respect to the weight rather than the functions themselves?
Ah nevermind, its because you want to optimize the weights w_3 and w_4, so you just take their derivative to get step size and so forth... im so dumb haha! Im assuming that in the next part then you will optimize the weights w_1 and w_2 by also connecting them to the derivative of the loss function with respect to the weights, so itll be a huge bonkers chain rule in action
Yes! It will be totally bonkers with chain rule action. :)
Why do you not average the derivative of the SSR (the gradient). What I mean by average is dividing the derivative of the SSR by the number of training examples. I read online that this is more common practice unless we are doing stochastic gradient descent. I was a little bit confused as this was not clarified. Thanks for the video though it really helped me understand the topic better.
As the video shows, it works just fine without averaging the SSR. However, we have a relatively small dataset and that keeps the derivative from getting out of hand. If we had tons and tons of data, the SSR alone might lead to a massive derivative that's too big to be helpful, and averaging could help with that.
Many many thanks for your videos.
Glad you like them!
AMAZING BROOOOOOOOO
Thanks!
Dw_3 = (observed-predicted) * y1. The output is also a softplus activation. Why isn’t this derivative in the chainrule? Thank you!
We include the derivative of the SoftPlus activation function in the next video (part 2), when we optimize all of the weights and biases, including the ones to the left of the activation functions: ruclips.net/video/GKZoOHXGcLo/видео.html
Sir, one question I have. when you said we randomly select w3 and w4 from standard distrib in the first iteration, that is any values from standard distrib table or we select no's w.r.t. given dataset?
In this example I selected random value from a standard normal distribution. This is a normal distribution with mean = 0 and standard deviation = 1 and is completely independent of the data.
Notes: 2:31 6:14 15:57 the "y"s are calculated based on other weights (w1 and w2)
:)
In "d SSR/ d Predicted", is Predicted a single value like Predictedi (with index i ) or a collection of values as i can range from 1 to 3?
A collection of values. You can tell if you keep watching the video and see how it is used.
@@statquest thank you for the prompt reply, Josh! you are the best!
My regards to the friendly folks of the genetics department of University of North Carolina at Chapel Hill
Thanks!
BAM!! I finally understand
but....
Am I correct to say that if I was optimizing 3 weights and biases at the same time i would do gradient descent in a function with 3 dimensions (1 for each weight and bias)??
Yes
The God! Please do NLP also
Noted
Hi Josh ! Love your videos, could you please explain why normal distribution is used to initialise w3 and w4 or else if you have already uploaded a video on normal distribution can you tag it
It's just a standard way to do it. However, you can use uniform distributions or other distributions if you would like. One thing people like about the normal distribution is that changing the standard deviation for each hidden layer can make it easier to train deeper models (models with lots of hidden layers).
Small question: since we fiddle with all (or part) of the parameters at once, and for example bias is dependent on weights on the graph, does that mean they fight with each other? Can something be done about it? Like we calculate the derivatives for current forward pass, ok, but then changing all parameters at once to what the think is optimal might throw off everything, since they can't communicate in any way, how does it not explode?
In my video on gradient descent, I show how to optimize two parameters at the same here: ruclips.net/video/sDv4f4s2SB8/видео.html In that video, we're trying to fit a straight line to some data points and are using gradient descent to find the best values for two parameters, the y-axis intercept and the slope. If you watch, you'll see a fancy graph, where one axis represents different values for the y-axis intercept and another axis represents different values for the slope. When we optimize both at the same time, we take a step towards a better intercept on that axis and take a step towards a better slope on that axis, which is different, and doesn't affect the one the intercept is on. So the parameters don't fight each other because each one gets its own axis to work on. That being said, we can still get stuck in a local minimum, but it's like progress in one parameter can be negated by progress in another.
@@statquest Ah, this makes a lot of sense now, I think I know why it was misleading for me - in the end all you see a numerical value, the error, but behind the scenes the partial derivatives take apart the loss function in their own domains, so it's not just one number. Thank you for very descriptive response!
@@madghostek3026 bam! Your question is actually a very good one and maybe one day I'll make a short video that explains it for everyone.
The nested chain rule.
:)
Please make a tutorial on Gaussian mixture model and EM algorithm
I'll keep that in mind.
@@statquest thanks..
It will be really helpful 🙂
when i plug the value -1.43 into the equation log(1 + e**x) i get 0.093. should I use the base 10 for log or a different one?
In statistics, data science, machine learning and almost all programming languages, the default base for the log function is 'e', and that's what I use here.
@@statquest Thanks, this was very helpful.
i dont know how u feel but i feel very bad that your are not recognized as much as you should be (from india )
Thanks!
@@statquest urs is the first channel I purchased member ship and God I am the most conjuze(resistant. In spending money)person I ever saw
@@kiranchowdary8100 WOW! That is awesome!!! Thank you so much for your support. It means a lot to me that you care enough to contribute.
The right way to learn, textbooks and lectures should be obsolete
bam! :)
Just wanted to clarify. Is the output given at the end of each pass an actual function or just a set of 3 points (summed from y1 and y2)? Thanks!
What time point, minutes and seconds, are you asking about?
@@statquest Basically I'm just confused about if the final curve approximating the 3 points is a "curve" as in a polynomial, or just a set of 3 points. Because when we add the two activation functions, you talked about adding them at each point as if we were adding the equations for the lines themselves, in order to get the final line. But it seems like instead we're just adding the y values at each input (the 3 given inputs) rather than a line itself..?
@@statquest At 4:03 for example.
@@dahirou_harden The adding is done for all possible x-axis coordinates (or input values), and thus, we are adding the lines themselves, not just the 3 points. The points (or circles) on the lines are just to illustrate the concept of adding y-axis values, and do not to limit the adding to just those points.
So what is the clever part of back prop? Why does it have a special name and it isn't just called "gradient estimation"? How does it save time? It looks like it just calculates all derivatives one by one
Backpropagation refers to how the gradient is calculated. Gradient Descent is how the gradient is used.
Hi, how to understand back? not forward or other direction? I mean the video is nice, but didn't explain to clear why backward is important. Why not forward?
got it. At the back point, the derivative is much simpler than the derivatives at the front. So, as we would like to go from simple to hard, we'd choose from back to front. That's why it's backpropagation, which is discussed in the next video.
BR
bam! :)
Triple BAM!!! That's what I said when I knew my girl, married her and got children :)
:)