Prof...stop ...stop...it's already dead! Oh BP you thought you were this tough complex thing and then you met Prof. Justin Johnson who ended you once and for all! The internet is 99.99% garbage but content like this makes me so glad that it exists. What a masterclass! What a man!
Sir, you are amazing! I've wasted hours reading and watching internet gurus on this topic, and they could not explain it at all, but your lecture worked!
At 58:56 prof Johnson tells something huge imho , the final equation is not formed by jacobians , finally I got it..simply the best explanation on the backprop .Thank you prof Johnson
I don't get how back propagation tutorials by 3B1B, StatQuest, etc, get so much praise, but neither of them are as succinct as you were in those first two examples. Fuck that was simple.
19:38 Shouldn't 0.39 be 0.4 and 0.59 be 0.6 -- not sure where the rounding errors have creeped in. 49:45 would it not be much easier to use Einstein index notation?
Our upstream gradient was -0.53 right? And now we need the local gradient of e^-x which is -e^-x and -e^-(-1)= -0.36. So upstreamgrad(-0.53) multiplied with local grad (-0.36) is 0.1949 which is approximately 0.2. So 0.2 is not local grad it is local multiplied with upstream
I don't think beaver's answer is quite right. The upstream gradient is -0.53. But the local gradient comes from the function e^x not e^-x. The derivative of e^x is e^x. Now we plug in the input which is -1 and we get e^-1 as the local gradient. This is approximately 0.37. Now that we have the local gradient we just multiply it with the upstream gradient -0.53 which results in approximately -0.20.
Great lecture thank you. I have a question, would be great if anyone could clarify. When you first introduce vector valued backpropagation, you have the example showing 2 inputs to the node, each input is a vector of DIFFERENT dimension - when would this be the case in a real scenario? I thought the vector formulation was so that we could compute the gradient for a batch of data (e.g. 100 training points) rather than running backprop 100x. In that case the input vectors and output vectors would always be of the same dimension (100). Thanks!
Can anyone explain 1:08:05? dL/dx1 should be next to dL/dL, not L when it is subject to function f2'. Thereby back propagation cannot connect fs and f's.
No, the operation is not relu, its calculation of the downstream gradient. since last row of jacobian is 0 meaning that changes in that value does not affect the output, so 0.
For the point of passing or killing the value of the upstream matrix, you have to decide pass or kill by looking at the input matrix, here that is [ 1, -2, 3, -1] so looking at -1, we will kill that value from the upstream matrix, so 0.
It is actually muchhhhhh more simpler than the way he used to explain. I believe he was redundant and too many symbols that hides the beauty of the underneath reason of the algorithm and the math behind it. It all could have been explained in less amount of time.
Prof...stop ...stop...it's already dead! Oh BP you thought you were this tough complex thing and then you met Prof. Justin Johnson who ended you once and for all!
The internet is 99.99% garbage but content like this makes me so glad that it exists. What a masterclass! What a man!
I work in ML and am doing review for interviews, this lecture is extremely thorough!
this lecture is an example of a perfect technical lecture
Sir, you are amazing! I've wasted hours reading and watching internet gurus on this topic, and they could not explain it at all, but your lecture worked!
I 've just watched 30 minutes, but I 'm so excited to comment here that it's definately the best course for back propagation!!!!
Dr. JJ, you sly sun of a gun. This is one of the best things ever. 47:39, the way he asks if it is clear. It is damn clear man. Well Done!
Amazing! One of the best Backprop explanation out there!
Best lecture ever on explanation of backpropagation in math
Future reference for anybody, but I think there's a typo @ 50:24. It should be dz/dx * dL/dz when using chain rule to find dL/dx
Finally!! I understood how to apply backpropagation. Thank you sir! Thank you!
At 58:56 prof Johnson tells something huge imho , the final equation is not formed by jacobians , finally I got it..simply the best explanation on the backprop .Thank you prof Johnson
finally some coverage on backprop with tensors
10:02. Dr. Johnson means, "right to left" not "left to right"
At around 18:20 shouldn't the original equation have a w_2 term that gets added to w_0*x_0+w_1*x_1?
w2 is a bias
I don't get how back propagation tutorials by 3B1B, StatQuest, etc, get so much praise, but neither of them are as succinct as you were in those first two examples. Fuck that was simple.
how come u are getting the value of e^x as -0.20. Could u explain
You earned a like, a comment and a subscriber ... what an explanation .
19:38 Shouldn't 0.39 be 0.4 and 0.59 be 0.6 -- not sure where the rounding errors have creeped in.
49:45 would it not be much easier to use Einstein index notation?
22:22 the local gradient should be "[1-sigma(1.00)]*sigma(1.00)" where 1.00 is the input to the sigmoid-fcn block
Such an amazing lecture with easy-to-understand examples!
wooooowww.. what a superb lecture on backpropagation. simply amazing.
Such awesome and intuitive explaination!
@49:44 - Mistake in dL/dx formula - 2nd operand should be dL/dz (not dL/dx)
The good thing about these lectures is that finally Dr.Johnson has more time to speak compared to cs231n !
1:03:00 should the dimension of grad x3 / x2 be D2 x D3?
Thank you very much! I really enjoy this lecture! Hello from Russia with love :)
19:14 Can someone explain computing the local gradient of exponential function. I mean how the result -0.2 comes? I'm lost there!!!
Our upstream gradient was -0.53 right? And now we need the local gradient of e^-x which is -e^-x and -e^-(-1)= -0.36. So upstreamgrad(-0.53) multiplied with local grad (-0.36) is 0.1949 which is approximately 0.2. So 0.2 is not local grad it is local multiplied with upstream
@@beaverknight5011 got it, thank you so much!
@@zainbaloch5541 you are welcome, good luck with your work
I don't think beaver's answer is quite right. The upstream gradient is -0.53. But the local gradient comes from the function e^x not e^-x. The derivative of e^x is e^x. Now we plug in the input which is -1 and we get e^-1 as the local gradient. This is approximately 0.37. Now that we have the local gradient we just multiply it with the upstream gradient -0.53 which results in approximately -0.20.
Great lecture thank you. I have a question, would be great if anyone could clarify. When you first introduce vector valued backpropagation, you have the example showing 2 inputs to the node, each input is a vector of DIFFERENT dimension - when would this be the case in a real scenario? I thought the vector formulation was so that we could compute the gradient for a batch of data (e.g. 100 training points) rather than running backprop 100x. In that case the input vectors and output vectors would always be of the same dimension (100). Thanks!
19:00 where did w2 come from?
Can anyone explain 1:08:05? dL/dx1 should be next to dL/dL, not L when it is subject to function f2'. Thereby back propagation cannot connect fs and f's.
Amazing courses!
46:16, shouldn't dl/dx be 4, 0, 5, 9 instead of 4, 0, 5, 0?
No, the operation is not relu, its calculation of the downstream gradient. since last row of jacobian is 0 meaning that changes in that value does not affect the output, so 0.
For the point of passing or killing the value of the upstream matrix, you have to decide pass or kill by looking at the input matrix, here that is [ 1, -2, 3, -1] so looking at -1, we will kill that value from the upstream matrix, so 0.
this is extremly hard. but this is a great lecture for sure. you are awesome Mr Johnson
Top notch content
Beautifully done!
I finally understand backprop!
Masterpiece!!!!
Can anyone clarify the computation of hessian matrix in detail ?
54:51 when my cd player gets stuck on a old eminem track
great video.thanks for posting it
THANK YOU SO MUCH! finally not shallow and excellent explanation.
Why is he calculating derivatives relative to the inputs?
Amazing!
45:00 Jacobean matrix does not have to be diagonal right?
Correct. That was unique to the ReLU function.
The "local gradient slices" in his discussion at 53:00 are slices of a more complex Jacobian.
👍👍👍👍
thanks, good god, best wish to you.
Awesome explanation of Backpropagation! Amazing slides! Much better than CS231n.
terrible sound quality !
It is actually muchhhhhh more simpler than the way he used to explain. I believe he was redundant and too many symbols that hides the beauty of the underneath reason of the algorithm and the math behind it. It all could have been explained in less amount of time.
do you have any resources that explain this better?
it seems the coughing guy got the china virus at that time
I'm pretty sure you'd be called out as racist back in the days of your comment. Now that it's almost proven to be a china virus...