Part 3 will be on backpropagation. I had originally planned to include it here, but the more I wanted to dig into a proper walk-through for what it's really doing, the more deserving it became of its own video. Stay tuned!
I'm only 12 minutes into this video right now, but I just wanted to say how much I appreciate the time and spacing you give to explaining a concept. You add pauses, you repeat things with slightly different wording, and you give examples and zoom in and out, linking to relevant thought processes that might help trigger an "a-ha" moment in the viewer. Many of these "hooks" actually make me understand concepts I've had trouble grasping in Maths, all because of your videos and the way you choose to explain things. So thanks! You're helping me a lot to become a smarter person. :)
Not only the videos themselves are great on this channel but the lists of the supporting materials are amazing too! Drives me down a breathtaking rabbit hole every time! Thank you!
I'm an IT student, and we have an Assignment on exactly this topic. We even have to use the MNIST data set. I have to say, this is absolutely lifesaving and I can not thank you enough Grant. What you do here is something that only a handful of people on this planet can do, explain and visualize rather complicated topics beautifully and simple. So from me and A LOT of students all around the globe, thank you so so much
Yes, it is just extremely good, in an objective way. He is brilliant at it, and spends a lot of time on each video. If there is an explanation of something by 3blue1brown, you will not find anything explaining it nearly as good.
Unlike most teachers of subjects like this, this gentleman seems to be genuinely concerned that his audience understands him, and he makes a concerted and highly successful effort to convey the ideas in a cogent, digestible and stimulating form.
@@redflipper992 I don't think you understand what concerted means. Stop trying to act smart and think you're better than everyone here. Be humble. You are irrelevant in the big picture.
@@tehbonehead No-one is. We're trapped in three dimensions. That's why you could never imagine what a 4-dimensional cube looks like. Making a 4-dimensional projection of a 13000-dimensional object isn't significantly better than 3 dimensions. If you meant to be humorous I hope someone gets a chuckle, because I didn't. Then your effort won't be in vain. Have a nice day 🙂
my professor has explained this in 3 lectures for about 6-7 hours. 3B1B explained it in 30 mins and it is much more clearer. I can now visualize and understand the what/why/how behind the basic deep learning algorithms. Really appreciate it!!!
I just sat through a 3 day ML accelerator class and you series did a far better job at explaining them with 4 twenty minute videos. Well done mate. Really appreciate it. Thank you
this youtuber is the best in maths and engineering in general i have never been so astounded for how easy learning machine learning can be, without having to take in bunch of complex topics that doesnt add to the discussion. Like most of the courses try to make you understand various different complex topics and by the time you finished it, you will ahve forgotten mostly about machine learning. Why dont you just explain the catch for each concept then allow us learn it in depth afterwards like these channels only explaining the concepts with both ease of learning and depths are the best.
I love you so much. I'm taking multivariate calculus and I'm doing some neural network work right now, and none of my teachers have the passion nor the capability to teach as well as you. You help me keep my passion for learning alive
I think what puts this material apart from the competition is the authors intuition of the focal points where the audience might loose the plot. Then he takes a patient and systematic turn to reiterate what have been learned so far to reinforce the basics to decrease the cognitive leap needed to grasp the next step. This ability is in my experience pretty unique.
That end comment with Lisha Li really points out how important it is to put a lot of effort into gathering and creating good and structured data sets. I know it's cliché to state "garbage in, garbage out", but these findings put very precise context and weight to this particular issue.
Some thoughts on the results: 1. 14:01 The weights for the first layer _seem_ to be meaningless patterns when viewed individually, but combined, they do encode some kind of sophisticated pattern detection. That particular pattern detection isn't uniquely specified or constrained by this particular set of weights on the first layer; rather, there are infinitely many ways that the pattern detection scheme can be encoded in the weights of this single layer. These infinite other solutions can be thought of as the set of matrices that are row-equivalent to the 16x700ish matrix where each row is the set of weights for each of the neurons on this layer. You can rewrite each of the rows as a linear combination of the set of current rows, while possibly still preserving the behavior of the whole NN by performing a related transformation to the weights of the next layer. In this way, you can rewrite the patterns of the first layer to try and find an arrangement that tells you something about the reasoning. Row reduction in particular might produce interesting results! 2. 15:10 I think I understand the reason why your NN produces a confident result - it's because it was never trained to understand what a number _doesn't_ look like. All of the training data, from what I can tell, is numbers associated with 100% confident outputs. You'd want to train it on illegible handwriting, noise, whatever you expect to feed it later, with a result vector that can be interpreted as 0% confidence, by having small equal weights, having all weights to zero, or maybe an additional neuron that the NN uses to report "no number".
Quite hard to read your comment, but it seems that we have the same idea: the neural network _is_ detecting smaller elements like "edges and loops" (as video author puts it), assuming those pictures 14:01 are of the actual results. The next layer then starts combining these elements, and it seems that if you stare at it long enough you can almost start guessing what it might be doing, like combining row 1 column 2 = strong + row 3 column 1 = strong + row 3 column 4 = strong + row 2 column 4 = weak + row 1 column 4 = weak, you might be going toward an 8 on those counts, and depending on some other combination of pattern strengths it might be a 6 or 9 if there is absence of signal upper/right or lower/left given by some of those patterns which are matched with the input. This is almost certainly not accurate as an example, but it seems to be the theme of how it works. 16:05 _"... picking up on edges and ... not at all what the network is doing."_ This statement in the video seems wrong. P.S. If this first part above is true, then the neural network might be capable of drawing a 5 (15:39). You just have to extract that answer in the way that it is in there, which is a bit more involved that following it's normal operation for which it is built. If you look into what combination of patterns from the first layer output, in what strengths, leads to a number (5 for example), than you could superimpose those patterns unto each other, and that would be what this neural network thinks is that number. It shouldn't be too hard to write a function to the already trained network, to draw this out.
Math courses in my college are basically trash compared to your videos , finally now I understand how math is being applied in computer science . Thank you so much for teaching in such an illustrative way .
After watching your first video, I ended up drawing a "mock" neural network up on paper that would work on a 3x3 grid (after all what else are you supposed to do during a boring lecture class?). It was supposed to recognize boxes, x's, sevens, simple shapes, and I defined the 7 or so neurons that I thought it might need by hand. I did all the weighted sums and sigmoid functions on paper with calculator in hand. It took maybe an hour and a half to get everything straight but once I did, it worked. It guessed with fairly good accuracy that the little seven I "inputted" was a little seven. All that excitement because of your video. Later that evening and the next one, I tried to program the same function taking PNGs as inputs and definitions of the neurons and it honestly was only a little more rewarding. But now that I see what the hidden neurons *actually* look like, I only want to learn so much more. I expected the patterns to be messy, but I was really surprised to see that it really does almost look like just noise. Thank you for making these videos. I find myself suddenly motivated to go back to calculus class tomorrow and continue our less on gradients. There's just so much out there to learn and it's educators like you that are making it easier for curious individuals like me to get there.
That's so cool, thanks for sharing! I didn't expect anyone to actually go an play with it by hand, but simplifying down to a 3x3 grid seems really smart. Stay curious!
Cant believe you explained this so easily. I thought it would take me ages to wrap my head around what neural networks basically are. This is truly amazing explanation!
Bruh there's literally Reinforcement Learning where the Network is tortured by the researchers when it gets a wrong answer and the torture continues until it gets the right answer lol
Now it's easier to explain. He couldn't have made a video like this back then, both because RUclips didn't exist, and all the relevant stuff would be in technical papers...
The neural networks themselves generally feel the same, so at least we won't be alone in that sentiment. Assuming thinking rocks and metal count as good companionship at any rate.
When I watch your videos, all I want to do is keep going. Thanks for motivating me, as well as others, with your amazing content. I really appreciate it.
Who writes any more??? That horse left the barn a long time ago. Besides, we are no longer doctors. Courtesy of insurance companies we are "providers". (The same strategy of devalued identities has long been used by invading armies to anonymize those being conquered, an apropos metaphor.)
@@johnbarbuto5387 not courtesy of insurance companies; courtesy of the fact that healthcare needs to be paid for. State systems are also payer systems.
This man is a living legend. I really sincerely believe he's one of the best "explainers" for many complex mathematical topics. I found your channel because of linear algebra, and now I'm relieved whenever I search for a topic and see one of your videos. You truly are the master of your league
for me it is. step 3b: make sure you are ready. you need to be 100% focused. step 1b: Watch it critically, be sure not to strengthen your miss believes. If it seems simpel and obvious I am probably misunderstanding it.
You sir deserve a noble, or an oscar...what an incredibly effective method of teaching. thank you so very much!!! NO! BAD COMPUTER! made me crack up! lol
Perfect Gradient Descent explaiantion Best one indeed ! 12:30 Training the Neural Nets with 13,000 weights, MNIST data set Image classification case study.
Absolutely love your videos! I'm trying to show this video to my friends who doesn't know English so well, and I decided to make subtitles. Hope you approve them, I really want to spread word about your work.
Yes it is, because the dataset is veerry clean and CNN through Keras is very easy to implement, though you need to have huge background in math and code-writing (I'm a statistician so I have a little bit of both) :)
Hi 3B1B, as a technical physics student, beeing interested into this topic for quite a while now, and also enjoying your content for quit a while, i really wanna thank you for not only going into this topic particulary, but also for educating a relatively large audience with your informative videos and improving the interest into mathematical sciences for a lot of people including me, which is pretty important in my opinion! Keep it up!!
I saw this video when i know nothing and i had a lots of intuitions, I'm rewatching after studied a lot more and I'm still learning a lot. You're a great teacher
Once I graduate and start working, I’m gonna send you the money I owe you for watching all these videos. I’m doing BSEE for control systems so hopefully it works out.
You are the best science teacher I have every seen. If anyone upstairs is serious about our education system they should use your videos as baseline for how to teach properly; you never use a term that wasn't clearly defined prior, you use analogies perfectly, and you tie new technical info back to the original concept thereby making sense how the new info fits together in the larger picture. If my high-school and college teachers were like you I would have done infinitely better at school.
Hi 3b1b i love your vids and they are one of the reasons why I know so much of how the universe works. Your channel inspires me to know more and you show the beauty of all of it. Please dont ever stop making these videos, you are making so many lives greater.
Hello....I would like to learn how to make these animations....I teach math, particularly statistics at the graduate level, and I find the way you make numbers illustrate the idea actually helps understand the flow of parameters in question in a much better manner . Do give me certain leads so that I can pick this up. I shouldn't have problems programming once I know which platform to work with
@Amagic potato, For the world of me, I cannot possibly figure out how watching 3blue1brown's videos helped you attain that enlightenment about How The Universe Works...
you know something? your channel is the best math teaching I have ever laid eyes upon. it is brilliant. beyond amazing. everything is meticulously choreographed to perfection. I WISH I could have learned math from someone like you when I was younger, and I am so happy I get to have the occasional brush with your videos. they are sublime.
Would just take a moment here to appreciate the sheer brilliance of Grant on this series. I would not have reached a decent level of NN w/o these explanations and so would so many other people. Single best series on NN / Math out there in general.
after looking though many udemy, orelilly and other youtube videos finally found this one - beginner friendly but on a profound enough level, explained in a comprehensible way, that does not lose you in the middle because it jumps from abc to hard concept - this channel is a gem. thank you!
Pause and ponder?! Every 10 seconds I stop for a minute of thinking! And on all of your videos! This time I had a time I thought for like half an hour.
+Patrik Banek it did help me ! But I was already familiar with these concepts. If you don't understand, watch the video again and look for different sources of explanation it will help you :)
+Patrik Banek You're welcome :) If you specifically don't get how the gradient can help reduce the error, you should learn what is the point of a derivative in a simple variable function and dig into multivariable calculus and optimization !
I'm doing my Masters in applied Econ right now and we briefly went over Neural Networks in my advanced econometrics class. Some of the terminology was a bit different and I felt like I could understand it decently in office hours with my professor, but this was still a great resource to solidify my understanding of the concept. (Also we looked at the MNIST dataset in class too) :D
At the 16 minute mark, I got sucker punched. After having gone through this and the previous video on machine learning and just loving how an art student like myself is enjoying math such as this and feeling like I'm making some progress..... You tell me that this is all stuff that was figured out in the 80s and I'm like...... Oh Come On! Lol!
@Dark Aether What do you even mean by that? Right now, we are living in a moment in which scientific knowledge is being acquired at the fastest rate we have ever seen. The number of active scientists right now, as you might expect, is the largest in history. Now, if you are talking about "raw" intelligence... well, I'm pretty sure evolving into creatures with a noticeable higher intelligence is going to take a long, long, long time.
"Our growth mindset is reflected when we think always if we can do better!" You are always awesome. Thanks for all the informative videos that imparts a lot of fundamental knowledge to people like me.
Once you learn more math it will be meaningful. I guess if you dont understand this video then something is missing in your calculus and/or linear algebra
I am so happy I discovered this today. I ignored all (well, most) of the math, and I still came away with a pretty solid understanding of how it works. Your explanations are ridiculously clear; you have a gift.
@@MOHANKUMARAPGPBatch No. Calclus can always be applied and your idea of a frequency is horrible since it can easily be represented by many other methods like time or transforming it. And anyway, that's not how a biological machine works. The "decimals" in math serve no real purpose in reality cause everything in our universe is quantumized.
@@nullbeyondo still the time representation will not be continuous since the irrational values will not be included in the domain. I think you should read more about it. A lot more.
Thanks a lot, at 2:45 AM in the morning, sitting peacefully in in the hostel of my institute, you actually cleared a lot of things up in the first two videos. This is the first time I understood to some extent what gradient descents and weights and cost functions were all about. Looking forward to continuing this journey of learning on your awesome channel 😃
Suprised u pulled this off real well. I am 26 years old and working in google for a 1.3 crore job in IT. I am about to be a millionaire all because of u . Like seriously u are the most helpfull person in my life.
That XKCD comic is how a lot of people consider neural networks to work :P Great video again, 3B1b! Edit: Waiting eagerly for your ConvNets and LSTMs :D
Humans have a habit of seeing images in random data, such as clouds, craters on the Moon or Mars or hearing voices in random radio static. Is this similar to identifying a 5 in a field of random data?
well for human you see things from the shape or outline that looks like a particular object, kinda like the "See a digit by the loops and lines it has" ideal for this system. So it's all notthe same as that of picking a number and being very confident about it from a static.
But is that what our brain is doing? Is it looking at a static, or are our neurons going .... ok, straight line... then round edge... another round edge..... hmmm that looks like the other 5s I seen... than triggering memory banks to look for other 5s. and again compare.
No. As explained in the video the network been (more) punished for providing multiple answers than single output wrong answers. That means a multiple answer does not exists as an option for the trained network, i.e. the set of output pattern it has been trains to respond with does not contain multiple choices. That is to say, the alternative answer "I dont know" or "maybe this or that" does not exists for the networks as an answer. Regarding clouds or craters, this is not "random data", the shapes we recognize are real and can be agreed upon to exists. This is not the case with noise, i.e. random data. Per definition random data does not not contains pattern and that is why noise carries no meaning to our brains. Regarding hearing voices in random static, I would suspect you only would hear voices if there is a pattern (signal) of some form which the brain pics up on and tries to make sense of. How prone you are to hear an actual voices might depend on how your brain be trained, i.e. biased, to detect voices (for instance if you believe one can communicate with ghosts you might be more prone to hear voices were others hears none). Because in the end, detecting meaning, i.e. label stuff, is all about being biased towards a certain interpretation of reality. So to conclude, the "reality" for the neural network in the video is biased, or limited, towards a singe neuron output and anything it "perceives" will get a response as such. However, human brains are a little bit more complex and biased differently, i.e. wired up in unique ways, which makes up for the diversity in believes and reasoning among people.
@@seditt5146 Not likely. Nobody knows. Brain s truly bizarre and the connections between cells are actually differently wired. No ask yourself: how does a brain know it exist, how are feelings developed etc. Brain is still a mystery.
i found your video on jacobians on khan academy at first i was like i don't know this new person he's not sal and then i read the comments and found out it was you !! it was a pleasant surprise
What about given it just noice and make it learn that it is no digit? Could that improve the output? Maybe also other images that is no digits. It is not only what things are that we can learn from but also that they are not.
Rofl falafel, this is why we're watching it in the first place. If we could we would, honey. I'll give this a go once I learn more, right now I'm mooching off the really smart kids in the year above until I learn a bit more. This channel is really fantastic and seems to be some kind of frequency illusion phenomenon. Your essence of calculus series came right as we started actually building up intuitions for what is going on in calculus, and now this!
Very interesting! I agree you should try it. The vast majority of all possible 26x26 images look random or noisy so my thoughts are that it might learn to properly classify only a small subset of all possible noise (the kind used in the training data) and the rest will still activate random neurons.
The problem is with the problem itself, not the cost function etc. If you give it random input, the *actual* 'correct' answer, like you would get from a human, would be 'none of the above/indecipherable', not 'well I can see this much of these aspects of these different numbers in here'. So you need an additional output to match the additional input you give it. Otherwise it's like when you ask someone "okay, is Jerry's favorite color black or white?" They make a guess, and then you're like "no, it was splork, a color I just made up you big dumb idiot". The problem isn't with the guesser (or in this case, the mechanics of the network and cost function) but with the fact that the available outputs do not match the possible inputs. So one of the two requires adjustment, not something else.
+Niles Black The problem is not with the problem, it's with the training data, because it COULD output "utter garbage", it just never got rewarded for doing so.
3:253:365:21 5:36 why gradient descent is necessary (b/c closed-form solution isn't always available) 5:51 gradient descent 6:24 different starting points (initially random as weights are initially assigned randomly) could lead to different local minima 6:42 making stepsize proportional to slope helps prevents overshooting (stepsize = learning rate * derivation, so it's already there) 10:58 a correction here: a negative derivative indicates we should increase to minimize loss function, instead of decrease. 15:23
Man , it feel so good to learn everything in zero shot now.. the neural networks , gradient decent , backpropagation . I used to get frustrated with lot of challenging concepts.. cuz I did not know maths , and AI terms.. but now after learning it for year it feels worth learning. Thanks to 3Blue guy.. whatever course he touched is worth all lectures combined i can't say. Its just pure core concept with animation. Quality at par
Your videos are awesome in the truest sense of the word. Hence, I thought it worthy to mention something which was slightly confusing to me from the videos. At 18:05, I think that the concept of deep neural networks generalizing to a randomly labelled image set can be explained better. When I first saw this part of the video, I thought that shuffling of labels for images maintained the 1:1 mapping of the image classes to labels, albeit the false ones. In such a case, the learning would definitely not result to be any different than if the 1:1 mapping was correct. The part of the video in question should bring out the fact (as intended in the concerned paper) that, after random shuffling of labels, the image class to label mapping was 1:Many i.e. two dog images can have labels 'cow' and 'lion' respectively and the network would still be able to achieve very good training accuracy, showing that it is failing to learn features general to dogs but memorizing the image to labels mapping (which does pose a very potent argument to re-think generalization) Thanks for making these. :)
Thank you for pointing this out! It's fairly confusing. Quite obviously the simple act of calling things by different names would not change anything. I get the jist of what you mean but I still don't fully understand, I should probably read the paper haha!
Hey, I know this comment is one year old but you are the only one to point out the same confusion I had! I still don't quite get it, when you say 'the network would still be able to achieve very good training accuracy' do you mean it still recognizes a picture of a dog as a dog even though the labels are randomized?
I have the same question regarding the recognition of the dog picture even though the labels can be randomized to be other things like 'cow' and 'lion' in a 1 to many mapping, does it mean that the majority of the dog pictures were labelled correctly, with the exception of some labelled incorrectly? Otherwise, wouldn't the network not learn anything if most of the labels are incorrect?
"Is it... **Learning**?!" (How terrifying) -"It's just calculus" "Even worse!" I love the realism, where people would rather have a potential robot uprising than have calculus
This is the first video I found about AI that actually explains the how, instead of just dumbing everything down and leaving just as confused as before watching. Thank you!
I've watched many videos and done some reading on how neural networks work (learn), but I couldn't find a satisfactory explanation until I watched this video. Your examples, analogies, visuals... were just perfect. Thank you so much.
![The moment we stopped understanding AI [AlexNet]](/img/1.gif)
Part 3 will be on backpropagation. I had originally planned to include it here, but the more I wanted to dig into a proper walk-through for what it's really doing, the more deserving it became of its own video. Stay tuned!
Can you provide some link to the training set?
I'm quite new and I'm trying to learn this "Hello World" of NN,
Did you consider to do a part about phase functioned neural networks? Would be awesome!
As the part progresses,
we're getting closer to seeing that "lena.jpg" picture
You are doing very good job , lots of hard work behind this video , i salute your hard work thanks
So have you decided to do more of these videos? There is a line with CNNs and LSTMs in video series...
I'm only 12 minutes into this video right now, but I just wanted to say how much I appreciate the time and spacing you give to explaining a concept. You add pauses, you repeat things with slightly different wording, and you give examples and zoom in and out, linking to relevant thought processes that might help trigger an "a-ha" moment in the viewer. Many of these "hooks" actually make me understand concepts I've had trouble grasping in Maths, all because of your videos and the way you choose to explain things. So thanks! You're helping me a lot to become a smarter person. :)
Shrooblord +
yes.
Totally agree!
Absolutely agree
Couldn't agree more
One of youtube's highest quality content channels! Chapeau
Kosio Varbenov +
True!
'chapeau'
agree
Why you write "chapeau"? Hat in French. What does hat in that sentence do?????Be honest tho
Not only the videos themselves are great on this channel but the lists of the supporting materials are amazing too! Drives me down a breathtaking rabbit hole every time! Thank you!
I'm an IT student, and we have an Assignment on exactly this topic. We even have to use the MNIST data set. I have to say, this is absolutely lifesaving and I can not thank you enough Grant. What you do here is something that only a handful of people on this planet can do, explain and visualize rather complicated topics beautifully and simple. So from me and A LOT of students all around the globe, thank you so so much
Yes, it is just extremely good, in an objective way. He is brilliant at it, and spends a lot of time on each video. If there is an explanation of something by 3blue1brown, you will not find anything explaining it nearly as good.
@@vsiegel bro you guys are so much smarter than i am im jealous
@@johndough510 If you are thinking about how smart you are, you are probably smarter than you think. No worries.
@@vsiegel thanks for being so cool about it man, hope you have a good one
@@johndough510 I'm kill
Unlike most teachers of subjects like this, this gentleman seems to be genuinely concerned that his audience understands him, and he makes a concerted and highly successful effort to convey the ideas in a cogent, digestible and stimulating form.
TOTALLY agreed
concerted with whom? I don't think you understand how to use that word.
he is a true proffesor!
@@redflipper992 I don't think you understand what concerted means. Stop trying to act smart and think you're better than everyone here. Be humble. You are irrelevant in the big picture.
@@redflipper992I read that as "concerned"
"But we can do better! Growth mindset!" at 5:18 .... a wholesome intellectual i love to see it
Dissapointed you did not animate a 13000-dimensional graph. Would make things easier
No.... No it would not....
A 2-dimensional projection of a 13000-dimensional graph would probably look like a pile of garbage.
If 3Blue1Brown cannot animate a 13000-dimensional graph, then noone can.
@@mrwalter1049 You're not thinking fourth-dimensionally!!
@@tehbonehead No-one is. We're trapped in three dimensions. That's why you could never imagine what a 4-dimensional cube looks like.
Making a 4-dimensional projection of a 13000-dimensional object isn't significantly better than 3 dimensions.
If you meant to be humorous I hope someone gets a chuckle, because I didn't. Then your effort won't be in vain.
Have a nice day 🙂
my professor has explained this in 3 lectures for about 6-7 hours. 3B1B explained it in 30 mins and it is much more clearer. I can now visualize and understand the what/why/how behind the basic deep learning algorithms. Really appreciate it!!!
Same!
You are changing the world, shaping humanity. I wish you and your team, happy and peaceful life. This is a noble profession, god bless you guys.
My math career is over. Once I learned about gradient descent, it was all downhill from there.
I had a similar fate, except mine went negatively uphill.
just remember there are people in an even lower minima than you are.
Yea, but making random choices makes you eventually reach the bottom.
It's all planar vectors from here.
You just need to take a few steps back and turn your life around.
The visualizations are so great. Appreciate the thoughtfulness of these videos
I just sat through a 3 day ML accelerator class and you series did a far better job at explaining them with 4 twenty minute videos. Well done mate. Really appreciate it. Thank you
this youtuber is the best in maths and engineering in general
i have never been so astounded for how easy learning machine learning can be, without having to take in bunch of complex topics that doesnt add to the discussion.
Like most of the courses try to make you understand various different complex topics and by the time you finished it, you will ahve forgotten mostly about machine learning. Why dont you just explain the catch for each concept then allow us learn it in depth afterwards
like these channels only explaining the concepts with both ease of learning and depths are the best.
could you pin which videos are you referring to?
I love you so much. I'm taking multivariate calculus and I'm doing some neural network work right now, and none of my teachers have the passion nor the capability to teach as well as you. You help me keep my passion for learning alive
Incompetence galore.
I think what puts this material apart from the competition is the authors intuition of the focal points where the audience might loose the plot. Then he takes a patient and systematic turn to reiterate what have been learned so far to reinforce the basics to decrease the cognitive leap needed to grasp the next step. This ability is in my experience pretty unique.
That end comment with Lisha Li really points out how important it is to put a lot of effort into gathering and creating good and structured data sets. I know it's cliché to state "garbage in, garbage out", but these findings put very precise context and weight to this particular issue.
Haha, "weight".....get it?
StiffWood True. This also becomes ethically important in medical applications of AI, where poor input can create racist AI and the like.
Hands down, I have never seen anyone explain mathematics so beautifully the way he does. Kudos!
i have no words to describe how thankful i am. thank you so much for such great content.
Some thoughts on the results:
1. 14:01 The weights for the first layer _seem_ to be meaningless patterns when viewed individually, but combined, they do encode some kind of sophisticated pattern detection. That particular pattern detection isn't uniquely specified or constrained by this particular set of weights on the first layer; rather, there are infinitely many ways that the pattern detection scheme can be encoded in the weights of this single layer.
These infinite other solutions can be thought of as the set of matrices that are row-equivalent to the 16x700ish matrix where each row is the set of weights for each of the neurons on this layer. You can rewrite each of the rows as a linear combination of the set of current rows, while possibly still preserving the behavior of the whole NN by performing a related transformation to the weights of the next layer. In this way, you can rewrite the patterns of the first layer to try and find an arrangement that tells you something about the reasoning. Row reduction in particular might produce interesting results!
2. 15:10 I think I understand the reason why your NN produces a confident result - it's because it was never trained to understand what a number _doesn't_ look like. All of the training data, from what I can tell, is numbers associated with 100% confident outputs. You'd want to train it on illegible handwriting, noise, whatever you expect to feed it later, with a result vector that can be interpreted as 0% confidence, by having small equal weights, having all weights to zero, or maybe an additional neuron that the NN uses to report "no number".
2 is a painfully easy mistake to make, because it requires the human assembling the programming data to think outside the box.
Quite hard to read your comment, but it seems that we have the same idea: the neural network _is_ detecting smaller elements like "edges and loops" (as video author puts it), assuming those pictures 14:01 are of the actual results. The next layer then starts combining these elements, and it seems that if you stare at it long enough you can almost start guessing what it might be doing, like combining row 1 column 2 = strong + row 3 column 1 = strong + row 3 column 4 = strong + row 2 column 4 = weak + row 1 column 4 = weak, you might be going toward an 8 on those counts, and depending on some other combination of pattern strengths it might be a 6 or 9 if there is absence of signal upper/right or lower/left given by some of those patterns which are matched with the input. This is almost certainly not accurate as an example, but it seems to be the theme of how it works. 16:05 _"... picking up on edges and ... not at all what the network is doing."_ This statement in the video seems wrong.
P.S. If this first part above is true, then the neural network might be capable of drawing a 5 (15:39). You just have to extract that answer in the way that it is in there, which is a bit more involved that following it's normal operation for which it is built. If you look into what combination of patterns from the first layer output, in what strengths, leads to a number (5 for example), than you could superimpose those patterns unto each other, and that would be what this neural network thinks is that number. It shouldn't be too hard to write a function to the already trained network, to draw this out.
3:38 you missed the chance of using the meme *"AI: I've found an output, but at what cost?"*
Math courses in my college are basically trash compared to your videos , finally now I understand how math is being applied in computer science . Thank you so much for teaching in such an illustrative way .
After watching your first video, I ended up drawing a "mock" neural network up on paper that would work on a 3x3 grid (after all what else are you supposed to do during a boring lecture class?). It was supposed to recognize boxes, x's, sevens, simple shapes, and I defined the 7 or so neurons that I thought it might need by hand. I did all the weighted sums and sigmoid functions on paper with calculator in hand. It took maybe an hour and a half to get everything straight but once I did, it worked. It guessed with fairly good accuracy that the little seven I "inputted" was a little seven. All that excitement because of your video.
Later that evening and the next one, I tried to program the same function taking PNGs as inputs and definitions of the neurons and it honestly was only a little more rewarding. But now that I see what the hidden neurons *actually* look like, I only want to learn so much more. I expected the patterns to be messy, but I was really surprised to see that it really does almost look like just noise.
Thank you for making these videos. I find myself suddenly motivated to go back to calculus class tomorrow and continue our less on gradients. There's just so much out there to learn and it's educators like you that are making it easier for curious individuals like me to get there.
That's so cool, thanks for sharing! I didn't expect anyone to actually go an play with it by hand, but simplifying down to a 3x3 grid seems really smart. Stay curious!
Nyhilo +
3Blue1Brown +
Were you able to do all of that by watching this video series alone? Please can you add other resources you referred? Thanks!
.....go anD play... AND AN+D
how can anyone dislike these videos? very detailed, accurate explanations and cool animations. Keep up the good work!!
stupids are everywhere.
Cant believe you explained this so easily. I thought it would take me ages to wrap my head around what neural networks basically are. This is truly amazing explanation!
Did anybody else feel bad for the network after he called the output utter trash?😢
I know, especially with those sad computer eyes. Tragic
Yes, specially when he called it bad computer :(
we are all utter trash for future robots
Bruh there's literally Reinforcement Learning where the Network is tortured by the researchers when it gets a wrong answer and the torture continues until it gets the right answer lol
Lmao.
I wish someone would have introduced this to me at a young age back in the 90s. I had no idea neural network have existed for so long
Now it's easier to explain. He couldn't have made a video like this back then, both because RUclips didn't exist, and all the relevant stuff would be in technical papers...
@@lopezb Also, it was a really niche field that didn't show that much promise.
Also, from my understanding, modern neural networks are very different from the one in the 90s
It's a college graduate course.
In the 90s NN fell into disfavor.
Probably, even a 5 year old would master deep learning when taught in this way. What a video man ! Awesomeeeeeeeee
How confident I feel in my current knowledge of neural networks:
15:41 - *"Uh...I'm really more of a multiple choice kinda guy."*
The neural networks themselves generally feel the same, so at least we won't be alone in that sentiment.
Assuming thinking rocks and metal count as good companionship at any rate.
When I watch your videos, all I want to do is keep going. Thanks for motivating me, as well as others, with your amazing content. I really appreciate it.
This video is truly a work of art. The animations are mesmerizing. Thank you for all your work, Grant!
We need AI to recognise what the doctors write
whoever designs the algorithm and engineers the software deserves a nobel peace prize
@@frankchen4229 LOL
Who writes any more??? That horse left the barn a long time ago. Besides, we are no longer doctors. Courtesy of insurance companies we are "providers". (The same strategy of devalued identities has long been used by invading armies to anonymize those being conquered, an apropos metaphor.)
@@johnbarbuto5387 why so mad
@@johnbarbuto5387 not courtesy of insurance companies; courtesy of the fact that healthcare needs to be paid for. State systems are also payer systems.
"I'm more of a multiple choice guy" LOL
This man is a living legend. I really sincerely believe he's one of the best "explainers" for many complex mathematical topics. I found your channel because of linear algebra, and now I'm relieved whenever I search for a topic and see one of your videos. You truly are the master of your league
Watching 3b1b
Step 3b. Drop Everything
Step 1b. Watch religiously.
oh the accuracy of this comment
My god I relate
for me it is.
step 3b: make sure you are ready. you need to be 100% focused.
step 1b: Watch it critically, be sure not to strengthen your miss believes. If it seems simpel and obvious I am probably misunderstanding it.
+
counting level: Parker
You sir deserve a noble, or an oscar...what an incredibly effective method of teaching. thank you so very much!!!
NO! BAD COMPUTER! made me crack up! lol
A balon di or too
@@shakhaoathossain5032 XD good one.
@@shakhaoathossain5032 add a grammy in there too hahaha
The fact it was recorded digitally meant he said that to a real computer.😭
AND SO DID YOU!!! 😠
Perfect Gradient Descent explaiantion Best one indeed ! 12:30 Training the Neural Nets with 13,000 weights, MNIST data set Image classification case study.
Absolutely love your videos! I'm trying to show this video to my friends who doesn't know English so well, and I decided to make subtitles. Hope you approve them, I really want to spread word about your work.
So uhhhh, did you just say that this was the "Hello World!" of neural networking.....
@Winston Mcgee "p r e t t y m u c h i t"
Well, the data are the hello world, because it is the first image set that people take to test or learn about an algorithm.
Because you don't need all those knowledge to build a network that can read digits. It's easy with Keras even my grandmother can do it
Yes it is, because the dataset is veerry clean and CNN through Keras is very easy to implement, though you need to have huge background in math and code-writing (I'm a statistician so I have a little bit of both) :)
Use a package. People already did the hard stuff, you just have to call their function / use their models
I’m a grad student in physics and i wanted to thank you. It is insane to find such good videos on such advanced subjects !
The graphics of this video is absolutely stunning! Thank you for your work ♡
Hi 3B1B, as a technical physics student, beeing interested into this topic for quite a while now, and also enjoying your content for quit a while, i really wanna thank you for not only going into this topic particulary, but also for educating a relatively large audience with your informative videos and improving the interest into mathematical sciences for a lot of people including me, which is pretty important in my opinion! Keep it up!!
Keep coming back to this series and sharing it with so many people. This whole channel is truly a gift. Thank you so much for making these!
"It's actually just calculus."
"Even worse!"
i'm dead
Actually, AI just lie to people of the world
so far
calculus is the easiest form of math
@@the.abhiram.r
I wish you are joking
This is mindblowingly interesting and extremely well explained. Thank you!
I saw this video when i know nothing and i had a lots of intuitions, I'm rewatching after studied a lot more and I'm still learning a lot. You're a great teacher
Once I graduate and start working, I’m gonna send you the money I owe you for watching all these videos. I’m doing BSEE for control systems so hopefully it works out.
You explained in less than 4 minutes something that took me several days to understand from other sources. You are awesome!
You are the best science teacher I have every seen. If anyone upstairs is serious about our education system they should use your videos as baseline for how to teach properly; you never use a term that wasn't clearly defined prior, you use analogies perfectly, and you tie new technical info back to the original concept thereby making sense how the new info fits together in the larger picture. If my high-school and college teachers were like you I would have done infinitely better at school.
Hi 3b1b i love your vids and they are one of the reasons why I know so much of how the universe works. Your channel inspires me to know more and you show the beauty of all of it. Please dont ever stop making these videos, you are making so many lives greater.
Thanks so much Amagic, I'll do my best.
Hello....I would like to learn how to make these animations....I teach math, particularly statistics at the graduate level, and I find the way you make numbers illustrate the idea actually helps understand the flow of parameters in question in a much better manner . Do give me certain leads so that I can pick this up. I shouldn't have problems programming once I know which platform to work with
@Amagic potato, For the world of me, I cannot possibly figure out how watching 3blue1brown's videos helped you attain that enlightenment about How The Universe Works...
+charlie dexter
He has all of the code for his animations (written in python) here: github.com/3b1b/manim
Your videos with such wonderful LaTeX animations are just as high level as a BBC awarded documentaries. Very impressive to say the least.
I agree. I need to learn more about this LaTex animations
you know something? your channel is the best math teaching I have ever laid eyes upon.
it is brilliant. beyond amazing.
everything is meticulously choreographed to perfection.
I WISH I could have learned math from someone like you when I was younger, and I am so happy I get to have the occasional brush with your videos. they are sublime.
Thanks for the part 2, I was waiting for it impatiently.
Would just take a moment here to appreciate the sheer brilliance of Grant on this series. I would not have reached a decent level of NN w/o these explanations and so would so many other people. Single best series on NN / Math out there in general.
after looking though many udemy, orelilly and other youtube videos finally found this one - beginner friendly but on a profound enough level, explained in a comprehensible way, that does not lose you in the middle because it jumps from abc to hard concept - this channel is a gem. thank you!
I love how he has people like Desmos and Markus Persson supporting him on patreon
WOO BEEN WAITING FOR THIS
Possibly one of the best mini-courses on ML anywhere. Clearly explained concepts, beautiful post-production. kudos
Love the eye/ pi animations! :) Gets me smiling amidst all the complexities of Neural Networks 😲
Pause and ponder?! Every 10 seconds I stop for a minute of thinking! And on all of your videos!
This time I had a time I thought for like half an hour.
This is simply the best channel on RUclips
great explanation, thank you:)
did you honestly understand any of this did this video honestly help you comprehend something
+Patrik Banek it did help me ! But I was already familiar with these concepts. If you don't understand, watch the video again and look for different sources of explanation it will help you :)
Okay thanks for the advice
+Patrik Banek You're welcome :) If you specifically don't get how the gradient can help reduce the error, you should learn what is the point of a derivative in a simple variable function and dig into multivariable calculus and optimization !
@ 3:39
Wow... I didn't realize 3B1B likes to bully neural networks. That was ruthless.
Yes, it's a jungle in there.
@- RedBlazerFlame -
' Oversimplified' reference
There's a training method called Reinforcement Learning where you literally torture the Network when it gets the wrong output lol
I can't describe the moment of pure joy I had when I noticed you put the label of "tease" on Fermat at 18:05
I'm doing my Masters in applied Econ right now and we briefly went over Neural Networks in my advanced econometrics class. Some of the terminology was a bit different and I felt like I could understand it decently in office hours with my professor, but this was still a great resource to solidify my understanding of the concept. (Also we looked at the MNIST dataset in class too) :D
At the 16 minute mark, I got sucker punched. After having gone through this and the previous video on machine learning and just loving how an art student like myself is enjoying math such as this and feeling like I'm making some progress..... You tell me that this is all stuff that was figured out in the 80s and I'm like...... Oh Come On! Lol!
Honestly, the 80s is actually very recent compared to other stuff. In math undergrad I was learning concepts from the 1800s lmao
@@apuapustaja2047 yup! Math is older than all of us... it's our discoveries that are "new"~
Honestly, don’t feel discouraged. General Relativity was formulated almost a century ago, but that doesn’t mean it is easier to understand.
@Dark Aether some definitely could say that~
@Dark Aether What do you even mean by that? Right now, we are living in a moment in which scientific knowledge is being acquired at the fastest rate we have ever seen. The number of active scientists right now, as you might expect, is the largest in history. Now, if you are talking about "raw" intelligence... well, I'm pretty sure evolving into creatures with a noticeable higher intelligence is going to take a long, long, long time.
"Our growth mindset is reflected when we think always if we can do better!" You are always awesome. Thanks for all the informative videos that imparts a lot of fundamental knowledge to people like me.
I was looking forward to this video all week.
Me in part 1: Ah I think I'm starting to understand this whole thing.
Me in part 2: Nevermind...
excatly
Once you learn more math it will be meaningful. I guess if you dont understand this video then something is missing in your calculus and/or linear algebra
Rewatch it 3 times. I did
I am so happy I discovered this today. I ignored all (well, most) of the math, and I still came away with a pretty solid understanding of how it works. Your explanations are ridiculously clear; you have a gift.
Pretty sure at 11:03 the weights should either start at w1 or end at w13,001.
Gah! Good catch.
This just goes to show that off-by-one errors really do happen to the best ;)
+thesecondislander
Well, next to cache invalidation and naming things it's one of the two big problems in computer science :P
It gained weight.
So my little pony, whats the appeal?
15:19 seems interessting, just like you have to train your own (biological) NN to draw a human face, although you saw millions of them
Hakim R thats a very good analogy
This might be relevant: look up general adversarial networks
This is, like, literally the most important video on the internet.
3:40 "NO! Bad computer!"
AJ Nelson Bad boy!
I felt so sorry for the computer...
"To say that more mathematically..."
i heard eric cartman say it.
+
This series is fantastic and just the right level of being complex but understandable. Thanks!
When something is amazing, it will look like
with your work. Even your bg voice is totally understandable, legit and direct
10:10 biological neurons are continuous-valued as well: their firing frequency varies.
still, the frequency cannot be decimal right ? so its still discrete input where calculus cannot be applied.....
@@MOHANKUMARAPGPBatch No. Calclus can always be applied and your idea of a frequency is horrible since it can easily be represented by many other methods like time or transforming it. And anyway, that's not how a biological machine works. The "decimals" in math serve no real purpose in reality cause everything in our universe is quantumized.
@@nullbeyondo still the time representation will not be continuous since the irrational values will not be included in the domain. I think you should read more about it. A lot more.
BEST VISUAL THERAPY IN MY LIFE
Thanks a lot, at 2:45 AM in the morning, sitting peacefully in in the hostel of my institute, you actually cleared a lot of things up in the first two videos. This is the first time I understood to some extent what gradient descents and weights and cost functions were all about. Looking forward to continuing this journey of learning on your awesome channel 😃
Draw a 5 for me.
"I am a more of a multiple choice guy"
Could you please just drop everything else you are doing and do these videos full time for the rest of your life.
But his videos are a result of his other interests. XD
@@iLoveTurtlesHaha Just like you saying 'XD' is......
Sadly people still need to work to fricking live.
@@thibauldnuyten2891 Wouldn't he be rich off these videos though? I mean they all have millions of views and he almost has 4m subs
Suprised u pulled this off real well. I am 26 years old and working in google for a 1.3 crore job in IT. I am about to be a millionaire all because of u . Like seriously u are the most helpfull person in my life.
3:41 Oh that poor little network... Say something good to it so it's happy again!
That XKCD comic is how a lot of people consider neural networks to work :P
Great video again, 3B1b!
Edit: Waiting eagerly for your ConvNets and LSTMs :D
And it certainly isn't wrong btw
You do such a great job at explaining complex topics in a "simple" way, great job
Humans have a habit of seeing images in random data, such as clouds, craters on the Moon or Mars or hearing voices in random radio static. Is this similar to identifying a 5 in a field of random data?
well for human you see things from the shape or outline that looks like a particular object, kinda like the "See a digit by the loops and lines it has" ideal for this system. So it's all notthe same as that of picking a number and being very confident about it from a static.
But is that what our brain is doing? Is it looking at a static, or are our neurons going .... ok, straight line... then round edge... another round edge..... hmmm that looks like the other 5s I seen... than triggering memory banks to look for other 5s. and again compare.
No. As explained in the video the network been (more) punished for providing multiple answers than single output wrong answers. That means a multiple answer does not exists as an option for the trained network, i.e. the set of output pattern it has been trains to respond with does not contain multiple choices. That is to say, the alternative answer "I dont know" or "maybe this or that" does not exists for the networks as an answer.
Regarding clouds or craters, this is not "random data", the shapes we recognize are real and can be agreed upon to exists. This is not the case with noise, i.e. random data. Per definition random data does not not contains pattern and that is why noise carries no meaning to our brains.
Regarding hearing voices in random static, I would suspect you only would hear voices if there is a pattern (signal) of some form which the brain pics up on and tries to make sense of. How prone you are to hear an actual voices might depend on how your brain be trained, i.e. biased, to detect voices (for instance if you believe one can communicate with ghosts you might be more prone to hear voices were others hears none). Because in the end, detecting meaning, i.e. label stuff, is all about being biased towards a certain interpretation of reality.
So to conclude, the "reality" for the neural network in the video is biased, or limited, towards a singe neuron output and anything it "perceives" will get a response as such. However, human brains are a little bit more complex and biased differently, i.e. wired up in unique ways, which makes up for the diversity in believes and reasoning among people.
That falls under the imagination bracket,it's more like changing the value of the pixels instead of the weights or biases to get a desired output.
@@seditt5146 Not likely. Nobody knows. Brain s truly bizarre and the connections between cells are actually differently wired. No ask yourself: how does a brain know it exist, how are feelings developed etc. Brain is still a mystery.
This is the best educational channel on RUclips by a long mile.
This is so so much better Information visualization than any of my teachers eher had! Plus your calm Voice and humour. I just Like it!
I wish I had these video when I was first learning. Damn you Tom Mitchell and your formal explanations.
i found your video on jacobians on khan academy at first i was like i don't know this new person he's not sal and then i read the comments and found out it was you !! it was a pleasant surprise
Dude you did in 20 minutes what my professor couldn't in 3 hours, Thanks a lot !
That's not true. He probably spent hours and hours to design the animations , edit the video and to write the script as smoothly as possible.
What about given it just noice and make it learn that it is no digit? Could that improve the output? Maybe also other images that is no digits. It is not only what things are that we can learn from but also that they are not.
Try it!
Rofl falafel, this is why we're watching it in the first place. If we could we would, honey. I'll give this a go once I learn more, right now I'm mooching off the really smart kids in the year above until I learn a bit more. This channel is really fantastic and seems to be some kind of frequency illusion phenomenon. Your essence of calculus series came right as we started actually building up intuitions for what is going on in calculus, and now this!
Very interesting! I agree you should try it.
The vast majority of all possible 26x26 images look random or noisy so my thoughts are that it might learn to properly classify only a small subset of all possible noise (the kind used in the training data) and the rest will still activate random neurons.
The problem is with the problem itself, not the cost function etc.
If you give it random input, the *actual* 'correct' answer, like you would get from a human, would be 'none of the above/indecipherable', not 'well I can see this much of these aspects of these different numbers in here'. So you need an additional output to match the additional input you give it.
Otherwise it's like when you ask someone "okay, is Jerry's favorite color black or white?" They make a guess, and then you're like "no, it was splork, a color I just made up you big dumb idiot".
The problem isn't with the guesser (or in this case, the mechanics of the network and cost function) but with the fact that the available outputs do not match the possible inputs. So one of the two requires adjustment, not something else.
+Niles Black
The problem is not with the problem, it's with the training data, because it COULD output "utter garbage", it just never got rewarded for doing so.
3:25 3:36 5:21
5:36 why gradient descent is necessary (b/c closed-form solution isn't always available)
5:51 gradient descent
6:24 different starting points (initially random as weights are initially assigned randomly) could lead to different local minima
6:42 making stepsize proportional to slope helps prevents overshooting (stepsize = learning rate * derivation, so it's already there)
10:58 a correction here: a negative derivative indicates we should increase to minimize loss function, instead of decrease.
15:23
Man , it feel so good to learn everything in zero shot now.. the neural networks , gradient decent , backpropagation . I used to get frustrated with lot of challenging concepts.. cuz I did not know maths , and AI terms.. but now after learning it for year it feels worth learning. Thanks to 3Blue guy.. whatever course he touched is worth all lectures combined i can't say. Its just pure core concept with animation. Quality at par
Your videos are awesome in the truest sense of the word. Hence, I thought it worthy to mention something which was slightly confusing to me from the videos.
At 18:05, I think that the concept of deep neural networks generalizing to a randomly labelled image set can be explained better. When I first saw this part of the video, I thought that shuffling of labels for images maintained the 1:1 mapping of the image classes to labels, albeit the false ones. In such a case, the learning would definitely not result to be any different than if the 1:1 mapping was correct. The part of the video in question should bring out the fact (as intended in the concerned paper) that, after random shuffling of labels, the image class to label mapping was 1:Many i.e. two dog images can have labels 'cow' and 'lion' respectively and the network would still be able to achieve very good training accuracy, showing that it is failing to learn features general to dogs but memorizing the image to labels mapping (which does pose a very potent argument to re-think generalization)
Thanks for making these. :)
+Sagar Jounkani I think you are right.
Thank you for pointing this out! It's fairly confusing. Quite obviously the simple act of calling things by different names would not change anything. I get the jist of what you mean but I still don't fully understand, I should probably read the paper haha!
Hey, I know this comment is one year old but you are the only one to point out the same confusion I had! I still don't quite get it, when you say 'the network would still be able to achieve very good training accuracy' do you mean it still recognizes a picture of a dog as a dog even though the labels are randomized?
I have the same question regarding the recognition of the dog picture even though the labels can be randomized to be other things like 'cow' and 'lion' in a 1 to many mapping, does it mean that the majority of the dog pictures were labelled correctly, with the exception of some labelled incorrectly? Otherwise, wouldn't the network not learn anything if most of the labels are incorrect?
"Is it... **Learning**?!" (How terrifying)
-"It's just calculus"
"Even worse!"
I love the realism, where people would rather have a potential robot uprising than have calculus
Change the gradient based optimization to genetic algorithm and its "evolving"
@@sagadegeminis9722 english please?
This is the first video I found about AI that actually explains the how, instead of just dumbing everything down and leaving just as confused as before watching. Thank you!
I am so happy that vsauce recommended this channel because it is amazing!
4:33 "And that's a complicated thing!"
Blues: *shuffle nervously*
I've watched many videos and done some reading on how neural networks work (learn), but I couldn't find a satisfactory explanation until I watched this video. Your examples, analogies, visuals... were just perfect. Thank you so much.
Thank you for providing us with such nice content. Keep it up.