I’m in ML since 2013 and have to say: wow… you and your team do really deserve praise for solid research and delivery. I’ll bookmark this video to point people to. Thank you
The span and depth of topics covered with a eye on technical details is truly awesome and rare. Smart commenters point out the occasional inaccuracies (understandable for the span of topics) which benefits everyone as well.
In the 80's Hinton, LeCun, Schmidhuber and others invented backpropagation, CNNs (convolutional NNs), then RNNs, LSTMs in the 90's but it was still very niche area of study with "limited potential" because NNs always performed a bit worse than other methods, until couple breakthroughs in speech recognition and image classification in the end of 00's. 2012 AlexNet brought instant hype to CNNs, which was followed by one-liners critically improving quality and stability of training: better initial values, sigmoid -> relu, dropout, normalization (forcing values to be in certain range), resnet (just adding values of previous layer to the next one). That allowed to train models so much bigger and deeper that they started to dominate everything else by sheer size. Then came Transformer in 2017 that allowed to treat basically any input as a sequence of tokens and scaling hypothesis which brought us to the present time with "small NNs" being "just" several billion parameters. Between 2012 and now also been extreme progress with hardware for running these networks, optimizing the precision (it turned out that you don't need 32bit float numbers to train/use NNs, lower possible is 1 bit, good amount is 4 bit integer which is 100x faster in hardware), new instructions, matmuls, sparsity, tensor cores and systolic arrays and what not to get truly insane speedups. For comparison, AlexNet was trained on 2 GTX580 so it was about 2.5TFLOPs of compute. This year we have ultrathin and light laptops with 120TOPs and server cards with 20000TOPs and biggest clusters are in the range of 100 000 such cards, so in total 1 billion times more compute thrown at the problem than 12 years ago. And 12 years ago it was 1000x more than at the start of the century, so, we got about a trillion times more compute to make neural networks work and we still not anywhere close to be done. Of course, early pioneers had no chance without that much compute.
I love seeing Minsky come up as I have a (tenuous) connection to him as he is my academic "great great grand advisor." In that, my PhD's advisor's PhD advisor's PhD advisor's PhD advisor was Minsky. Unfortunately, stories about him never got passed down, I only have a bunch of stories with my own advisor, and his advisor, so it is interesting seeing what he was up to.
Thanks Jon, as someone who tinkered with neural nets in the 1980s and 90s, this history connects the evolutionary dots and illuminates the evolution/genesis of those theories & tools we were working with... J
The one name missing from this from my high-school memory is Norbert Weiner, author of "Cybernetics". I do remember a circa 1980 effort of mine to understand the implication to my area of training (medicine) of rule-based AI. The Mycin program (infectious disease diagnosis and management) sited at Stanford could have been the seed crystal for a very useful application of the symbol-based methods. It wasn't maintained and expanded after its initial development. Took too long to do data input and didn't handle edge cases or apply common sense. It was, however, very good at difficult "university level specialist" problems. I interviewed Dr Shortliffe and his assessment was that AI wouldn't influence the practice of medicine for 20-30 years. I was hugely disappointed. At the age of 30 I thought it should be just around the corner. So here it is 45 years later and symbolic methods have languished. I think there needs to be one or more "symbolic layers" in the development process of neural networks. For one thing it would allow insertion of corrections and offer the possibility of analyzing the "reasoning".
Interesting that Claude Shannon's observations on the meaning of information being reducible to binary came about at virtually the same time as the early neural networks papers. Edit - The Mathematical Theory of Communication by Shannon was published in 1948. Also, Herb Simon was an incredible mind.
Thanks for bringing back memories of the class I took from Minsky and Papert (short, not long a in pronouncing his name) in 1969 just when the book had come out. You filled in some of the back story that I wasn't aware of.
Gosh, I remember studying physiology in the late 60s when human nervous system understanding was still in the relative dark ages - for instance plasticity was still unknown, and they taught us that your nerves stopped growing at a young age and that was it. But I had no idea how far they'd come with machine learning in the Perceptron - already using tuneable weighted responses simulatong neurons? Wow! If they could have licked that multilayer problem it would have sped things up quite a bit. You mentioned the old chopped up planaria trick - are you familiar with the work of Dr Miachel Levin? His team is carrying the understanding of morphogenisis to new heights - amazing stuff! Thank you kindly for your videos! Cheers.
There must have been a speech of Demis Hassabis on 14 Nov 2017 in the late morning at the Society of Neuroscience in Washington. In this Keynote lecture where he told the audience that AI is nothing more than applied Brain Science. He must have said (I only have the translated German wording) 'First we solve the problem and understand whats intelligence (possibly the more German usage of the word) and then we solve all the other problems'. The 6000-8000 People must have been extremely quiet knowing what this young man already has achieved. Unfortunately I never found the video. (Source: Manfred Spitzer).
The difficulty with Minsky's adamant focus on symbolic logic was his failure to recognize that the vast majority of biological sensory processing is dedicated to creating meaningful, logically usable symbolic representations of a complicated physical world. Minsky’s position thus was a bit like saying that once you understand cream, you have all you need to build a cow.
As always I love your videos, the depth of knowledge and the people that comment, as they all have interesting stories about what is in your videos. So one of the descendants of the symbolic movement was cognitive architectures like SOAR and ACT-R from Newell's theories of Cognition. Symbolic systems are not gone and they perform many tasks that Neural Networks don't do well. However Neural Networks do something so much better than Cognitive Systems, and that is in getting all the data and knowledge of the world into the neural network, and being able to extract it out. There is no way you can program all of that as rules in symbolic systems. There will be a merger of both systems so they can perform better reasoning and cognitive tasks in the next iteration of all of this. We are really just at the beginning, standing on the shoulders of Giants.
@@luciustarquiniuspriscus1408 MAMBA is already a valid alternative to transformers, and it is some kind of variant of linear recurrent neural networks. Also I do not see how could we avoid recurrent neural networks for music generation, they or their variants seem like a perfect fit for this very specific generation task.
@@luciustarquiniuspriscus1408 The SSM/Mamba papers already address this. In fact you can train a GPT-3 like small model using Mamba right here and now, with excellent performance (both in terms of training speed and outputs). With "infinite attention" (well, limited by the capacity of the hidden state vector).
The Einstein, Oppenheimer, Bohr, Feynman, Schroeder and Heisenbergs of a.i.. McCulloch-Pitts neuron network, Rosenblatt's training paradigm, took 70 years to get to "here" and should be acknowledged. I remember as a little kid in the 70s reading articles on different people leading the symbolic movement, and thinking "none of them really seem to know or have conviction in what they're campaigning for".
Disagree. I worked on natural language NNs in the 80s (a failure). Now I'm using what we should no longer call LLMs to do real work, replacing older specialised AIs and improving accuracy. This is for horizon scanning. We are finding valuable insights for our government.
@@endintiers it’s the credibility issue that’s the problem. I’ve stopped using LLMs altogether even for coding, because it gets things incorrect often and there are usually better solutions.
Super happy i found you again! Your content is off the charts amazing! I wish i could patreon you up- im in my 50's autistic af and i dont have an income. Appreciate you.. p.s. i thought you said Rosenblat died in a tragic coding accident! Lmfao. Love the flatworms! Keep on keeping your valuble perception turned on!!
It surprises me how "little" progress we have made in that time. Pretty much every other discipline has made incredible leaps in the past 60-70 years, yet AI is still nowhere near the human brain. Obviously an early perceptron is infinitely worse than a modern LLM, but AGI doesnt really feel any closer than back then.
@@lbgstzockt8493 if you're comparing what a few smart computer geeks did over 80 years to what mother nature did over 3-ish billion years, then I would argue it's not surprising AT ALL that we haven't simulated a human brain yet...
It's pronounced Pæpert not Pāpert, and that was a bit of a confusing way to present the XOR function: since you set it up with a XY plot, you should have put the Inputs on different axes with the values in the middle. Other than that, great video as always 😊
What a fantastic vide tracing the history of Neural Nets. It would also be really useful if you could put up the links to the papers mentioned in the video in the description.
Good summary. Ultimately, even today, there are no functionally useful and agreed definitions of intelligence and thinking. Maybe we're unintentionally approaching this from the back, by making things that sort of work, then later figuring out what's really going on (not yet!)
I took a generic algorithms and neural networks module at university. In the exam we would train and solve simple neutral networks on paper with a calculator. Good fun, this was in 2000.
I think there's one guy that doesn't receive much attention, Demis Hassabis. I knew him as the founder of the game company that made Republic: The Revolution, but he then went on to take a PhD in Neuroscience. I wondered why. Now it makes sense. He founded DeepMind
hebb's postulate are stilling being taught in neuroscience,only difference is a time element is being added its now called "spike time dependent plasticity"
Wow, didn't know they researched that back then, so long ago. Thank you for educating me on this matter. Today, AI is amazing already. I developed a USB reader/ tester with GPT4. The code that it wrote was spot on. The rest was just electronics, an amazing tool.
Very nice and well scoped contextualization about the developement of NNs ... I know that the video is about an specific branch of computer science ... but the seminal work for AI research was not Alan Turing papers ... the seminal work for AI and Computer science is George boole's The Laws of Thought (1854) which contains Boolean algebra.
@@leannevandekew1996 typical bot. trying to deflect. Your profile is AI generated and you look just like adult content bots i see all over the platform.
Thank you for this segment, and the asianometry (strange name 🙂) channel. Lots of interesting stuff. I have worked in the field of neural networks for many years, and what I find most striking is how the field has been plagued by researchers who project the most abstract brain functions onto mechanisms with negligible complexity (here Rosenblatt 's hyperboles about a couple of linear units with dynamics weights). This is bad for the image of the field (especially to the general public who finances these research) but also for the field itself where new ideas have to fight these oversimplifications to be recognized. The work by people like Stephen Gross and Stanislas Dehaene (who once in a talk at the Montreal Neuronal Institute in the 2000s likened the process of a human becoming conscient of some stimulus to a printer that turns on to print a document) are unfortunate examples. But global warming will probably make this a moot point anyway: human society's inability to manage the responsibilities that come from the technology that our brain has allowed us to develop (together with the profit-at-all-cost economic model used to exploit it) will destroy us before we can understand the organ that made it possible. What a shame...
I completed a machine learning course from Cornell a little before genAI became really popular and we had to learn how to code a basic Perceptron. The rare times I see it mentioned, I always feel like I have some special, unique insight. (I definitely do not!)
Hi! Great fan of your channel! :) However this time I'm a bit puzzled. Several years ago I read somewhere Marvin Minsky talking about how he build this (awesome in my opinion) mechanical neural network. Since that time I was sure his network was the first. However here you talk about neural network build almost decade later and call it the first one... You mentioned that Marvin Minsky did some nerual network research previously, but he left. Ok, fine, so why his neural network that was bult before perceptron is not the first one in your opinion? :) Maybe next video? :) Also - to my knowledge Turings paper was published in 1937, not 36. In year 1936 Alonso Church published his paper related to Entscheidungsproblem. I don't know who was the second one that came up with theory of gravity or relativity, we don't usually remember them. But for some reason we remember Turing for being second in something :) Just fun fact :)
There's more about this early history of artificial intelligence in this 1988 book: Graubard, S. R. (Ed.). (1988). The artificial intelligence debate: False starts, real foundations. MIT Press. The chapters also detail a very interesting discussion of related general philosophical problems and limitations of the time.
very forwards thinking people from a time when most people couldn't even comprehend computers. i know a lot of people born after this period who are only slightly older than me that can't even handle windows 10 and don't believe computers existed when they were young. and i am talking about people born in the 1970s here not boomers. yet you had these geniuses born in the late 19th century or early 20th century who made it all possible .
When I first read about N.N. around 2000 this was still the state of the matter 30 years later. When I was at university NN were no topic. Then after 2010 things suddenly changed. Training Data and Flops had become available.
I needed a video to do the dishes, after spending a day making pedagogical materials on Stable Diffusion. Now I will rewind and delight myself seeing this video carefully. *Thank you*
Before the multi-layer perceptron, statisticians used that exact same model with sigmoid activation functions and called the process "edge regression." The statisticians knew how to "train" the model using second-order multivariant optimization and "weight decay" methods, which were vastly superior to the ad hoc backpropagation methods that neural network researchers were still using as late as the 1980's. The neural net guys were blinded by their unwarranted certainty that they were onto something new.
I studied the perceptron in the '70s. My conclusion was, the hardware was not up to the task. Using a matrix of photoresistors as the input proved the design principle but couldn't bring to a working prototype.
I built a multilayer perceptron from 4096 operational amplifiers in the '80s. It was a kind of analog computer where the weighting values could be set with electronic potentiometers controlled from a digital computer, digital port, and the freely reorganizable switch matrix could be set with electronic switches, how many layers to have and in what matrix. It beat the digital machines of the time in real-time speech recognition and live speech generation. Nowadays I want to integrate this into an IC and sell it as an analog computer. We tried to model the back propagation method using the PROLOG language, but the machines at that time were very slow. It took 40 years for the speed and memory size of machines to reach the level where this could be realized. The so-called scientific paper work was very far from the practical solutions realized then and now, there is a couple of decades behind in theory, because a lot of technical results achieved in practice could not be published.
I find it ironic/devious that Minsky criticized Perceptrons for inability to scale. None of the little toy systems that came out of MIT or Yale (Schank) scaled.
I was just watching this video by Formosa TV and their piece of on the founder of SuperMicro. Just curious if there is any interest in the SuperMicro or its founder (for the English speaking folks that don't understand Mandarin hehe).... just thought I'd ask. 🙂
Wow thats strange timing I'm in the middle of reading the book " The Dream Machine by Mitchell Waldrop" Have you read that yet ? Its about these exact same people
Beyond the usual kudos to Jon and his research, the primary reason much of this area has, and will continue to come up short of what a human brain can do has to do with scale and complexity. The human brain is WAY more complex than even the combination of the current LLM systems. Sitting at over 50,000 cells and 150,000,000 synaptic connections per cubic mm (Science: A petavoxel fragment of human cerebral cortex reconstructed at nanoscale resolution), and using less than 300W/hr. of energy per day, the brain is a formidable piece of wetware. With advances of computer hardware and software we'll get there some day. But, not for a long time. Until then, remember that all the current LLMs are 100% artificial and 0% intelligent.
I see what you do there. 3030 year: History of AI essey: As we know, our new hyperdeepinnercurlingdoubleflashing neural network almost working, we need few more tiny touches and literally 2 extra layers to show it awesomeness in next 3031year. And again, and again.
It is not enough to digitally model the most common LLM for Artificial Intelligence today, it doesn't matter if it is 1-bit or 1 TRIT 1.58b = log2(3), it has to be done with working ANALOG hardware! If software, then Machine Learning (algorithm) If hardware, then Learning Machine (Hardware that is better and faster than an algorithm)
When I eat pulled pork I gain none of the pig’s memories. I do however gain an essence of the wood (tree) used. The next day I am able to impart this wood knowledge in the men’s room at work. Ahh, hickory with a bit of cherry.
I enjoy this channel more than I understand the subjects covered. I worry that AI will be a garbage in garbage out product. It seems like a product forced upon me and I don't like it at all. Thanks for the video.
Your Audio is always WAY too low compared to everything else online ... have to turn speakers up too high and then back down after your videos... No other notes , Love your videos .,... nobody else is going to teach us this nerdy shit =)
@@Frostbytedigital Yes, totally fascinating. In many cases, you just have to see the equivalence in mathematical form of sum of products form like in 2:35. In some cases, you just sub in a complex number for the most part. I modeled a polychromatic light particle in our recent wave-based patent where we expand the equation of a photon c=lamda*nu as a sum of products. I filled it under my company "Calective." Also look up "Higher dimensional quantum computing" by Sabre Kais and Barry Sanders.
![BLACK BAG - Official Trailer [HD] - Only in Theaters March 14](http://i.ytimg.com/vi/Du0Xp8WX_7I/mqdefault.jpg)
I’m in ML since 2013 and have to say: wow… you and your team do really deserve praise for solid research and delivery. I’ll bookmark this video to point people to. Thank you
He's great ! dad was a chip designer.. go figure :) amazing backlog of content sir
Especially chips..
The span and depth of topics covered with a eye on technical details is truly awesome and rare. Smart commenters point out the occasional inaccuracies (understandable for the span of topics) which benefits everyone as well.
Dude, try living through all of this.
He has a team? I assumed he is just grinding and great at this.
He is independent AI researcher
Please continue the story. A cliffhanger like that deserves a sequel!
Seriously, this was a truly impressive video and I learned new things from it.
In the 80's Hinton, LeCun, Schmidhuber and others invented backpropagation, CNNs (convolutional NNs), then RNNs, LSTMs in the 90's but it was still very niche area of study with "limited potential" because NNs always performed a bit worse than other methods, until couple breakthroughs in speech recognition and image classification in the end of 00's. 2012 AlexNet brought instant hype to CNNs, which was followed by one-liners critically improving quality and stability of training: better initial values, sigmoid -> relu, dropout, normalization (forcing values to be in certain range), resnet (just adding values of previous layer to the next one). That allowed to train models so much bigger and deeper that they started to dominate everything else by sheer size. Then came Transformer in 2017 that allowed to treat basically any input as a sequence of tokens and scaling hypothesis which brought us to the present time with "small NNs" being "just" several billion parameters.
Between 2012 and now also been extreme progress with hardware for running these networks, optimizing the precision (it turned out that you don't need 32bit float numbers to train/use NNs, lower possible is 1 bit, good amount is 4 bit integer which is 100x faster in hardware), new instructions, matmuls, sparsity, tensor cores and systolic arrays and what not to get truly insane speedups. For comparison, AlexNet was trained on 2 GTX580 so it was about 2.5TFLOPs of compute. This year we have ultrathin and light laptops with 120TOPs and server cards with 20000TOPs and biggest clusters are in the range of 100 000 such cards, so in total 1 billion times more compute thrown at the problem than 12 years ago. And 12 years ago it was 1000x more than at the start of the century, so, we got about a trillion times more compute to make neural networks work and we still not anywhere close to be done. Of course, early pioneers had no chance without that much compute.
2nd that 😊
"The Second Neural Networks"
Your videos sre always well worth the time to watch them, thanks!
Got this recommended to me after getting my first digit recognition program working. The neural networks know I’m learning about neural networks
One of the top education channels on RUclips for sure
I love seeing Minsky come up as I have a (tenuous) connection to him as he is my academic "great great grand advisor." In that, my PhD's advisor's PhD advisor's PhD advisor's PhD advisor was Minsky. Unfortunately, stories about him never got passed down, I only have a bunch of stories with my own advisor, and his advisor, so it is interesting seeing what he was up to.
The Society of Mind.
Thanks Jon, as someone who tinkered with neural nets in the 1980s and 90s, this history connects the evolutionary dots and illuminates the evolution/genesis of those theories & tools we were working with... J
The one name missing from this from my high-school memory is Norbert Weiner, author of "Cybernetics". I do remember a circa 1980 effort of mine to understand the implication to my area of training (medicine) of rule-based AI. The Mycin program (infectious disease diagnosis and management) sited at Stanford could have been the seed crystal for a very useful application of the symbol-based methods. It wasn't maintained and expanded after its initial development. Took too long to do data input and didn't handle edge cases or apply common sense. It was, however, very good at difficult "university level specialist" problems. I interviewed Dr Shortliffe and his assessment was that AI wouldn't influence the practice of medicine for 20-30 years. I was hugely disappointed. At the age of 30 I thought it should be just around the corner. So here it is 45 years later and symbolic methods have languished. I think there needs to be one or more "symbolic layers" in the development process of neural networks. For one thing it would allow insertion of corrections and offer the possibility of analyzing the "reasoning".
Your storyline is decades long, so how old are you? 😮
@@honor9lite1337 7.5 decades
This is the best documentary on this topic i have ever seen. Its so well researched, its like doing the whole wikipedia dive
"A human being without life" hurts too much.
Avg person in 2024
Interesting that Claude Shannon's observations on the meaning of information being reducible to binary came about at virtually the same time as the early neural networks papers.
Edit - The Mathematical Theory of Communication by Shannon was published in 1948. Also, Herb Simon was an incredible mind.
Thanks for bringing back memories of the class I took from Minsky and Papert (short, not long a in pronouncing his name) in 1969 just when the book had come out. You filled in some of the back story that I wasn't aware of.
That book set AI back by decades.
Thanks!
Gosh, I remember studying physiology in the late 60s when human nervous system understanding was still in the relative dark ages - for instance plasticity was still unknown, and they taught us that your nerves stopped growing at a young age and that was it.
But I had no idea how far they'd come with machine learning in the Perceptron - already using tuneable weighted responses simulatong neurons? Wow!
If they could have licked that multilayer problem it would have sped things up quite a bit.
You mentioned the old chopped up planaria trick - are you familiar with the work of Dr Miachel Levin? His team is carrying the understanding of morphogenisis to new heights - amazing stuff! Thank you kindly for your videos! Cheers.
There must have been a speech of Demis Hassabis on 14 Nov 2017 in the late morning at the Society of Neuroscience in Washington. In this Keynote lecture where he told the audience that AI is nothing more than applied Brain Science. He must have said (I only have the translated German wording) 'First we solve the problem and understand whats intelligence (possibly the more German usage of the word) and then we solve all the other problems'. The 6000-8000 People must have been extremely quiet knowing what this young man already has achieved. Unfortunately I never found the video. (Source: Manfred Spitzer).
Studying in the late 60's? Even my dad was born in late 70's, how old are you?
I love listening to this with our current modern context
The difficulty with Minsky's adamant focus on symbolic logic was his failure to recognize that the vast majority of biological sensory processing is dedicated to creating meaningful, logically usable symbolic representations of a complicated physical world.
Minsky’s position thus was a bit like saying that once you understand cream, you have all you need to build a cow.
5:14 Look at this guy, throwing out Principia Mathematica without even name-dropping its author. 😂
It's nothing new. Ton of people do that.
Its not Isaac Newton if thats what you were thinking. It's Russell and Whitehead.
Well that's no good. I can't think of a terrible pun on their names!
@@theconkernator Thanks! I was definitely thinking Newton.
@@theconkernatoryeah? 😮
You always bring up interesting topics. Keep it up, it's great job 👍.
this was great. A more in depth one will be awesome. The fall and rise of the perceptron. Going from single to multiple layers.
As always I love your videos, the depth of knowledge and the people that comment, as they all have interesting stories about what is in your videos. So one of the descendants of the symbolic movement was cognitive architectures like SOAR and ACT-R from Newell's theories of Cognition. Symbolic systems are not gone and they perform many tasks that Neural Networks don't do well. However Neural Networks do something so much better than Cognitive Systems, and that is in getting all the data and knowledge of the world into the neural network, and being able to extract it out. There is no way you can program all of that as rules in symbolic systems. There will be a merger of both systems so they can perform better reasoning and cognitive tasks in the next iteration of all of this. We are really just at the beginning, standing on the shoulders of Giants.
Recurrent Neural Networks are about to make a HUGE comeback.
@@luciustarquiniuspriscus1408 MAMBA is already a valid alternative to transformers, and it is some kind of variant of linear recurrent neural networks.
Also I do not see how could we avoid recurrent neural networks for music generation, they or their variants seem like a perfect fit for this very specific generation task.
xLSTM FTW
@@luciustarquiniuspriscus1408 The SSM/Mamba papers already address this. In fact you can train a GPT-3 like small model using Mamba right here and now, with excellent performance (both in terms of training speed and outputs). With "infinite attention" (well, limited by the capacity of the hidden state vector).
I absolutely love your channel. I look forward to your new videos. Thank you. I’ve learned sooo much 🥰
The Einstein, Oppenheimer, Bohr, Feynman, Schroeder and Heisenbergs of a.i.. McCulloch-Pitts neuron network, Rosenblatt's training paradigm, took 70 years to get to "here" and should be acknowledged. I remember as a little kid in the 70s reading articles on different people leading the symbolic movement, and thinking "none of them really seem to know or have conviction in what they're campaigning for".
Having worked in ML and have done a lot with the perceptron. It feels like we're right where we started, promising the world, providing, not much...
I guess lonely people will be happy when chat-bots can render audio replies.
This is what real researchers say while companies go as far to boast their cat food is powered by AI.
AI winter.
Disagree. I worked on natural language NNs in the 80s (a failure). Now I'm using what we should no longer call LLMs to do real work, replacing older specialised AIs and improving accuracy. This is for horizon scanning. We are finding valuable insights for our government.
@@endintiers it’s the credibility issue that’s the problem. I’ve stopped using LLMs altogether even for coding, because it gets things incorrect often and there are usually better solutions.
SEGA creating an AI computer in 1986 is crazy
Super happy i found you again! Your content is off the charts amazing! I wish i could patreon you up- im in my 50's autistic af and i dont have an income. Appreciate you.. p.s. i thought you said Rosenblat died in a tragic coding accident! Lmfao. Love the flatworms! Keep on keeping your valuble perception turned on!!
I am probably not the first, but i am surprised at how far back the idea of artificial "intelligence" goes to.
It surprises me how "little" progress we have made in that time. Pretty much every other discipline has made incredible leaps in the past 60-70 years, yet AI is still nowhere near the human brain. Obviously an early perceptron is infinitely worse than a modern LLM, but AGI doesnt really feel any closer than back then.
@@lbgstzockt8493 if you're comparing what a few smart computer geeks did over 80 years to what mother nature did over 3-ish billion years, then I would argue it's not surprising AT ALL that we haven't simulated a human brain yet...
We've been here before
Before the universe..
@@goldnutter412 ?
The first idea of machines that could think appeared in the 17th century as Leibniz's mill concept
It's pronounced Pæpert not Pāpert, and that was a bit of a confusing way to present the XOR function: since you set it up with a XY plot, you should have put the Inputs on different axes with the values in the middle. Other than that, great video as always 😊
What a fantastic vide tracing the history of Neural Nets. It would also be really useful if you could put up the links to the papers mentioned in the video in the description.
Good summary. Ultimately, even today, there are no functionally useful and agreed definitions of intelligence and thinking. Maybe we're unintentionally approaching this from the back, by making things that sort of work, then later figuring out what's really going on (not yet!)
Damn we're really going full cycle. We've been hearing eerily similar things from people in similar roles as folks in this video.
Please do a video on the decline of British manufacturing, it would be greatly appreciated
Nobody cares about the Brits
Wow! well explained, and a way more engaging story than what I expected.
Amazing that some of this work was done at Dartmouth during the days of 'Animal House'
Very interesting topic
People are like transistors, its how they are connected that makes all the difference
I took a generic algorithms and neural networks module at university.
In the exam we would train and solve simple neutral networks on paper with a calculator.
Good fun, this was in 2000.
I think there's one guy that doesn't receive much attention, Demis Hassabis. I knew him as the founder of the game company that made Republic: The Revolution, but he then went on to take a PhD in Neuroscience. I wondered why. Now it makes sense. He founded DeepMind
A significant moment is when the instructor leaves the plane & says "do a couple takeoffs & landings".
PERCEPTRON 😍😍
hebb's postulate are stilling being taught in neuroscience,only difference is a time element is being added its now called "spike time dependent plasticity"
Wow, didn't know they researched that back then, so long ago. Thank you for educating me on this matter. Today, AI is amazing already. I developed a USB reader/ tester with GPT4. The code that it wrote was spot on. The rest was just electronics, an amazing tool.
I'm the PERCEPTRON
Thank you for making this movie.
Very nice and well scoped contextualization about the developement of NNs ... I know that the video is about an specific branch of computer science ... but the seminal work for AI research was not Alan Turing papers ... the seminal work for AI and Computer science is George boole's The Laws of Thought (1854) which contains Boolean algebra.
Would love to hear you talk about Doug Lenat's Cyc and what ever happened to that approach to AI
Terrific
Worth reading up on Walter Pitts! Interesting life and work
In 1996 neural networks were touted as predicting pollution from combustion sources without any need for chemical or visual monitoring.
looks like a bot
@@alexdrockhound9497 Why'd you write "channel doesn't have any conte" on your channel ?
@@leannevandekew1996 typical bot. trying to deflect. Your profile is AI generated and you look just like adult content bots i see all over the platform.
@@alexdrockhound9497 You totally are.
I would stalk you just as you said.
Thank you for this segment, and the asianometry (strange name 🙂) channel. Lots of interesting stuff. I have worked in the field of neural networks for many years, and what I find most striking is how the field has been plagued by researchers who project the most abstract brain functions onto mechanisms with negligible complexity (here Rosenblatt 's hyperboles about a couple of linear units with dynamics weights). This is bad for the image of the field (especially to the general public who finances these research) but also for the field itself where new ideas have to fight these oversimplifications to be recognized. The work by people like Stephen Gross and Stanislas Dehaene (who once in a talk at the Montreal Neuronal Institute in the 2000s likened the process of a human becoming conscient of some stimulus to a printer that turns on to print a document) are unfortunate examples. But global warming will probably make this a moot point anyway: human society's inability to manage the responsibilities that come from the technology that our brain has allowed us to develop (together with the profit-at-all-cost economic model used to exploit it) will destroy us before we can understand the organ that made it possible. What a shame...
Love it. Awesome summary.
I completed a machine learning course from Cornell a little before genAI became really popular and we had to learn how to code a basic Perceptron. The rare times I see it mentioned, I always feel like I have some special, unique insight. (I definitely do not!)
Hi! Great fan of your channel! :)
However this time I'm a bit puzzled. Several years ago I read somewhere Marvin Minsky talking about how he build this (awesome in my opinion) mechanical neural network. Since that time I was sure his network was the first. However here you talk about neural network build almost decade later and call it the first one... You mentioned that Marvin Minsky did some nerual network research previously, but he left. Ok, fine, so why his neural network that was bult before perceptron is not the first one in your opinion? :) Maybe next video? :)
Also - to my knowledge Turings paper was published in 1937, not 36. In year 1936 Alonso Church published his paper related to Entscheidungsproblem. I don't know who was the second one that came up with theory of gravity or relativity, we don't usually remember them. But for some reason we remember Turing for being second in something :) Just fun fact :)
There's more about this early history of artificial intelligence in this 1988 book: Graubard, S. R. (Ed.). (1988). The artificial intelligence debate: False starts, real foundations. MIT Press.
The chapters also detail a very interesting discussion of related general philosophical problems and limitations of the time.
very forwards thinking people from a time when most people couldn't even comprehend computers. i know a lot of people born after this period who are only slightly older than me that can't even handle windows 10 and don't believe computers existed when they were young. and i am talking about people born in the 1970s here not boomers. yet you had these geniuses born in the late 19th century or early 20th century who made it all possible .
When I first read about N.N. around 2000 this was still the state of the matter 30 years later.
When I was at university NN were no topic.
Then after 2010 things suddenly changed. Training Data and Flops had become available.
Thank you
I love how cheesy that name is. "The Perceptron!" Is it one of the good transformers or the bad ones??
Working in physics one of the first things I learned is half the names are just "-tron" and it always makes me giggle
The title should be "the rise and rise of the perceptron" since it's the basic idea behind nearly all modern AI.
I needed a video to do the dishes, after spending a day making pedagogical materials on Stable Diffusion. Now I will rewind and delight myself seeing this video carefully. *Thank you*
Great video :)
Before the multi-layer perceptron, statisticians used that exact same model with sigmoid activation functions and called the process "edge regression." The statisticians knew how to "train" the model using second-order multivariant optimization and "weight decay" methods, which were vastly superior to the ad hoc backpropagation methods that neural network researchers were still using as late as the 1980's. The neural net guys were blinded by their unwarranted certainty that they were onto something new.
I studied the perceptron in the '70s. My conclusion was, the hardware was not up to the task. Using a matrix of photoresistors as the input proved the design principle but couldn't bring to a working prototype.
As I recall, AMD's zen uarch also use some form of perceptron as their BTB or branch prediction
great video love the look back at currently used technology
I built a multilayer perceptron from 4096 operational amplifiers in the '80s. It was a kind of analog computer where the weighting values could be set with electronic potentiometers controlled from a digital computer, digital port, and the freely reorganizable switch matrix could be set with electronic switches, how many layers to have and in what matrix.
It beat the digital machines of the time in real-time speech recognition and live speech generation. Nowadays I want to integrate this into an IC and sell it as an analog computer.
We tried to model the back propagation method using the PROLOG language, but the machines at that time were very slow.
It took 40 years for the speed and memory size of machines to reach the level where this could be realized.
The so-called scientific paper work was very far from the practical solutions realized then and now, there is a couple of decades behind in theory, because a lot of technical results achieved in practice could not be published.
I was just studying this subject in university 😮 one of the many lectures was about the history of IA
What a coincidence
Well I learned a whole lot from this video. TY 👏
0:12 Is that Turing on the right
I find it ironic/devious that Minsky criticized Perceptrons for inability to scale. None of the little toy systems that came out of MIT or Yale (Schank) scaled.
Great great video. But the McCulloch-Pitts Neuron didn't use weights. You displayed diagrams showing weights whenever you mentioned it.
I was just watching this video by Formosa TV and their piece of on the founder of SuperMicro. Just curious if there is any interest in the SuperMicro or its founder (for the English speaking folks that don't understand Mandarin hehe).... just thought I'd ask. 🙂
Wow thats strange timing I'm in the middle of reading the book " The Dream Machine by Mitchell Waldrop" Have you read that yet ? Its about these exact same people
As a fellow 4 hour sleeper. Any advice on brain health. I'm getting paranoid.
Minsky & Papert and the XOR problem was the invention of deep learning
Yuxi in the Wired, any audio essay?
Beyond the usual kudos to Jon and his research, the primary reason much of this area has, and will continue to come up short of what a human brain can do has to do with scale and complexity. The human brain is WAY more complex than even the combination of the current LLM systems. Sitting at over 50,000 cells and 150,000,000 synaptic connections per cubic mm (Science: A petavoxel fragment of human cerebral cortex reconstructed at nanoscale resolution), and using less than 300W/hr. of energy per day, the brain is a formidable piece of wetware.
With advances of computer hardware and software we'll get there some day. But, not for a long time. Until then, remember that all the current LLMs are 100% artificial and 0% intelligent.
I see what you do there. 3030 year: History of AI essey: As we know, our new hyperdeepinnercurlingdoubleflashing neural network almost working, we need few more tiny touches and literally 2 extra layers to show it awesomeness in next 3031year. And again, and again.
the 'boating accident' is peculiar
Russian vessels close to the Belgian coast (in international waters) are being closely watched these days...
ELIZA: "Cat got your tongue?" I had a Boolean Logic class @ LSU. Very interesting.
MLPs are very prominent still. Also Attractor NNs are very popular, transformers would not exist without ANNs.
FYI, XOR is pronounced “ex-or” like “ECK-sor”
Part 2 when?
Every time i hear word multifaceted i think of chatgpt.
Great entry, very à propos!
Minsky is currently frozen, waiting for return after his untimely death in 2016 or so
It is not enough to digitally model the most common LLM for Artificial Intelligence today, it doesn't matter if it is 1-bit or 1 TRIT 1.58b = log2(3), it has to be done with working ANALOG hardware!
If software, then Machine Learning (algorithm)
If hardware, then Learning Machine (Hardware that is better and faster than an algorithm)
Is every cell conscious?
When I eat pulled pork I gain none of the pig’s memories. I do however gain an essence of the wood (tree) used. The next day I am able to impart this wood knowledge in the men’s room at work. Ahh, hickory with a bit of cherry.
Idk about you but when I eat pulled pork I feel like a pig
This guy has been messing with us for so long I can't tell if the "references and sources go here" is a bit, or a mistake.
Didn't I build this in Alpha Centauri?
I enjoy this channel more than I understand the subjects covered. I worry that AI will be a garbage in garbage out product. It seems like a product forced upon me and I don't like it at all. Thanks for the video.
An explanation of perceptrons and where did they go???...
Get the heck out of my head, I was literally just wondering about this.
asianometry my goat
My CPU is a neural-net processor; a learning computer.
Your Audio is always WAY too low compared to everything else online ... have to turn speakers up too high and then back down after your videos...
No other notes , Love your videos .,... nobody else is going to teach us this nerdy shit =)
The first neutral network theory is/was valid. The computing hardware is catching up. All the equations are valid with wave or wave states.
ALL the equations from the original perceptron are correct if you sub in waves or wave states? That's fascinating. Any sources?
@@Frostbytedigital Yes, totally fascinating. In many cases, you just have to see the equivalence in mathematical form of sum of products form like in 2:35. In some cases, you just sub in a complex number for the most part. I modeled a polychromatic light particle in our recent wave-based patent where we expand the equation of a photon c=lamda*nu as a sum of products. I filled it under my company "Calective." Also look up "Higher dimensional quantum computing" by Sabre Kais and Barry Sanders.
🎉
Now people use neural network to draw furry pictures🙂
symbolic AI was a neat idea ... rip