Hinton's Advice: #1 - Read the literature but don't read too much of it. #2 - For creative researchers, I think what you want to do is to read a little bit of the literature and notice something that you think everybody is doing wrong and contrarian in that sense, you look at it and it just doesn't feel right and then figure out how to do it right, and when people tell you that's no good....just keep at it. And I have a very good principle for helping people keep at it. Which is, either your intuitions are good or they're not. If your intuitions are good, you should follow them and you will eventually be successful. If your intuitions are not good, it doesn't matter what you do. #3 - Never stop programming! #4 - I think one should reach enough to start building intuitions and then trust your intuitions and go for it. And, don't be too worried if everybody else says is nonsense. #5 - If you think its a really good idea and other people tell you it's complete nonsense, then you know you're really on to something. #6 - one good piece of advice for new grad students, see if you can find an adviser who has beliefs similar to yours because if you work on stuff that your adviser feels deeply about you'll get a lot of good advice and time from your adviser.
I attended one of his lecture in Washington University,. He is extremely humble and a great speaker. He is the Einstein of our generation. I am really looking forward to his upcoming paper on Capsule theory. God ! it gives me a feeling like standing in queue for the latest iPhone. LOL !
00:30 How Hinton’s fascination with the brain led him to explore AI 3:33 The story behind his seminal 1986 paper on backpropagation that he wrote with David Rumelhart 8:16 The invention Hinton is still most excited about 12:55 Hinton’s work on ReLU activations 25:35 How Hinton’s understanding of AI has changed over the decades 15:27 Hinton’s thoughts on the relationship between the brain and backpropagation 19:09 Hinton’s work on dealing with multiple time scales 20:43 Hinton’s ongoing research on capsules 29:40 Hinton’s advice for someone who wants to break into AI 37:15 Hinton’s thoughts on the paradigm shift in AI
This is very interesting in terms of Prof. Hinton's historical perspective on his interest and development of many algorithms. Also, Prof Ng contribution to making everyone dreams become a reality from learning ML and applying in their prospective jobs. Thanks to both!!
Dear Andrew, first of all I wanted to thank you and the entire coursera team for the great academic contribution they give us. Secondly, I wanted to ask you to put the subtitles where possible so that we can enjoy 100% Spanish and South American speaking. finally ask you to include the course of neural networks of Jeoffrey Hinton since when you search through the search engine of coursera it does not find it. I will thank you for correcting this error because when I tried to search for the courses offered by the University of Toronto, I did not have a positive success either. Thanks again to you and the coursera team for the great innovative academic contribution and scientific and technological research.
"Finally Andrew Ng bought a decent mic!" - nah.. have to edit myself. At around 27:00 and on he returns to toilet/pillow-filtered sound. That's just Andrew Ngs trademark!
Thanks for this interview, Andrew! Hey, fellow learners, if anyone is interested in joining my weekly Machine Learning Research Paper reading and discussion group, let me know!
Three years ago I took Ng's Machine Learning Coursera class and it really got me hooked on the subject. Then in the spring of 2017 I took Hinton's Neural Networks for Machine Learning Coursera class and it was a big step in my understanding the subject. Hinton's class goes 15 weeks and is a little intimidating in both depth and breadth. I have the math background so that really helped. I guess the bottom line is that if you're going to study the subject, study math through partial differential equations.
Fav quote, little after minute 30... "Either your intuition is good or its not. If its good you should follow them and eventually you will be successful. If they are no good it doesn't matter what you do. There's no point in not trusting your intuitions."
I so agree with Hinton: have an idea and go for it. I took this approach with something other than AI, but it also worked. What do I mean? I mean, even though my idea wasn't revolutionary and totally worthwhile, I LEARNED A LOT by just going for it and programming the heck out of it. The practical experience I gained served me well--very well--in my first jobs. Remember: your purpose is to learn, and you can do that following your intuition--which is fun--or following someone else's--which is less fun.
Interesting and informative. Thanks. But, on the last part: I believe that the symbolic approach of AI should not be totally abandoned by the researchers. One reason, by no means the only one, is that it gives AI a less of a "black box" approach than neural networks.
symbolic.... total baloney! Symbols and indeed mathematics in general are emergent entities of the brain. AI if it is to replicate the workings of the brain has to dig deeper, be more subatomic, less molecular as therein lies the route to a multitasking, general intelligence.
that advice wasn't clear to me...is the advice to never stop programming cuz if a "bad student" doesn't make the idea work then if you don't lose your edge on programming then you can make work yourself?
Funny but important (?): “You either have good intuitions or don't. If you have good intuitions you’ll eventually be successful but if you have bad intuition no matter what you do it will suck, so follow your intuitions”-paraphrase of Hinton's research advice. What do people think?
I think his intuition comes from his experience in diverse disciplines. He mentioned that he tried physics, physiology, psychology and philosophy during his undergrad. Intuition isn't some magical ability you're born with. It builds as you get exposed to information - similar to how variation in training data is beneficial to a good machine learning model.
if u live long enoguh bad become good. trouble is we have short life span... so if you suspect u have bad intuiation , look deeper into your heart to find a better one which yiu can use to explain everything happen to you and around the world. Then u should arrive on some decent intuition.
Based on Hinton's descriptions of capsules, is it possible he has overlooked manifolds? It is pertinently the behaviour of manifolds, where solutions or sub manifolds (i.e. latent vectors on the states of particular concepts in the input space - some latent z is entailed by some factor distribution : {position, scale…}) are observed to lie in local patches of the global manifold, that engender that particular factors may be learned; for example pixels in the neighbourhood of some other pixels may signify transformations on that same pixel, while other neighbourhoods may be disentangled from the sampled latent vectors of the aforesaid pixel altogether, (i.e. other pixel data and their transformation data are separable from the events of the pixel discussed above)
Hinton surely did not overlook manifolds. The "routing by agreement" bit is where the main difference is; The way traditional CNNs learn means that they would not generalise as well as capsule nets; a lot of the information is lost / duplicated. Capsule nets (with this focus on "concepts") would be biased to learn more re-usable concepts, with different properties.
힌튼의 조언 : # 1-문헌을 읽되 너무 많이 읽지 마십시오. # 2-창의적인 연구자들에게, 사람들이 좋지 않다고 말할 때 .. 그냥 계속하십시오. 그리고 저는 사람들이 그것을 지키도록 돕는 아주 좋은 원칙을 가지고 있습니다. 즉, 직관이 좋거나 그렇지 않습니다. 당신의 직관이 좋다면 당신은 그것을 따라야 하며 결국 성공할 것입니다. 당신의 직감이 좋지 않다면 당신이 무엇을 하든 상관 없습니다. # 3-프로그래밍을 멈추지 마십시오! # 4-직관을 쌓기 시작할 만큼 충분히 도달 한 다음, 직감을 믿고 실행해야 한다고 생각합니다. 그리고 다른 사람들이 말도 안된다고 해도 너무 걱정하지 마세요. # 5-당신이 그것이 정말 좋은 아이디어라고 생각하고 다른 사람들이 그것이 완전히 넌센스라고 말한다면, 당신은 당신이 정말로 무언가를 하고 있다는 것을 압니다. # 6-새로운 대학원생을 위한 좋은 조언 한 가지, 당신과 비슷한 신념을 가진 조언자를 찾을 수 있는지 확인하세요. 당신이 조언자가 당신에 대해 깊이 느끼는 일을 하면 좋은 조언과 시간을 많이 얻을 수 있기 때문입니다.
38:19 people who thought that thoughts were symbolic expressions made a huge mistake.. what comes in is a string of words, and what comes out is a string of words, and because of that, strings of words are the obvious way to represent things, so they thought what must be in between was a string of words or something alike.. Hinton thinks there's nothing like a string of words in between, he thinks thinking of it as of some kind of language is as silly as the idea that understanding the layout of a spacial scene must be in pixels :))
WOW I can't believe Hinton et al came up with an algorithm for STDP decades before neuroscientists came up with the concept....He is talking about this algorithm that came in the late 1980s, while STDP arrived on the scene in the early-mid 2000s
Hinton: "Thoughts are great big vectors, and big vectors have causal powers, they cause other big vectors" Me: "Thought Curvature" paper : www.academia.edu/25733790/Causal_Neural_Paradox_Thought_Curvature_Aptly_the_transient_naive_hypothesis
Oh yeah!? then how come elon musk, stef. hawking, sam harris and even f**g bill gates, seriously talk about it as an existential threat!? surely they know something the rest of us don't :))
mukuste yeah it's ridiculous...though AGI is a very interesting topic, as Neil D Tyson noted, we are far from being on the path of something like that happening...so till then AI is just another tool to help us live better
Seriously doubt Geoffrey Hinton considers himself a hero - more like Dr. Frankenstein now. He's doing his part to spread the word on the dangers of reliance on AI.
YESSS! A technical tour-de-force! The Godfather! Gotta love the consistent elephant in the room though. AI inevitability: Human intelligence becomes obsolete for any purpose beyond the banal. Luckily, the onset of autonomous AGI (consciousness optional) solves this permanently. ;-[ Intelligence is THE differentiator on earth; let's relentlessly pursue / design Humanity's successor as Apex Predator. GAN indeed. GONE, more likely. Just keep kicking the can down the road, you clever hand-wringers. Learning leads to Knowledge, Knowledge is Power, Power corrupts, absolute Power corrupts absolutely. A clearly sub-optimal general (Human) outcome, driven by an optimising, self-adjusting, closed-loop feed-back system. Brilliantly myopic.
Hinton's Advice:
#1 - Read the literature but don't read too much of it.
#2 - For creative researchers, I think what you want to do is to read a little bit of the literature and notice something that you think everybody is doing wrong and contrarian in that sense, you look at it and it just doesn't feel right and then figure out how to do it right, and when people tell you that's no good....just keep at it. And I have a very good principle for helping people keep at it. Which is, either your intuitions are good or they're not. If your intuitions are good, you should follow them and you will eventually be successful. If your intuitions are not good, it doesn't matter what you do.
#3 - Never stop programming!
#4 - I think one should reach enough to start building intuitions and then trust your intuitions and go for it. And, don't be too worried if everybody else says is nonsense.
#5 - If you think its a really good idea and other people tell you it's complete nonsense, then you know you're really on to something.
#6 - one good piece of advice for new grad students, see if you can find an adviser who has beliefs similar to yours because if you work on stuff that your adviser feels deeply about you'll get a lot of good advice and time from your adviser.
The 2nd one!
#2 👏 #TomášMikolov
29:59 this is where Hinton actually says that
Good principals to follow. Regardless of what you are doing.
I attended one of his lecture in Washington University,. He is extremely humble and a great speaker. He is the Einstein of our generation. I am really looking forward to his upcoming paper on Capsule theory. God ! it gives me a feeling like standing in queue for the latest iPhone. LOL !
00:30 How Hinton’s fascination with the brain led him to explore AI
3:33 The story behind his seminal 1986 paper on backpropagation that he wrote with David Rumelhart
8:16 The invention Hinton is still most excited about
12:55 Hinton’s work on ReLU activations
25:35 How Hinton’s understanding of AI has changed over the decades
15:27 Hinton’s thoughts on the relationship between the brain and backpropagation
19:09 Hinton’s work on dealing with multiple time scales
20:43 Hinton’s ongoing research on capsules
29:40 Hinton’s advice for someone who wants to break into AI
37:15 Hinton’s thoughts on the paradigm shift in AI
Thank you!
Even Andrew looks super-exciting in the video, imagine the viewers. :)
Two legends talking on a domain they help build. No surprise it turned out so informative. Thank u for making this video.
Utterly fascinating. Professor Hinton is inspiring.
This is very interesting in terms of Prof. Hinton's historical perspective on his interest and development of many algorithms. Also, Prof Ng contribution to making everyone dreams become a reality from learning ML and applying in their prospective jobs.
Thanks to both!!
Dear Andrew, first of all I wanted to thank you and the entire coursera team for the great academic contribution they give us. Secondly, I wanted to ask you to put the subtitles where possible so that we can enjoy 100% Spanish and South American speaking. finally ask you to include the course of neural networks of Jeoffrey Hinton since when you search through the search engine of coursera it does not find it. I will thank you for correcting this error because when I tried to search for the courses offered by the University of Toronto, I did not have a positive success either. Thanks again to you and the coursera team for the great innovative academic contribution and scientific and technological research.
"Finally Andrew Ng bought a decent mic!" - nah.. have to edit myself. At around 27:00 and on he returns to toilet/pillow-filtered sound. That's just Andrew Ngs trademark!
Thanks for this excellent interview. I'm totally geeking out at this interview!
Best research advice ever!! "Read the literature, but not too much of it."
Thanks, Andrew Ng for the interviews.
This interview is highly insightful and helpful for deep learners, recommended watching!
Thank you for this wonderful interview.
Thanks for the amazing interview and for the sharing!
Excellent interview...and very good advises from Hinton!
Geoggrey Hinton and Andrew NG, ML Titans!
Great interview, but it's so deep. I wish I could understand more of it...
thank you so much for doing this interview
Thanks for this interview, Andrew! Hey, fellow learners, if anyone is interested in joining my weekly Machine Learning Research Paper reading and discussion group, let me know!
Huge thanks to Andrew and Geoffrey for this interview!
Thanks for introducing us to the heroes of DL like Geoffrey
Andrew looking at Hinton Like how we all see Andrew when learning from him.. Huge respect for everyone who contributed to AI. :)
Three years ago I took Ng's Machine Learning Coursera class and it really got me hooked on the subject. Then in the spring of 2017 I took Hinton's Neural Networks for Machine Learning Coursera class and it was a big step in my understanding the subject. Hinton's class goes 15 weeks and is a little intimidating in both depth and breadth. I have the math background so that really helped. I guess the bottom line is that if you're going to study the subject, study math through partial differential equations.
Most of the backpropagation is just add-subtract-multiply, but the tricky step is to calculate the updated weights using gradient descent.
Fav quote, little after minute 30... "Either your intuition is good or its not. If its good you should follow them and eventually you will be successful. If they are no good it doesn't matter what you do. There's no point in not trusting your intuitions."
38:00 onwards, brilliant philosophy of AI from this legend.
This was great. Love hearing his more far out ideas
honoured to listen to this discussion , subhanallah
"When you have something you think is a good idea and other people think it's a complete nonsense, then you know you are onto something"
Fascinating !
I so agree with Hinton: have an idea and go for it. I took this approach with something other than AI, but it also worked. What do I mean? I mean, even though my idea wasn't revolutionary and totally worthwhile, I LEARNED A LOT by just going for it and programming the heck out of it. The practical experience I gained served me well--very well--in my first jobs. Remember: your purpose is to learn, and you can do that following your intuition--which is fun--or following someone else's--which is less fun.
I love Andrew so much. :)
Interesting and informative. Thanks.
But, on the last part: I believe that the symbolic approach of AI should not be totally abandoned by the researchers. One reason, by no means the only one, is that it gives AI a less of a "black box" approach than neural networks.
symbolic.... total baloney! Symbols and indeed mathematics in general are emergent entities of the brain. AI if it is to replicate the workings of the brain has to dig deeper, be more subatomic, less molecular as therein lies the route to a multitasking, general intelligence.
listening to Geoff is something really fancy and super exciting.
Legend!
This guys' intuitions puts everyone else to shame.
GreatUnwashedMass 0
Nice and humble atheist
Its also his clarity of writing I think. His papers are easier to understand for me than a lot of others.
I wish someone post the link to all the paper mentioned in the interview
NEVER STOP PROGRAMMING!! dope
Amazing
Never stop programming !
At 32:08 : Never stop programming!
that advice wasn't clear to me...is the advice to never stop programming cuz if a "bad student" doesn't make the idea work then if you don't lose your edge on programming then you can make work yourself?
You only understand the tiny details when you can write a program to implement the whole thing yourself.
And stop copy pasting and make mistakes and fix the bugs I guess !! :)
BTW, that auto-generated caption is so creepy accurate
Godfather 🙌🙌🙌
Enlightened !!
Yes, I learned from his tutorial in Coursera in data science and technology
I think the difference between wake and sleep is during sleep it is in the testing phase and during wake it is the operative phase of learning.
The legend looks so innocent
"The most beautiful one is the work I do between Terrence Sejnowski on Boltzmann Machines"
Anybody know if anything was published about his stack of auto-encoders backprop idea?
Would love to hear more about it
Anton He's talking about layer-wise pretraining.
Funny but important (?): “You either have good intuitions or don't. If you have good intuitions you’ll eventually be successful but if you have bad intuition no matter what you do it will suck, so follow your intuitions”-paraphrase of Hinton's research advice.
What do people think?
Don't give up (but in a smart way) seems to be better advice and I agree.
You know where he borrowed it from?
I think his intuition comes from his experience in diverse disciplines. He mentioned that he tried physics, physiology, psychology and philosophy during his undergrad. Intuition isn't some magical ability you're born with. It builds as you get exposed to information - similar to how variation in training data is beneficial to a good machine learning model.
if u live long enoguh bad become good. trouble is we have short life span... so if you suspect u have bad intuiation , look deeper into your heart to find a better one which yiu can use to explain everything happen to you and around the world. Then u should arrive on some decent intuition.
Wow - the capsule concept is pretty close to the thousand brains idea!
Very inspiring interview.
AI is changing the world fundamentally anyway.
Legend.
Based on Hinton's descriptions of capsules, is it possible he has overlooked manifolds?
It is pertinently the behaviour of manifolds, where solutions or sub manifolds (i.e. latent vectors on the states of particular concepts in the input space - some latent z is entailed by some factor distribution : {position, scale…}) are observed to lie in local patches of the global manifold, that engender that particular factors may be learned; for example pixels in the neighbourhood of some other pixels may signify transformations on that same pixel, while other neighbourhoods may be disentangled from the sampled latent vectors of the aforesaid pixel altogether, (i.e. other pixel data and their transformation data are separable from the events of the pixel discussed above)
Hinton surely did not overlook manifolds.
The "routing by agreement" bit is where the main difference is; The way traditional CNNs learn means that they would not generalise as well as capsule nets; a lot of the information is lost / duplicated. Capsule nets (with this focus on "concepts") would be biased to learn more re-usable concepts, with different properties.
I watched the video as a part of Deep Learning Specialization. Sadly, it's way way over my head to comprehend much of what was said in the video.
35:08 our relationship to computers has changed.. instead of programming them, we show them, and they figure it out
36:04 :D
Top ten anime crossovers
힌튼의 조언 :
# 1-문헌을 읽되 너무 많이 읽지 마십시오.
# 2-창의적인 연구자들에게, 사람들이 좋지 않다고 말할 때 .. 그냥 계속하십시오. 그리고 저는 사람들이 그것을 지키도록 돕는 아주 좋은 원칙을 가지고 있습니다. 즉, 직관이 좋거나 그렇지 않습니다. 당신의 직관이 좋다면 당신은 그것을 따라야 하며 결국 성공할 것입니다. 당신의 직감이 좋지 않다면 당신이 무엇을 하든 상관 없습니다.
# 3-프로그래밍을 멈추지 마십시오!
# 4-직관을 쌓기 시작할 만큼 충분히 도달 한 다음, 직감을 믿고 실행해야 한다고 생각합니다. 그리고 다른 사람들이 말도 안된다고 해도 너무 걱정하지 마세요.
# 5-당신이 그것이 정말 좋은 아이디어라고 생각하고 다른 사람들이 그것이 완전히 넌센스라고 말한다면, 당신은 당신이 정말로 무언가를 하고 있다는 것을 압니다.
# 6-새로운 대학원생을 위한 좋은 조언 한 가지, 당신과 비슷한 신념을 가진 조언자를 찾을 수 있는지 확인하세요. 당신이 조언자가 당신에 대해 깊이 느끼는 일을 하면 좋은 조언과 시간을 많이 얻을 수 있기 때문입니다.
God has spoken.
Nice!
38:10 a thought is just a great big vector of neural activity
38:19 people who thought that thoughts were symbolic expressions made a huge mistake.. what comes in is a string of words, and what comes out is a string of words, and because of that, strings of words are the obvious way to represent things, so they thought what must be in between was a string of words or something alike.. Hinton thinks there's nothing like a string of words in between, he thinks thinking of it as of some kind of language is as silly as the idea that understanding the layout of a spacial scene must be in pixels :))
youtube have only one like button :(
@ 8:34 what was Geoffrey mentioning ? Which algorithm ?
boltzmann machines
Great
18:00 2007 ignored Hinton and Bengio picked it up layer on
Hey And , Do you still accept donations
by any chance, I am hoping for 720p videos from now on.
10:49
WOW I can't believe Hinton et al came up with an algorithm for STDP decades before neuroscientists came up with the concept....He is talking about this algorithm that came in the late 1980s, while STDP arrived on the scene in the early-mid 2000s
0:19 Godfather 😎
25:33
I wish my neurones could remember what they were doing when they pop out of a recursive call...
All rise for the Godfather
Hinton: "Thoughts are great big vectors, and big vectors have causal powers, they cause other big vectors"
Me: "Thought Curvature" paper : www.academia.edu/25733790/Causal_Neural_Paradox_Thought_Curvature_Aptly_the_transient_naive_hypothesis
no question about AI and singularity!? sure he would've had a super funny answer :))
The singularity is a pure scifi idea that actual AI researchers have no time for.
Oh yeah!? then how come elon musk, stef. hawking, sam harris and even f**g bill gates, seriously talk about it as an existential threat!? surely they know something the rest of us don't :))
None of them are AI researchers. They're laypeople on this issue.
mukuste yeah it's ridiculous...though AGI is a very interesting topic, as Neil D Tyson noted, we are far from being on the path of something like that happening...so till then AI is just another tool to help us live better
indeed
alien language to me right now, still it's great haha
Concretely
Still there are 17 dislikes, don't know why.. may be outliers...
Andrew "I see" Ng
Seriously doubt Geoffrey Hinton considers himself a hero - more like Dr. Frankenstein now. He's doing his part to spread the word on the dangers of reliance on AI.
YESSS! A technical tour-de-force! The Godfather!
Gotta love the consistent elephant in the room though. AI inevitability: Human intelligence becomes obsolete for any purpose beyond the banal. Luckily, the onset of autonomous AGI (consciousness optional) solves this permanently. ;-[
Intelligence is THE differentiator on earth; let's relentlessly pursue / design Humanity's successor as Apex Predator. GAN indeed. GONE, more likely. Just keep kicking the can down the road, you clever hand-wringers.
Learning leads to Knowledge, Knowledge is Power, Power corrupts, absolute Power corrupts absolutely. A clearly sub-optimal general (Human) outcome, driven by an optimising, self-adjusting, closed-loop feed-back system. Brilliantly myopic.
damn that's deep lol
@@aniketphatak951 Shallow: OK, you got it.
The reflections of the screen in his glasses look like googly eyes.
Google eyes! Connected to Google Brain :)
Absolutely love this, but Andrew, please stop saying "I see" constantly