Links to papers in video: @3:58 Machine learning for fluid mechanics Brunton, Noack, Koumoutsakos, Ann. Rev. Fluid Mech 52:477--508, 2020 www.annualreviews.org/doi/pdf/10.1146/annurev-fluid-010719-060214 @5:04 Efficient collective swimming by harnessing vortices through deep reinforcement learning Verma, Novati, Koumoutsakos, Proc. Nat. Acad. Sci. 115(23):5849--5854, 2018 www.pnas.org/content/115/23/5849 @6:57 Automating turbulence modelling by multi-agent reinforcement learning Novati, Lascombes de Laroussilhe, Koumoutsakos, Nat. Mach. Int. 3:87--96, 2021 www.nature.com/articles/s42256-020-00272-0 @8:47 A review of Deep Reinforcement Learning for fluid mechanics, Garnier, Viquerat, Rabault, Larcher, Kuhnle, Hachem, 2019 arxiv.org/abs/1908.04127 @9:57 Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control Rabault, Kuchta, Jensen, Reglade, Cerardi, J. Fluid Mech. 865, 2019 doi.org/10.1017/jfm.2019.62 @10:56 Reinforcement learning for bluff body active flow control in experiments and simulations Fan, Yang, Wang, Triantafyllou, Karniadakis, Proc. Nat. Acad. Sci. 117(42), 2020 doi.org/10.1073/pnas.2004939117 @11:50 Fluid directed rigid body control using deep reinforcement learning Ma, Tian, Pan, Ren, Manocha, SIGGRAPH 2018 gamma.cs.unc.edu/DRL_FluidRigid/ @13:26 Autonomous helicopter flight via Reinforcement Learning Ng, Kim, Jordan, Sastry, NeurIPS 2004 papers.nips.cc/paper/2003/file/b427426b8acd2c2e53827970f2c2f526-Paper.pdf @13:26 An Application of Reinforcement Learning to Aerobatic Helicopter Flight Abbeel, Coates, Quigly, Ng, NeurIPS 2007 proceedings.neurips.cc/paper/2006/file/98c39996bf1543e974747a2549b3107c-Paper.pdf @13:26 Autonomous helicopter aerobatics through apprenticeship learning Abeel, Coates, Ng, Int J Rob Res 2010 journals.sagepub.com/doi/abs/10.1177/0278364910371999 @14:02 Learning to fly like a bird Tedrake, Jackowski, Cory, Roberts, Hoburg, Int. Symp. Rob. Res. 2009 groups.csail.mit.edu/robotics-center/public_papers/Tedrake09.pdf @14:58 Control of a Quadrotor with Reinforcement Learning Hwangbo, Sa, Siegwart, Hutter, IEEE Rob Aut 2(4) 2017 arxiv.org/abs/1707.05110 @15:22 Learning to soar in turbulent environments Reddy, Celani, Sejnowski, Vergassola Proc. Nat. Acad. Sci. 113(33, 2016 www.pnas.org/content/113/33/E4877 @16:31 Learning to Fly: Computational Controller Design for Hybrid UAVs with Reinforcement Learning Xu, Du, Foshey, Li, Zhu, Schulz, Matusik, SIGGRAPH 2019 people.csail.mit.edu/jiex/papers/LearningToFly/index.html
You mentioned that your wife is a neuroscientist and indicated that animal reward is internal. Can you by chance provide a reference article that would describe this in greater detail?
I'm thinking about Jim Lovell and Apollo 13. After the explosion, he had to relearn flying the craft in a new configuration. He said it would go left when he wanted to go right. But they did it. This kind of work could save lives when we suddenly find ourselves in a new place we didn't count on.
Great video! One thing I'm curious about - my understanding is that reinforcement learning is difficult in practice b/c it's hard to build a simulated environment which matches reality. But here, it seems that issue has been managed, since these drones are flown in the real circumstance. So, are our turbulence simulations really that good or are there other clever tricks here? In general, do you see the accuracy of the simulation as the primary limiting factor in reinforcement learning?
So these simulations (like in the case of the fish) are essentially used as RL environments, correct? Are these used within the framework of gym or what kind of toolset is used?
I guess the advantage of traditional methodologies is that they come with mathematically proven guarantees on stability and performance, though no doubt they have disadvantages, too.
Dear Sir, thank you for your channel and your book. These are a real treasure trove of very useful tools for the ‘honest mind’ of the 21st century. I was wondering whether you could direct me to resources regarding genetic algorithms. I have a couple of finance related optimisation problems involving discrete variables (essentially inclusion or exclusion in a portfolio subject to certain constraints) and I believe the genetic algorithms would be the most suitable in these cases. I would be extremely grateful for any guidance/suggestion on this. Many thanks and best regards from sunny Singapore.
At 2:47 you said that in biological systems, the reward comes in the form of dopamine. But how is that dopamine released in the body? Wouldn't it be according to the environment we are in (or when we tell our mind that we are in a positive environment)?. Let me know.
3 unavailable videos are hidden? Doctor Brunton, so these 3 videos are still not open to public? But I still appreciate your first 5 videos which helps me understand reinforcement learning much more deeply. Thanks!
Links to papers in video:
@3:58 Machine learning for fluid mechanics
Brunton, Noack, Koumoutsakos, Ann. Rev. Fluid Mech 52:477--508, 2020
www.annualreviews.org/doi/pdf/10.1146/annurev-fluid-010719-060214
@5:04 Efficient collective swimming by harnessing vortices through deep reinforcement learning
Verma, Novati, Koumoutsakos, Proc. Nat. Acad. Sci. 115(23):5849--5854, 2018
www.pnas.org/content/115/23/5849
@6:57 Automating turbulence modelling by multi-agent reinforcement learning
Novati, Lascombes de Laroussilhe, Koumoutsakos, Nat. Mach. Int. 3:87--96, 2021
www.nature.com/articles/s42256-020-00272-0
@8:47 A review of Deep Reinforcement Learning for fluid mechanics,
Garnier, Viquerat, Rabault, Larcher, Kuhnle, Hachem, 2019
arxiv.org/abs/1908.04127
@9:57 Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control
Rabault, Kuchta, Jensen, Reglade, Cerardi, J. Fluid Mech. 865, 2019
doi.org/10.1017/jfm.2019.62
@10:56 Reinforcement learning for bluff body active flow control in experiments and simulations
Fan, Yang, Wang, Triantafyllou, Karniadakis, Proc. Nat. Acad. Sci. 117(42), 2020
doi.org/10.1073/pnas.2004939117
@11:50 Fluid directed rigid body control using deep reinforcement learning
Ma, Tian, Pan, Ren, Manocha, SIGGRAPH 2018
gamma.cs.unc.edu/DRL_FluidRigid/
@13:26 Autonomous helicopter flight via Reinforcement Learning
Ng, Kim, Jordan, Sastry, NeurIPS 2004
papers.nips.cc/paper/2003/file/b427426b8acd2c2e53827970f2c2f526-Paper.pdf
@13:26 An Application of Reinforcement Learning to Aerobatic Helicopter Flight
Abbeel, Coates, Quigly, Ng, NeurIPS 2007
proceedings.neurips.cc/paper/2006/file/98c39996bf1543e974747a2549b3107c-Paper.pdf
@13:26 Autonomous helicopter aerobatics through apprenticeship learning
Abeel, Coates, Ng, Int J Rob Res 2010
journals.sagepub.com/doi/abs/10.1177/0278364910371999
@14:02 Learning to fly like a bird
Tedrake, Jackowski, Cory, Roberts, Hoburg, Int. Symp. Rob. Res. 2009
groups.csail.mit.edu/robotics-center/public_papers/Tedrake09.pdf
@14:58 Control of a Quadrotor with Reinforcement Learning
Hwangbo, Sa, Siegwart, Hutter, IEEE Rob Aut 2(4) 2017
arxiv.org/abs/1707.05110
@15:22 Learning to soar in turbulent environments
Reddy, Celani, Sejnowski, Vergassola Proc. Nat. Acad. Sci. 113(33, 2016
www.pnas.org/content/113/33/E4877
@16:31 Learning to Fly: Computational Controller Design for Hybrid UAVs with Reinforcement Learning
Xu, Du, Foshey, Li, Zhu, Schulz, Matusik, SIGGRAPH 2019
people.csail.mit.edu/jiex/papers/LearningToFly/index.html
You mentioned that your wife is a neuroscientist and indicated that animal reward is internal. Can you by chance provide a reference article that would describe this in greater detail?
It is so pleasant to see someone doing their work so passionately! You are an outstanding professor, Dr Brunton!
We need more professors and lecturers like you Dr. Brunton. You made academic publications more interesting to study !
you're so incredibly apt at explaining such mind-blowing phenomenon while tying them into interesting and novel areas of study! Thank YOU!
This channel is pure gold.
Thanks!
the best teacher I havve ever seen on youtube . greetings and regards from INDIA !!!!
Wow, thank you!
I'm thinking about Jim Lovell and Apollo 13. After the explosion, he had to relearn flying the craft in a new configuration. He said it would go left when he wanted to go right. But they did it. This kind of work could save lives when we suddenly find ourselves in a new place we didn't count on.
I love these kinds of examples, where humans are able to rapidly re-learn on the fly.
The legend of control engineers 🙏 thanx
Thanks!
I really enjoy the enthusiasm you show when delivering the topics! Perfect and outstanding...
Thank you very much for sharing you great papers and knowledge!
My pleasure!
It makes me want to join Washington University
I'm enrolled in prof. Brunton's April offering at UW, "machine learning for fluids". Will let you know how it is
@@harv609 is this offered as an online course?
Awesome, apply to our program in mechanical engineering!
@@kevalan1042 most courses at UW are offered remotely because of covid, so yea. It's offered as an online course this term
@@Eigensteve are you planning to offer the course on a MOOC platform such as Coursera or edX?
Thank you professor, it's great to watch your videos!
Glad you like them!
Dear prof,
I really admire it.. Your videos make to do research on AI (DRL) in CFD
Thank you professor.
Using the thermals to climb reminds me of the finite horizon, energy optimal trajectory video you just posted.
Fantastic Video
please put the links of videos and papers you mentioned!
Great point. Just pinned a comment with links to all papers referenced.
Great video! One thing I'm curious about - my understanding is that reinforcement learning is difficult in practice b/c it's hard to build a simulated environment which matches reality. But here, it seems that issue has been managed, since these drones are flown in the real circumstance. So, are our turbulence simulations really that good or are there other clever tricks here? In general, do you see the accuracy of the simulation as the primary limiting factor in reinforcement learning?
This makes me really want to get into UW's CS program. Fingers crossed!
Hey Steve, thanks for the great video and the paper highlight ;)
Thank you, fantastic video!
Glad you liked it!
Thank you for making this video
My pleasure!
Great video and content!
Would be really nice if you share how you are creating this video with content overlayed on screen.
So in the case of biological systems the reward is actually just a function of the state ?
Well it is somehow internally motivated, since it is your own body that is dumping the dopamine.
a great video! Thank you
Glad you liked it!
This is so cool !
RL fort the win !
Fantastic job big thanks
Thank you too!
Amazing video thank you
Thank you too!
So these simulations (like in the case of the fish) are essentially used as RL environments, correct? Are these used within the framework of gym or what kind of toolset is used?
here we go!
Do you think these AI control techniques will replace current robust controller design techniques such as mu-synthesis, etc.?
I guess the advantage of traditional methodologies is that they come with mathematically proven guarantees on stability and performance, though no doubt they have disadvantages, too.
I don't think so, but they serve different purposes. Eventually I see RL leveraging more traditional control techniques more effectively.
Dear Sir, thank you for your channel and your book. These are a real treasure trove of very useful tools for the ‘honest mind’ of the 21st century. I was wondering whether you could direct me to resources regarding genetic algorithms. I have a couple of finance related optimisation problems involving discrete variables (essentially inclusion or exclusion in a portfolio subject to certain constraints) and I believe the genetic algorithms would be the most suitable in these cases. I would be extremely grateful for any guidance/suggestion on this. Many thanks and best regards from sunny Singapore.
Dear professor please prepare an education about using Q learning PID contriller
Steve, have you ever thought about creating a MOOC?
I have, but I like keeping everything 100% open on RUclips. But check out databookuw.com for more information on syllabi, homework, etc.
At 2:47 you said that in biological systems, the reward comes in the form of dopamine. But how is that dopamine released in the body? Wouldn't it be according to the environment we are in (or when we tell our mind that we are in a positive environment)?. Let me know.
Yes, but it is delivered internally, based on our internal perception of the external state.
@@Eigensteve but isn’t that equivalent to our brain thinking that we’ve got some positive response in the environment?
Well done!
Thanks!
Question: Are the fluid equations solved using a deep learning, or is it done through more traditional solvers like finite element?
11:42
yeah, those people are called pitchers ...
Hah, good call...
@@Eigensteve steeeeeRiiiiiik
Could u talking about architecture robot interactive/creative and AI
Probably SpaceX uses RL to land their rockets in the ships. It would be very nice to know something about that, however it should be secret.
Interesting... secret RL!
Pretty sure they use classical control. RL doesn't make a lot of sense in that particular case
@@seremetvlad could you explain why you think it doesn't make sense? I'm curious to understand
Perfectly 🎉
Nice
Thanks
Don't worry, we'll find the mystery behind this 'one person' solved by tomorrow 10'am.
e-fish-ently, i got it
Nice :)
3 unavailable videos are hidden?
Doctor Brunton, so these 3 videos are still not open to public? But I still appreciate your first 5 videos which helps me understand reinforcement learning much more deeply. Thanks!