This russian doll of dichotomies has always been a mind bender, often it seems the literature has nebulous definitions and the boundaries aren't so clear. Thank you for the great insights in this lecture, the graphic is superb.
Superb. One of the things I always I struggle with when learning something is having a well structured map in my head of the topic and subtopics and this does an extremely good job of doing that. Many thanks.
Great video! Though imho the on/off-policy distinction explained at 14:24 might be a bit misleading. I believe both on/off policy can explore sub-optimal actions with something like epsilon-greedy.
I found several definitions from internet sources and papers, and this video is quite enlightening and confirms what they conclude. Very much Thank you sir.
It's been a great series of videos on RL! I'm updating my research interests and now I want to combine MPC with RL in such a way that the resulting control structure can be safely implemented and has some stability guarantees. Thank you very much!
At 14:20 to 14:41, you state On-Policy is always playing the best game possible. Is this approach the same as greedily picking the best action at each state? If so, would On-Policy algorithms not include exploration such as epsilon-greedy? The way I understood On-Policy vs Off-Policy here is that On-Policy is purely exploitation whereas Off-Policy is both exploration & exploitation. Am I misunderstanding it? Thanks!
I was also a bit confused at that point. But, I think you have probably misunderstood it since, for RL, we have to always ensure exploration & exploitation. I feel that what I understand is completely the opposite to Prof. Steve's description: on-policy uses the to be improved policy to select an action (meaning not always the best action), while the off-policy uses a different policy to decide which action to take (you may always choose the best action). I hope Steve will elaborate more on it. :)
The main difference between on-policy and off-policy methods is the way the generated data is used during learning. If the method uses the data from the current policy, it is on-policy learning. But if the method uses data from another policy (e-greedy or older policies like replays), it is an off-policy method.
The distinction between On-Policy and Off-Policy explained in this video seems to be different from other sources on the internet. I'm trying to get my head around reinforcement learning and I have noticed that different people have different understandings of certain concepts. Model-free and Model-based are also given a different distinction by others, and this really throws me off. I'm not saying the explanation in this video is incorrect, but that there are different explanations elsewhere and I'm not sure which one is correct.
Very nice overview video! There is a small typo in the non-linear dynamic equation, the superflous dt on the right. Regarding how MPC fit in the whole DP framework, I remember Prof. Bertsekas was presenting it as a way to approximate cost-to-go online.
I have Energy data and I need to implement RL on these data (Inverter) to achieve the best result (when charge/discharge battery, when is the best time to feed in grid,etc.) which algorithm should I use for that ?
Thanks for the wonderful videos. It would be great if you add real code to the end of the main videos. it would be very easier to understand with detail.
I am a happy owner of you "Data Driven Science and Engineering" book. That fact that there will be much more content on RL in the 2nd edition is really good news! Will there also be a print version of the 2nd edition of your and Kutz' Book?
At 20:07 Steve said about the "model of system" that if we have a "model of system" we use the model-base, and if we don't have a "model of system" just use model-free. So, can you explain me more about "model of system". What is it? Example? and Why? Thank you so much.
Model of the system means you know every possible state of the system, every possible next state and the actions that take you there (& with what probability) as well as the rewards earned after every state & action pair. You can have a model of a maze, you CANT have a model of a chess game (unless you make assumptions about the opposing players strategy).
oil lamp except the clear liquid is a gradient and the bubbles are parameters that perform an action. then its p>p' goes to q, then flipflops p q>q'. p'=p or q'=q if the AI improves at a particular task. two randoms cancel each other out on an error function and acts like an implicit rolles theorem without explicitly stating d/dx=0.
I was reading 1 journal article and found that the author claimed following a model-free RL problem but they have used Markov Decision process to model the problem? they have not mentioned probabilities for states. What does that mean? Also there is another paper which used probabilities for state transitions and solved the problem using Q-learning, so it's all confusing again.
Why is "Actor Critic" assigned (only) to the left side? Isn't it (also) a combination of gradient free and gradient based algorithms, e.g. the Critic is a DQN and the Actor is a Deep Policy Network?
Great series! Can you make a contro bootcamp like series for non-linear control theory? Would love to see some simplified explanations for topics like PDE backstepping, reference governors, lyapanov stability criteria etc.
Thanks so much for such great video. Can you please tell where does Proximal Policy Optimization (PPO) fit in these categories. For my case a small game, I know that I will be using model free RL, but not able to decide what else to use apart from Q-Learning.
Hi Mr. Brunton, Your videos are impressive and thank you for making the content. A small suggestion though, it will be better for us to navigate if you make separate playlists with orders in a particular content.
I enjoy your broad strokes topics. I was wondering can an AI write to its memory once it learns or discovers something new ? Or it doesn't work like that.
Nice lectures and lots of stuffs to learn. Thanks for sharing. Are the On policy and Off policy somehow related to exploitation and exploration concept?
Off plolicy..on policy is slightly confusing here...isnt off policy, a setup where you have the prior data and cant continuously interact with the environment
Dear professor please explain to us how to use reinforcement learning to tune pid controller gains I’m looking forward to hearing from you Sincerely mohammad
Dear professor, please explain to us about how to use reinforcement learning to tune pid gains ❤️ Im looking forward to hearing from you Sincerely mohammad
I deeply appreciate the quality of knowledge you are providing to the community. please continue to democratise knowledge.
This russian doll of dichotomies has always been a mind bender, often it seems the literature has nebulous definitions and the boundaries aren't so clear. Thank you for the great insights in this lecture, the graphic is superb.
Thanks!
Superb. One of the things I always I struggle with when learning something is having a well structured map in my head of the topic and subtopics and this does an extremely good job of doing that. Many thanks.
Great video! Though imho the on/off-policy distinction explained at 14:24 might be a bit misleading. I believe both on/off policy can explore sub-optimal actions with something like epsilon-greedy.
OK, this year is gonna be better than I thought. Thanks, professor!
I found several definitions from internet sources and papers, and this video is quite enlightening and confirms what they conclude. Very much Thank you sir.
Wow! Steve, you've managed to break this all down into bite-sized chunks. Thank you 🙏
0:00 Intro
3:00 Background
7:54 Model & Model-Free Reinforcement Learning (RL)
8:29 Markov Decision Process (MDP)
10:25 Nonlinear Dynamics
13:02 Gradient & Gradient-Free RL
14:05 Off-Policy (Q Learning) & On-Policy (SARSA) RL
17:23 Policy Gradient Optimization
18:05 Deep RL
You helped me during my undergrad, now you're an inspiration to me during my masters.
Finally! Thank you for you posting. Can't wait to see the whole playlist.
I just started getting back into RL so this comes at a perfect time! Looking forward 👌
Please make a separate playlist for reinforcement learning :-)
Good call -- will do
@@Eigensteve please put the videos in order, the current order is not correct. But great content.
You really invest perfectly the time in your lessons, very very useful! great series!
Thanks! :)
Such a high quality course and a free book in description? You're awesome!!
Really really great overview for those new to learning about reinforcement learning! Thanks so much!
I am really grateful for your eye-opening videos, especially this one
Thanks!
Fantastic opening video. You're a talented teacher and I appreciate this content. Looking forward to watching the entire series.
It's been a great series of videos on RL! I'm updating my research interests and now I want to combine MPC with RL in such a way that the resulting control structure can be safely implemented and has some stability guarantees. Thank you very much!
At 14:20 to 14:41, you state On-Policy is always playing the best game possible. Is this approach the same as greedily picking the best action at each state? If so, would On-Policy algorithms not include exploration such as epsilon-greedy?
The way I understood On-Policy vs Off-Policy here is that On-Policy is purely exploitation whereas Off-Policy is both exploration & exploitation. Am I misunderstanding it?
Thanks!
I was also a bit confused at that point. But, I think you have probably misunderstood it since, for RL, we have to always ensure exploration & exploitation. I feel that what I understand is completely the opposite to Prof. Steve's description: on-policy uses the to be improved policy to select an action (meaning not always the best action), while the off-policy uses a different policy to decide which action to take (you may always choose the best action).
I hope Steve will elaborate more on it. :)
Coming to this video after a while. Really great video, thank you!!
An excellent arrangement of a very tough topic, logical and in the proper flow, keep up the very good job
Thank you.
The main difference between on-policy and off-policy methods is the way the generated data is used during learning. If the method uses the data from the current policy, it is on-policy learning. But if the method uses data from another policy (e-greedy or older policies like replays), it is an off-policy method.
Needed this..... Thank you Professor
What an amazing start to the new year! 😍
The dichotomy break down are so awesome...
The distinction between On-Policy and Off-Policy explained in this video seems to be different from other sources on the internet. I'm trying to get my head around reinforcement learning and I have noticed that different people have different understandings of certain concepts. Model-free and Model-based are also given a different distinction by others, and this really throws me off. I'm not saying the explanation in this video is incorrect, but that there are different explanations elsewhere and I'm not sure which one is correct.
Great channel. You have a knowledge in the field that I was looking for that is mix on AI/NN and control theory.
Dear steve its amazing category to classify the reinforcement learning thanks alot
There is a typo at 10:08, the dynamic model in continuous time should be dx/dt=f(x,u,t) only
Good catch, thanks!
Thanks, Sir, Please add some robotics-related examples in the upcoming series also.
Thanks for the suggestion!
Great teacher and master of the art!
Very nice overview video! There is a small typo in the non-linear dynamic equation, the superflous dt on the right. Regarding how MPC fit in the whole DP framework, I remember Prof. Bertsekas was presenting it as a way to approximate cost-to-go online.
Good catch on the typo! And interesting perspective on MPC -- thanks!
Great didactic, congratulations! I used to confuse myself frequently when dealing with these concepts.
Great synthetic and dense video ! Thank you very much for sharing !
I have Energy data and I need to implement RL on these data (Inverter) to achieve the best result (when charge/discharge battery, when is the best time to feed in grid,etc.) which algorithm should I use for that ?
Dear Steve, great explanation. However, just wanna confirm: I thought Actor-Critic is a model-free model?
Actor critic should be in the policy gradient optimization no?
The heart of AI is reinforcement learning, it is the only most interesting in whole AI/ml. Basically original AI .
Thanks professor 🤝👍
this explanation is just beautiful! Thx so much
Good stuff! You need to remove the "dt" on the right side of your nonlinear dynamics equation.
Great presentation 👏
Thanks for the wonderful videos. It would be great if you add real code to the end of the main videos. it would be very easier to understand with detail.
I am a happy owner of you "Data Driven Science and Engineering" book. That fact that there will be much more content on RL in the 2nd edition is really good news! Will there also be a print version of the 2nd edition of your and Kutz' Book?
Thanks! Yes, the print edition should be out sometime later this year.
This is amazing. Thank you, steve!
At 20:07 Steve said about the "model of system" that if we have a "model of system" we use the model-base, and if we don't have a "model of system" just use model-free. So, can you explain me more about "model of system". What is it? Example? and Why? Thank you so much.
Model of the system means you know every possible state of the system, every possible next state and the actions that take you there (& with what probability) as well as the rewards earned after every state & action pair. You can have a model of a maze, you CANT have a model of a chess game (unless you make assumptions about the opposing players strategy).
I would like to have RL as a career , and you would be the best lecturer for a kickstart.
oil lamp except the clear liquid is a gradient and the bubbles are parameters that perform an action. then its p>p' goes to q, then flipflops p q>q'. p'=p or q'=q if the AI improves at a particular task. two randoms cancel each other out on an error function and acts like an implicit rolles theorem without explicitly stating d/dx=0.
I was reading 1 journal article and found that the author claimed following a model-free RL problem but they have used Markov Decision process to model the problem? they have not mentioned probabilities for states. What does that mean? Also there is another paper which used probabilities for state transitions and solved the problem using Q-learning, so it's all confusing again.
Great overview, which is just what I need, thank you sir!
What is difference between deterministic policy and stochastic policy?
hi steve, I think actor critic are usually considered model-free
The link for the new chapter of the 2nd edition of the book is not working for me. Can someone post the correct link in the comments?
Yeah it doesnt work :(
Maybe use databookuw.com/databook.pdf
Thank you, this link is working, but it still shows the 1st edition of the book I think.
@@csalahuni Shoot, sorry, here is the chapter: faculty.washington.edu/sbrunton/databookRL.pdf added to description too
@@Eigensteve tnx professor 🙏
Why is "Actor Critic" assigned (only) to the left side? Isn't it (also) a combination of gradient free and gradient based algorithms, e.g. the Critic is a DQN and the Actor is a Deep Policy Network?
I just have one doubt, is A3C a model free one?
I am a beginner in RL field... i think A2C is an actor critic algo as well (and It's a model-free RL)
Sir, it model free algorithm uses Marakove decision proce (MDP)?
Great series! Can you make a contro bootcamp like series for non-linear control theory? Would love to see some simplified explanations for topics like PDE backstepping, reference governors, lyapanov stability criteria etc.
Would love to do a bootcamp on this -- maybe a goal for the new year! :)
Thank you! Happy New Year!
Thanks so much for such great video. Can you please tell where does Proximal Policy Optimization (PPO) fit in these categories.
For my case a small game, I know that I will be using model free RL, but not able to decide what else to use apart from Q-Learning.
Dear steve we can use ls algorithm instead of gradient algorithm isn’t it
Hi Mr. Brunton,
Your videos are impressive and thank you for making the content. A small suggestion though, it will be better for us to navigate if you make separate playlists with orders in a particular content.
Thnx for video professor.
I enjoy your broad strokes topics. I was wondering can an AI write to its memory once it learns or discovers something new ? Or it doesn't work like that.
" ABSTRACT IS ABSTRACT TO YOU VARIABLLES " @ THE PROFF 7:23
thanks for your great videos
Excellent. What other topics will be included in the 2nd edition?
Updates throughout, all code in Python and Matlab (with R and Julia online), and new chapters on RL and physics informed machine learning
Steve, you are amazing.
Nice lectures and lots of stuffs to learn. Thanks for sharing. Are the On policy and Off policy somehow related to exploitation and exploration concept?
Great as always.
Great job Prof. Steve. How about multi-agent based DRL especially graph learning based RL. That can be a remarkable addition to your playlist.
Hello Professor, where will DDPG algorithm sit in this chart?
Can anyone explain RL by comparing it with ML mathematically? I know much about ML but getting trouble understanding RL.
Off plolicy..on policy is slightly confusing here...isnt off policy, a setup where you have the prior data and cant continuously interact with the environment
Dear professor please explain to us how to use reinforcement learning to tune pid controller gains
I’m looking forward to hearing from you
Sincerely mohammad
Dear professor, please explain to us about how to use reinforcement learning to tune pid gains ❤️
Im looking forward to hearing from you
Sincerely mohammad
Thank you!
Thank you for sharing.
Will we see the programs teaching each other? (like chess)
Actor critic is model based ???? I dont think so.
Sound level is low.
Excellent sunmary
Bated breath. No, really.
Hurry uuuuh-uuuup!