I hope you will continue making videos forever. Your explanations are the best I've ever seen anywhere + the wide choice of topics gives me food for thought when dealing with my own optimization problems.
A thing I absolutely like is how palatable you make these concepts, not too mathematical/theoratical and not overly simplified, just the right balance ( € - greedy is set right 😉)
What an amazing explanation. I am taking a Machine Learning Course and he tried to explain the concept using Bandits but couldn't quite really grasp it in detail. I understood what we are trying to figure out but wasn't quite their yet. You have made it so much easier. Kudos to You Brother.
Love your videos, the quality just keeps going up! PS. the name of the slot machine is "One-armed bandit", because of the long arm-like lever that you pull to play.
It would be great if you made a whole playlist where you explain the statistics for machine learning by explaining the formulas in an intuitive way like you do (you make me understand them all). For example, explain the various distributions and their meaning, statistical tests (p-value), etc. Thank you so much for the work you do and the knowledge you share!
we need to a person like you to democratize these important concepts cannot express how grateful i am to understand these important concepts which i have struggled in the past.
multi-armed bandit is a misnomer really... it should be multi-one-armed-bandit problem. slot machines were called one-armed bandits because they have a single arm that is pulled, and the odds of winning are stacked against the player making them bandits. the goal is not so much about finding out which to play, which would become more apparent given enough plays, but instead to determine which mix of N plays to spread out across the group, settling in on the best mix to achieve exploration in balance against exploiting the best returning bandit. i am a career research scientist pioneering in this field for 40 years... i am always reviewing videos to back-share with students and learners and YOURS have Returned the greatest value for my Exploration, and I will be Exploiting YOURs by sharing them the most with my students. its the best compliment i can think of. cheers. dr vogt ;- )
Well said, needed a refresher after not seeing this for a while and this nailed it. Hopefully you've gone into more advanced topics like MAB reinforcement learning
Thanks for the great explanation. What is the essential difference between contextual bandit (CB) problem vs multi-arm bandit (MB) problem? How does the difference impact the strategy?
2396 was the happiness for that specific case, where restaurant #2 was chosen to exploit. 330 is the (approximate) average regret for every case. So 3000 - 2396 would be correct if you were only talking about that unique case.
Thank you so much for the clarity in this video! However, I thought the regret for the exploit-only strategy would be 3,000 - 2396 = 604. Kindly clarify.
Slot machines were not called bandit but one-arm bandit (they "stole" your money and the bulky box with one lever on its side kind of looked like a one-arm man. So the name of this problem is kind of a pun, a slot machine with more than one levers you can pull (here three) is a multi-armed bandit. ;-)
Thanks a lot for this video! Just one thing I would like to find out here is where we store the result of our learning? like some policy or parameter to be updated?
You've just made a very good point. One strategy I did not note is an epsilon-greedy strategy where the probability of explore in the beginning is very high and then it goes to 0 over time. This would likely be a good idea.
Great videos ! Thanks for your clarification. It's much clearer for me now. But I just wonder how you calculate the 330 regret in the case of exploitation only ?
Good question. You can get that number by considering all possible cases of visiting each restaurant on the first three days. Something like, consider the probability that of the first three days of visits, what is the probability that restaurant 1 is best, vs. probability restaurant 2 is best, etc. You can do this via pencil and paper but I'd recommend writing a simple computer simulation instead.
@@ritvikmath Thank you for this prompt response. I think I get the idea from the epsilon greedy formula (option number 3 in the example). Thank you a lot, your video is really helpful :)
I knew everything from the start. Ate at the same place for 299 days and got pretty bored. So watched youtube and found this video. Now I am stuck at this same restaurant on the 300th day to minimize my regret. Such a paradox. Just kidding. Amazing explanation and example.
Related to regret, we never really know the true distributions (since we can only infer from taking samples). Would you basically just use your estimated distributions at the end of the 300 days as the basis for calculating regret?
enough exploration for good youtube lecture on ml. i should keep exploit this guy. 0 regret guaranteed :)
I hope you will continue making videos forever. Your explanations are the best I've ever seen anywhere + the wide choice of topics gives me food for thought when dealing with my own optimization problems.
Thank you :) I'm happy to help
If he makes videos forever, we'll get zero regrets.
@@ritvikmathdon't let this channel die man
Your explanations, didactics, and dynamism are amazing, way better than several university professors. Well done!
A thing I absolutely like is how palatable you make these concepts, not too mathematical/theoratical and not overly simplified, just the right balance ( € - greedy is set right 😉)
What an amazing explanation. I am taking a Machine Learning Course and he tried to explain the concept using Bandits but couldn't quite really grasp it in detail. I understood what we are trying to figure out but wasn't quite their yet. You have made it so much easier. Kudos to You Brother.
Bro I completed my CS degree with your help and now I got accepted for master and you are still here to help. You are a true man, thx mate
After watching 5 videos, finally I found the best lecture teller for this topic. The examples are great, Thanks.
Love your videos, the quality just keeps going up!
PS. the name of the slot machine is "One-armed bandit", because of the long arm-like lever that you pull to play.
....And the bandit bc it’s the WORST odds in every casino
i guess the slot machine is a bandit cause it keeps robbing money from the players.
It would be great if you made a whole playlist where you explain the statistics for machine learning by explaining the formulas in an intuitive way like you do (you make me understand them all). For example, explain the various distributions and their meaning, statistical tests (p-value), etc. Thank you so much for the work you do and the knowledge you share!
This is the best explanation I have come across so far for the Upper Bound Confidence concept. Thank you!
Thanks Ritvik! this is the best explanation I have come across so far!
Perfect Explanation!
Awesome! Thank you! You helped me a lot!
we need to a person like you to democratize these important concepts cannot express how grateful i am to understand these important concepts which i have struggled in the past.
What a great and easy to understand explanation of MAB - thank you for this!!!!
Your teaching method is highly appreciated. Please make lectures on statistics and machine learning algorithms
U r just awesome ,any person who doesn't have any knowledge of Reinforcement learning can understand,Keep up the spirit...cheers
I just realized that I need to explore more to maximize my happiness. Thank you Multi-Amed Bandit :)
What an amazing explanation! Thank you so much. Keep making such videos.
Great video, and it's really nice listening to you! Thank you :)
Love your videos. To understand the average regret value for exploitation, which extra material should we refer to? Why not 604?
Thank you so much, I passed my exam thanks to your explanation :)
Glad it helped!
multi-armed bandit is a misnomer really... it should be multi-one-armed-bandit problem. slot machines were called one-armed bandits because they have a single arm that is pulled, and the odds of winning are stacked against the player making them bandits. the goal is not so much about finding out which to play, which would become more apparent given enough plays, but instead to determine which mix of N plays to spread out across the group, settling in on the best mix to achieve exploration in balance against exploiting the best returning bandit. i am a career research scientist pioneering in this field for 40 years... i am always reviewing videos to back-share with students and learners and YOURS have Returned the greatest value for my Exploration, and I will be Exploiting YOURs by sharing them the most with my students. its the best compliment i can think of. cheers. dr vogt ;- )
Nicely done.
Thanks so much for explaining this in detail !!
You are so welcome!
Subscribed since few days, your videos are more than excellent! Amazing skill for teaching, thanks a lot.
Awesome, thank you!
This is more than enough for me
Thanks!
Best example ever!!!
Well said, needed a refresher after not seeing this for a while and this nailed it. Hopefully you've gone into more advanced topics like MAB reinforcement learning
I have explored and finally decided that I am going to exploit you!
*Subscribed*
I am new to your channel. You have a talent in teaching my friend. I enjoy your content a lot. Thanks.
Thanks!
Great explanation, can you leave a link to the code, which you used in simulations ?
Thanks! I have a follow up video on Multi-Armed Bandit coming out next week and the code will be linked in the description of that video. Stay tuned!
I'm grateful to you because of this great tutorial.
This is so cool! Thanks for your clear explanation.
The way you explain is stunning, what a awesome lesson.
This is amazing !
Perfectly explained. Genius.
This was a useful supplement to my read of Reinforcement Learning by Sutton & Barto. Thanks.
Glad it was helpful!
i cannot thank you enough for makin this excellent vid!
Awesome cool technique just got hooked to this
Thanks, your work is really awesome.
Thank you too!
Simple and accurate. That is it. Thanks!!!
Why 330 is the response in the explotation example? Should t be;
3000-2396=604??
Thanks, it was quite useful, heading to your Thompson Sampling video :)
Amazing video!
Well explained!
Thanks a lot. Very insightful!
Nice explanation!
Thank you for a great explanation!!
Great video
thanks man, this is truly helpful! 6 min at 2x and I got it all
Great to hear!
This is so clear to me. Thank you for making this video!
Wow, great example and amazing explanation!
Amazing explanation, very clear, thank you Sr
You explained so good
WOW! That's was brilliant! Thank you!
You are very good! Please explore more this topic. Also include the code and explain it
Thanks! Very good explanation!
My exam is in 2 days and I'm so close to graduating with the highest grades.
Thanks for your help!
very clear and simple explaination!
Glad it was helpful!
Very clear explanation. Thanks for this video.
Best explanation!!
Glad you think so!
really nice job! thank you
Well explained! Thank you!
Excellent explanation!
Thanks a lot! Really good representation!
It was awesome technique
👍👍 thanks
thanks for your words!
awesome video ! thanks so much
excellent explanation!!! thanks
crystal clear explanation worth a subscription for more👌
Thanks for the great explanation. What is the essential difference between contextual bandit (CB) problem vs multi-arm bandit (MB) problem? How does the difference impact the strategy?
Brilliant
Very nice explanation, thanks!
Glad it was helpful!
Hi! Thank you for your video. I have a question at 6:28. Why the roh is not simply 3000 - 2396?
2396 was the happiness for that specific case, where restaurant #2 was chosen to exploit. 330 is the (approximate) average regret for every case.
So 3000 - 2396 would be correct if you were only talking about that unique case.
@@senyksia Hey, what do you mean by average regret for every case? I'm still having trouble wrapping my head around this step. Thanks!
@Bolin WU I know it's 8 months already but I wanted to know whether you got the answer or not. I also have the same doubt.
Thanks! I really wish the RLBook authors could explain the k-armed bandit problem as clearly as you do, their writing is really confusing.
Thank you so much for the clarity in this video!
However, I thought the regret for the exploit-only strategy would be 3,000 - 2396 = 604.
Kindly clarify.
Cool explanation. Can you also talk about Upper Confidence Bound Algorithm relating to this?
Good timing! I have a video scheduled about UCB for Multi-Armed Bandit. It will come out in about a week :)
Really good video
Thanks!
Very helpful. How is the regret 300 in the second case? Shouldn't it be 3000 - 2396 = 604?
Slot machines were not called bandit but one-arm bandit (they "stole" your money and the bulky box with one lever on its side kind of looked like a one-arm man.
So the name of this problem is kind of a pun, a slot machine with more than one levers you can pull (here three) is a multi-armed bandit. ;-)
Wow I did not know that, thanks !!
Thanks a lot for this video!
Just one thing I would like to find out here is where we store the result of our learning? like some policy or parameter to be updated?
Nicely explained!
Assuming a finite horizon (known beforehand), aren't you (in expectation) better off doing all the exploration before starting to exploit?
You've just made a very good point. One strategy I did not note is an epsilon-greedy strategy where the probability of explore in the beginning is very high and then it goes to 0 over time. This would likely be a good idea.
Hello Ritvik
This was a very helpful video. You have explained a concept so simply. Hope you continue making such informative videos.
Best wishes.
Thanks so much!
Great videos ! Thanks for your clarification. It's much clearer for me now. But I just wonder how you calculate the 330 regret in the case of exploitation only ?
Good question. You can get that number by considering all possible cases of visiting each restaurant on the first three days. Something like, consider the probability that of the first three days of visits, what is the probability that restaurant 1 is best, vs. probability restaurant 2 is best, etc. You can do this via pencil and paper but I'd recommend writing a simple computer simulation instead.
@@ritvikmath Thank you for this prompt response. I think I get the idea from the epsilon greedy formula (option number 3 in the example). Thank you a lot, your video is really helpful :)
I love this vid! It would be great if you could also do more videos on online learning and regret minimization 😆😆😆
Great explanation
Thanks for the vid boss. How exactly did you calculate the average rewards for the Exploit Only and Epsilon-Greedy strategies though?
*insert taleb nassim talking about the possibility that a meal kills you or changes you whole life* or something like that : p
can you share the calculation for the regret in case of exploitation only?
Would you say exploit only strategy is the same as the eplore-then-commit strategy (also know as explore-then-exploit)?
I knew everything from the start. Ate at the same place for 299 days and got pretty bored. So watched youtube and found this video. Now I am stuck at this same restaurant on the 300th day to minimize my regret. Such a paradox. Just kidding. Amazing explanation and example.
Coolest prof ever! 😎
haha!
Could you explain the difference between the MAB problem and the ranking and selection problem? Thanks
Related to regret, we never really know the true distributions (since we can only infer from taking samples). Would you basically just use your estimated distributions at the end of the 300 days as the basis for calculating regret?
Sir, video on softmax approach.
❤
What ML books do you recommend or use?
Why didn't you discuss the best strategy ... Bayesian Bandits with uniform priors coming from a Beta distribution