fantastic video, man I was so confused for some reason when my lecturer was talking about it, not supposed to be hard iguess, just how exactly it worked this video helped fill in the details
Great video but how can we use policy iteration for a MDP when the state space grows considerably with each action? I know there’s various methods of approximation for policy iteration but I just haven’t been able to find anything, do you have any resources on this?
the state value function Bellman equation includes the policy action probability at the beginning of the equation which you did not consider in your equation. any reason why?
The living reward is 0, not 0.72. 0.72 is the V at time 2 for grid square (3,3). Use the 0.72 value to update grid squares (2,3) and (3,2) at time step 3.
Instead of starting in square (3, 3), you start in squares (3, 2) and (2, 3). After that, you do the same calculations to get 0.78. The optimal reward in square (3, 2) would be to go up, so the equation will look like: 0.8[0 + 0.9(0.72)] + 0.1[0 + 0.9(0)] + 0.1[0 + 0.9(-1)] = 0.43. The optimal reward in square (2, 3) would be to go right, so the equation will look like: 0.8[0 + 0.9(0.72)] + 0.1[0 + 0.9(0)] + 0.1[0 + 0.9(0)] = 0.52.
I think there is a value V for (2,3) at V2-- it is 0. You get that value taking the "left" action and bumping into the wall, thereby avoiding the -1 terminal state. What action could you take that would result in a value of .09?
Remember you are taking the max of action values. So for (2,3), the max action is to move left, which may result in (2, 3) or (3,3) or (1, 3). The value is all 0.
Hi, thank you for the explanation. Can you please explain how you got 0.78 for (3,3) in 3rd iteration (V3) ? According to bellman equation, I got the value 0.8 * (0.72 + 0.9 * 1) + 0.1 * (0.72 + 0.9 * 0) + 0.1 * (0.72 + 0.9 * 0) = 1.62. Please correct where I got wrong. Assignment due tomorrow :(
@@maiadeutsch4424 Thank you so much for the detailed explanation. This was really helpful. I was not considering the agent's own discounted value when going towards the wall and coming back.
@@maiadeutsch4424 hey ! nice explaination but can you tell if we will get a table regarding probabilities like 0.8 0.1 0.3 etc for going right left up visa versa
For the first moment, you do not need to calculate terminal states and get +1, -1 for them. its wrong ! we have things like terminal state in grid world. use it.
Great video! Walking through the first few its of the VI on a gridworld problem helped me to understand the algorithm much better!
My bald teacher will talk about this for 2 hours and I won’t understand anything. This helps a lot
lmfaooo
I seached many many times to find this solution, and finally I found. Thank you.
kes
this is better explantation than my teacher from MIT, thanks
you're not from MIT lol
Best video on the topic I have seen so far, to the point and well explained! Kudos to you brother!
nice explaination
great video!! thanks!!
fantastic video, man I was so confused for some reason when my lecturer was talking about it, not supposed to be hard iguess, just how exactly it worked this video helped fill in the details
great explanation ! thanks.
Nice job, thanks
Thank you so much! My professor explained this part a bit too fast so I got confused, but this makes a lot of sense!
Helped me learn it. Thank you.
This is helpful, thank you
I see the values at V3 are for gamma only, shouldn't they be for gamma squared?
Great video but how can we use policy iteration for a MDP when the state space grows considerably with each action? I know there’s various methods of approximation for policy iteration but I just haven’t been able to find anything, do you have any resources on this?
For v1, would the two terminal states not be 0.8, since you have to multiply by the probability to get the expected value?
Remember, we're taking the sum and as all probabilites add to 1, we get 0.8x1 + 0.1x1 + 0.1x1 = 1, same for -1
@@aidan6957 thanks, can you send me the timestamp since I no longe have it?
9:06 Why when iterating v2, the values of the all other squares are 0's? Shouldn't the squares near the terminal states have non-zero value?
Can you provide example of policy iteration too
If you also suffer from the vague explanation in GT's ML course, here comes Upenn to rescue you!
Literally why I'm here. CS7641 has been pretty good so far but the RL section was honestly crap in the lectures IMO.
Yes ! Finally found suck a video! Yay!
the state value function Bellman equation includes the policy action probability at the beginning of the equation which you did not consider in your equation. any reason why?
Thank God, get RL videos from an Indian....
In second iteration V = 0.09 ?
nice
why would you substitute value of +1 in equation in green? the formula says it should the V(S') and not reward value!!!
According to bellman equation, I got the value 0.8 * (0.72 + 0.9 * 1) + 0.1 * (0.72 + 0.9 * 0) + 0.1 * (0.72 + 0.9 * 0) = 1.62. Please correct where I got wrong.
The living reward is 0, not 0.72. 0.72 is the V at time 2 for grid square (3,3). Use the 0.72 value to update grid squares (2,3) and (3,2) at time step 3.
Please can you explain how you got the 0.78 in V3.?
do you understand :( if yes please explain for me
Thanks for the video. In v3, how do you get 0.52 and 0.43?
Instead of starting in square (3, 3), you start in squares (3, 2) and (2, 3). After that, you do the same calculations to get 0.78. The optimal reward in square (3, 2) would be to go up, so the equation will look like: 0.8[0 + 0.9(0.72)] + 0.1[0 + 0.9(0)] + 0.1[0 + 0.9(-1)] = 0.43. The optimal reward in square (2, 3) would be to go right, so the equation will look like: 0.8[0 + 0.9(0.72)] + 0.1[0 + 0.9(0)] + 0.1[0 + 0.9(0)] = 0.52.
@@HonduranHunk How can we calculate to have 0.78. please help me sir
In V2 why is it that there is no Value for (2,3)? Doesnt the presence of -1 give it a value of 0.09. I am confused there.
lol same. any confirmations?
I think there is a value V for (2,3) at V2-- it is 0. You get that value taking the "left" action and bumping into the wall, thereby avoiding the -1 terminal state. What action could you take that would result in a value of .09?
Remember you are taking the max of action values. So for (2,3), the max action is to move left, which may result in (2, 3) or (3,3) or (1, 3). The value is all 0.
Good!
quick question: 6:10, is the R(s,a,s_prime) always 0 in the example.
yes it is fixed to zero
@@cssanchit except in terminal states
🙏🙏🏿
Cant understand how it it 0.52
Is it just me or are the subscripts in the wrong directions?
There shouldn’t be any value for the terminal state… my god…
Hi, thank you for the explanation. Can you please explain how you got 0.78 for (3,3) in 3rd iteration (V3) ? According to bellman equation, I got the value 0.8 * (0.72 + 0.9 * 1) + 0.1 * (0.72 + 0.9 * 0) + 0.1 * (0.72 + 0.9 * 0) = 1.62. Please correct where I got wrong. Assignment due tomorrow :(
+1
@@maiadeutsch4424 Thank you so much for the detailed explanation. This was really helpful. I was not considering the agent's own discounted value when going towards the wall and coming back.
@@maiadeutsch4424 we don't need to multiply 0.1??
@@maiadeutsch4424 hey ! nice explaination but can you tell if we will get a table regarding probabilities like 0.8 0.1 0.3 etc for going right left up visa versa
@@maiadeutsch4424 hey there nice explanation, but for the cases with 10% chance it should be 0.1*(0 + 0.9*V_previter).
Seriously people cant explain this in a easy way. Same for this video
For the first moment, you do not need to calculate terminal states and get +1, -1 for them. its wrong !
we have things like terminal state in grid world. use it.
what do you mean?