Видео

The problem I have with these videos is that they always assume the "terrain" is a like a bowl. However, I test on problem where there are 5 parameters, and the "terrain" is more like the Grand Canyon. The path is extremely narrow and the "walls" are steep. Non of these algorithms work very well by themselves. I have also found that a line search really helps more than any of the mentioned methods. I did gives this video a thumbs up because it is one of the best on this topic. BTW, for my data the scipy.optimize.least_squares works MUCH better. Memory is cheap,

Wow super complete information now I'm subscribed now on 😮

Great video! Though I'm having trouble running parts of the code that involves calling classes from other modules. Is there anything I can do about it? This is the error message:- ImportError: cannot import name 'AgentNN' from 'agent_nn'

Excellent

“Only two things”, think you forgot one bud

you look more gujarati/marwari than bengali

bhalo video TY

Hi Sourish, the video was very helpful I found the following config on Amazon, how would you rate it. Plan to run some Ollama models and few custom projects leveraging smaller size LLMS Cooler Master NR2 Pro Mini ITX Gaming PC- i7 14700F - NVIDIA GeForce RTX 4060 Ti - 32GB DDR5 6000MHz - 1TB M.2 NVMe SSD

Thank you for such easy, simple, and great explanation. I searched quick overwiev how Adam is working and found your video. Actually I am training DRL Reinforce Policy Gradient algorithm with theta parameters as weights and viases from CNN, where exactly Adam is involved. Thanks again, very informative.

This feels like a step by step tutorial, great job! I’m building my RTX 4070 Ti machine learning PC soon, can’t wait!

This video is amazing! You covered most important topic in ML, with all major optimization algorithms. I literally had no idea about Momentum, NAG, RMSprop, AdaGrad, Adam. Now, I have a good overview of all, will deep dive in all of them. Thanks for the video! ❤

Woah what a great video! And how you are helping people on the comments kind of has me amazed. Thank you for your work!

Fantastic video for those learning about the topic. Thank you!

10:19 What a weird formula for NAG! It's much easier to remember a formulation where you always take antigradient. You want to *add* velocity and take gradient with *minus* . The formula just changes to V_t+1 = b V_t - a grad(Wt + b V_t) W_t+1 = W_t + V_t+1 It's more intuitive and more similar to standard GD. Why would anyone want to change these signs? How often do you subtract velocity to update the position? Do you want to *add* gradient to update V right after you explained we want to subtract gradient in general to minimize the loss function? It makes everything twice as hard and just... wtf...

Great clear nd thorough content. I look forwrard to seeing more! 🤓

I echo other comments; this is such a great video and you can see the effort put in, and you present your knowledge really well. Keep it up :)

Thanks for the great explanations! The graphics and benchmark were particularly useful.

I tried using momentum for a 3SAT optimizer token i worked on in 2010. Doesn't help with 3SAT since all variables are binary. It's cool that it works with NNs though!

Im new to coding im generell. how do i make a python file like you did?

check out fracm optimizer (Chen et al. - An Adaptive Learning Rate Deep Learning Optimizer Using Long and Short-Term Gradients Based on G-L Fractional-Order Derivative)

very clearly explained - thanks

is the code available?

What is the total Cost of this Setup?

This video is super helpful my god thank you

I might just wanna attend GTC next year, nice video!

the implementation has more information than a whole semester of my MSc in AI

Sorry did I misunderstand something or did you say SGD when it was only GD you talked about? When was stochastic elements discussed?

I am the only one to not understand the RMS propagation math formula? What is the gradient squared is it per component or is the Hessian? How do you divide a vector by another vector? Could someane explain me please.

what a title 😂

Very Clear Explanation! Thank you. I especially appreciate the fact that you included the equations.

I wonder if we could use the same training loop NVIDIA used in the DrEureka paper to find even better optimizers.

The intuition behind why the methods help with convergence is a bit misleading imo. The problem is not in general with slow convergence close to optimum point because of a small gradient, that can easily be fixed with letting step size depend on gradient size. The problem that it solves is when the iterations zig-zag because of large components in some directions and small components in the direction you actually want to move. By averaging (or similar use of past gradients) you effectively cancel out the components causing the zig-zag.

Absolutely loved the graphics and intensive paper based proof of working of different optimizers , all in the same video. You just earned a loyal viewer.

The “problem” the Adam algorithm in this case is presented to solve (the one with local and global minima) is simply wrong - in small amounts of dimensions this is infact a problem, but the condition for the existence of a local minima grows more and more strongly with the amount of dimensions. So in practice, when you have millions of parameters and therefore dimensions, local minima that aren’t the global minima will simply not even exist, the probability for such existence is simply unfathomably small.

This Server is a dream 😄

I used to have networks where the loss was fluctuating in a very periodic manner every 30 or so steps and I never knew why that happened. Now it makes sense! It just takes a number of steps for the direction of Adam weight updates to change. I really should have looked this up earlier.

why not using a metaheuristic approach?

Great video dude!

Gemini 1.5 Pro: This video is about optimizers in machine learning. Optimizers are algorithms that are used to adjust the weights of a machine learning model during training. The goal is to find the optimal set of weights that will minimize the loss function. The video discusses four different optimizers: Stochastic Gradient Descent (SGD), SGD with Momentum, RMSprop, and Adam. * Stochastic Gradient Descent (SGD) is the simplest optimizer. It takes a step in the direction of the negative gradient of the loss function. The size of the step is determined by the learning rate. * SGD with Momentum is a variant of SGD that takes into account the history of the gradients. This can help the optimizer to converge more quickly. * RMSprop is another variant of SGD that adapts the learning rate for each parameter of the model. This can help to prevent the optimizer from getting stuck in local minima. * Adam is an optimizer that combines the ideas of momentum and adaptive learning rates. It is often considered to be a very effective optimizer. The video also discusses the fact that different optimizers can be better suited for different tasks. For example, Adam is often a good choice for training deep neural networks. Here are some of the key points from the video: * Optimizers are algorithms that are used to adjust the weights of a machine learning model during training. * The goal of an optimizer is to find the optimal set of weights that will minimize the loss function. * There are many different optimizers available, each with its own strengths and weaknesses. * The choice of optimizer can have a significant impact on the performance of a machine learning model.

Very nicely explained. Wish you brought up the relationship between these optimizers and numerical procedures though. Like how vanilla gradient descent is just Euler's method applied to a gradient rather than one derivative.

7hrs of work per day, that's pretty sweet. Wondering how many hours your Chinese colleagues do...

Gem of a channel!

what a good video, I watched it and bookmarked so I can come back to it when I understand more about the topic

I dont know what I did for youtube to randomly bless me with this gem of a channel, but keep your work up man. I love your content, its nice to see people with similar passions.

This is so cool, im definitely gonna try this when I get my hands on some extra hardware. Amazing video. I can also imagine this must be pretty awesome if youre some sort of scientist/student at a university that needs some number crunching machine since youre not limited to being at your place or some pc lab.

Just found out your channel. Instant follow 🙏🏼 Hope we can see more Computer Science content like this. Thank you ;)

I need help. I tried using the code and the trials are being saved somewhere, but I can't find it. can you tell me where it is getting stored at? Edit: I found it. it was stored in the C:User\(UserName)\AppData\Local\Temp folder.

Why do you move your head so much

I remembered when my teacher gave me assignment on optimizers I have gone through blogs, papers and videos but everywhere I see different formulas I was so confused but you explained everything at one place very easily.

love that title haha