Derivation of Recursive Least Squares Method from Scratch - Introduction to Kalman Filter

#kalmanfilter #estimation #controlengineering #controltheory #mechatronics #adaptivecontrol #adaptivefiltering #adaptivefilter #roboticsengineering #roboticslab #robotics #electricalengineering #controlengineering #pidcontrol #roboticseducation
If you need help with your professional engineering problem, or you need to develop new skills in the fields of control, signal processing, embedded systems, programming, optimization, machine learning, robotics, etc., we are here to help. We provide professional engineering services as well as tutoring and skill development services. We have more than 15 years of industry, research, and university-level teaching experience. Describe your problem and we will send you a quote for our services. The contact information is [e-mail]
It takes a significant amount of time and energy to create these free video tutorials. You can support my efforts in this way:
- Buy me a Coffee: www.buymeacoff...
- PayPal: www.paypal.me/...
- Patreon: www.patreon.co...
- You Can also press the Thanks RUclips Dollar button
The webpage accompanying this video is given here:
aleksandarhabe...
In this video tutorial and in the accompanying web tutorial, we explain how to derive a recursive least squares method from scratch. The recursive least squares method is a very important method since it serves as the basis of adaptive control, adaptive estimation, Kalman filter, and machine learning algorithms. In this video, we start with the measurement equation, and by formulating the cost function that sums the variances of the estimation error, and by solving this cost function we obtain the recursive least squares gain matrix. We also derive an expression for the propagation of the estimation error covariance matrix.

[@aleksandarhaber]

If you need help with your professional engineering problem, or you need to develop new skills in the fields of control, signal processing, embedded systems, programming, optimization, machine learning, robotics, etc., we are here to help. We provide professional engineering services as well as tutoring and skill development services. We have more than 15 years of industry, research, and university-level teaching experience. Describe your problem and we will send you a quote for our services. The contact information is [e-mail]
It takes a significant amount of time and energy to create these free video tutorials. You can support my efforts in this way:
- Buy me a Coffee: www.buymeacoffee.com/AleksandarHaber
- PayPal: www.paypal.me/AleksandarHaber
- Patreon: www.patreon.com/user?u=32080176&fan_landing=true
- You Can also press the Thanks RUclips Dollar button

[@gj5450]

Hello, I'm a college student majoring in aerospace engineering in Korea. This video helped me a lot in learning recursive least squares in probability and random variables classes. Thank you so much for the perfect explanation!!

[@aleksandarhaber]

Glad it was helpful!

[@kakunmaor]

The best explanation I've ever seen on the subject

[@aleksandarhaber]

Thank you for the encouraging comment!

[@aleksandarhaber]

The webpage accompanying this video is given here:
aleksandarhaber.com/introduction-to-kalman-filter-derivation-of-the-recursive-least-squares-method-with-python-codes/

[@pytydy]

Thanks for the video and the write-up. In Eq (26), in the second term of the last formula, e_{k-1}^T should be e_{k-1}.

[@aleksandarhaber]

@@pytydy thank you, this is corrected I think

[@mehdifrotan4141]

Thanks for the great work that you are doing!!!

[@aleksandarhaber]

Thank you Mehdi!

[@maxwellsdaemon7]

At 26:56, the derivative formulas in (36), (37) and (38), X should be K (or K should be X).
Anyway, I've always wanted to understand the Kalman filter, thanks for making this video.

[@aleksandarhaber]

Yes, you are correct, I will correct this in the post I wrote. Thank you very much for noticing this and informing me!

[@andresariaslondono7003]

Excellent video. Very well explained !!!! I followed your code and replicated in Matlab, It works great !! I had to be aware of the dimensions of the matrices and vectors. For example: the kalman matrix in this case is a column vector of three elements. The covariance matrix is a diagonal matrix whose dimensions corresponds to nxn where n is the number of variables to estimate (correct me if I am wrong); and last but not the least the Ck depends on the nature of the system. Thank you very much

[@aleksandarhaber]

Glad it helped!

[@user-nh8mu8se6f]

Very excellent videos and posts, I learned kalman filter with your tutorial, thanks so much for your great contribution. By the way, I noticed a small mistake in the equation numbering in the post. In the sentences 'By substituting (49) in (20),' and 'We substitute (47) in (49)' the equation number should be 33 instead of 49.

[@aleksandarhaber]

Thank you ERIC X! I will double-check these typos and correct them in the tutorial.

[@lamaabdullah1937]

Thank you for your effort this is what I'm looking for!

[@aleksandarhaber]

Thank you Lama. Like and subscribe please.

[@lamaabdullah1937]

Yes already done ✅

[@aleksandarhaber]

@@lamaabdullah1937 thank you!

[@TheProblembaer2]

Thank you!

[@aleksandarhaber]

Glad it helps!

[@zhengrongshang1571]

Thanks!

[@aleksandarhaber]

Thank you very much for your donation! I really appreciate it!

[@arjunmore7545]

Thanks 😇

[@aleksandarhaber]

Glad it helps!

[@michaelbaudin]

Thank you for the explanations. Don't you think that "Iterative least squares" would be a better name?

[@aleksandarhaber]

In signal processing, control engineering, and ML, we usually call this method recursive least squares or in short RLS. Maybe in some books, you will also find the name "iterative least squares method". Call it as you wish, as long as you know what it is and how to use it. It is one of the most fundamental methods in control engineering, and especially in system identification and adaptive control.

[@jackhughman9583]

Hello, Thank you very much for the wonderful explanation. I just couldn't understand how the derivation [Eq. no. 42] answer has the term 2*KkPk-1(Ck)^t unlike the formula gives X*B^t + X*B. Similar for Eq no. 43.If you could explain it would be very helpful. Thank you.

[@aleksandarhaber]

because B is symmetric in our case, in our case, B should be C_{k}P_{k-1}C_{k}^{T} (double check the derivative formula since I do not have time to do that now). Since P_{k-1} is symmetrix, you can figure out that if you take the transpose of C_{k}P_{k-1}C_{k}^{T} you will exactly obtain C_{k}P_{k-1}C_{k}^{T}

[@ly3282]

excellent video！btw could you please list the references you used for making this video?Could you please suggest any textbooks on this topic(RLS，RLS with forgetting factor)?

[@aleksandarhaber]

The best book on Kalman filtering for beginners is (in my opinion) "Optimal State Estimation: Kalman, H Infinity, and Nonlinear Approaches", by Dan Simon. My lecture is partly based on this book. Then, RLS is extensively covered in the book: Linear Estimation by Kailath and Sayed, and in System Identification: Theory for the User, by Ljung.