Singular Value Decomposition (SVD): Dominant Correlations
HTML-код
- Опубликовано: 22 авг 2024
- This lectures discusses how the SVD captures dominant correlations in a matrix of data.
These lectures follow Chapter 1 from: "Data-Driven Science and Engineering: Machine Learning, Dynamical Systems, and Control" by Brunton and Kutz
Amazon: www.amazon.com...
Book Website: databookuw.com
Book PDF: databookuw.com/...
Brunton Website: eigensteve.com
This video was produced at the University of Washington
![SVD: Image Compression [Matlab]](http://i.ytimg.com/vi/QQ8vxj-9OfQ/mqdefault.jpg)
![SVD: Image Compression [Matlab]](/img/tr.png)
I cannot put in words how much talent you have for teaching. It is just impressive. Thank you so much.
If they had a Nobel prize for teaching on RUclips, this guy would be one of the top contenders
ruclips.net/video/0Ahj8SLDgig/видео.html
You do these fantastic explanations with your original behind-the-whiteboard technique AND you're giving the book this information comes from for free?
??????????
What a gift you are to the internet, thank you so very much. I love your teaching style.
Wow, thank you!
This guy is the refreshed version of professor Gilbert Strang. Professor Steve Brunton, you're an amazing linear algebra teacher. Glad I found these videos. Going to recommend to my entire computational linear algebra classmates.
Thank you so much
Strang will always be the GOAT.
this series is like a gift, thank you so much. Will definitely buy the book when I can :)
Hope you enjoy it! (check out databookuw.com/databook.pdf until then)
I could complain about these videos being to fast-paced and technical for a more novice audience, or that the explanations are not very thorough, or even about the idiosynchratic notation (m for number of columns, like come on matrices are mxn by default). But it still wouldn't change the fact that these videos motivated me to learn more about SVD and served as a invaluable resource for getting introduced to the maths I need for my job. Thank you. I hope there is more content in the making.
Usually some of the explanations can be skipped using Einstein notation but it is often ignored how very educational it can be to explicitly walk though the process. Respect!
I'm highly critical of so-called RUclips "educators." I just watched several on SVD from MIT and Stanford, all of which were garbage. But this... this is art in its purest form. You are a scholar among scholars! Absolutely beautiful to watch unfold. Thank you!!!
Brunton is unparalleled until others do the clear board as well. re MIT tho..Strang is no joke, don't sleep on him. abstruse but distilled well, passionate dude yet elderly linear algebra notable
@@phdrxakadennyblack4098 No doubt. If I could possess the mind of a modern mathematician, Strang is in my top 5, without question. But this video really emphasized the simple elegance of a good TEACHER.
@@ethan6708 Foucault and Baudrillard always made me feel smarter like I learned something significant that they pondered long and hard over. When it is so good that the lecture and the concepts stick clearly in your head for years and years, that to me is the measure of a great teacher.
I hope SB can address the latest stuff I see like the math behind KAN networks as opposed to MLPs, or how ternary filters are so good at approximating so efficiently cutting through data without losing information, if he hasn't already.
Mit and likes are either simplistic to superficial and wasting much time, or like teaching to seniors... Strang is passionate but teach in a unique way, after that you are on your own studies or restart another similar course
I learned SVD 4 years ago, and nobody ever explained it so well! Will recommend this series to every math student!
OMG.... These videos really moved me. I haven't had the feeling to learn knowledge with excitement and refreshment, thank you so much for being such a good teacher. I feel the beauty of math.
Professor Brunton is great. I saw his SVD slightly covered in his in-class lecture. Now he dedicate his time to make this video lecture. Super!
a real genius can dissect complex subject matter to explain it in a simple and intuitive way. and this guy is just a splendid example of such genius
Engaged delivery with exceptional content clarity. If Mr Brunton isn't tenured it would be an indictment of the US university system. Cheer!
I don't think I would ever see this sort of intuitive explanation of U and V matrices in terms of Eigen values and Eigen vectors. Thanks a lot Professor for helping humanity in understanding SVD.
the best grad school prof ever!
One of the best things to have happen in Jan 2020 is seeing you share more of your knowledge! Much Appreciated!
This is one of the best Stuffz I've seen on the internet.....explicit and was taught with so much passion, rigor, Aggressiveness in presentation, clear intuition etc. With these stepwise approach, even the blind is set and equipped to be a Genius and solve real time problems. You don't want to know how grateful I am. Thanks
Sir, do you even have a slightest idea what a superawesomely-superawesome teacher you are?!!!
Thank you, SO MUCH!
🙏🙏🙏
That was so clear and well explained. It's really connecting a lot of dots in the maths I've had to learn for modelling in engineering
Every one is applauding the wonderful lesson, which is very true!!
BUT No one is applauding Steve's wonderful ability to write inverted.
One of the best RUclips explanation i have ever seen.
This is so good. We need more teachers like this!
wow these tutorials needs high level of attention!
thanks for the extraordinary lectures
Your descriptions here are, by far, the best I have seen. Maybe it is because you teach in the way I learn but these have been really useful.
Thank you for this fantastic series, definitely among the best educational content i have come across on youtube!
thanks a lot ❤.. youve made someone in Lebanon get SVD in a very clear way :)))....
Amazing lecture, thank you for making the SVD series! I appreciate the importance of SVD much more after watching your lectures. Thank you again!
this video is on a different level of teaching, thanks
Wow, thank you. I just found your channel today, and I'm grateful for these intuitive explanations. Cheers!
Thank you for making such excellent learning material freely available, you are a godsend.
You are the the best teacher I have seen
Dominant correlations? More like "Dang good pieces of information!" Being serious, this is some of the absolute best content on RUclips.
Thanks a lot.
I haven't watched the series yet, but I already know it's fantastic.
I already liked your classroom course on the same topic.
This lecture cleared a misunderstanding I had about SVD. Thank you so much !
Glad it was helpful!
I am not clear on how X^tX is a correlation matrix, which I understand to be a matrix of correlation coefficients between variables. Really enjoying this series.
Yeah that's my problem too! The video is incredible but I don't get why those matrixes are correlations matrixes
It technically isn't a correlation matrix. If you center each variable (column) by subtracting a mean, then X'X yields a covariance matrix (if you divide the whole result by n-1). If you normalize each column in X by its standard deviation, then X'X yields a correlation matrix. I think he leaves out these details to simplify.
@@neurochannels ty .. that seems plausible
best video on SVD topic, concept crispy clear, and the video is movie quality, really enjoy
Your lectures are treasure for us ...........👍👍❤️❤️
So nice of you!
Thank you so much ! This seemed so complicated and you made It so clear and easily understandable , amazing !
Thank you so much for uploading these amazing lectures!
God Bless you, Steve because you make a big difference in how to teach ML salut from Brazil
Thank you for the great video!
Couple questions:
1. Is the column space of the matrix X what people usually refer to as the feature space?
2. How does it relate to the standard basis on R^n?
3. When you take the inner products in (X^T)(X), is that the inner product in R^n? or some other inner product? i.e pearsons correlation on the space of random variables.
First SVD I've ever liked and like spreading it.
you just got a genius for teaching complex things.
This is as good as it gets, thank you.
I got the book last week, and the Python implementation from the website is a great addition. Thanks again for these priceless videos.
Very good videos you are making! Keep it up!
I honestly understood nothing at all when my professor tried to explain SVD and stuff...i really don't like the lecture style of my university because it is basically: yeah the SVD is a thing and you have the decomposition of input and you can use it and i mean it is easy to see that this are eigenvalues and this are the eigenvectors and you easily get to that formula
Like this is how the professor says it..but without even writing something down or showing anything..or if he writes something it is literally the sentences he said. So i was just like: wtf is this and how does it work and why are these eigenvalues and why this formula and stuff. And sure at a university you should always ask if something is unclear but there is always waaaaay to much to ask about simply because the explanations are a big amount of BS.
And now watching this i understood sooo much..and it even only took 11 minutes..i really wish my university would explain like that :(
anyways i am glad there are people like you so in my freetime i just learn all the stuff on my own 🤷♂️
greatly put. again, linear algebra is magic.
Making my research ever so meaningful! Thank you.
As always, It is really nice to have more prof. Brunton content to learn from. This is the nudge I needed to get your new book and dive deeper to data-driven modeling and machine learning control. Thanks, prof. Brunton. I just finished the series, I am really looking forward to the rest of it.
If the [X]T[X] represents the correlation between the columns, does that mean that the diagonal has the largest elements? And does [X][X]T represent the correlation between the row? Can that be interpreted in any meaningful way?
You're awesome. I wish I could watch this video a few years earlier.
Great thanks professor!!! At last I understood what SVD is
For this amazing videos you are going to heaven.
Thank you, your explanation is beautiful.
THAT WAS A W E S O M E!
Thank you very much for these amazing lecture videos, I bought the book after watching a few of them. I am also very happy that you made it available as an ebook as well, it works great on my Kindle :)
Incredibly well explained. Thank you!
Your teaching is so impressive and awesome.
RUclips should add in option to put multiple likes
For sure, the best explanation I'd seen about the whole topic. I'm very surprised on the production and staging. Prof. Brunton seems to be writing in a transparent surface with actual pens. However, it's not possible to film it from the perspective all we are seeing unless he is doing mirror writing, like Leonardo Davinci. Involved mathematics, fluent speaking and mirror writing, at the same time! too much, even for a sharp mind. So, how was done? Maybe the scene is reflected in a physical mirror and the camera is pointing that mirror, and some linear algebra to correct the perspective 🙂, or maybe everything is advanced postprocessing. Please, we want to know how did you shoot this fantastic video. Greetings from Spain!
This video is incredible. Such a great explanation
Thank you so much! So clear, this is what you need to be a good engineer/scientist in any fields.
very clear. I love it professor
Thank you so much, your videos helped a lot !
Excelent content, really love it! thanks
thank you very much! you video is most intuition!!!😊
Very clean and clear explanation of the computational process for SVD. However, it doesn’t explain much the intuitive concepts of SVD in terms of what it is conceptually as a decomposition of a linear transformation with basis change.
Why would you say U is the eigenvectors of the column space of the data when U is corresponding to X @ X.T which is the correlation between rows of X.
Great explanation! thanks for sharing
My pleasure!
Nicely explained again
I am greatful for such a good series of videos with you Steve as your are really talented. I have a question that might be basic : to obtain a proper correlation matrix, one should center X and reduce it afterwards ? I asked chatgpt but trust you more on this ^^'
I know this specific video is about intuition, I'm just asking to confirm my understanding thanks :)
Thanks, I'm glad you like them! Yes, typically you would center X (subtract the column-wise mean or row-wise mean first, and then take inner products with either all pairs of columns, or all pairs of rows). We discuss this in databookuw.com/databookV2.pdf if you want more details.
Ok super thanks!@@Eigensteve
I think, If nxm X has n features of m faces, then (X-p)(X-p)^T , where p is nx1 vector of row means(here p is broadcast from nx1 to nxm), is the nxn covariance matrix of the features among those faces.
(X-q)^T (X-q), where q is the 1xm row vector of column means (again broadcast to nxm), is the mxm covariance between the faces among those features.
After dividing by the number of data points - 1.
Brilliant. I think there is one little error at 6.10. Sigma has the size mxn, so Sigma Transpose is of the size nxm. They both have sigma1,..,sigma_m in the diagonal. So instead of Sigma^2 we get another mxm matrix. It doesn't change end result,but I thought it was important to mention to avoid confusion.
Beautifully explained :)
Why are the values of the correlation matrix inner products of the columns of X and not of the rows of X? Don't we want to see the correlations between different features of the faces, not between the different faces themselves?
Great question. We dig into this more in the other videos in the playlist. But short answer is that you can look at the row-wise or column-wise correlation matrices, and they will have the same non-zero eigenvalues (really quite surprising when you first think about it). Both of these can be used to extract the SVD.
@@Eigensteve ahh ok thanks so much for the reply! Will watch the whole playlist now. Great videos btw
Thank you for the great content, I have a question:
At 3:54 you say "if these are people's faces then the ij-th entry of this matrix ( X^t X ) is the inner product between person i and person j 's face, so if there's a value of this matrix that is large that means that those two people had a large inner product, their faces are similar, they have the same basic face structure. If you have a small value of this inner product that means that they are nearly orthogonal and they're very different faces".
The inner product (as usually defined for a Euclidean space) of two vectors gives you an idea of the "similarity" of the two vectors, in this case each vector is a "list" of numbers each representing a pixel of a picture of a face. The idea of "if the two vectors are similar then the two faces they represent are similar" supposes that the pictures were somehow taken such that the distribution of the values of the pixels was related to the facial characteristic, correct? Otherwise it's not clear to me how could the similarity between two faces be "detected" (for example two photos of the same face in different lighting/position would be associated to two very different vectors).
Thanks!
Just to note, it is not the correlation matrix but the covariance matrix which is gained by the X^T X etc. Otherwise great video, I really needed this type of intuition for SVD:s.
I know that your post is a year old, but I was surprised that you seem to have been the only person that noticed the "correlation"/"covariance" confusion. Too bad that Steve Brunton did not respond to your post.
This is gold.
Hi. Great explanantion! Just had a slight confusion. Isnt it like Inner product signifies similarity but wont be able to give a correlation since 'technically' correlation could mean a measure of linear reltionship between 2 variables (That is, how much a value cahnges in response to the other)? Bu tin the video, it is marked as a correlation.
Thanks for the video!
Does the data have to be centered/standardized for transpose(X)*X to be the "correlation matrix" ?
Great interpretations!
Great lecture Steve. You say that we usually want the economy SVD. In what situations would we want to compute the full svd please?
Awesome videos!
I performed XT*X for V and X*XT for U on a matrix X. But X does not equal to the U*S*VT. I even converted the values of U and V to orthogonal. Still does not equal to the orignal X. What am I doing wrong?
I just cannot not pushing the thumbs up button thanks
Hi Prof,
at 6:43 , it is substituted U^ (U^)T = I. But U^ is an economy matrix you, earlier it is mentioned that they are no more unitary matrices once truncated. Please clarify this point.
In the previous lecture, he mentioned that for truncated matrices, U'U=I, but it is not the case that UU'=I. In this lecture, he only made use of the first fact. So you are correct they are not "full" unitary, but you only need "half" unitary for this proof to go through.
Awesome!!
The class is just incredible, and you are a very good teacher. But I just want to know, how do you write it in front of you and the text and math symbols are not inverted to us? Thats a nice magic trick.
Good video but it would also be handy if you marked the array sizes of the SVD factors (both full & economy), not just those of the correlation matrix factors.
great video! thanks so much!i have a question... what if you have data where m>>n? so my datapoints outnumber the parameters i have evaluated for each of them. can you still interpret the SVD such that the columns in U and V are the eigenvectors of the row and column correlation matrix, respectively?
ah sorry! i think that may have been a silly question! i think i'll just switch the rows and columns of my data so that n>>>m and just pull out whichever U or V would be more meaningful for my data!
thank you so much
In the book (p. 13) you say:
"_This provides an intuitive interpretation of the SVD, where the columns of
U are eigenvectors of the correlation matrix XX* and columns of V are eigenvectors of X*X".
Isn't the rows of
U are eigenvectors of the correlation matrix XX* and rows of V are eigenvectors of X*X, is it?
Very impressive and didatic.
Great video, thanks for it, but I have a question regarding your explanation of positive semidefinite in minute 4:36. You said that X*X is symmetric and positive semidefinite because the elements are inner products. Nevertheless, according to the definition of an inner product, it is only positive definite when and it does not occur for all the elements of that matrix. Can anyone explain to me if that was a mistake in the video or if I am wrong, please?
Hi! Yes, off-diagonal elements of a matrix can be negative, but this does not mean that the matrix is not positive semi-definite. Positive semi-definite ensures that the eigenvalues are not negative. You can google about the Gram matrix :)
At first I also had this question )
Hello professor, many thanks for the video's. I have watching both your electrical engineering (laplace transform) videos, as well as this one on SVD. I am bit confused where the approximation you state (towards the end of the video) comes from. Is the original SVD itself approximate ? In the case of truncated SVD, we are just dropping the columns that would be zero'd anyway correct ?, so at what point of time do things become approximate.
Hi, so I'm just wondering, does a column in X here represent a feature column, and the rows represent the different samples, or do the columns represent the different samples and the rows represent the different features?
Hi. What isn't a hat V there since we are using the truncated SVD? Thanks.
The class is amazing. thanks. I just wonder where you store the python code for this class
Glad you like it! Python code is at databookuw.com or at github.com/dynamicslab/databook_python
is V the single vectors of the correlation matrix or the data matrix? 8:12
Mind blowing
Why dot product of X is correlation matrix? Did we substructure the mean somewhere?
Is it a guarantee that X^T X is positive semi-definite?