I looked at many blogs and explanations but none of them got to the practical usage of LoRA and showed exactly how it is used in practice. This is exactly what I was looking for.
These videos are life-saver for my upcoming interview. So in depth and shows the code as well. Easy to follow along. Way better than my university. Thank you so much for the uploads.
I'm genuinely impressed by the content and presentation you've crafted for the ML/AI community. The way you've structured the presentation is both user-friendly and cohesive, allowing for a gradual and understandable flow of information.
Is it like the base model is stored in 4bit and as the data (X vector) passes through the layer that layer is first dequantized and then the matrix multiplication is done (X*W)? And the same thing for LoRA as well? and after we get Y (by adding output of lora and base layer) the W and LoRA layers are again quantized back to 4bit? and Y is passed on to next layer? Also, if the LoRA is at the base of the model, does that mean to update the parameters of this LoRA we need to calculate the gradients of loss wrt all the W and LoRA matrices above it?
Thank you for the great video. Now I am wondering how LLM or diffusion trined with LoRA. These models have many layers which attention, dense, fully connected.. how does LoRA adapted for this? In the digit example, number '9' becomes better result after LoRA adaption. But other numbers accuracy become much worse. Is this natural for LoRA apdation? or We can make all number accurately with LoRA (which is trained for 9)? Thank you very much!
In LoRA, Wupdated = Wo + BA, where B and A are decomposed matrices with low ranks, so i wanted to ask you that what does the parameters of B and A represent like are they both the parameters of pre trained model, or both are the parameters of target dataset, or else one (B) represents pre-trained model parameters and the other (A) represents target dataset parameters, please answer as soon as possible
To push loRa to its efficient limit, does it make sense to find the rank of the original weight matrices by finding statistically significant singular values with Marchenko-Pastur Law to choose the rank of the LoRA matrices?
Hello everyone, I have been in Deep Learning space for about some time now..Sometimes when I come across new codes like this in the video..I just can't keep up many of things in code feels new and gets overwhelming to understand How do I bridge this gap effectively?
Hi! Putting the AB matrix on the GPU while the rest of the model on the CPU still has one problem: the loss. I have never tried it, but I believe PyTorch would complain when it tries to compute the loss (which involves both the frozen weights and the AB matrix). You can try using my notebook (freely available on my GitHub) and comment with the result of the experiment :D
Thanks for the reply. So in Lora also we need to store the pretrained weights in the GPU. Also can you make a detailed video on Flash Attention and Retentive Transformer.
As of now, LoRA is used for fine-tuning. For reducing the "size" of the model, there are quantization techniques. I'll make a video about them in the future. Have a nice day!
Thanks for you great work! May I ask a question? When Lora is enabled, the accuracy is 84.3% which is much worse than the original accuracy. So is it really beneficial when we enable Lora?
Of course the accuracy may degrade depending on the rank of the LoRA matrices, because the model has less parameters and so, less degrees of freedom. But it's not a rule: an overparameterized model may not suffer at all from degradation when using LoRA.
Thank you for the very cool video! Can you suggest any ways that we can use to combine the finetuned and the pretrained models so they can perform well on all digits?
🎉Top tier content!, thank you, I was looking at the net results for the other digits in your demo and realized they were worse off, then thought about it a bit more deeply, it looks like you trained a single B and A matrix and added to all layers, where I think an improvement would be a separate BA matrix for each layer. Curious your thoughts on this?
Hi @tljstewart Actually, in my code we train 3 different pairs of A and B, one for each of the layers. That's why I call the method "register_parameterization" method 3 times, one for each of the layers. Each A and B matrix has a different dimensions, because the dimensions of each layers are different. Usually we can't know which layer we should fine tune or not, unless we have a clue on what each layer may be doing (this can be said only for very specific architectures like the Transformer).
Ah thanks @umarjamilai I reviewed the code again, and it appears you do freeze the original model and train a Lora matrix for each layer. That leads me to a couple questions, how do you save the Lora weights and then how would you load them back in, for sharing say on hugging face? Just for an example, how might you load stable diffusion then load a Lora programmatically?
i think you should save the lora paramters beside the orginal model, and during inference stage, you can enable, or even add different lora to get the fine tuned result@@tljstewart
Can you do a video on finetuning a multimodal LLM (Video-LlaMA, LLaVA, or CLIP) with a custom multimodal dataset containing images and texts for relation extraction or a specific task? Can you do it using open-source multimodal LLM and multimodal datasets like video-llama or else so anyone can further their experiments with the help of your tutorial. Can you also talk about how we can boost the performance of the fine-tuned modal using prompt tuning in the same video?
Cool video mainly due to the topic. Sometimes, I had to rewind backwards, bacuase I could not get something, mainly why the reduction rank was 2 - is this just a chosen parameter?
Hi! The rank of the matrix is a hyper-parameter and in my PyTorch implementation, I had chosen a rank of 1. The lower the parameter, the lower the size of the matrix, but also the higher the loss of "precision", because the matrix may have an intrinsic dimension higher than the chosen hyper-parameter. If it doesn't make sense to you, I suggest you read what is the rank of the matrix and how dimentionality reduction works in PCA. That should give you the math background.
@@umarjamilai ok i just thought about it and uhhh, yeah, i dont see how, i had a misconception in my head, i forgot that the input data goes through the weights one layer at a time hence the output of layer 1 is the input of layer 2, plus they have activations functions that might make the process non-linear, my bad, have a nice day.
@@weiyaoli6977 That's the number of parameters due to LoRA, which is the size of the two matrices. When you save the model, you save the two matrices separately, so you only need to consider the size of each separately and sum them together. When you use LoRA, on the other hand, you need to multiply the two matrices.
As usual the full code and slides are available on my GitHub: github.com/hkproj/pytorch-lora
🥰🥰
thank you 感谢你的分享！
thanks a lot
I looked at many blogs and explanations but none of them got to the practical usage of LoRA and showed exactly how it is used in practice. This is exactly what I was looking for.
These videos are life-saver for my upcoming interview. So in depth and shows the code as well. Easy to follow along. Way better than my university. Thank you so much for the uploads.
Perfect.
Genius.
Simple.
To the point.
Theoretical.
Practical.
I have seen so many videos on LoRA, none of them contained this kind of explanation.
Thanks for the video!!!
I'm genuinely impressed by the content and presentation you've crafted for the ML/AI community. The way you've structured the presentation is both user-friendly and cohesive, allowing for a gradual and understandable flow of information.
One of the best explanation out there. Also the hands on code piece was just awesome!
Clear, simple and coincise. You rock Umar!
I have to say one of the best youtube channels out there. And thanks for sharing the code!
Rock solid content once again. From scratch implementations are soo beneficial.
As always, one of the best explainer on RUclips
Thank you so much for your explanations, they are helping me out tremendously during my master thesis work!
exceptional video , best 1hr spent on wkend
This channel is the best, 😊❤
This is very nicely explained, Thanks Umar
Another awesome video, you're a gem. Thank you for your work, do keep making these kind of videos on the latest research papers.
simple use case and clear explanation thanks for this please do more of this like implementing from scratch videos
Umar thank you for the content, really helps to grasp what the concepts are .
So you assume there is no activation function used neither in the layers contained in the frozen W layer nor in the lower representation AB layer?
Such a great RUclips channel. Keep the great work!!!
this is the best video on LoRA.
Very good and clear explanation, thank you!
Your videos are pure gold
Amazing video, everything was well explained, Is just what I was looking for, explanations and coding, thank you so much!
best explanation about lora, thank you...
This is such quality content! Thank you!
Great as usual!
this was flawless. FLAWLESS!
Thanks!
22:57 Why other digits accuracy can be decrease? I dont understand here.
Amazing explanation! Thank you.
Loved your explanation
Great Explanation!
Great video, really impressed by the video and channel, deservers a like.
Thanks!
LOVE YOUR VIDEOS LOVE YOUR KITTY'S VOICE! I MISS MY KITTY YOUR KITTEN EXACTLY MEOWS LIKE MINE WHEN I USED TO RECORD!!!! Thanks🖤👑🖤.
😸😸😸
This was insanely clear!
Very good explanation. Thank you!
How to load and save the lora weights can anyone explain?
Is it like the base model is stored in 4bit and as the data (X vector) passes through the layer that layer is first dequantized and then the matrix multiplication is done (X*W)? And the same thing for LoRA as well? and after we get Y (by adding output of lora and base layer) the W and LoRA layers are again quantized back to 4bit? and Y is passed on to next layer?
Also, if the LoRA is at the base of the model, does that mean to update the parameters of this LoRA we need to calculate the gradients of loss wrt all the W and LoRA matrices above it?
perfect, this really helps me a lot
so while inferencing, we ll only use a and b? but arent a and b matrices used to handle only digit 9?
I was going insane until I came across this amazing LoRA video ,an oasis for me, Can you also explain QLoRA?
Thank you for the great video.
Now I am wondering how LLM or diffusion trined with LoRA.
These models have many layers which attention, dense, fully connected.. how does LoRA adapted for this?
In the digit example, number '9' becomes better result after LoRA adaption.
But other numbers accuracy become much worse.
Is this natural for LoRA apdation? or We can make all number accurately with LoRA (which is trained for 9)?
Thank you very much!
In LoRA, Wupdated = Wo + BA, where B and A are decomposed matrices with low ranks, so i wanted to ask you that what does the parameters of B and A represent like are they both the parameters of pre trained model, or both are the parameters of target dataset, or else one (B) represents pre-trained model parameters and the other (A) represents target dataset parameters, please answer as soon as possible
To push loRa to its efficient limit, does it make sense to find the rank of the original weight matrices by finding statistically significant singular values with Marchenko-Pastur Law to choose the rank of the LoRA matrices?
Hello everyone, I have been in Deep Learning space for about some time now..Sometimes when I come across new codes like this in the video..I just can't keep up many of things in code feels new and gets overwhelming to understand
How do I bridge this gap effectively?
That Great. Thank You. You are the god!!
Hi Umair ,is instruction fine tuning and full fine tuning both are same
Great Presentation. I just want to know are there open source libraries for LLM models so that I can fine tune them.
Was hoping you would combine LORA in your stable diff video.
For fine-tuning, I have a question suppose we store the pre-train matrix in a cpu and load the AB matrix in the gpu for fine-tuning. Will this work?
Hi! Putting the AB matrix on the GPU while the rest of the model on the CPU still has one problem: the loss. I have never tried it, but I believe PyTorch would complain when it tries to compute the loss (which involves both the frozen weights and the AB matrix). You can try using my notebook (freely available on my GitHub) and comment with the result of the experiment :D
Thanks for the reply. So in Lora also we need to store the pretrained weights in the GPU.
Also can you make a detailed video on Flash Attention and Retentive Transformer.
so amazing tutorial
Please do a video where you show the process from scratch so we can do this with voice models ✊🏼
Hi a question, can we use lora to just reduce the size of a model and run inference, or we have to always do the fintuning
?
As of now, LoRA is used for fine-tuning. For reducing the "size" of the model, there are quantization techniques. I'll make a video about them in the future.
Have a nice day!
Thanks for you great work! May I ask a question? When Lora is enabled, the accuracy is 84.3% which is much worse than the original accuracy. So is it really beneficial when we enable Lora?
Of course the accuracy may degrade depending on the rank of the LoRA matrices, because the model has less parameters and so, less degrees of freedom. But it's not a rule: an overparameterized model may not suffer at all from degradation when using LoRA.
Great job!
Amazing Explanation.
Thank you for the very cool video! Can you suggest any ways that we can use to combine the finetuned and the pretrained models so they can perform well on all digits?
Why are two matrices, A&B needed instead of just one?
Because you the multiplication of the two matrices produces the original one.
Genius !
Great video! Would you consider doing a tutorial on finetuning llama2 7b model using lora?
🎉Top tier content!, thank you, I was looking at the net results for the other digits in your demo and realized they were worse off, then thought about it a bit more deeply, it looks like you trained a single B and A matrix and added to all layers, where I think an improvement would be a separate BA matrix for each layer. Curious your thoughts on this?
Hi @tljstewart
Actually, in my code we train 3 different pairs of A and B, one for each of the layers. That's why I call the method "register_parameterization" method 3 times, one for each of the layers. Each A and B matrix has a different dimensions, because the dimensions of each layers are different.
Usually we can't know which layer we should fine tune or not, unless we have a clue on what each layer may be doing (this can be said only for very specific architectures like the Transformer).
Ah thanks @umarjamilai I reviewed the code again, and it appears you do freeze the original model and train a Lora matrix for each layer. That leads me to a couple questions, how do you save the Lora weights and then how would you load them back in, for sharing say on hugging face? Just for an example, how might you load stable diffusion then load a Lora programmatically?
i think you should save the lora paramters beside the orginal model, and during inference stage, you can enable, or even add different lora to get the fine tuned result@@tljstewart
@umarjamilai ^^
Can you do a video on finetuning a multimodal LLM (Video-LlaMA, LLaVA, or CLIP) with a custom multimodal dataset containing images and texts for relation extraction or a specific task? Can you do it using open-source multimodal LLM and multimodal datasets like video-llama or else so anyone can further their experiments with the help of your tutorial. Can you also talk about how we can boost the performance of the fine-tuned modal using prompt tuning in the same video?
Cool video mainly due to the topic. Sometimes, I had to rewind backwards, bacuase I could not get something, mainly why the reduction rank was 2 - is this just a chosen parameter?
Hi! The rank of the matrix is a hyper-parameter and in my PyTorch implementation, I had chosen a rank of 1. The lower the parameter, the lower the size of the matrix, but also the higher the loss of "precision", because the matrix may have an intrinsic dimension higher than the chosen hyper-parameter. If it doesn't make sense to you, I suggest you read what is the rank of the matrix and how dimentionality reduction works in PCA. That should give you the math background.
Good explaining. 👍👍
why dont they lora the entire model's weights both the original and the changes ?
How would you LoRA the original weights?
@@umarjamilai ok i just thought about it and uhhh, yeah, i dont see how, i had a misconception in my head, i forgot that the input data goes through the weights one layer at a time hence the output of layer 1 is the input of layer 2, plus they have activations functions that might make the process non-linear, my bad, have a nice day.
thank you :)
15:30 🤣🤣The comment is hilarious, rich boy net
why b + a not b * a
Where did you read B + A? 🤔
d=1000, k=5000, p=5000(original). lora: 1000*1+1*5000=6000. so from the formula it is A*B. why A+B here? Thanks@@umarjamilai
@@weiyaoli6977 That's the number of parameters due to LoRA, which is the size of the two matrices. When you save the model, you save the two matrices separately, so you only need to consider the size of each separately and sum them together. When you use LoRA, on the other hand, you need to multiply the two matrices.