The legend returns, Always excited for your videos. I am an international student at Shanghai Jiao Tong daxue. Your videos have given me a very strong foundation of transformers. Much blessings your way
I humbly request you to make videos on how to build a career in machine learning and AI. I am a huge fan of your videos and i thank you for all the knowledge that you have shared
Hi! I will for sure make a video in the future about my personal journey. I hope that can help more people in navigating their own journeys. Have a nice day!
I believe the most evident insight of DPO is to change a RL problem to an equivalent MLE, while the optimal reward model is guarranteed by the human input as definition. That's the meat. But the efficiency depends still on the human annotater's consistency.
Thanks for the great video! Very intuitive explanation and particular thanks for the code examples. Question: at 37:41, how do we know that the solving the optimization problem will yield the pi_*? Is there a guaranteed unique solution?
Please check the paper I linked in the description for a complete derivation of the formula. It is also done in the DPO paper, but in my opinion the other paper is better suited for this particular derivation.
Great explanation but I have some doubts, please help 36:50 in Ldpo π* was replaced with π theta... why π theta is considered as optimal policy? 44:13 You said "each hidden state contains information about itself and all the tokens that comes before it", but this is applicable only to decoder part of the transformer. So this transformer layer is actually a decoder layer? like GPT
Hello Umar, Great as usual, however why do you say at 46:11, that you need to sum log probabilities up? The objective function is the expectation of logarithm of the difference of two weighted log probabilities ratios. I don't get what do you want to sum up exactly? Thank you
Amazing video. Would it be possible for you to explain video-transformers or potential guesses at how Sora works? Another exciting idea is explaining GFlowNets
Thanks for the video. Do you know any way on how we can create a dataset for DPO training. I currently have only question, answer pairs. Is it fine if i take y_w as answer and y_l as some random text(which would obviously have lower preference than answer) and then train it?
Hi Umar, If I use LoRA for fine-tuning a chat model with DPO loss, what should I use as a reference model? - The chat model applied LoRA - Or the chat model itself without LoRA?
Considering LoRA is just a way to "store" fine-tuned weights with a smaller computation/memory footprint, the model WITHOUT LoRA should be used as the reference model.
@@umarjamilai With my limited GPU, I can only fine-tune by combining a 4-bit-quantized model + LoRA. Surprisingly, using just the 4-bit model leads to NaN weight updates after one batch. But once LoRA is added, my loss updates smoothly without any problems.
Thank you SO much for the quick answer and your excellent video. I did get the hang of DPO loss and be able to implement DPO loss + training loop with vanilla PyTorch code.
According to the Hugging Face library, yes, looks like you need a dataset with prompt and two answers, one is called the "chosen" one and the other is the "rejected" one. I'm pretty sure there are ways to convert more than two preferences into a dataset of two preferences.
@@umarjamilai thank you! Yes of course. I am just wondering why it wouldn’t help to have more than 1 rejected for 1 accepted. I guess the formula does not consider this case but may add value.
I enjoy your videos umar on my phone while commuting or sitting in a coffe. Only the small fint on a phone is tiring me ..if you make them a bit bigger that will be better
The legend returns, Always excited for your videos. I am an international student at Shanghai Jiao Tong daxue. Your videos have given me a very strong foundation of transformers. Much blessings your way
我们在领英联系吧，我有个微信小群，你可以参加
@@umarjamilai我也想加
@@umarjamilai我看到b站也有你的账号
same here from Canada!
I can't thank you enough @umarjamilai
@@umarjamilai 我也想加
Legend is back, the GOAT, if my guess is right next will be ORPO or Q*
Actually, the next video is going to be a totally new topic not related specifically to language models. Stay tuned!
@@umarjamilai waiting
wow your explanation is so clear and complete... you are godsend, keep doing it. Sei un fenomeno
Your explanations are on a whole another level, compared to whatever else you can find online. Keep up the amazing work and thank you!
Thanks for making these videos. Concise and clear
Thank you! It's very clear explaination. It helps for reading the original paper. Looking forward to new topics.
Amazing! Great job once again!
Very clear explanations!! Please, continue making such good videos!
I humbly request you to make videos on how to build a career in machine learning and AI. I am a huge fan of your videos and i thank you for all the knowledge that you have shared
Hi! I will for sure make a video in the future about my personal journey. I hope that can help more people in navigating their own journeys. Have a nice day!
Thanks so much Umar, always learn a lot from your video!
Great video, thank you very much for this!
These lectures are amazing. Thank you!
Valeu!
Awesome, thank you so much for putting this out, super helpful!
You explained very clearly. Thanks!
Thanks!
This is very, very good. Thank you!
New video🎉 can't wait to watch. Although having used DPO in production for a while now!
I believe the most evident insight of DPO is to change a RL problem to an equivalent MLE, while the optimal reward model is guarranteed by the human input as definition. That's the meat. But the efficiency depends still on the human annotater's consistency.
Best explanation so far !!!!...
Awesome Video. please Continue.
Thanks for the great video! Very intuitive explanation and particular thanks for the code examples. Question: at 37:41, how do we know that the solving the optimization problem will yield the pi_*? Is there a guaranteed unique solution?
Please check the paper I linked in the description for a complete derivation of the formula. It is also done in the DPO paper, but in my opinion the other paper is better suited for this particular derivation.
Great explanation
but I have some doubts, please help
36:50 in Ldpo π* was replaced with π theta... why π theta is considered as optimal policy?
44:13 You said "each hidden state contains information about itself and all the tokens that comes before it", but this is applicable only to decoder part of the transformer. So this transformer layer is actually a decoder layer? like GPT
My Kind Request Please Increase volume little bit , just little bit. Otherwise your videos Outstanding . Best I can say.
Hello Umar,
Great as usual, however why do you say at 46:11, that you need to sum log probabilities up? The objective function is the expectation of logarithm of the difference of two weighted log probabilities ratios. I don't get what do you want to sum up exactly? Thank you
Thank you very much for this video, please make ORPO as well.
Love from India sir, you are a legend 😊😊
Enjoyed the style in which the video is presented. Which video editor/tools do you use to make your videos? Thanks.
I use PowerPoint for the slides, Adobe Premiere for video editing
@@umarjamilai What do you use to draw on your slides? I am assuming you connected an ipad to your screen.
Amazing explanation. Would it be possible to make a video on the theory and implementation of automatic differentiation (autograd).
Amazing Video! Please do one on SPIN (Self Play Fine-tuning) as well
Thanks for your lecture. I wonder could you explain the vision language models
Thank you for the video ! Can you provide the video that explains AgentQ training in details ?
love ur videos umar !!
Amazing video. Would it be possible for you to explain video-transformers or potential guesses at how Sora works? Another exciting idea is explaining GFlowNets
what is the ppt software you are using sir ?
Brilliant!!!!!!
Thanks for the video. Do you know any way on how we can create a dataset for DPO training. I currently have only question, answer pairs. Is it fine if i take y_w as answer and y_l as some random text(which would obviously have lower preference than answer) and then train it?
The potential problem that I think could happen is that having random text may decrease the loss and the policy may not even change much
Hi Umar,
If I use LoRA for fine-tuning a chat model with DPO loss, what should I use as a reference model?
- The chat model applied LoRA
- Or the chat model itself without LoRA?
Considering LoRA is just a way to "store" fine-tuned weights with a smaller computation/memory footprint, the model WITHOUT LoRA should be used as the reference model.
@@umarjamilai With my limited GPU, I can only fine-tune by combining a 4-bit-quantized model + LoRA. Surprisingly, using just the 4-bit model leads to NaN weight updates after one batch. But once LoRA is added, my loss updates smoothly without any problems.
Thank you SO much for the quick answer and your excellent video. I did get the hang of DPO loss and be able to implement DPO loss + training loop with vanilla PyTorch code.
great explaination, thanks. how about the recent work: KTO: Model Alignment as Prospect Theoretic Optimization? can you compare it with DPO?😁
Thank you very much for the explanation.
I had one questions. Are the dataset of preferences always made of two and only two answers?
According to the Hugging Face library, yes, looks like you need a dataset with prompt and two answers, one is called the "chosen" one and the other is the "rejected" one. I'm pretty sure there are ways to convert more than two preferences into a dataset of two preferences.
@@umarjamilai thank you! Yes of course. I am just wondering why it wouldn’t help to have more than 1 rejected for 1 accepted. I guess the formula does not consider this case but may add value.
I love ur videos so much. please make a video about mamba or mamba vision
There's already a video about Mamba, check it out
great video. love from india.
Thank you. can you make video about liquid neural network?
thank you so much for your effort! could you make a video about tokenizers like BPE and sentencepiece from scratch? I would be very appreciate of it!
thank you
I enjoy your videos umar on my phone while commuting or sitting in a coffe. Only the small fint on a phone is tiring me ..if you make them a bit bigger that will be better
Sorry for the trouble, I'll keep it in mind for the next videos!
Great 👍
“digital biscuits”, lol