Thanks for breaking this down so well. As a privacy lawyer, this is the first time someone has managed to explain this to me in a way that I can actually understand!
As a student, I do really appreciate your vedios. On the Internet, there's barely the guide about how to code a complete model, most of the vedio is talking about the theory. But if we don't walk through the code, we couldn't understand and use the model well. So I subscribe your channel to see your vedios about how to code a model from scratch, it's really awesome! Very clear and comprehensible. Can't wait to see Coding a BERT from scratch!
Thanks Umar, I really appreciate your time and effort you put in creating these videos, very appreciate to create video coding BERT from scratch with PyTorch.
Best explanations in such a beautiful manner. Please share video on GPT also. Would be highly thankful for such. It's great to learn from you. Thanks a lot Sir ☺☺. I highly request to share video on GPT.
Thanks for providing a good quality of video as always. As a newbie of computer vision tasks, I still have found myself struggling with training and inference of source code with dataset. Would you please upload an informative video to show and explain the entire process of how we can do this with an open source code? Thanks.
Hello Umar. Could you please make a video coding BERT, MLM and NSP from scratch like the previous video on Transformer? That would be very helpful to us.
I think I followed your explanation of how BERT can be fine-tuned for Q&A, but still I am amazed how that can work. For example, fine-tuning, so that it knows the financial capital of China is Shanghai doesn't mean that it knows the financial capital of Indonesia.
Of course. The LLM will only know the concepts it is taught. If you never mention Indonesia in the training or fine-tune dataset, the LLM will never know what is Indonesia or its capital Jakarta.
@umarjamilai sir, I have some confusion, as you mentioned about MLM and NSP, how does that work in code? Does it pass one sentence (A) with some masking (and loss be called L) and simultaneously guess sentence B from sentence A (and loss for that be called L') and then train itself minimising L+L'? Like does it work on one loop like: for i in sthg: Fill MLM Evaluate loss in guessing word Guess sentence B from current filled sentence Evaluate loss in guessing sentence Minimise the total loss? Is there something that I can learn (as tutorials) for understanding how filling in the blanks and guessing next sentence works?
@@AdityaRaut765 Hi! It is calculated as two separate task, for which the loss is summed up as L1 + L2. I saw many different implementations of BERT online and so far the most trustworthy is the Hugging Face one.
Isnt the value vector for each Q,K,V should be different? Or say it is not necessary to be equal? And the product of the softmax function should be dot product again with Value Tensor V?
Thank you Umar for this great video! One confusion I have is about CLS token at 47:49, you mentioned we should use it because it can attend to all other words, since the first row don't have any 0 values(in the orange matrix). But wouldn't it also hold true for other tokens/rows in this matrix? Since this organge matrix is derived from softmax(QKt/sqrt(d)), while calculating this orange matrix, each row(word) in Q will multiple with every column of Kt matrix. Wouldn't this process enable each word to interact with all the rest words of the input? Sorry if I missed anything.
Thank you, Umar, for the video. I genuinely appreciate and enjoy your content. I'm looking forward to your implementation video. I have a question. At 6:14, you mentioned that the output of the Encoder would be a sequence of 10 tokens. In the case of BERT, wouldn't the output be simply the contextualized embeddings of the input token (rather than next token of sequence)? Thank you
Hi! It depends on which task BERT has been fine tuned for. If you fine tune it for the next token prediction task, it will return 10 tokens, of which the last one is the next token. If you fine tune it for another task, then the output will still be 10 tokens (since a transformer is a sequence-to-sequence network), but you need to interpret the output differently based on the task.
I have trouble understanding last part - question answering. You train model to say where is start where is end of the question - ok - but when/where you are training (fine-tuning) it to actually answer properly for some custom domain knowledge?
In Self-Attention the Q, K and V are always the same, that's why it's called self-attention. When K, V are coming from somewhere else (and are different from Q), in that case we talk about cross-attention.
@@HowToTechChannel If you refer to the parameters Wq, Wk and Wv, yes they're separate and trainable parameters. But they all receive as input the same sequence in Self-Attention.
In 12:56 you mentioned the idea of Cosine function to denote similarities between words. Can you tell me in which research paper this idea is most skill-fully mentioned.
If you could make a video on one of the finetuning tasks with example dataset and finetuning BERT on it, that basically would complete this lecture as one video contains every single information.
Shouldn't for Q&A tesk, linear layer should be 3 classes for each token, because for all other tokens apart from T10 and T27, there should be another class which will say NOT START/END.
Nope. Since we have two linear layers, each token will either be "Start/NOT_Start" or "End/Not_End". This means each token can be in 4 possible states: "Start - End", "Not_Start - End", "Start - Not_End" and "Not_Start - Not_End". This also covers the case in which the same token is the start and the end token at the same time, because the softmax score for that token will be the highest for both linear layers for the "Start" and the "End" class.
@@umarjamilai Oh here by 2 linear layers you mean parallel layers. I assumed 2 linear layers as in series like an MLP. Also, if I think now, we don't require two separate layers right, we can use a single layer with output as 2 neurons, where each neuron will be binary START/NOT_START and other will be END/NOT_END.
In the framework of scaled dot-product attention, particularly within the context of Transformer architectures, a key consideration is the interaction between the query (q), key (k), and value (v) matrices. This mechanism typically involves scaling the product of q and k by the square root of the dimensionality (d), followed by the application of the softmax function, and subsequently multiplying this result with the v matrix. A critical aspect of this process is the role of the attention heads in processing tokens. If we consider a scenario where each attention head is exposed to the entirety of the token set throughout the training process, how might this influence the effectiveness and efficiency of the model? This question assumes particular relevance in a setting where the embedding dimension is fixed at 512, and the model is dealing with a substantial number of tokens, potentially in the range of 10,000. Could such a comprehensive exposure of all heads to all tokens potentially compromise the model's performance, or are there mechanisms within the architecture that mitigate this concern?
To be honest, within the context of multihead attention, all the tokens are exposed to each head, but each head is only working with a different part of the embedding of each token. This is exactly what happens in the vanilla Transformer and also in BERT and works wonderfully. For example, if the embedding size is 512 and we have 4 attention heads, the first attention head will see the first 128 dimensions of each token, that is, the range [0...127], the second head will see the range [128...255], the third the next 128 dimensions and so on... So I don't understand your question: the transformer is already working like this, and so does BERT. In BERT, particularly, each token attends also to tokens coming after it in the sentence, which is called its "right context". Hope my explanation clarifies your doubts
@@umarjamilai First of all, I'd like to express my gratitude for your insightful videos. They've been a great resource as I embark on my PhD journey in generative AI in Spain. Though I'm originally from Pakistan, I've been living in Spain since I was nine years old. Now, regarding my query about the Transformer model: I appreciate your explanation of the multihead attention mechanism and how each attention head interacts with a different segment of the token's embedding. However, my question specifically focuses on the logic behind multiplying the attention weights by the value matrix (V). While I understand that the product of the key (K) and query (Q) matrices indicates the relationship between tokens, the rationale behind subsequently multiplying this result by V isn't clear to me. Could you please clarify the purpose or reasoning behind this step in the attention mechanism?
A Foundation Model is a large language model, just trained on a massive amount of data that only big tech can afford (Meta, Google, etc.). Foundation Models have been pre-trained on a variety of data, so they can easily be fine tuned for a specific task or even used with zero shot prompting on unseen tasks.
Hi! I didn't understand what is a mistake... The whole video I have referenced the first line of the Chinese poem. Sometimes, even for different inputs, I reference the same matrix so that people know we are talking about the same concept.
you were using seq of China.... but table was not referring to that. in fact you said relation of EOS but table was confusing as it didnt refer to the same seq you were basing on. All in all i loved all your vidoes its just this one appeared out of sync in that period rest its cool. already watching your 2 hr video :)
At the point 25:00, he was just trying to explain causal mask concept with regards to the self attention in case of usual vanila transformer where the tokens doesn't interact with token which comes after it and hence made them -inf before applying softmax and ultimately become 0 after softmax is applied to them. But, It doesn't apply in case of BERT which particularly use MLM concept where by masking certain tokens at either side. Again, this is my understanding from his explanation. Once @Umar does coding, it would be more clearer I believe.
At 19:20 you say that Key, Query and Value matrices are identical to each other. I think that is wrong. Those matrices are calculated using different linear layers with learnable parameters. Also, if they were identical as you said, the 10x10 matrix that you show at 21:17 should have been symmetrical along the diagonal.
I think I misworded it. The input sequence to the Wq, Wk and Wv matrix is the same during Self-Attention, but their projections are different. So it all depends on what you call Q, K and V... The input or the output of these three linear projections? The inputs are the same (it's the same sequence), while the output is obviously different.
@@umarjamilai Well, not so obviously because you say Q, K and V are the same matrices in the video. And then you multiply Q with K^T, then with V. You also never mention a projection. So that part of information is missing and misleading for a not knowing viewer.
@@HowToTechChannel Also found it missleading. Using Encoder architecture for 'Next word prediction' task brings concerns. I have watched Llama 2.0 videos and built my understanding of many ideas like Multi Head Attention etc. based on them, hopefully they were prepared more carefully :(
Hi Umar, gret video, around 19:37 you say that Q,K, V are identical copies, but is it actually the case? If Q=K, then QK^T would be a symmetric matrix, but it doesn't look like it is in the result of the matrix product shown later. Probably I'm just missing something here. Thanks again for the great explanation!
One of the best videos for BERT in youtube
Thanks for breaking this down so well. As a privacy lawyer, this is the first time someone has managed to explain this to me in a way that I can actually understand!
AMAZING summarisation of BERT and all of the intuitiveness behind it!
Hello you are one of the best teacher I have found in my life.
Love the way you explain things, so clear. Excellent Work!
Man! You know how to explain these topics! Please continue uploading these great videos for the good of the community! Really appreciate it!
i was trying to get the full picture of this for so long… seeing this felt just like getting a days long headache gone. thank you!
As a student, I do really appreciate your vedios. On the Internet, there's barely the guide about how to code a complete model, most of the vedio is talking about the theory. But if we don't walk through the code, we couldn't understand and use the model well. So I subscribe your channel to see your vedios about how to code a model from scratch, it's really awesome! Very clear and comprehensible. Can't wait to see Coding a BERT from scratch!
Thank you sir you explain these topics in great detail.Please keep uploading , excellent work.
This video is awesome! Worth watching multiple times as refresher. Please keep up the good work!🎉🎉🎉
I'm coming here from the trasformers video, and again really really good and detailed explanations! Keep up the good work and thank you!
Thanks Umar, I really appreciate your time and effort you put in creating these videos, very appreciate to create video coding BERT from scratch with PyTorch.
explained much clearer than my prof. Great!
Excellent video sir! Can't wait to see coding BERT from scratch.
Excellent BERT deep-dive video!!
Thanks Umar. I really appreciate your time and effort you put in creating these videos. I am anxiously waiting for the implementation video.
Great primer or BERT. Excellent illustrations, and you explained the concepts very well.
We need more videos from you sir..
Like explaining more papers and LLMs
thank you man! The best explanation i've ever seen
I've been using a Bert models summary endpoint and it's pretty good! Bert is underrated AF.
Man, you are an excellent teacher. I must say.
Thank you for explaining bert in layman's language 👍
Great Explanation😍
Thank you so much Jamil, it's really helps a lot!!😁
谢谢你! Hope you enjoy the life in China.
thanks a lot, I love all your presentations, please talk about GPT and other models as well.
Nice narration
Brilliant video, thank you for sharing.
Wow , what an explanation!!Thanks a lot
Thanks for this video :)
Best explanations in such a beautiful manner. Please share video on GPT also. Would be highly thankful for such. It's great to learn from you. Thanks a lot Sir ☺☺. I highly request to share video on GPT.
Great video, it’s explained really well 😊
Waiting for the BERT implementation from scratch !
Best video of BERT
Thanku for information video.....
after this video , I got a idea for my work❤...,
best video explanation by far
Thank you for your feedback. Make sure to like and subscribe! Hopefully you'll love my future as much as this one.
Please create more content it is helping us lot.....thanks for video
Amazing explanations!
very clear! thank you!
your Chanel is pretty awesome
Excellent work!
Hey bro, great and amazing work. Please do a video about coding BERT from scratch
Thanks for providing a good quality of video as always. As a newbie of computer vision tasks, I still have found myself struggling with training and inference of source code with dataset. Would you please upload an informative video to show and explain the entire process of how we can do this with an open source code? Thanks.
Could you please make a video on Complete Gpt architecture, different versions....your explanations are really good!!!!
Great as usual ❤❤
Great video explanation, could you please explain how the embeddings are generated from the token ids for the positional encodings are added?
Thanks Umar Jamil,
不客气🤓
Great video. Thank you. Could you please do coding BERT from scratch? I'm very curious to learn how to implement MLM and NSP tasks in PyTorch
thanks sir
Best Explained, can you please add some more videos regarding different different architecture related bert and transformers
Hello Umar. Could you please make a video coding BERT, MLM and NSP from scratch like the previous video on Transformer? That would be very helpful to us.
I think I followed your explanation of how BERT can be fine-tuned for Q&A, but still I am amazed how that can work. For example, fine-tuning, so that it knows the financial capital of China is Shanghai doesn't mean that it knows the financial capital of Indonesia.
Of course. The LLM will only know the concepts it is taught. If you never mention Indonesia in the training or fine-tune dataset, the LLM will never know what is Indonesia or its capital Jakarta.
@umarjamilai sir, I have some confusion, as you mentioned about MLM and NSP, how does that work in code? Does it pass one sentence (A) with some masking (and loss be called L) and simultaneously guess sentence B from sentence A (and loss for that be called L') and then train itself minimising L+L'? Like does it work on one loop like:
for i in sthg:
Fill MLM
Evaluate loss in guessing word
Guess sentence B from current filled sentence
Evaluate loss in guessing sentence
Minimise the total loss?
Is there something that I can learn (as tutorials) for understanding how filling in the blanks and guessing next sentence works?
@@umarjamilaior is it calculated as MLM first and then NSP (like two different loops?)
@@AdityaRaut765 Thanks for your question, i am too, waiting someone can answer this
@@AdityaRaut765 Hi! It is calculated as two separate task, for which the loss is summed up as L1 + L2. I saw many different implementations of BERT online and so far the most trustworthy is the Hugging Face one.
Isnt the value vector for each Q,K,V should be different? Or say it is not necessary to be equal? And the product of the softmax function should be dot product again with Value Tensor V?
Thank you Umar for this great video! One confusion I have is about CLS token at 47:49, you mentioned we should use it because it can attend to all other words, since the first row don't have any 0 values(in the orange matrix). But wouldn't it also hold true for other tokens/rows in this matrix? Since this organge matrix is derived from softmax(QKt/sqrt(d)), while calculating this orange matrix, each row(word) in Q will multiple with every column of Kt matrix. Wouldn't this process enable each word to interact with all the rest words of the input? Sorry if I missed anything.
Awesome explanation.
Cls token is important just because there is no zero values with others token?
Thanks for your efforts, I would love if you can make a tutorial on how to code OCR from scratch. Thanks
Thanks!
Thank you that was really helpful, can you olease do vision language models.
Thank you, Umar, for the video. I genuinely appreciate and enjoy your content. I'm looking forward to your implementation video.
I have a question. At 6:14, you mentioned that the output of the Encoder would be a sequence of 10 tokens. In the case of BERT, wouldn't the output be simply the contextualized embeddings of the input token (rather than next token of sequence)? Thank you
Hi! It depends on which task BERT has been fine tuned for. If you fine tune it for the next token prediction task, it will return 10 tokens, of which the last one is the next token. If you fine tune it for another task, then the output will still be 10 tokens (since a transformer is a sequence-to-sequence network), but you need to interpret the output differently based on the task.
Danke!
I have trouble understanding last part - question answering. You train model to say where is start where is end of the question - ok - but when/where you are training (fine-tuning) it to actually answer properly for some custom domain knowledge?
Fantastic content, like always. Just worth noting that Q and K (and V) don't necessarily have to be the same in self attention
In Self-Attention the Q, K and V are always the same, that's why it's called self-attention. When K, V are coming from somewhere else (and are different from Q), in that case we talk about cross-attention.
@@umarjamilai No, Q K and V are not the same. They are computed separately with trainable parameters.
@@HowToTechChannel If you refer to the parameters Wq, Wk and Wv, yes they're separate and trainable parameters. But they all receive as input the same sequence in Self-Attention.
In 12:56 you mentioned the idea of Cosine function to denote similarities between words. Can you tell me in which research paper this idea is most skill-fully mentioned.
is BERT outdated now?
No, BERT is very important on setence similarity tasks, and still well used.
What you said for the CLS, isn’t it also true for the other tokens too? Aren’t their values too getting multiplied 768 times similarly?47:34
Thanks, most of it great! But the Q&A fine-tuning is confusing.
Please do a video on GPT and LLaMA too....!!
I have two videos on LLaMA, check them out ;-)
I like it
excellent explanation of BERT , Thank you Umar. Can you suggest how to implement this for a NER type of task
In the future I plan to make a video on how to code BERT from scratch, but it's gonna take some time :D
If you could make a video on one of the finetuning tasks with example dataset and finetuning BERT on it, that basically would complete this lecture as one video contains every single information.
Shouldn't for Q&A tesk, linear layer should be 3 classes for each token, because for all other tokens apart from T10 and T27, there should be another class which will say NOT START/END.
Nope. Since we have two linear layers, each token will either be "Start/NOT_Start" or "End/Not_End". This means each token can be in 4 possible states: "Start - End", "Not_Start - End", "Start - Not_End" and "Not_Start - Not_End". This also covers the case in which the same token is the start and the end token at the same time, because the softmax score for that token will be the highest for both linear layers for the "Start" and the "End" class.
@@umarjamilai Oh here by 2 linear layers you mean parallel layers. I assumed 2 linear layers as in series like an MLP.
Also, if I think now, we don't require two separate layers right, we can use a single layer with output as 2 neurons, where each neuron will be binary START/NOT_START and other will be END/NOT_END.
@umarjamilai Then in this case, how do we ensure end_token is always start_token?
Please tell us about AliBi using the example of embedding in Llama 🙃 Instead of rotary coding 🙂
Bert has embedding vector of 768 and 1024 size but positional embeddings of 512, how does that work? I am confused
BERT has 512 positions. 512 is not the size of the positional embedding vector. It means the max sequence length can be up to 512 tokens.
In the framework of scaled dot-product attention, particularly within the context of Transformer architectures, a key consideration is the interaction between the query (q), key (k), and value (v) matrices. This mechanism typically involves scaling the product of q and k by the square root of the dimensionality (d), followed by the application of the softmax function, and subsequently multiplying this result with the v matrix. A critical aspect of this process is the role of the attention heads in processing tokens. If we consider a scenario where each attention head is exposed to the entirety of the token set throughout the training process, how might this influence the effectiveness and efficiency of the model? This question assumes particular relevance in a setting where the embedding dimension is fixed at 512, and the model is dealing with a substantial number of tokens, potentially in the range of 10,000. Could such a comprehensive exposure of all heads to all tokens potentially compromise the model's performance, or are there mechanisms within the architecture that mitigate this concern?
To be honest, within the context of multihead attention, all the tokens are exposed to each head, but each head is only working with a different part of the embedding of each token. This is exactly what happens in the vanilla Transformer and also in BERT and works wonderfully. For example, if the embedding size is 512 and we have 4 attention heads, the first attention head will see the first 128 dimensions of each token, that is, the range [0...127], the second head will see the range [128...255], the third the next 128 dimensions and so on...
So I don't understand your question: the transformer is already working like this, and so does BERT. In BERT, particularly, each token attends also to tokens coming after it in the sentence, which is called its "right context".
Hope my explanation clarifies your doubts
@@umarjamilai First of all, I'd like to express my gratitude for your insightful videos. They've been a great resource as I embark on my PhD journey in generative AI in Spain. Though I'm originally from Pakistan, I've been living in Spain since I was nine years old. Now, regarding my query about the Transformer model: I appreciate your explanation of the multihead attention mechanism and how each attention head interacts with a different segment of the token's embedding. However, my question specifically focuses on the logic behind multiplying the attention weights by the value matrix (V). While I understand that the product of the key (K) and query (Q) matrices indicates the relationship between tokens, the rationale behind subsequently multiplying this result by V isn't clear to me. Could you please clarify the purpose or reasoning behind this step in the attention mechanism?
What's the dimension in most LLMs like mixtral
Hi Umar ,please let me know the diff between Language models vs foundation models
A Foundation Model is a large language model, just trained on a massive amount of data that only big tech can afford (Meta, Google, etc.). Foundation Models have been pre-trained on a variety of data, so they can easily be fine tuned for a specific task or even used with zero shot prompting on unseen tasks.
I would better if you cover coding part as well..
Hit the like button if you are impatiently waiting for " code bert from scratch" video🎉
25:00 your input sequence is different and you are explaining matrix for different sequence!! isnt it a mistake ?
Hi! I didn't understand what is a mistake... The whole video I have referenced the first line of the Chinese poem. Sometimes, even for different inputs, I reference the same matrix so that people know we are talking about the same concept.
you were using seq of China.... but table was not referring to that. in fact you said relation of EOS but table was confusing as it didnt refer to the same seq you were basing on. All in all i loved all your vidoes its just this one appeared out of sync in that period rest its cool. already watching your 2 hr video :)
At the point 25:00, he was just trying to explain causal mask concept with regards to the self attention in case of usual vanila transformer where the tokens doesn't interact with token which comes after it and hence made them -inf before applying softmax and ultimately become 0 after softmax is applied to them. But, It doesn't apply in case of BERT which particularly use MLM concept where by masking certain tokens at either side. Again, this is my understanding from his explanation. Once @Umar does coding, it would be more clearer I believe.
I’ll use the restroom beef or Ethel come
At 19:20 you say that Key, Query and Value matrices are identical to each other. I think that is wrong. Those matrices are calculated using different linear layers with learnable parameters.
Also, if they were identical as you said, the 10x10 matrix that you show at 21:17 should have been symmetrical along the diagonal.
I think I misworded it. The input sequence to the Wq, Wk and Wv matrix is the same during Self-Attention, but their projections are different. So it all depends on what you call Q, K and V... The input or the output of these three linear projections? The inputs are the same (it's the same sequence), while the output is obviously different.
@@umarjamilai Well, not so obviously because you say Q, K and V are the same matrices in the video. And then you multiply Q with K^T, then with V. You also never mention a projection. So that part of information is missing and misleading for a not knowing viewer.
@@HowToTechChannel Also found it missleading. Using Encoder architecture for 'Next word prediction' task brings concerns. I have watched Llama 2.0 videos and built my understanding of many ideas like Multi Head Attention etc. based on them, hopefully they were prepared more carefully :(
You are a legend
Tomorrow
Thanks!
The explanation was amazing, thanks a lot 🙂.
Thanks for the lucid explanation. In the Q&A case, how does BERT know that the answer is Shanghai?
Hi Umar, gret video, around 19:37 you say that Q,K, V are identical copies, but is it actually the case? If Q=K, then QK^T would be a symmetric matrix, but it doesn't look like it is in the result of the matrix product shown later. Probably I'm just missing something here. Thanks again for the great explanation!
Superb explanation, love your channel :)
what a awesome video!!!!!!!
谢谢你！
Very good guys!
Thanks for your wonderful tutorial, waiting for the code part👀👀