To learn more about Lightning: lightning.ai/ Support StatQuest by buying my books The StatQuest Illustrated Guide to Machine Learning, The StatQuest Illustrated Guide to Neural Networks and AI, or a Study Guide or Merch!!! statquest.org/statquest-store/
This explanation is essential for anyone looking to understand how ChatGPT works. While more in-depth exploration is necessary to grasp all the intricacies fully, I believe this explanation couldn't be better. It's exactly what I needed.
Haven't had a single stats course in over 3 years but I still keep up with this channel from time to time! Neural networks are way more complex than what I've ever had to deal with, but you manage to break down even these topics into bite size pieces...Bam!!
Was stuck on a stupid detail about the architecture for a couple of hours. Your elaborate illustrations helped me make it clear in my mind right away! Thanks, keep going :)
This video is proof that repetition is prime when teaching advanced concepts. I've watched many similar videos in the past and could never get all of these numbers to finally make sense in my mind. With your previous transformer video, I was getting closer but somewhat got lost again with the QVK values. Having to this second video to watch in a row made it clearer for me what all these numbers do and why we need them.
Thank you so much! It is an amazing video and I haven't seen a video teaching AI/ML techniques like this anywhere! You're talented. And my research areas span Efficient LLM (LoRA, Quantization, etc). It cannot be better if I can see those concepts
I just want to say you are AMAZING. Thank you so much. I would personally love to see a video on backprop to train this, or even just training an RNN since we saw multi dim training, but not training once we get the state machine / unrolling involved. Loved the whole series 🎉
Thanks! I have notes for training an RNN, but the equations get big really fast. That said, it really is the exact same techniques presented in other videos, just a lot more of them.
Hey Josh, I’ve been really digging your videos! They’re not only informative and helpful for my studies, but they’re also super entertaining. In fact, you’ve played a big part in my decision to continue pursuing AI Engineering. Could you please do a video about low-rank adaptation(LoRA). I am not good with that.
Hello Josh, i am enjoying your videos as they are helping me so much with my studies as well as entertaining me. You are kinda a reason i decided to continue studying bioinformatics. Since you are covering chatGTP and stuff now, could you maybe make a video about AlphaFold architecture in the future? I understand it might not be your topic of interrest, but i would love to lear in more deeply (pun intended). Thanks either way!
Delighted to watch one of the Most Brilliant videos. Hats off. Will join the channel tomorrow, first thing. Meanwhile do you have a one on Probability Density Function.
Hello Josh, thank you again for your video ! I had one question concerning training the model on next token prediction: As training data, would you use "What is statquest " or "What is statquest awesome" ? What I mean by that, is when training the model by feeding it an input prompt such as "What is statquest ", do you also feed the model the word that comes after it (for calculating the loss), here "awesome" ?
The training inputs were "What is statquest awesome", and the labels were "is statquest awesome ". I'm working on a video that goes through how to code a transformer and how to prepare the training data. Hopefully it will be out soon.
@@statquest Thank you for your answer. I see that the decoder also learns to embed the input then (here, on the input , the label is "awesome"). I'm looking forward to your next vide !
Thanks again Josh! I noticed that many GPTs are decoder only. Thanks for clarifying! BTW saw that Yannic had a video on history rewrites. Probably not a topic for this channel, but still pretty cool 😁
Endless thanks to your awesome explanation. Would you mind clarifying the next confusion to me? You've mentioned that the normal transformer uses masked self-attention only during training and normal self-attention during inference, while according to other resources, including your iterations through the examples (I think you followed the masked self-attention mechanism), and in 34:20, how are we supposed to know the next tokens during training (parallel training) while we are restricted by the mask? and thank you in advance.
What an awesome video and channel😁👍. Would you consider doing a video on deep q learning models? I believe everyone would benefit from a video on such a fundamental topic. Thank you for your invaluable work🤩
Thanks for the great video, Josh. I got a question for you. What should drive my decision on which model to choose when facing a problem? In other words, how to choose between an Encoder-Decoder transformer, Decoder-only transformer or Encoder-only transformer? For instance, why ChatGPT was based on a Decoder-only model, and not on a Encoder-Decoder model or an Encoder-only model (like BERT, which has a similar application)
Well, the reason ChatGPT choose Decoder-Only instead of Encoder-Decoder was that it was shown to work with half as many parameters. As for why they didn't use an Encoder-Only model, let me quote my friend and colleague, Sebastian Raschka: "In brief, encoder-style models are popular for learning embeddings used in classification tasks, encoder-decoder-style models are used in generative tasks where the output heavily relies on the input (for example, translation and summarization), and decoder-only models are used for other types of generative tasks including Q&A." magazine.sebastianraschka.com/p/understanding-encoder-and-decoder#:~:text=In%20brief%2C%20encoder%2Dstyle%20models,other%20types%20of%20generative%20tasks
Hey, fantastic video as usual! Getting hard to find new ways to compliment, haha. Just one quick question since you mentioned positional encoding. When generating embeddings from GPT embedding models (e.g., text-embedding-3-large), do the embeddings contain both positional encoding layer and masked-self-attention info in the numbers?
Hi there! Thank you for the awesome video. It is very helpful and clear. I have a question about the part around 34:10, where you are talking about how normal transformer uses Masked Self-Attention in training. I did not quite understand how that is Masked Self-Attention (i.e. looking at the tokens before). Is the Decoder still looking at the input tokens, or is it looking at the output tokens that come before it? Thank you very much!
During training we know what the output should be, but we don't let the decoder look at tokens that come after the one it is currently processing. So if the output is "my name is fred", then, when we calculate attention for "name", the decoder can't use "is" and "fred". What's going on here might be more obvious if you look at how the math is done with matrix algebra in this video: ruclips.net/video/KphmOJnLAdI/видео.html (and if you are not already familiar with matrix algebra, see: ruclips.net/video/ZTt9gsGcdDo/видео.html )
Had a couple of questions regarding word embedding: - Why do we represent each word using two values? Couldn't we just use a single one? - What is the purpose of the linear activation function, can't we just pass the summation straight to the embedder output? Thanks for the video!
1) Yes. In these examples I use 2 because that's the minimum required for the math to be interesting enough to highlight what's really going on. However, usually people us 512 or more embedding values. 2) Yes. The activation functions serve only to be a point where we do summations.
Hi Josh, great video, as always. I was wondering if you would also make a video about Encoder-only Transformers, like Google's BERT for instance, which can also be used for a great variety of tasks.
I don't understand why we need the residual connections.... =''( isn't the word and position encoded values information already included in the masked self-attention values? or is most of the information lost so we need to directly add the word and position encoded values?
In theory you do not need them, but in practice they make it much easier to train large neural networks since each component can focus on it's own thing without having to maintain the information that came before it.
Hey Josh ! Would you mind making videos about graph neural networks ( GNN ) or graph convolutional network ( GCN ), and most importantly, the graph Attention Network ( GAT ) ? I have briefly gone over the maths these days, I already knew the matrix manipulation stuff but I think with your help, it would be much clear like your Transformer series, especially on the attention mechanism in the graph attention network ( GAT ), many Thanks 🙏🏻🙏🏻🙏🏻🙏🏻 appreciated !
GNNs really only have two main elements to them, the aggregate function and the update function. The different choices of these two functions give rise to the different variants, GCN, GAT, etc.
Thank you, Josh, for yet another excellent video on GPT. I find myself slightly puzzled regarding the input and output used to train the Decoder-only transformer in your example. In the normal Transformer model, the training input would be "what is statquest ," and the output would be "awesome ." However, in the case of the Decoder-only model, as far as I understand, the training input remains "what is statquest ," but the output becomes "what is statquest awesome ." Could you help to clarify this? If my understanding is correct, I'm wondering how the Decoder-only transformers know when to stop during inference, considering that there are two tokens within the generated response.
Because the first is technically part of the input, we just ignore it during inference. Alternatively, you could use a different token to indicate the end of the input.
Hi Josh! Thank you for your clearly explained work! You did a great job. I'm a big fan of you! I have a question about the embedding position. Here, you use two activation functions to encode the word to a vector, so that is why for the position embedding, you only use two sine and cosine to encode each word. Am I right? I found from the previous video: Transformer the foundation of chatGPT, you use four activation functions to encode each word, and later, you use four sine and cosine squiggles for the position embedding.
Each word embedding value needs a positional encoding value. So the number of sine and cosine squiggles that you use depends on the number of embedding values that you create for each token.
Hi there! Came to YT in hope you had a nice video of Rank Robustness. Would be amazing, if you wanted to make a video about it! Keep it up! Also: nice Dinosaurs!
Another amazing video! So the fully connect layer basically maps two dimensional vectors to 5 dimensional vectors, in this case, is the size of the vocab(collection of tokens). Is that correct?
@@statquestGreat! I have another question: How does a fully connected neural network map vectors representing different features (dimensions) to vectors that represent the indices (dimensions) of tokens in the vocabulary?
@@harryliu1005 I'm not sure I understand your question since the video shows exactly how the fully connected layer works. If you're not already familiar with the basics of neural networks, check out these videos: ruclips.net/video/CqOfi41LfDw/видео.html and ruclips.net/video/83LYR-1IcjA/видео.html
Josh, why in min 19:34 the similarity value for "Statquest" is lower for itself than for the word "What"? It shouldn't be larger to itself as is the self-similarity?
Wow this series culminating in a perfect explanation of GPT is the most magnificent piece of education in the history of mankind. Explaining the very climax of data science in this understandable step-by-step way so I can say that I understood it should earn you the noble prize in education! I am so grateful that you never used linear algebra in and of your videos. Professors at university don't understand that using linear algebra prevents everyone from actually understanding what is going on but only learning the formula. I have an exam in Data Science on Friday in a week. Can you make a quick video about spectral clustering by Wednesday evening? I will pay you 250$! :)
Hello Josh! First of all thank you for this great video, as usual it's very simplified and straightforward. However, I have a little question. I saw your videos on transformers and this one, but every time I feel like the output is already there waiting to be embedded and then predicted. I mean that why the answer can't be "great" in stead of "awesome", what was the probablities given by the model for "great" and for "awesome" to make the final prediction. Here I gave the example of one extra word (great) but in real life it's the whole dictionary of words that can be predicted. So when generating the output, does it compute the "query" and "key" of the whole dictionary of words and then hopefully the right word has the best softmax probability? Thanks in advance for the clarification.
no, you only calculate the queries, keys and values for the input tokens and the output as it is generated. However, in practice, instead of training on just a few phrases, we train on the entire wikipedia. As a result, the transformer can be much more expressive.
Hi Josh, excellent video. I only recently found you but your channel is amazing. It looks like you inadvertantly copied over the example value for "is" from the value for "what" starting around 1925 and this continues forward through the video. I mostly say this in case you ever adopt these notes directly into a book.
Chat gpt is still a dialog system at its heart and has many different models which it gets results from . It softmaxes the outpits acording to the intent , ... So intent detection plays a large role in the chatgpt response .. the transformers are doing major works .. its super interested despite bqttling away with vb net !
@@statquest Yeah. Can you please explain - Note :- If we were training the Decoder-only transformer, then we would use the fact that we made a mistake to modify weights and biases. In contrast when we are just using the model to generate the responses, then it doesn't really matter what words come out right now.
Thank you so much for another great video! I did have a question -- I'm confused about why you can train word embeddings with only linear activation functions because I thought that linear activation functions wouldn't allow you to learn non-linear patterns in the data, so why wouldn't you just not use an activation function at all in that case or use only one?
For word embeddings specifically, we want to learn linear relationships among the words. This is illustrated in my video on word embeddings: ruclips.net/video/viZrOnJclY0/видео.html And, technically, when coding a linear activation function, you just omit the activation function.
Thank you for the incredible content. Josh, quick question for you. I didn't see you mention vertically stacking the decoders in a way where the output of one decoder is the input for the next. From the 'Illustrated Transformer' page (I can't link b.c. youtube won't let me) it seems to be a core aspect of transformers. Thanks again.
Personally, I wouldn't call that a core aspect. Unlike the ability to layer attention units, feeding the output of one decoder into the input of another in a stack, did not influence how the decoder-transformer (or even an encoder-decoder transformer) was designed. In contrast, the ability to layer attention had a big influence on how the transformer was designed.
That makes sense. I just took another look at the Attention is All You Need paper, and it corroborates your explanation. Thank you. Request: a video on the practicalities of creating and training a production LLM. The data volume, the number of parameters and how the production architecture differs from the simplified educational model provided in this video. I think this would allow the audience to better understand what simplifications were (rightly) made for the purpose of explication. Also thank you so much for what you do. You are creating some of the best educational content on the internet. I am so jealous of this upcoming generation for having teachers like you :)
@@jacksonrudd3886 That's right - in the Attention is all you need paper, they just mention the stacking (N=6) in passing and don't spend any time on it. And I'm planning on making the exact video that you want me to make. It may take some time, but it's in the works.
After painstaking research on this topic, I have realized that all the information is not in the transformer itself. The transformer is just the neural network (i.e., the processor or function). The information is actually in the embedding model. The sequences are all stored there. When training the transformer, the word embedding model needs to be fully trained or trained with the same data. This allows the learned embeddings to fit the transformer model. Here, overfitting is beneficial as the model will be a closed model after training. The transformer can be retrained (tuned) to your specific task. The main task should be text generation corpus/tuned for question and answer. The learned embeddings model can actually generate data as well as the transformer neural network, making it a rich model. I also noticed that to get an effective word embeddings model, the optimization or "fine-tuning" should involve classified and labeled data. This allows clusters of similar terms to appear, as well as emotionally aware embeddings. The transformer is essentially performing an "anything in, anything out" operation, forcing the input and output to converge despite any logical connection between them. Given an input, it produces the desired outcome. As many similar patterns are entered, it will fit this data using the masking strategy. Transformers need to be trained as masked language models to be effective, highlighting the drawbacks of LSTM and RNN models which cannot produce the same results. The more distanced the output is from the input, the more layers and heads you need to deploy.
@@iProFIFA I mean the most important area of focus should be the embeddings model as this is the part which stores the data after training ... The transformer is just the neural network function. .. Also.... Chat gpt is primarily a "chat bot" so all inputs are fed to its Intent detection module , IE classify the user intention ... To send the query to the correct model for a response ... Right now the transformer is seen to be able to fit anything to anything model .... As long as you give an input and target to the model and enough samples the "function (transformer)" can fit the data to the model ... Hence the first model to train is the text generation model to give it many possibilitys to generate some form of text given a seed .... Then tune the model to your specific task , IE question and answer or the New "instruct" style model ... IE : for a true code generation model . The first corpus should be text books , tutorials , forum posts , mass source repos .... This will give it a massive base to generate some form of code given a seed ... Then to pass in the instruct code models , IE build a binary tree , and provide the specified code , ... It can be ANY code language .... Because the model is a transformer , later we can add another fine tune , for specific code translation ... IE csharp examples with Thier equivalent python and rust and javascript etc ... .. now this model can be retrained until it "overfits it's domain model" .... By adding general language corpus and basic questions answer modelling "after" you can now ask question and pursues project objectives ... Because this was originally trained with code model it's preferred output will be code .... The sealed language model outputted .. this is also the issue as all the data is in the embedding model you train it with ... So ... . This embedding model is the most important , as neural networks can fit any numerical data .. Which is the best model for word embeddings ... Fasttext , glove , skipgram ? Are there others ? What's the best optimiser for these models ? Are they interchangeable ? .. Finally , the transformers successes of late are due to the deployment of layers and heads .. this increase give the perception of adding dimensionality(randomness( tweakable settings (all weights and biases) but this is due to attempting to use a single model to fit all tasks ... If you have models for code gen. And others for picture gen , and others for chat gen, then others for business and local domain knowledge . Then it's better to have an intent model In front of all of these dedicated models . And direct the query to the appropriate models and take a softmax of the output to choose the correct output ! ... Again optimising the intent detector with the input and expected output and correct classification of the requested task .... Hopefully I explained it a bit better this time ! Lol .
@@xspydazx yo, your way of writing is really confusing and it seems like you went on a tangent so it’s hard to follow. Also you shouldn’t put spaces before commas and periods.
Hi Josh, I have a nagging question: if I understand your explanation correctly, in order to predict the next token, we first compute Keys, Queries and Values for the prompt tokens (or more generally - the prompt and the current set of completion tokens), THEN we use them to compute Masked Self-Attention and Residual Connections, which finally are passed through the fully-connected layer to predict (generate) the next token. If that is true, it would seem that in order to generate the completion, all we need to do is (conceptually) "place" the prompt at the output, compute ONLY Keys and Values for the prompt, then use these Keys and Values when computing Masked Self-Attention and Residual Connections for the completion tokens ONLY. There doesn't seem to be any need for computing Masked Self-Attention and Residual Connections for the PROMPT tokens when generating the completion tokens. In other words, ONLY the prompt Keys and Values are sufficient to provide the full context to compute the Masked Self-Attention and Residual Connections for the completion tokens. However, in your explanation you did go through computing Masked Self-Attention and Residual Connections for the prompt and (seemingly) re-generating the prompt tokens before moving on to generating the completion tokens. This part was unclear and appeared unnecessary to me. When asked, ChatGPT insists that "In decoder-only transformer models like GPT-3 or GPT-4, the prompt tokens are indeed fully processed through all the layers, INCLUDING the masked self-attention mechanisms and residual connections" and that "Keys and Values are derived from the hidden states AFTER incorporating the masked self-attention and residual connections". This would suggest that the Keys and Values are somehow computed AFTER (based on, using) Masked Self-Attention and Residual Connection and not vice versa as explained in your lecture. I understand that you might have simplified the design and/or reordered the computations for brevity/simplicity of explanation, but I would appreciate a clarification if the Masked Self-Attention and Residual Connections indeed need to be computed for the PROMPT tokens in addition to their Keys and Values in order to generate the completion tokens. Thanks so much!
@@statquest Thanks for your response. This makes sense for training, but in my question I was focusing on the situation during inference (I failed to state it clearly, sorry!)
@@statquest Does it mean that computing the Masked Self-Attention and Residual Connections for the PROMPT tokens is unnecessary during inference, since only their Keys and Values appear to be needed to compute the Masked Self-Attention and Residual Connections for the COMPLETION tokens? That is my (mis)understanding based on your video, but ChatGPT disagrees, stating the masked Self-Attention and Residual Connections for the PROMPT tokens are also needed when generating the COMPLETION tokens.
@@rbryll2002 I believe that it is done, and that is how it is described in the original manuscript. However, it is possible that you could skip that step. If you would like, you can play with the code to see what happens. I explain the code here: ruclips.net/video/C9QSpl5nmrY/видео.html and and you can get it to play around with here: github.com/StatQuest/decoder_transformer_from_scratch
I didn't really understand where the fully connected layer came from and how it is connected to the input values. It seems independent of the previous layers and somehow outputs the input words
Hey ,if we are feeding the predicted output as next input after doing the embedding,positional encoding and masked attention with previous values ,why did you take "awesome" as the next input even when the model predicted it as "what" .i didn't get you here.
@@SaiPragnaanMarala Because we are still processing the input prompt. During this stage, it doesn't matter what tokens the transformer generates because they are all discarded since we already know what the prompt is. However, once we start generating a response to the prompt, then we can't discard any of the tokens that we generate and have to use them (unless we are training the model).
hello..first off thank you for this great content I had a question/s could you give an example of how the embedding neural network is trained? i.e. what is the input and output in the embedding neural network during training? The neural networks I have worked with statements that go along the lines of "given a set of pixels determine whether the picture is a cat or not".. I do not know what the equivalent is with embedding neural networks and follow up question..can the embedding neural network be the same for an encoder-decoder model and a decoder-decoder model?
1) We don't train the embedding layer separately from the rest of the transformer. So the inputs are what you see here as well as the ideal outputs that we use for training. 2) Once trained, yes.
20:56 I think the value of the word "is" is miswritten it should be 1.1 , 0.9 not 2.9,1.3 it should not be same with the value of word 'what' Thank you for your videos btw ur explanation is awesome.
That is correct. In order to illustrate how a decoder-only transformer worked, I had to make the model as simple as possible, and, as a result, some of the nuance in the values for the weights was lost.
I feel it’s a bit misleading that it seems the tokens of the input sequence is fed in one by one, and that when you put in the first token, it predicts the second token but just ignores it, where in reality it feeds the entire sequence to predict the next target token, and on next iteration, you append the input sequence with the target token as input, and predicts the second target token, and so on. Right?
I didn't understand just one part: how are the weights to calculate Q, K and V for each word in the sentence calculated? Is it also an optimization process? If so, how is the loss function calculated?
At 5:08 I say that all of the Weights in the entire transformer are determined using backpropagation. Specifically, we use cross entropy as the loss function. For more details about cross entropy, see: ruclips.net/video/6ArSys5qHAU/видео.html and ruclips.net/video/xBEh66V9gZo/видео.html
Great video. Do I understand correctly DNN which is responsible for word embedding, it not only converts the token to its representation as a numeric vector, but already predicts as the next word should be returned ?
In a transformer the embedding layer alone does not predict the next word because it wasn't specifically trained to do that the way a stand alone word embedding layer (like word2vec) would.
@@statquest But if we train the whole model at the same time, then backpropagation does not change the weights of the network responsible for word embedding in such a way that they learn to predict the next word? Or don't we train this first network while learning ?
@@JanKowalski-dm5vr It might. But the whole model, word embeddings and attention and everything, is trained to predict the next word, or translate, or whatever it's trained to do. So it's hard to say exactly what the word embedding layer will learn.
Hi Josh! I'm a little bit confused about the whole idea of generating the input first, compare it to the actual input and use to modify weights and biases in the training phase. I cannot find anywhere on the internet that it is mentioned. All I see is that the masked self attention is used on the input sequence to make contextualized version of each word and then, they are used to generate the target tokens. Nowhere can I find that generating the input sequence and compare it to the actual input is part of the process. Can you please clarify?
@@raminziaei6411 The idea of comparing the predicted input sequence to the known input sequence comes from the original manuscript that describes decoder only transformers, GENERATING WIKIPEDIA BY SUMMARIZING LONG SEQUENCES. They say: "Since the model is forced to predict the next token in the input as well as [the output] error signals are propagated from both input and output time-steps during training."
@@statquest Thanks Josh. A related question would be "Does this only hold true if we are considering causal decoder transformers where we have masked self-attention for both input and output sequences? For prefix decoder transformers, where the input has bidirectional self-attention (full self-attention) and the output has masked self-attention, it should not hold true. Is that correct? I mean if the input has bidirectional self-attention, there is no point in predicting the next token in the input, since it has already seen the whole input sequence.
@@raminziaei6411 I think "Encoder-Only Transformers", like BERT, use full self-attention on the input, even though they still can predict the input. However, I don't know for sure.
It is a bit inaccurate here, it has both relation with pizza and oven, but due to its sampling, once it sampled taste after it, it will keep generating a sentence about pizza it can also sample cooks and complete with "cooks fast and never overdo" afterwards. thats why a temperature is here.
In an encoder-decoder transformer encoder was trained in English and decoder was trained in Spanish which made it possible to do translations. But here, only English is used for both encoding and decoding which makes it impossible to convert the English encoding to Spanish output. So here, would we used both language datasets combined to train the model to enable it to do translations as well?
Usually the tokens are just fragments of words, instead of entire words. This gives the decoder-only transformer more flexibility in terms of the vocabulary, since it can form new works it was never even trained on by combining the tokens in new ways. In this way, you can train a decoder-only transformer to translate english to spanish.
To learn more about Lightning: lightning.ai/
Support StatQuest by buying my books The StatQuest Illustrated Guide to Machine Learning, The StatQuest Illustrated Guide to Neural Networks and AI, or a Study Guide or Merch!!! statquest.org/statquest-store/
Bruh. This channel is criminally underrated.
Thanks!
🎉🎉🎉
Love your work!
Bam!
Well then criminals should rate it more highly
@@rickymort135I laughed. ⭐️
This explanation is essential for anyone looking to understand how ChatGPT works. While more in-depth exploration is necessary to grasp all the intricacies fully, I believe this explanation couldn't be better. It's exactly what I needed.
Thanks! I have a video that shows how all of these calculations are done using matrix algebra coming out soon.
And I thought you'd stop at ChatGPT. Thanks for never stopping to learn and teach!
Thank you!
Yes it's a good series
Haven't had a single stats course in over 3 years but I still keep up with this channel from time to time! Neural networks are way more complex than what I've ever had to deal with, but you manage to break down even these topics into bite size pieces...Bam!!
Thank you so much!!!
Quests on attention, transformer and decoder only transformer are of immeasurable value! Thank you so much! Keep the quests coming!
Thanks, will do!
Was stuck on a stupid detail about the architecture for a couple of hours. Your elaborate illustrations helped me make it clear in my mind right away! Thanks, keep going :)
Glad it helped!
Congrats on 1 million subs statquest!! All the Love from Korea!!
Thank you very much!!! :)
Thanks!
Thank you so much for supporting StatQuest!!! BAM! :)
謝謝！
Thank you so much for supporting StatQuest!!! TRIPLE BAM! :)
YOU ARE THE BEST TEACHER EVER JOSHH!! I wish you can feel the raw feeling we feel when we watch your videos
bam! :)
Oh my. Thanks for the recap, it was so necessary for this video. It made the concept extremely clear.
Glad it was helpful!
This video is proof that repetition is prime when teaching advanced concepts. I've watched many similar videos in the past and could never get all of these numbers to finally make sense in my mind. With your previous transformer video, I was getting closer but somewhat got lost again with the QVK values. Having to this second video to watch in a row made it clearer for me what all these numbers do and why we need them.
BAM! :)
Thank you so much! It is an amazing video and I haven't seen a video teaching AI/ML techniques like this anywhere! You're talented. And my research areas span Efficient LLM (LoRA, Quantization, etc). It cannot be better if I can see those concepts
Glad it was helpful!
Bedankt
HOORAY!!!! Thank you so much for supporting StatQuest! BAM! :)
I just want to say you are AMAZING. Thank you so much. I would personally love to see a video on backprop to train this, or even just training an RNN since we saw multi dim training, but not training once we get the state machine / unrolling involved. Loved the whole series 🎉
Thanks! I have notes for training an RNN, but the equations get big really fast. That said, it really is the exact same techniques presented in other videos, just a lot more of them.
Liking this video before i even start watching it as i know the content is simply brilliant!🎉
bam!
Triple BAM ❤❤👌👌
Hooray! :)
Hey Josh, I’ve been really digging your videos! They’re not only informative and helpful for my studies, but they’re also super entertaining. In fact, you’ve played a big part in my decision to continue pursuing AI Engineering. Could you please do a video about low-rank adaptation(LoRA). I am not good with that.
Thanks! I'll keep that in mind.
It summarized how GPT style of transformer architecture works and also helps us to understand how ChatGPT generates the text. Simply beautiful
Thank you!
Woahh, this is actually cool. We appreciate it a lot Josh!
Thanks!
This the most brilliant explanation that I have seen!!!!!! You are just awesome!!!!
Wow, thanks!
This is the only video on youtube that explains how such a complicated thing works so simply.
bam! :)
Hello Josh, i am enjoying your videos as they are helping me so much with my studies as well as entertaining me. You are kinda a reason i decided to continue studying bioinformatics. Since you are covering chatGTP and stuff now, could you maybe make a video about AlphaFold architecture in the future? I understand it might not be your topic of interrest, but i would love to lear in more deeply (pun intended). Thanks either way!
I'll keep that in mind.
BAM... You really killed it. Thanks for your explanation.
Thank you!
awesome, really helpful. Can't wait for another exciting episode!!
More to come!
Your videos are awesome! I've never thought I could learn machine learning in such an easy way. Love from china
Thank you!
Hey Josh! You're a gift for this planet 😍 so thanks this awsome explanations..
Wow, thank you!
I immediately rushed to amazon and purchased your book. Will get it in few days.
Hooray!!! I also have a new book coming out in early January. It's all about neural networks.
Delighted to watch one of the Most Brilliant videos. Hats off. Will join the channel tomorrow, first thing. Meanwhile do you have a one on Probability Density Function.
All of my videos are organized on this page: statquest.org/video-index/
Thanks, All are good, may be I could not find a one on P Density Function. Could you please point me out that specific video.
Wonderful Explanation! With great Visualisations!!!
Thank you!
Amazing Explanation! Double Bam 😊👍
Thank you! 😃
Another great video as always! Would be amazing if you could continue with Masked Language Models such as BERT in the future!
I'll keep that in mind.
your series almost save me...love from China💥
Happy to help!
Hello Josh, thank you again for your video !
I had one question concerning training the model on next token prediction:
As training data, would you use "What is statquest " or "What is statquest awesome" ?
What I mean by that, is when training the model by feeding it an input prompt such as "What is statquest ", do you also feed the model the word that comes after it (for calculating the loss), here "awesome" ?
The training inputs were "What is statquest awesome", and the labels were "is statquest awesome ". I'm working on a video that goes through how to code a transformer and how to prepare the training data. Hopefully it will be out soon.
@@statquest Thank you for your answer. I see that the decoder also learns to embed the input then (here, on the input , the label is "awesome").
I'm looking forward to your next vide !
incredible! this is such a clear explanation. thank you!
Thank you!
Thanks again Josh! I noticed that many GPTs are decoder only. Thanks for clarifying!
BTW saw that Yannic had a video on history rewrites. Probably not a topic for this channel, but still pretty cool 😁
Interesting!
Endless thanks to your awesome explanation. Would you mind clarifying the next confusion to me? You've mentioned that the normal transformer uses masked self-attention only during training and normal self-attention during inference, while according to other resources, including your iterations through the examples (I think you followed the masked self-attention mechanism), and in 34:20, how are we supposed to know the next tokens during training (parallel training) while we are restricted by the mask?
and thank you in advance.
We only apply masking to the attention mechanism. For more details, see: ruclips.net/video/KphmOJnLAdI/видео.html
@@statquest Understood, thank you very much for your quick answer. bam! :)
Sir Josh, Thank you for making this public. May God Bless you.
Thank you!
Greate explanation. It help me a lot. A million heart for u!!
Thank you!
A thorough explanation 😀
Thanks!
@@statquest if possible, could you please do a video on structural differences between llama and GPT?
@@ruksharalam173 I'll keep that in mind.
What an awesome video and channel😁👍. Would you consider doing a video on deep q learning models? I believe everyone would benefit from a video on such a fundamental topic. Thank you for your invaluable work🤩
I'll keep that in mind.
Thanks for the excellent explanation!
You are welcome!
Perfect video! Quick question, how are you drawing your lines? This line style is awesome!
I do everything in "keynote".
This is a god level YoutTube channel
:)
Very good video, thank you!
Thank you!
would love to learn about bidirectional transformers next ;-)
I'll keep that in mind.
@@statquest Pleeeeease, Josh!
Thanks for the great video, Josh. I got a question for you. What should drive my decision on which model to choose when facing a problem? In other words, how to choose between an Encoder-Decoder transformer, Decoder-only transformer or Encoder-only transformer? For instance, why ChatGPT was based on a Decoder-only model, and not on a Encoder-Decoder model or an Encoder-only model (like BERT, which has a similar application)
Well, the reason ChatGPT choose Decoder-Only instead of Encoder-Decoder was that it was shown to work with half as many parameters. As for why they didn't use an Encoder-Only model, let me quote my friend and colleague, Sebastian Raschka: "In brief, encoder-style models are popular for learning embeddings used in classification tasks, encoder-decoder-style models are used in generative tasks where the output heavily relies on the input (for example, translation and summarization), and decoder-only models are used for other types of generative tasks including Q&A." magazine.sebastianraschka.com/p/understanding-encoder-and-decoder#:~:text=In%20brief%2C%20encoder%2Dstyle%20models,other%20types%20of%20generative%20tasks
Hey, fantastic video as usual! Getting hard to find new ways to compliment, haha.
Just one quick question since you mentioned positional encoding. When generating embeddings from GPT embedding models (e.g., text-embedding-3-large), do the embeddings contain both positional encoding layer and masked-self-attention info in the numbers?
I believe it's just the word embeddings.
What a wonderful video!!! BTW, When will you publish your CD? I will buy it too😄Thanks!
BAM! Thank you!
that's perfect. Can you do more lectures on LLMs? Thanks a lot.
I'll keep that in mind.
great vids, any chance you could make videos on Q-Learning, Deep Q-Learning, and other RL Topics! Keep up the good work!
I hope to.
Great Video. If possible, please do a video on model fine-tuning techniques like PEFT/LoRA
I'll definitely keep that in mind.
Hi there! Thank you for the awesome video. It is very helpful and clear. I have a question about the part around 34:10, where you are talking about how normal transformer uses Masked Self-Attention in training. I did not quite understand how that is Masked Self-Attention (i.e. looking at the tokens before). Is the Decoder still looking at the input tokens, or is it looking at the output tokens that come before it? Thank you very much!
During training we know what the output should be, but we don't let the decoder look at tokens that come after the one it is currently processing. So if the output is "my name is fred", then, when we calculate attention for "name", the decoder can't use "is" and "fred". What's going on here might be more obvious if you look at how the math is done with matrix algebra in this video: ruclips.net/video/KphmOJnLAdI/видео.html (and if you are not already familiar with matrix algebra, see: ruclips.net/video/ZTt9gsGcdDo/видео.html )
Had a couple of questions regarding word embedding:
- Why do we represent each word using two values? Couldn't we just use a single one?
- What is the purpose of the linear activation function, can't we just pass the summation straight to the embedder output?
Thanks for the video!
1) Yes. In these examples I use 2 because that's the minimum required for the math to be interesting enough to highlight what's really going on. However, usually people us 512 or more embedding values.
2) Yes. The activation functions serve only to be a point where we do summations.
Can you please make a video about GNN? You are reaaallyy good at explaining
I'll keep that in mind.
Thanks for clear explanation
Glad it was helpful!
Hi Josh, great video, as always. I was wondering if you would also make a video about Encoder-only Transformers, like Google's BERT for instance, which can also be used for a great variety of tasks.
I'll keep that in mind.
I don't understand why we need the residual connections.... =''( isn't the word and position encoded values information already included in the masked self-attention values? or is most of the information lost so we need to directly add the word and position encoded values?
In theory you do not need them, but in practice they make it much easier to train large neural networks since each component can focus on it's own thing without having to maintain the information that came before it.
@@statquest Bam! Thank you!
I really enjoyed this video!
Thank you!
Thank you sir.
thanks!
Thank you for a deep explanation, I have a question , does the size of output layer equal the size of the vocabulary ?
yes
If the size of the vocabulary is 5 million, then I need 5 million neurons at the output layer ?
@@laythherzallah3493 Yes. But usually vocabulary sizes are much smaller (in the range of tens of thousands of tokens, rather than millions).
Hey Josh ! Would you mind making videos about graph neural networks ( GNN ) or graph convolutional network ( GCN ), and most importantly, the graph Attention Network ( GAT ) ? I have briefly gone over the maths these days, I already knew the matrix manipulation stuff but I think with your help, it would be much clear like your Transformer series, especially on the attention mechanism in the graph attention network ( GAT ), many Thanks 🙏🏻🙏🏻🙏🏻🙏🏻 appreciated !
I'll keep that in mind.
GNNs really only have two main elements to them, the aggregate function and the update function. The different choices of these two functions give rise to the different variants, GCN, GAT, etc.
thank you! the video is really nice
Glad you liked it!
Thank you, Josh, for yet another excellent video on GPT. I find myself slightly puzzled regarding the input and output used to train the Decoder-only transformer in your example. In the normal Transformer model, the training input would be "what is statquest ," and the output would be "awesome ."
However, in the case of the Decoder-only model, as far as I understand, the training input remains "what is statquest ," but the output becomes "what is statquest awesome ." Could you help to clarify this? If my understanding is correct, I'm wondering how the Decoder-only transformers know when to stop during inference, considering that there are two tokens within the generated response.
Because the first is technically part of the input, we just ignore it during inference. Alternatively, you could use a different token to indicate the end of the input.
We dont have to ask gpt to know stat quest is awesome reply from gpt BAM!!! BAM!! BAM!!
BAM! :)
Hi Josh! Thank you for your clearly explained work! You did a great job. I'm a big fan of you! I have a question about the embedding position. Here, you use two activation functions to encode the word to a vector, so that is why for the position embedding, you only use two sine and cosine to encode each word. Am I right? I found from the previous video: Transformer the foundation of chatGPT, you use four activation functions to encode each word, and later, you use four sine and cosine squiggles for the position embedding.
Each word embedding value needs a positional encoding value. So the number of sine and cosine squiggles that you use depends on the number of embedding values that you create for each token.
@@statquest Thank you for your explanation ! Have a nice day!
Hi there!
Came to YT in hope you had a nice video of Rank Robustness. Would be amazing, if you wanted to make a video about it!
Keep it up!
Also: nice Dinosaurs!
Thanks!
Another amazing video! So the fully connect layer basically maps two dimensional vectors to 5 dimensional vectors, in this case, is the size of the vocab(collection of tokens). Is that correct?
Yes, exactly!
@@statquestGreat! I have another question: How does a fully connected neural network map vectors representing different features (dimensions) to vectors that represent the indices (dimensions) of tokens in the vocabulary?
@@harryliu1005 I'm not sure I understand your question since the video shows exactly how the fully connected layer works. If you're not already familiar with the basics of neural networks, check out these videos: ruclips.net/video/CqOfi41LfDw/видео.html and ruclips.net/video/83LYR-1IcjA/видео.html
You didn't use the innocent, cozy, soft bear for softmax 🧸😢 _(in most of the parts)_
Good point!!! I think I need a smaller bear. :)
Josh, why in min 19:34 the similarity value for "Statquest" is lower for itself than for the word "What"? It shouldn't be larger to itself as is the self-similarity?
Unfortunately, this example is really too simple to really show off the nuance of what the actual numbers represent.
@@statquest:( wak wak. Thanks though :)
Wow this series culminating in a perfect explanation of GPT is the most magnificent piece of education in the history of mankind. Explaining the very climax of data science in this understandable step-by-step way so I can say that I understood it should earn you the noble prize in education! I am so grateful that you never used linear algebra in and of your videos. Professors at university don't understand that using linear algebra prevents everyone from actually understanding what is going on but only learning the formula.
I have an exam in Data Science on Friday in a week. Can you make a quick video about spectral clustering by Wednesday evening? I will pay you 250$! :)
Thanks! If I could make a video on anything in a week, that would be a miracle. Unfortunately, all of my videos take forever to make.
Hello Josh! First of all thank you for this great video, as usual it's very simplified and straightforward.
However, I have a little question. I saw your videos on transformers and this one, but every time I feel like the output is already there waiting to be embedded and then predicted. I mean that why the answer can't be "great" in stead of "awesome", what was the probablities given by the model for "great" and for "awesome" to make the final prediction. Here I gave the example of one extra word (great) but in real life it's the whole dictionary of words that can be predicted. So when generating the output, does it compute the "query" and "key" of the whole dictionary of words and then hopefully the right word has the best softmax probability?
Thanks in advance for the clarification.
no, you only calculate the queries, keys and values for the input tokens and the output as it is generated. However, in practice, instead of training on just a few phrases, we train on the entire wikipedia. As a result, the transformer can be much more expressive.
Another great session, thank you!!! Quick question, how do we decide what numbers to use for the Keys and Values?
For the weights? Those are determined with backpropagation: ruclips.net/video/IN2XmBhILt4/видео.html
Hi Josh, excellent video. I only recently found you but your channel is amazing.
It looks like you inadvertantly copied over the example value for "is" from the value for "what" starting around 1925 and this continues forward through the video.
I mostly say this in case you ever adopt these notes directly into a book.
Thank you! I've corrected my notes and do, in fact, plan on including it in a book soon!
Incredible!
Thank you!
Chat gpt is still a dialog system at its heart and has many different models which it gets results from . It softmaxes the outpits acording to the intent , ... So intent detection plays a large role in the chatgpt response .. the transformers are doing major works .. its super interested despite bqttling away with vb net !
Great explanation! Btw, what is the manuscript that first described the original GPT?
I believe it is called "Improving Language Understanding by Generative Pre-Training"
Hi Josh, thanks a ton for making such a simple video on such a complex topic. Can you please explain what do you mean when you say "
Your comment is missing the quote that you have from the video. Could you retype it in?
@@statquest Yeah. Can you please explain - Note :- If we were training the Decoder-only transformer, then we would use the fact that we made a mistake to modify weights and biases. In contrast when we are just using the model to generate the responses, then it doesn't really matter what words come out right now.
Thank you so much for another great video! I did have a question -- I'm confused about why you can train word embeddings with only linear activation functions because I thought that linear activation functions wouldn't allow you to learn non-linear patterns in the data, so why wouldn't you just not use an activation function at all in that case or use only one?
For word embeddings specifically, we want to learn linear relationships among the words. This is illustrated in my video on word embeddings: ruclips.net/video/viZrOnJclY0/видео.html And, technically, when coding a linear activation function, you just omit the activation function.
Awsome as always from you !! now we only need en real tutorial with python to creat a mini transformer model. hops it is on the making as my wish list
Working on it!
Thank you for the incredible content. Josh, quick question for you. I didn't see you mention vertically stacking the decoders in a way where the output of one decoder is the input for the next. From the 'Illustrated Transformer' page (I can't link b.c. youtube won't let me) it seems to be a core aspect of transformers. Thanks again.
Personally, I wouldn't call that a core aspect. Unlike the ability to layer attention units, feeding the output of one decoder into the input of another in a stack, did not influence how the decoder-transformer (or even an encoder-decoder transformer) was designed. In contrast, the ability to layer attention had a big influence on how the transformer was designed.
That makes sense. I just took another look at the Attention is All You Need paper, and it corroborates your explanation. Thank you.
Request: a video on the practicalities of creating and training a production LLM. The data volume, the number of parameters and how the production architecture differs from the simplified educational model provided in this video. I think this would allow the audience to better understand what simplifications were (rightly) made for the purpose of explication.
Also thank you so much for what you do. You are creating some of the best educational content on the internet. I am so jealous of this upcoming generation for having teachers like you :)
@@jacksonrudd3886 That's right - in the Attention is all you need paper, they just mention the stacking (N=6) in passing and don't spend any time on it. And I'm planning on making the exact video that you want me to make. It may take some time, but it's in the works.
Superrrb Awesome Fantastic video
Thank you!
After painstaking research on this topic, I have realized that all the information is not in the transformer itself. The transformer is just the neural network (i.e., the processor or function). The information is actually in the embedding model. The sequences are all stored there.
When training the transformer, the word embedding model needs to be fully trained or trained with the same data. This allows the learned embeddings to fit the transformer model. Here, overfitting is beneficial as the model will be a closed model after training. The transformer can be retrained (tuned) to your specific task. The main task should be text generation corpus/tuned for question and answer.
The learned embeddings model can actually generate data as well as the transformer neural network, making it a rich model. I also noticed that to get an effective word embeddings model, the optimization or "fine-tuning" should involve classified and labeled data. This allows clusters of similar terms to appear, as well as emotionally aware embeddings.
The transformer is essentially performing an "anything in, anything out" operation, forcing the input and output to converge despite any logical connection between them. Given an input, it produces the desired outcome. As many similar patterns are entered, it will fit this data using the masking strategy. Transformers need to be trained as masked language models to be effective, highlighting the drawbacks of LSTM and RNN models which cannot produce the same results.
The more distanced the output is from the input, the more layers and heads you need to deploy.
... wut?
@@iProFIFA I mean the most important area of focus should be the embeddings model as this is the part which stores the data after training ... The transformer is just the neural network function. ..
Also.... Chat gpt is primarily a "chat bot" so all inputs are fed to its Intent detection module , IE classify the user intention ... To send the query to the correct model for a response ...
Right now the transformer is seen to be able to fit anything to anything model .... As long as you give an input and target to the model and enough samples the "function (transformer)" can fit the data to the model ... Hence the first model to train is the text generation model to give it many possibilitys to generate some form of text given a seed .... Then tune the model to your specific task , IE question and answer or the New "instruct" style model ...
IE : for a true code generation model . The first corpus should be text books , tutorials , forum posts , mass source repos .... This will give it a massive base to generate some form of code given a seed ... Then to pass in the instruct code models , IE build a binary tree , and provide the specified code , ... It can be ANY code language .... Because the model is a transformer , later we can add another fine tune , for specific code translation ... IE csharp examples with Thier equivalent python and rust and javascript etc ... .. now this model can be retrained until it "overfits it's domain model" .... By adding general language corpus and basic questions answer modelling "after" you can now ask question and pursues project objectives ... Because this was originally trained with code model it's preferred output will be code ....
The sealed language model outputted .. this is also the issue as all the data is in the embedding model you train it with ... So ... .
This embedding model is the most important , as neural networks can fit any numerical data ..
Which is the best model for word embeddings ... Fasttext , glove , skipgram ? Are there others ? What's the best optimiser for these models ? Are they interchangeable ? ..
Finally , the transformers successes of late are due to the deployment of layers and heads .. this increase give the perception of adding dimensionality(randomness( tweakable settings (all weights and biases) but this is due to attempting to use a single model to fit all tasks ... If you have models for code gen. And others for picture gen , and others for chat gen, then others for business and local domain knowledge . Then it's better to have an intent model In front of all of these dedicated models . And direct the query to the appropriate models and take a softmax of the output to choose the correct output ! ... Again optimising the intent detector with the input and expected output and correct classification of the requested task ....
Hopefully I explained it a bit better this time ! Lol .
Weights an biases don't take up space (they are just a few numbers in a table) it's the embed model ..
@@xspydazx yo, your way of writing is really confusing and it seems like you went on a tangent so it’s hard to follow. Also you shouldn’t put spaces before commas and periods.
@@xspydazxare you talking about mixture of experts model in your last comment bro??
Hi Josh,
I have a nagging question: if I understand your explanation correctly, in order to predict the next token, we first compute Keys, Queries and Values for the prompt tokens (or more generally - the prompt and the current set of completion tokens), THEN we use them to compute Masked Self-Attention and Residual Connections, which finally are passed through the fully-connected layer to predict (generate) the next token. If that is true, it would seem that in order to generate the completion, all we need to do is (conceptually) "place" the prompt at the output, compute ONLY Keys and Values for the prompt, then use these Keys and Values when computing Masked Self-Attention and Residual Connections for the completion tokens ONLY. There doesn't seem to be any need for computing Masked Self-Attention and Residual Connections for the PROMPT tokens when generating the completion tokens. In other words, ONLY the prompt Keys and Values are sufficient to provide the full context to compute the Masked Self-Attention and Residual Connections for the completion tokens.
However, in your explanation you did go through computing Masked Self-Attention and Residual Connections for the prompt and (seemingly) re-generating the prompt tokens before moving on to generating the completion tokens. This part was unclear and appeared unnecessary to me.
When asked, ChatGPT insists that "In decoder-only transformer models like GPT-3 or GPT-4, the prompt tokens are indeed fully processed through all the layers, INCLUDING the masked self-attention mechanisms and residual connections" and that "Keys and Values are derived from the hidden states AFTER incorporating the masked self-attention and residual connections". This would suggest that the Keys and Values are somehow computed AFTER (based on, using) Masked Self-Attention and Residual Connection and not vice versa as explained in your lecture. I understand that you might have simplified the design and/or reordered the computations for brevity/simplicity of explanation, but I would appreciate a clarification if the Masked Self-Attention and Residual Connections indeed need to be computed for the PROMPT tokens in addition to their Keys and Values in order to generate the completion tokens. Thanks so much!
Using the Queries/Keys/Values from the prompt to regenerate the prompt speeds up training.
@@statquest Thanks for your response. This makes sense for training, but in my question I was focusing on the situation during inference (I failed to state it clearly, sorry!)
@@rbryll2002 Sure, if you wanted to optimize the process, you could skip generating tokens until you get to the response.
@@statquest Does it mean that computing the Masked Self-Attention and Residual Connections for the PROMPT tokens is unnecessary during inference, since only their Keys and Values appear to be needed to compute the Masked Self-Attention and Residual Connections for the COMPLETION tokens? That is my (mis)understanding based on your video, but ChatGPT disagrees, stating the masked Self-Attention and Residual Connections for the PROMPT tokens are also needed when generating the COMPLETION tokens.
@@rbryll2002 I believe that it is done, and that is how it is described in the original manuscript. However, it is possible that you could skip that step. If you would like, you can play with the code to see what happens. I explain the code here: ruclips.net/video/C9QSpl5nmrY/видео.html and and you can get it to play around with here: github.com/StatQuest/decoder_transformer_from_scratch
Hey Josh. I just want to ask what software you use to make the diagrams?
I use keynote and show some of my tricks here: ruclips.net/video/crLXJG-EAhk/видео.html
I didn't really understand where the fully connected layer came from and how it is connected to the input values. It seems independent of the previous layers and somehow outputs the input words
The illustration at 26:53 may make things easier to grasp. The residual connections are used as inputs to the fully connected layer.
Hey ,if we are feeding the predicted output as next input after doing the embedding,positional encoding and masked attention with previous values ,why did you take "awesome" as the next input even when the model predicted it as "what" .i didn't get you here.
What time point, minutes and seconds, are you asking about?
25:43 to 26:17 , the word "is" predicted next word as "what" but we still went on taking statquest as next word , why??
@@SaiPragnaanMarala Because we are still processing the input prompt. During this stage, it doesn't matter what tokens the transformer generates because they are all discarded since we already know what the prompt is. However, once we start generating a response to the prompt, then we can't discard any of the tokens that we generate and have to use them (unless we are training the model).
hello..first off thank you for this great content
I had a question/s
could you give an example of how the embedding neural network is trained? i.e. what is the input and output in the embedding neural network during training? The neural networks I have worked with statements that go along the lines of "given a set of pixels determine whether the picture is a cat or not".. I do not know what the equivalent is with embedding neural networks
and follow up question..can the embedding neural network be the same for an encoder-decoder model and a decoder-decoder model?
1) We don't train the embedding layer separately from the rest of the transformer. So the inputs are what you see here as well as the ideal outputs that we use for training.
2) Once trained, yes.
20:56 I think the value of the word "is" is miswritten it should be 1.1 , 0.9 not 2.9,1.3 it should not be same with the value of word 'what' Thank you for your videos btw ur explanation is awesome.
That is correct. Sorry for the typo! :)
The embedding values for "what" and "Statquest" are identical before the positional encoding. Is that not a typo ?
That is correct. In order to illustrate how a decoder-only transformer worked, I had to make the model as simple as possible, and, as a result, some of the nuance in the values for the weights was lost.
Brilliant
Thanks!
I feel it’s a bit misleading that it seems the tokens of the input sequence is fed in one by one, and that when you put in the first token, it predicts the second token but just ignores it, where in reality it feeds the entire sequence to predict the next target token, and on next iteration, you append the input sequence with the target token as input, and predicts the second target token, and so on. Right?
At 26:28 I state that each token in the prompt is processed simultaneously.
@@statquest gotcha. Thanks for the clarification, sensei.
I didn't understand just one part: how are the weights to calculate Q, K and V for each word in the sentence calculated? Is it also an optimization process? If so, how is the loss function calculated?
At 5:08 I say that all of the Weights in the entire transformer are determined using backpropagation. Specifically, we use cross entropy as the loss function. For more details about cross entropy, see: ruclips.net/video/6ArSys5qHAU/видео.html and ruclips.net/video/xBEh66V9gZo/видео.html
Great video. Do I understand correctly DNN which is responsible for word embedding, it not only converts the token to its representation as a numeric vector, but already predicts as the next word should be returned ?
In a transformer the embedding layer alone does not predict the next word because it wasn't specifically trained to do that the way a stand alone word embedding layer (like word2vec) would.
@@statquest But if we train the whole model at the same time, then backpropagation does not change the weights of the network responsible for word embedding in such a way that they learn to predict the next word? Or don't we train this first network while learning ?
@@JanKowalski-dm5vr It might. But the whole model, word embeddings and attention and everything, is trained to predict the next word, or translate, or whatever it's trained to do. So it's hard to say exactly what the word embedding layer will learn.
Hi josh. Would you mind making a video for retention network?
I'll keep that in mind.
Hi Josh! I'm a little bit confused about the whole idea of generating the input first, compare it to the actual input and use to modify weights and biases in the training phase. I cannot find anywhere on the internet that it is mentioned. All I see is that the masked self attention is used on the input sequence to make contextualized version of each word and then, they are used to generate the target tokens. Nowhere can I find that generating the input sequence and compare it to the actual input is part of the process. Can you please clarify?
What time point, minutes and seconds, are you asking about?
@@statquest The section "generating the next word in the prompt" in this video. mins 23-27
@@raminziaei6411 The idea of comparing the predicted input sequence to the known input sequence comes from the original manuscript that describes decoder only transformers, GENERATING WIKIPEDIA BY SUMMARIZING LONG SEQUENCES. They say: "Since the model is forced to predict the next token in the input as well as [the output] error signals are propagated from both input and output time-steps during training."
@@statquest Thanks Josh. A related question would be "Does this only hold true if we are considering causal decoder transformers where we have masked self-attention for both input and output sequences? For prefix decoder transformers, where the input has bidirectional self-attention (full self-attention) and the output has masked self-attention, it should not hold true. Is that correct? I mean if the input has bidirectional self-attention, there is no point in predicting the next token in the input, since it has already seen the whole input sequence.
@@raminziaei6411 I think "Encoder-Only Transformers", like BERT, use full self-attention on the input, even though they still can predict the input. However, I don't know for sure.
It is a bit inaccurate here, it has both relation with pizza and oven, but due to its sampling, once it sampled taste after it, it will keep generating a sentence about pizza
it can also sample cooks and complete with "cooks fast and never overdo" afterwards. thats why a temperature is here.
:)
In an encoder-decoder transformer encoder was trained in English and decoder was trained in Spanish which made it possible to do translations. But here, only English is used for both encoding and decoding which makes it impossible to convert the English encoding to Spanish output. So here, would we used both language datasets combined to train the model to enable it to do translations as well?
Usually the tokens are just fragments of words, instead of entire words. This gives the decoder-only transformer more flexibility in terms of the vocabulary, since it can form new works it was never even trained on by combining the tokens in new ways. In this way, you can train a decoder-only transformer to translate english to spanish.
Dude, can you make a video on state space models like Mamba? It's super interesting!
I'll keep that in mind.
Bam! @@statquest