To learn more about Lightning: lightning.ai/ Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/
Personally I find it more clear to link embedding to hidden class of words, i use character sheets as a netaphor, because what attention does is not looking at word but the description in its sheet, with each attention head focusing on different part of the description, which mean a word representation have multiple attention on different hidden class. Then at the end we look at the sheets transformed at each layer to find the next word. That also allows to explain multimodality, ie make sure image input and text input share the same description sheet.
Transformers needs more than one video, each part(multi H attention, word embeding(sin&cosine similarity),training &…) I was waiting for long to reach stat of art.
@@Tothefutureand I thought about doing it that way - and that was the original plan. But my video on Attention convinced me that most people would rather have a single video that has everything in it all at once. However, I've provided links in this video's description to full length videos on each topic you are interested in.
@@statquest oh you mentioned that you don't know why this number of head, that's hardware optimization, ie they can be split into gpu or memory pool or reduce bandwidth, such that they can parralelized or compute sequentially on resource starved machine.
As a Computer Science professor who teaches Machine Learning, this is probably my most anticipated video ever. I regularly use your videos to brush up on/review ML concepts myself and recommend them to my students as study aids. You explain these concepts in the clear, straightforward way that I aspire to. Thank you!
@@statquest our data analysis professor also uses your videos as references and recommends you almost every session haha. i learned about this amazing channel from him.
Its kinda hilarious that StatQuest videos give the impression they were menat for 5 year olds, yet are exploring legitimately complex topics. No jargon, no overcomplicated diagrams. Josh really tries to explain things and not show off his supirior understanding of neural networks. Thanks Josh!
I've just learned machine learning for a month, my major is a pure business student. I've been working as a Data Analyst for 2 months as the internship and I believe machine learning will be essential if I want to go further in this industry. Out of all tutorials videos I've watched, your videos brought up the clearest and most concise concepts for me to understand. All the videos walked me through from the series of neural network, back propagation, cross entropy with backward propagation, recurrent, LSTM and convolutional neural network, lastly, this video. Really appreciate for your understandings and amazing storytelling through your videos, your contents always make me eager to keep learning machine learning myself. Thanks a lot
This is the complet-est, precious-est, pur-est, brilliant-est video ever. Can't imagine how much work you've put into creating these illustrations. It's just brilliant. Hats off.
Not to be a buzz kill, but I suffered a bad Traumatic brain injury in my late 40's after being hit by an SUV while stopped on a motorcycle. I'm blessed i survived . At the time my job dealt with engineering and architecting IT solutions and I was looking forward to advancing my career into AI and Machine Learning. I was in a coma for a while and I lost lots of what i used to know. I know have Learning disabilities and memory issues. I have improved some over the last years, but If I'm being honest with myself, I wouldn't want me as an engineer, so I'm trying to move into management. I'm glad I ran across these videos . I purchased the .pdf books and notebooks today and I can honestly say they are well worth it. Josh, I'm so glad You created this material. Your books and notebooks etc.. are helping me slowly understand complex topics in hopes that I can stay relevant and continue to advance my career. Thanks again!
How a model like this was created is just beyond me. There’s so many different moving parts. You could write a whole book on the fully-connected network alone. Add in all the other stuff? Wow. Thank you, Josh, for explaining this so well!
Thanks! It's a little easier to understand how this model was created in the first place if you follow the whole Neural Networks playlist. You'll see how things changed, one step at a time, to eventually end up with a transformer: ruclips.net/video/CqOfi41LfDw/видео.html
I've been looking for tutorial about transformers for a long time. This is the smoothest tutorial. It does not hide any complexities(making me confident that I actually understand the concept instead of its dumbed down version for mortals that won't end up ever using the knowledge), but also does not get lost while explaining those complexities and clearly calls out what else I can learn about to understand the side concepts better. Super !!!
22:12 8 heads because 8 gpu clusters are common and hence can compute in parallel . The embedding dimension are 512 and that leaves each head has 64 query size. Great video 👍
@@jakob2946 curious aswell, I looked it up and it seems that its not true, every head is applied to all dimensions of the embedding. This also makes more sense to me since the word embeddings should be looked at as a whole. please correct me if I'm wrong
@@oliviervangoethem9365 I don't know about more recent transformers, but in the initial architecture each attention head is applied to a projection of input embeddings, with reduced dimensionality (in the original "Attention is all you need" paper: embeddings have a dimension of 512, and each of the 8 attention heads has a dimension of 64). The reason for this is spelled out in the original paper: "Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this."
its amazing how you make a 2 hours lecture in just 30 mins and explain it way better, after finishing this video and realizing what I just grasped, its mind blowing how you can make such complicated subject easy to understand. thank you very much!
For slowpokes like me: The example assumes total vacabulary size of 4 for each language. Thanks Josh for providing some of the best content on the subject! Finally the K, Q, V made clear sense
I try very much to understand transformer but I got 100% satisfaction from your video , your explanation is like unbelivable . I enjoy study very much when you taught any topic
Thanks, Josh, for keeping your promise to make a video about Transformers. I learned a lot and truly appreciate your effort in explaining this concept. I just placed an order to buy your book and made a donation to support the channel. I'm looking forward to more content on Machine Learning and hope to see videos about GPT and BERT models. ♥
Thank you so much!!! I really appreciate your support (TRIPLE BAM!!!). I hope to do the GPT video soon, but we'll see - the timeline is a little out of my control right now.
I finally decided to buy your book thinking there'd be transformers in the "Neural Network" section. But even if they're not, I'm glad it supports you. Your content is the best in popularisation that I've seen. It mainly helps me a lot to refresh and understand better than before to start my internship in AI after 1 year of gap year.
I just finished watching almost all the videos on this channel and i have to say that this is probably the best place to learn stats and machine learning. I also bought the ML book and it captures the essence of the style of teaching on this channel really well and is very handy to go back and quickly look up some details. You are doing great work!
Thanks to your engaging visualization and clear explanation, I've grasped the Stanford CS224n course! Your content is neatly condensed but doesn't miss a thing. It's like you've taken all the complex concepts and served them up on a platter. Triple Bam!
words cannot describe how happy I am to be able to watch this video. You really helped me with my studies. It is you who made me so interested in AI and think that I am actuaaly able to understand what is going on. Thank you for your simplified models. They really help when larning more complex stuff on this or that theme. But everytime there's a theme I do not know, the first thing I do is go to statquest. Thank you, Josh!
As a computer science student getting into the field of NLP, I really can't thank you enough for making a video that breaks down Transformer like this. Our uni doesn't go in depth about NLP related topics and with a very brief explanation they do, the uni expects us to have a full understanding about NLP. I can't thank you enough!
Thank you so much for this video. More than 35:years in IT, worked with punching cards, I started to believe I am too old for this new thing. It was not my age, but the way how the knowledge is offered which does not fit my learning style. Finding your video, listening and immediate insights, hearing the coins dropping, were only separated by minutes. Ordering your book is the least minimum I could do to say thank you.
I'm currently studying for job interviews and I don't know what I would do without you, thank you! When I get paid from my first job I will donate to you :)
That is a lot of stuff in a single video!! For those who are wondering, ChatGPT is a decoder only neural network, and the main diff between an encoder and a decoder is that a decoder uses masked attention - thus ChatGPT is essentially an autoregressive model. Notice how ChatGPT generates a response in sequential order, from left to right. Anyway, good stuff!
Yep - I'd like to make a GPT video just to highlight the explicit use of masking (the self attention in the decoder in this video used masking implicitly).
Definitely the clearest walkthru of Transformer. It's very good that you put heavy emphasis on the parallelizability of Transformer since IMO it is the most important feature that made Transformer so useful
@@statquest I actually had a doubt as I was going through, about the decoder part. In the masked multi head attention part of the typical transformer, what inputs do we provide? And is this part only used during training?
I've spent a good deal of time studying attention, the critical concept behind transformers. Don't anticipate a natural understanding of the Q, K, and V parameters. We aren't entirely certain about their function; we can only hypothesize. They could still function effectively even if we used four parameters instead of three. One crucial point to remember is that our intuitive understanding of neural networks (NNs) is far from complete. The matrices for Q, K, and V aren't static; they're learned via backpropagation over lengthy training periods, thus changing over time. As a result, it's not as certain as mathematical operations like 1+1=2. The same applies to the head count in transformers; we can't definitively state whether eight is a good number or not. We don't fully grasp what each head is precisely doing; we can only speculate.
Prof. Starmer, Thank You very much. You are an inspiration to all the aspiring Machine Learning Enthusiasts. Respect and Gratitude from India. #RESPECT
So nicely explained! I have searched for "how transformers work" but no one on youtube explained with both concept and math! Keep the good work going 😁👍
It's interesting how many different topics one has to contextualize to understand something like transformers! It's great to see all the math happening in detail and stop and ponder time-to-time while learning something as complex as transformers for the first time. It really helped build my intuition of a lot of building blocks for a transformer-based neural network. Thank you!
Your neural networks playlist including this video gave me an intuitive understanding of transformers in less than a week which is something that would have taken an entire semester otherwise. I stumbled onto them while searching for a better understanding of Q,K,V, which everyone seems to say is as simple as querying a database…but what does that even mean?? Your explanations are brilliant, and I will be sharing with everyone I know who wants to learn more about this topic. I look forward to future videos. Thank you!
I'm just starting learning about Machine Learning, and this video has been very clear as an introduction to learn concepts and move on. Thanks for great content, contiuining with side quests right now!
Thanks Josh, another great video, I’ve been following your channel for years now and your videos have massively helped me to change career so huge thanks. On to the transformer network, there’s something about the positional encoding that makes me feel a little uneasy. It feels we’ve gone through great effort to train a word embedding model that can cluster similar words together in n-dimensional word embedding space (where n can be very large, often 1,000). By then applying positional encoding before our self-attention, whilst you very clearly explained with your example how important adding this information to the model is, seems to me to mess up all the effort we put into word embedding to get similar words clustered together. The word pizza, instead of being positioned in the same place can now jump around word/positional embedding space. Instead of one representation of pizza in space, it can now move around to be in many different positions, and not move locally around its own 'area' but because we add the positional encoding to the word embedding, scaled equally, it can jump around a great deal of space. To me it would seem adding this much freedom to where the word pizza can be represented in space would make it much much harder to train the model. Is my understanding correct or is there something I’m missing?
I have a couple of thoughts on this. Maybe I should make a short video called "some thoughts about positional encoding". Anyway, here they are... Thought #1: Remember the positional encoding is fixed, so the word embedding values have to take them into account when training. For example, since all of the positional encoding value are between -1 and 1, it is possible that the word embedding values will have larger magnitudes and thus, not move around a lot when position is added to them. Thought #2: Because the periods of the squiggles get larger for larger embedding positions, after about the 20th position, the position encoding values end up alternating 1 and 0 (in other words, after the 20th position, the position encoding values are 1010101....) and it is in that space, from the 20th position to the 512th position (usually word embeddings have 512 or more positions) that the word embeddings are really learned, and that the first 20 positions are mostly just for position encoding.
I’ve been having some additional thoughts on this and think I may have another reason (or rather an example) why adding positional encoding to the word embedding vectors makes sense, Josh if you read this, feel free to shoot it down! Take the following sentence: “The weather is bad, but my mood is good”. In this sentence the first “is” refers to the weather, whereas the second "is" refers to my mood. Without positional encoding and only word embedding, the vector for “is” being passed into the attention unit will be the same for the two instances of the word in the sentence. If we don’t use masked self-attention and compare the word “is” to every word including itself in the sentence, then the output of the word “is” in the self-attention unit I believe should be the same for both instances. Therefore, the unit will struggle to successfully differentiate the relative meaning of the two words. By adding in positional encoding prior to the self-attention unit, we’re suddenly adding context to the word. The second “is” comes straight after the word “mood”, therefore the position vector we’re adding to each of the two words should be similar. However, because the word “weather” comes 6 words before the second “is”, the positional vector we add will be quite different. Presumably this difference helps a self-attention unit to differentiate the relative meanings of the two instances of the word “is”.
@@matthewhaythornthwaite9910 Another reason why you would want to add positional encoding instead of doing something else is that it preserves the dimensionality of the encoding. Imagine a theoretical encoding which is not added (like a one-hot encoding for each sequence location), and some linear (or non-linear for that matter) transform to combine word embedding and positinonal encoding. This is great in the sense that we do not polute the embedding space with "arbitrary" offsets, but now our input sequence has to be of a fixed shape. Addition of orthogonal sinusoids guarrantees a non-parametric, dimensionality preserving encoding which does not fix the number of inputs we can give to the network. By the way, I think there is an analogy between adding positional encoding to embeddings and adding residual/skip connections to network outputs. Imagine that we have a network that is represented by the function f(x) and we have some target function F(x) which we want the network to learn. Imagine now that we modify our network to compute the function f(x) = h(x) + x (where "h(x)" is the network in front of the skip connection "h(x) + x"). Here too we polute the output space of h(x) with the values of x. However the network f can still learn F, so long as the network h(x) learns the function h(x) = F(x) - x (such that f(x) = h(x) + x = F(x) - x + x = F(x)). I suppose for positional encoding something similar holds (altough it probably has to learn a much more difficult internal pattern), where the network is f(E(x)+q) learns to associate word embedding values E(x) which are "convolved" by some known offsets q and probably learns to deconvolve E(x) and q (into some abstract representation).Given that E(x) + q may in theory be (nearly) non-unique (i.e. E(x_1) + q_1 = approx E(x_2) + q_2) it might still be possible for the network to deconvolve the values into the correct inputs based on the context vector C which is calculable from the rest of the input sequence. I suppose one can't exclude that the network may never get this wrong, but in practical terms, it seems to work well enough.
One of the best explanation of encoder / decoder architecture. Esp. the self attention part. I really liked the way you colored Q,K, V to keep track of how things are moving . Looking forward to more such videos
Thanks! I've also got a video on Decoder-Only Transformers: ruclips.net/video/bQ5BoolX9Ag/видео.html and I'm working on one that shows the matrix algebra (color coded) of how these things are computed.
Hey @statquest - here is a quick suggestion. Another convenient way to explain positional encoding might be by drawing clock with minute and hour hand. Then - instead of sin() and cos() functions you could simply track the x and y coordinates of the tip of the minute and hour hand. It gives much more convenient intuition behind mechanics of the encoding. (a) it shows its repetitive nature (b) ties encoding position with sense of time (which is intuitive since speech is tied to time as well). Speech is the most common way we use language (c) it explains why we use both sin() and cos() functions (to track circular motion of the clock hand) (d) it provides intuition on why having two pair of sin() and cos() functions is better than just one
Transformers! More than meets the eye!? I think there is a lot of value in knowing this technology well! Thank you for your humor and learning support, I can't wait to return the favor!
I must say, after learning how GPT works I am even more blown away that this can work the way it works with the big online models. We just have to keep in mind that these companies are blowing tens of millions just for training. That's something else than me with a RTX 4090 and pytorch spending 12 cents of electricity and thinking I have unfathomable power 😂
I am coming to Josh after about 3 years, I love Josh Starmer style of teaching and it has not changed a bit. The good news is Josh is now teaching advanced Deep learnng concepts.
(Verse 1) Here we are with another quest, A journey through the world of stats, no less, Data sets in rows and columns rest, StatQuest, yeah, it's simply the best. (Chorus) We're diving deep, we're reaching wide, In the land of statistics, we confide, StatQuest, on a learning ride, With your wisdom, we abide. (Verse 2) From t-tests to regression trees, You make understanding these a breeze. Explaining variance and degrees, StatQuest, you got the keys. (Chorus) We're scaling heights, we're breaking ground, In your lessons, profound wisdom's found, StatQuest, with your sound, We'll solve the mysteries that surround. (Bridge) With bar charts, line plots, and bell curves, Through distributions, we observe, With every lesson, we absorb and serve, StatQuest, it's knowledge we preserve. (Chorus) We're traversing realms, we're touching sky, In the field of data, your guidance, we rely, StatQuest, with your learning tie, You're the statistical ally. (Outro) So here's to Josh Starmer, our guide, To the realm of stats, you provide, With StatQuest, on a high tide, In the world of statistics, we stride. (End) So get ready, set, quest on, In the realm of stats, dawn upon, StatQuest, till the fear's gone, Keep learning, till the break of dawn.
(just letting you know while cramming for my exam, ive used your videos for over 4 years now and some professors, including the ones from this class, use your videos for us to learn (either showing them to us or using screenshots from the videos with reference). Its so funny watching the reaction of people who arent familiar with your channel reacting to a soft cute bear as the soft max function and such haha Most people in portugal can speak english but still, many prefer to search in portuguese, so they never encountered your channel before. But I make sure to let all my friends know your content that helps us with some specific exam. I would never be able to thank you enough for the more than wonderful content and for your patience explaining questions we have in the comments)
Thank you very much! I'm glad to hear that my videos are helpful. Maybe one day I can go to Portugal and teach a lecture in person. That would be super fun. :)
Just wanted to say that I understood everything about LMs (thanks to your videos), except the part on transformers cuz the video wasn't out yet ahah. Well now that my dear squash teacher explained it, everything's clear. So really THANK YOU for your hard work and dedication, it made all the difference in my understanding of Neural Networks in general
These videos on machine learning (and statistics!) are incredible and empowering! Plus they make me laugh! You are the reason that I was able to keep my head above water in my Machine Learning courses as a pregnant grad student with limited time, money and energy. Thank you and I will definitely be checking out some merch!
Oh my god! Now it makes sense why training a transformer model takes FAR more compute than it does to run them! I can run them on my CPU, but training a big one may take weeks to months on entire GPU clusters! So many values to optimize and SO many nodes, inputs and outputs! That's insane! And it also shows why the models are SO large! There are SO many weights and biases, SO many connections and an entire world of vocabulary to store in it and connect up! This is SO impressive!
This is a lot of material for one video. But people wanted a single video, rather than a series of videos making incremental steps in learning, for transformers. Personally, I would have preferred a sequence of shorter videos, each focused on just one part. That said, there is something about seeing it all at once and getting that big picture. My book on neural networks (that I'm working on right now) will try to do both - take things one step at a time and give a big picture.
Thank you sir for this amazing video, it helped me last year in my NLP exam and now i'm refreshing my information's about transformers hoping to land an interview soon!
To learn more about Lightning: lightning.ai/
Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/
Personally I find it more clear to link embedding to hidden class of words, i use character sheets as a netaphor, because what attention does is not looking at word but the description in its sheet, with each attention head focusing on different part of the description, which mean a word representation have multiple attention on different hidden class. Then at the end we look at the sheets transformed at each layer to find the next word. That also allows to explain multimodality, ie make sure image input and text input share the same description sheet.
@@NeoShameMan Interesting.
Transformers needs more than one video, each part(multi H attention, word embeding(sin&cosine similarity),training &…)
I was waiting for long to reach stat of art.
@@Tothefutureand I thought about doing it that way - and that was the original plan. But my video on Attention convinced me that most people would rather have a single video that has everything in it all at once. However, I've provided links in this video's description to full length videos on each topic you are interested in.
@@statquest oh you mentioned that you don't know why this number of head, that's hardware optimization, ie they can be split into gpu or memory pool or reduce bandwidth, such that they can parralelized or compute sequentially on resource starved machine.
As a Computer Science professor who teaches Machine Learning, this is probably my most anticipated video ever. I regularly use your videos to brush up on/review ML concepts myself and recommend them to my students as study aids. You explain these concepts in the clear, straightforward way that I aspire to. Thank you!
Thank you! BAM! :)
Me too!
@@statquest our data analysis professor also uses your videos as references and recommends you almost every session haha. i learned about this amazing channel from him.
@@Daigandar That's awesome! BAM! :)
Well said. I do the same!
Its kinda hilarious that StatQuest videos give the impression they were menat for 5 year olds, yet are exploring legitimately complex topics. No jargon, no overcomplicated diagrams. Josh really tries to explain things and not show off his supirior understanding of neural networks. Thanks Josh!
Thank you! :)
Much like Richard Feynman.
BAM!
I've just learned machine learning for a month, my major is a pure business student. I've been working as a Data Analyst for 2 months as the internship and I believe machine learning will be essential if I want to go further in this industry. Out of all tutorials videos I've watched, your videos brought up the clearest and most concise concepts for me to understand. All the videos walked me through from the series of neural network, back propagation, cross entropy with backward propagation, recurrent, LSTM and convolutional neural network, lastly, this video. Really appreciate for your understandings and amazing storytelling through your videos, your contents always make me eager to keep learning machine learning myself. Thanks a lot
Thank you very much! I'm glad my videos are helpful.
@@statquest please upload videos on auto encoders and gated recurrent units also please
This is the complet-est, precious-est, pur-est, brilliant-est video ever. Can't imagine how much work you've put into creating these illustrations. It's just brilliant. Hats off.
Wow, thank you!
Did you just tokenize your comment?
@@lumiey I'm not sure I understand.
@@statquest He just separated words like complet, est, precious, est, pur, est... like tokenizer does (e.g. following -> follow, ing)
@@lumieyHaha
Not to be a buzz kill, but I suffered a bad Traumatic brain injury in my late 40's after being hit by an SUV while stopped on a motorcycle. I'm blessed i survived . At the time my job dealt with engineering and architecting IT solutions and I was looking forward to advancing my career into AI and Machine Learning. I was in a coma for a while and I lost lots of what i used to know. I know have Learning disabilities and memory issues. I have improved some over the last years, but If I'm being honest with myself, I wouldn't want me as an engineer, so I'm trying to move into management. I'm glad I ran across these videos . I purchased the .pdf books and notebooks today and I can honestly say they are well worth it. Josh, I'm so glad You created this material. Your books and notebooks etc.. are helping me slowly understand complex topics in hopes that I can stay relevant and continue to advance my career. Thanks again!
TRIPLE BAM!!! Thank you so much for supporting StatQuest and I wish you the best as you continue to learn about ML and Data Science! :)
How a model like this was created is just beyond me. There’s so many different moving parts. You could write a whole book on the fully-connected network alone. Add in all the other stuff? Wow.
Thank you, Josh, for explaining this so well!
Thanks! It's a little easier to understand how this model was created in the first place if you follow the whole Neural Networks playlist. You'll see how things changed, one step at a time, to eventually end up with a transformer: ruclips.net/video/CqOfi41LfDw/видео.html
@@statquest Thanks Josh! I’ll check out that playlist. What you’re doing is so special to the world, and humanity is so indebted.
@@statquest beautiful, thank you for the work
I've been looking for tutorial about transformers for a long time. This is the smoothest tutorial. It does not hide any complexities(making me confident that I actually understand the concept instead of its dumbed down version for mortals that won't end up ever using the knowledge), but also does not get lost while explaining those complexities and clearly calls out what else I can learn about to understand the side concepts better. Super !!!
Thank you very much! :)
22:12 8 heads because 8 gpu clusters are common and hence can compute in parallel . The embedding dimension are 512 and that leaves each head has 64 query size. Great video 👍
Awesome!
Thank you
Does the second part mean that each head only gets a portion of the embeddings?
@@jakob2946 curious aswell, I looked it up and it seems that its not true, every head is applied to all dimensions of the embedding. This also makes more sense to me since the word embeddings should be looked at as a whole. please correct me if I'm wrong
@@oliviervangoethem9365 I don't know about more recent transformers, but in the initial architecture each attention head is applied to a projection of input embeddings, with reduced dimensionality (in the original "Attention is all you need" paper: embeddings have a dimension of 512, and each of the 8 attention heads has a dimension of 64). The reason for this is spelled out in the original paper: "Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this."
This video is a treasure in a world where is explained in 2 minutes. Thank you Josh!
Thank you very much!
you are a blessing for anyone who is a visual learner. You have the gift to be able to explain complex topics in easy way.
Thank you!
BEST EVER VIDEO ABOUT CHAT GPT! I watched many videos but this video is just BAM!
Thank you!
its amazing how you make a 2 hours lecture in just 30 mins and explain it way better, after finishing this video and realizing what I just grasped, its mind blowing how you can make such complicated subject easy to understand. thank you very much!
Glad it helped!
This is just amazing ! I have no words, I came here to understand how transformers works and I am completely blown away
Thanks!
Thanks! This is the best explanation and illustration of a very complex topic ! Keep them coming..
Triple Bam!!! Thank you so much for supporting StatQuest!!!
For slowpokes like me: The example assumes total vacabulary size of 4 for each language. Thanks Josh for providing some of the best content on the subject! Finally the K, Q, V made clear sense
BAM! :)
Thank you from a fellow slowpoke
I try very much to understand transformer but I got 100% satisfaction from your video , your explanation is like unbelivable . I enjoy study very much when you taught any topic
Thanks and welcome!
I can't imagine how much effort it took for ai scientists to make this model. I really admire your illustration 😀
Thank you so much 😀!
Thanks, Josh, for keeping your promise to make a video about Transformers. I learned a lot and truly appreciate your effort in explaining this concept. I just placed an order to buy your book and made a donation to support the channel. I'm looking forward to more content on Machine Learning and hope to see videos about GPT and BERT models. ♥
Thank you so much!!! I really appreciate your support (TRIPLE BAM!!!). I hope to do the GPT video soon, but we'll see - the timeline is a little out of my control right now.
I try-harded deeply to understand transformers in few mouths ago, I can say that this video is a must have.
Thank you Josh
Glad it was helpful!
I finally decided to buy your book thinking there'd be transformers in the "Neural Network" section. But even if they're not, I'm glad it supports you. Your content is the best in popularisation that I've seen. It mainly helps me a lot to refresh and understand better than before to start my internship in AI after 1 year of gap year.
I'm starting a book on neural networks every soon.
I just finished watching almost all the videos on this channel and i have to say that this is probably the best place to learn stats and machine learning. I also bought the ML book and it captures the essence of the style of teaching on this channel really well and is very handy to go back and quickly look up some details. You are doing great work!
Wow, thanks!
Which book?
@@meirgoldenberg5638 I think he is referring to my book, The StatQuest Illustrated Guide to Machine Learning at statquest.org/statquest-store/
Thanks to your engaging visualization and clear explanation, I've grasped the Stanford CS224n course! Your content is neatly condensed but doesn't miss a thing. It's like you've taken all the complex concepts and served them up on a platter. Triple Bam!
Congratulations! TRIPLE BAM! :)
words cannot describe how happy I am to be able to watch this video. You really helped me with my studies. It is you who made me so interested in AI and think that I am actuaaly able to understand what is going on. Thank you for your simplified models. They really help when larning more complex stuff on this or that theme. But everytime there's a theme I do not know, the first thing I do is go to statquest. Thank you, Josh!
Hooray!!! Thank you very much!
And so we reach the state-of-art!! Congrats Josh :D
Hooray! :)
Triple bam.
2017...
The explanations in your videos are incredibly precise and efficient at the same time. There is nothing better to watch when learning any ML topic! 👍
Thank you very much! :)
I don't know how I'd learn DataScience/ML without this channel. Thanks so much for doing what you do!
Happy to help!
As a computer science student getting into the field of NLP, I really can't thank you enough for making a video that breaks down Transformer like this. Our uni doesn't go in depth about NLP related topics and with a very brief explanation they do, the uni expects us to have a full understanding about NLP. I can't thank you enough!
Thanks!
I'm following an advanced master of Artificial Intelligence. This whole NN playlist has saved me while studying for my exams! Thanks a bunch!
good luck!
Thank you so much for this video. More than 35:years in IT, worked with punching cards, I started to believe I am too old for this new thing. It was not my age, but the way how the knowledge is offered which does not fit my learning style. Finding your video, listening and immediate insights, hearing the coins dropping, were only separated by minutes. Ordering your book is the least minimum I could do to say thank you.
TRIPLE BAM!!! Thank you very much for your support! :)
I'm currently studying for job interviews and I don't know what I would do without you, thank you! When I get paid from my first job I will donate to you :)
Wow! Thank you!
That is a lot of stuff in a single video!! For those who are wondering, ChatGPT is a decoder only neural network, and the main diff between an encoder and a decoder is that a decoder uses masked attention - thus ChatGPT is essentially an autoregressive model. Notice how ChatGPT generates a response in sequential order, from left to right. Anyway, good stuff!
Yep - I'd like to make a GPT video just to highlight the explicit use of masking (the self attention in the decoder in this video used masking implicitly).
@@statquest Please do that video soon :) BAM
This is one of the most important videos in recorded history that every learner of AI must watch. Thank you sensei !
Thank you!
You're the only person on social media that can explain such complicated topics in an easy to understand manner. Keep up!
Thanks, will do!
After watching more than 50 of your videos, there's only one thing I can say: you're awesome. BAM!
Thank you very much! :)
Definitely the clearest walkthru of Transformer. It's very good that you put heavy emphasis on the parallelizability of Transformer since IMO it is the most important feature that made Transformer so useful
agreed!
you are the unique ML teacher guy in the world, and I don't think anyone can explain this thing like you. Thank you myself for knowing your channel!
Wow, thanks!
Man oh man the crazy timing .. I just watched your video on attention yesterday !! TRIPPLE BAAAAM
your rock josh thanks :D
BAM! :)
I had never struggled so much with understanding a concept before. But you cleared all the doubts. Thank you!
Glad it helped!
@@statquest I actually had a doubt as I was going through, about the decoder part. In the masked multi head attention part of the typical transformer, what inputs do we provide? And is this part only used during training?
@@pratyushrao7979 I actually talk about masking in my video on decoder-only transformers here: ruclips.net/video/bQ5BoolX9Ag/видео.html
I've spent a good deal of time studying attention, the critical concept behind transformers. Don't anticipate a natural understanding of the Q, K, and V parameters. We aren't entirely certain about their function; we can only hypothesize. They could still function effectively even if we used four parameters instead of three. One crucial point to remember is that our intuitive understanding of neural networks (NNs) is far from complete. The matrices for Q, K, and V aren't static; they're learned via backpropagation over lengthy training periods, thus changing over time. As a result, it's not as certain as mathematical operations like 1+1=2. The same applies to the head count in transformers; we can't definitively state whether eight is a good number or not. We don't fully grasp what each head is precisely doing; we can only speculate.
In visual transformers, we do understand what each head does. I guess heads trained on language are more difficult to interpret for us.
@@GreenCowsGames no, we don’t know that they do.
I've searched a lot about the transformers but seriously this is the best explanation I've ever got. Amazing!❤
Wow, thank you!
OMG, I waited for it for so long!!, thank you, Josh!
bam! :)
The most AWESOME 36 MINUTES - What an explanation of Transformers!
Thank you very much!!! BAM! :)
Best explanation I found in the whole Internet ! although I admit I needed 2 full turns. well done Josh !
Thanks! - Yes, this video packs in a ton of information, but I couldn't figure out any other way to make it work.
I like how he says "In this example we kept things super simple".
Great video, thank you!
Glad you liked it!
Prof. Starmer, Thank You very much. You are an inspiration to all the aspiring Machine Learning Enthusiasts. Respect and Gratitude from India. #RESPECT
Thank you very much!
So nicely explained! I have searched for "how transformers work" but no one on youtube explained with both concept and math! Keep the good work going 😁👍
Glad you liked it!
THIS is what I've been waiting for!
Thanks!
It's interesting how many different topics one has to contextualize to understand something like transformers! It's great to see all the math happening in detail and stop and ponder time-to-time while learning something as complex as transformers for the first time. It really helped build my intuition of a lot of building blocks for a transformer-based neural network. Thank you!
Glad you enjoyed it!
The positional encoding explanation truly was a BAM!
Hooray! :)
What a legend, there is no better channel than this!
Thank you!
Only videos like this can have "clearly explained" in the title.
bam!
Your videos have to be the best instance of teaching I have ever seen! Thank you for the amazing work!
Thank you!
Your neural networks playlist including this video gave me an intuitive understanding of transformers in less than a week which is something that would have taken an entire semester otherwise. I stumbled onto them while searching for a better understanding of Q,K,V, which everyone seems to say is as simple as querying a database…but what does that even mean?? Your explanations are brilliant, and I will be sharing with everyone I know who wants to learn more about this topic. I look forward to future videos. Thank you!
Thank you very much!!! I really appreciate it.
I'm just starting learning about Machine Learning, and this video has been very clear as an introduction to learn concepts and move on. Thanks for great content, contiuining with side quests right now!
Thank you very much!
HE HAS DONE IT YET AGAIN!
Thanks!
this is without a doubt the best explanation of transformers for a beginner out there. Thank you kind sir
Glad you liked it!
I owe my data science job to so many beautiful people on youtube, you are one of them.
Thank you
Wow, thank you!
Thanks Josh, another great video, I’ve been following your channel for years now and your videos have massively helped me to change career so huge thanks.
On to the transformer network, there’s something about the positional encoding that makes me feel a little uneasy. It feels we’ve gone through great effort to train a word embedding model that can cluster similar words together in n-dimensional word embedding space (where n can be very large, often 1,000).
By then applying positional encoding before our self-attention, whilst you very clearly explained with your example how important adding this information to the model is, seems to me to mess up all the effort we put into word embedding to get similar words clustered together. The word pizza, instead of being positioned in the same place can now jump around word/positional embedding space. Instead of one representation of pizza in space, it can now move around to be in many different positions, and not move locally around its own 'area' but because we add the positional encoding to the word embedding, scaled equally, it can jump around a great deal of space. To me it would seem adding this much freedom to where the word pizza can be represented in space would make it much much harder to train the model. Is my understanding correct or is there something I’m missing?
I have a couple of thoughts on this. Maybe I should make a short video called "some thoughts about positional encoding". Anyway, here they are...
Thought #1: Remember the positional encoding is fixed, so the word embedding values have to take them into account when training. For example, since all of the positional encoding value are between -1 and 1, it is possible that the word embedding values will have larger magnitudes and thus, not move around a lot when position is added to them.
Thought #2: Because the periods of the squiggles get larger for larger embedding positions, after about the 20th position, the position encoding values end up alternating 1 and 0 (in other words, after the 20th position, the position encoding values are 1010101....) and it is in that space, from the 20th position to the 512th position (usually word embeddings have 512 or more positions) that the word embeddings are really learned, and that the first 20 positions are mostly just for position encoding.
@@statquest Ah ok yeh that makes a lot of sense, thanks so much for taking the time to reply!
I’ve been having some additional thoughts on this and think I may have another reason (or rather an example) why adding positional encoding to the word embedding vectors makes sense, Josh if you read this, feel free to shoot it down! Take the following sentence: “The weather is bad, but my mood is good”. In this sentence the first “is” refers to the weather, whereas the second "is" refers to my mood. Without positional encoding and only word embedding, the vector for “is” being passed into the attention unit will be the same for the two instances of the word in the sentence. If we don’t use masked self-attention and compare the word “is” to every word including itself in the sentence, then the output of the word “is” in the self-attention unit I believe should be the same for both instances. Therefore, the unit will struggle to successfully differentiate the relative meaning of the two words. By adding in positional encoding prior to the self-attention unit, we’re suddenly adding context to the word. The second “is” comes straight after the word “mood”, therefore the position vector we’re adding to each of the two words should be similar. However, because the word “weather” comes 6 words before the second “is”, the positional vector we add will be quite different. Presumably this difference helps a self-attention unit to differentiate the relative meanings of the two instances of the word “is”.
@@matthewhaythornthwaite9910 That all sounds reasonable to me! BAM! :)
@@matthewhaythornthwaite9910 Another reason why you would want to add positional encoding instead of doing something else is that it preserves the dimensionality of the encoding. Imagine a theoretical encoding which is not added (like a one-hot encoding for each sequence location), and some linear (or non-linear for that matter) transform to combine word embedding and positinonal encoding. This is great in the sense that we do not polute the embedding space with "arbitrary" offsets, but now our input sequence has to be of a fixed shape. Addition of orthogonal sinusoids guarrantees a non-parametric, dimensionality preserving encoding which does not fix the number of inputs we can give to the network.
By the way, I think there is an analogy between adding positional encoding to embeddings and adding residual/skip connections to network outputs. Imagine that we have a network that is represented by the function f(x) and we have some target function F(x) which we want the network to learn. Imagine now that we modify our network to compute the function f(x) = h(x) + x (where "h(x)" is the network in front of the skip connection "h(x) + x"). Here too we polute the output space of h(x) with the values of x. However the network f can still learn F, so long as the network h(x) learns the function h(x) = F(x) - x (such that f(x) = h(x) + x = F(x) - x + x = F(x)).
I suppose for positional encoding something similar holds (altough it probably has to learn a much more difficult internal pattern), where the network is f(E(x)+q) learns to associate word embedding values E(x) which are "convolved" by some known offsets q and probably learns to deconvolve E(x) and q (into some abstract representation).Given that E(x) + q may in theory be (nearly) non-unique (i.e. E(x_1) + q_1 = approx E(x_2) + q_2) it might still be possible for the network to deconvolve the values into the correct inputs based on the context vector C which is calculable from the rest of the input sequence. I suppose one can't exclude that the network may never get this wrong, but in practical terms, it seems to work well enough.
One of the best explanation of encoder / decoder architecture. Esp. the self attention part. I really liked the way you colored Q,K, V to keep track of how things are moving . Looking forward to more such videos
Thanks! I've also got a video on Decoder-Only Transformers: ruclips.net/video/bQ5BoolX9Ag/видео.html and I'm working on one that shows the matrix algebra (color coded) of how these things are computed.
@@statquest are all these topics covered in your book ? Would love to read them in printed format
@@shaktisd They'll be in my next book.
@@statquest looking forward to the next edition.
谢谢！You saved my life man!
TRIPLE BAM!!! Thank you so much for supporting StatQuest!!! :)
The explanation of transformers that the Internet missed!
bam! :)
Danke!
TRIPLE BAM! Thank you so much for supporting StatQuest!
Hey @statquest - here is a quick suggestion. Another convenient way to explain positional encoding might be by drawing clock with minute and hour hand. Then - instead of sin() and cos() functions you could simply track the x and y coordinates of the tip of the minute and hour hand. It gives much more convenient intuition behind mechanics of the encoding.
(a) it shows its repetitive nature
(b) ties encoding position with sense of time (which is intuitive since speech is tied to time as well). Speech is the most common way we use language
(c) it explains why we use both sin() and cos() functions (to track circular motion of the clock hand)
(d) it provides intuition on why having two pair of sin() and cos() functions is better than just one
That's a great idea!
So kind of like a unit circle?
Transformers! More than meets the eye!? I think there is a lot of value in knowing this technology well! Thank you for your humor and learning support, I can't wait to return the favor!
Thanks!
I must say, after learning how GPT works I am even more blown away that this can work the way it works with the big online models. We just have to keep in mind that these companies are blowing tens of millions just for training. That's something else than me with a RTX 4090 and pytorch spending 12 cents of electricity and thinking I have unfathomable power 😂
:)
1 million subscribers and still taking the time to answer questions from his viewers. Absolute legend
BAM! :)
Bet it’s an AI bot answering 🤖
@@luisfernando5998 Nope - it's me. I really read all the comments and respond to as many as I can.
@@statquest do u have a team ? 🤔 how do u manage the time ? 🤯
@@luisfernando5998 It only takes about 30 minutes a day. It's not that big of a deal.
People like you are the positive side of the world
Thanks!
Great video! I'd love to see a pytorch + lightning tutorial on transformers similar to the LSTM video!
That's the plan!
I am coming to Josh after about 3 years, I love Josh Starmer style of teaching and it has not changed a bit. The good news is Josh is now teaching advanced Deep learnng concepts.
bam! :)
(Verse 1)
Here we are with another quest,
A journey through the world of stats, no less,
Data sets in rows and columns rest,
StatQuest, yeah, it's simply the best.
(Chorus)
We're diving deep, we're reaching wide,
In the land of statistics, we confide,
StatQuest, on a learning ride,
With your wisdom, we abide.
(Verse 2)
From t-tests to regression trees,
You make understanding these a breeze.
Explaining variance and degrees,
StatQuest, you got the keys.
(Chorus)
We're scaling heights, we're breaking ground,
In your lessons, profound wisdom's found,
StatQuest, with your sound,
We'll solve the mysteries that surround.
(Bridge)
With bar charts, line plots, and bell curves,
Through distributions, we observe,
With every lesson, we absorb and serve,
StatQuest, it's knowledge we preserve.
(Chorus)
We're traversing realms, we're touching sky,
In the field of data, your guidance, we rely,
StatQuest, with your learning tie,
You're the statistical ally.
(Outro)
So here's to Josh Starmer, our guide,
To the realm of stats, you provide,
With StatQuest, on a high tide,
In the world of statistics, we stride.
(End)
So get ready, set, quest on,
In the realm of stats, dawn upon,
StatQuest, till the fear's gone,
Keep learning, till the break of dawn.
THAT IS AWESOME!!! (what are the chords?)
@@statquest I V vi IV
It is so complex, ill have to watch this 2 3 times. But you did much better than anyone else. Thanks.
It can be made much simpler if you learn about neural networks first. You can do that with this play list: ruclips.net/video/CqOfi41LfDw/видео.html
We wanna more NLP material please, tiny bam !
:)
Thanks! You are the best
Thank you very much for supporting StatQuest! TRIPLE BAM!!! :)
You really explaining these concepts in a clear way! Will you do more explanation video on statistic like Cox model for survival ? Thanks! :)
I'll keep that in mind.
Penta BAM!!! All of your videos are extremely easy to understand in a peculiar way, they have helped me a lot, thank you very much.
Glad you like them!
(just letting you know while cramming for my exam, ive used your videos for over 4 years now and some professors, including the ones from this class, use your videos for us to learn (either showing them to us or using screenshots from the videos with reference). Its so funny watching the reaction of people who arent familiar with your channel reacting to a soft cute bear as the soft max function and such haha Most people in portugal can speak english but still, many prefer to search in portuguese, so they never encountered your channel before. But I make sure to let all my friends know your content that helps us with some specific exam. I would never be able to thank you enough for the more than wonderful content and for your patience explaining questions we have in the comments)
Thank you very much! I'm glad to hear that my videos are helpful. Maybe one day I can go to Portugal and teach a lecture in person. That would be super fun. :)
man i love you for this video thank you so much, there's absolutely no way i'd have understood all of this without your help
Glad I could help!
The most useful 35 minutes of my life !
bam! :)
@statquest you are a true professional and a master of your craft. The field of ML is getting a little stronger each day bc of content like this!
Thank you very much!
Best video on Transformer I have seen in RUclips! Amazing ! huge respect for you
Thank you so much 😀!
Very clear and builds up the concepts in a step by step manner, rather than starting with the overall architrcture.
Thanks!
This is the best of all that is available right now on Transformers. Thank you!!
Thank you!
Just wanted to say that I understood everything about LMs (thanks to your videos), except the part on transformers cuz the video wasn't out yet ahah. Well now that my dear squash teacher explained it, everything's clear. So really THANK YOU for your hard work and dedication, it made all the difference in my understanding of Neural Networks in general
Great to hear!
These videos on machine learning (and statistics!) are incredible and empowering! Plus they make me laugh! You are the reason that I was able to keep my head above water in my Machine Learning courses as a pregnant grad student with limited time, money and energy. Thank you and I will definitely be checking out some merch!
Thank you so much! I'm happy to hear my videos are useful. BAM! :)
Awesome and crystal clear explanation. Keep rocking.
Thank you!
As a Gen AI consultant, this video was amazing. Absolutely perfect
Thank you!
Oh my god! Now it makes sense why training a transformer model takes FAR more compute than it does to run them! I can run them on my CPU, but training a big one may take weeks to months on entire GPU clusters! So many values to optimize and SO many nodes, inputs and outputs! That's insane! And it also shows why the models are SO large! There are SO many weights and biases, SO many connections and an entire world of vocabulary to store in it and connect up!
This is SO impressive!
BAM! :)
Amazing as always. The internet is a better place because of humans like you. Thank you, Josh!
Wow, thank you!
Man, you are best teacher i have ever listen to. Thanks
It is first time i giving thumb up in every video in the playlist
Thank you very much! :)
Holy crap I’ve been waiting for this for months!!! Finally!
Hooray! :)
We have been waiting for this!! Awesome
Thank you very much!
This is your only video that I had to see twice to get complete idea of the topic. Transformers really is a decently tough topic.
This is a lot of material for one video. But people wanted a single video, rather than a series of videos making incremental steps in learning, for transformers. Personally, I would have preferred a sequence of shorter videos, each focused on just one part. That said, there is something about seeing it all at once and getting that big picture. My book on neural networks (that I'm working on right now) will try to do both - take things one step at a time and give a big picture.
@@statquest Nice. Waiting for the book in that case.
This might be the best video on deep learning I've seen period, rivalled only by 3Blue1Brown's content. Thank you so much!
Wow, thanks!
Wow Josh. You’re a genius, this was comprehensive and complete. Thank you again.
Glad it was helpful!
I feel so lucky that I just started learning Transformer and found out StatQuest made a video for it one day ago!
bam!
The best video that explains Transformer I've ever seen. Thanks for your contribution!
Thank you very much! :)
Thank you sir for this amazing video, it helped me last year in my NLP exam and now i'm refreshing my information's about transformers hoping to land an interview soon!
Good luck - let me know how it all goes.
U explained it really clear. It was the best transformer video i've watched.
Wow, thanks!
much awaited and anticipated video!!, Tysm
Thanks! :)