To learn more about Lightning: lightning.ai/ Support StatQuest by buying my books The StatQuest Illustrated Guide to Machine Learning, The StatQuest Illustrated Guide to Neural Networks and AI, or a Study Guide or Merch!!! statquest.org/statquest-store/ NOTE: A lot of people ask why the attention values are the same or are not what they might expect them to be based on the theory. The reason for this is that this model, and the dataset it was trained on, are both too simple to show any nuance in these numbers. The purpose of picking the simplest transformer model and the simplest dataset was so I could illustrate how each step was performed.
Personally I find it more clear to link embedding to hidden class of words, i use character sheets as a netaphor, because what attention does is not looking at word but the description in its sheet, with each attention head focusing on different part of the description, which mean a word representation have multiple attention on different hidden class. Then at the end we look at the sheets transformed at each layer to find the next word. That also allows to explain multimodality, ie make sure image input and text input share the same description sheet.
Transformers needs more than one video, each part(multi H attention, word embeding(sin&cosine similarity),training &…) I was waiting for long to reach stat of art.
@@Tothefutureand I thought about doing it that way - and that was the original plan. But my video on Attention convinced me that most people would rather have a single video that has everything in it all at once. However, I've provided links in this video's description to full length videos on each topic you are interested in.
@@statquest oh you mentioned that you don't know why this number of head, that's hardware optimization, ie they can be split into gpu or memory pool or reduce bandwidth, such that they can parralelized or compute sequentially on resource starved machine.
As a Computer Science professor who teaches Machine Learning, this is probably my most anticipated video ever. I regularly use your videos to brush up on/review ML concepts myself and recommend them to my students as study aids. You explain these concepts in the clear, straightforward way that I aspire to. Thank you!
@@statquest our data analysis professor also uses your videos as references and recommends you almost every session haha. i learned about this amazing channel from him.
Its kinda hilarious that StatQuest videos give the impression they were menat for 5 year olds, yet are exploring legitimately complex topics. No jargon, no overcomplicated diagrams. Josh really tries to explain things and not show off his supirior understanding of neural networks. Thanks Josh!
This is the complet-est, precious-est, pur-est, brilliant-est video ever. Can't imagine how much work you've put into creating these illustrations. It's just brilliant. Hats off.
I've just learned machine learning for a month, my major is a pure business student. I've been working as a Data Analyst for 2 months as the internship and I believe machine learning will be essential if I want to go further in this industry. Out of all tutorials videos I've watched, your videos brought up the clearest and most concise concepts for me to understand. All the videos walked me through from the series of neural network, back propagation, cross entropy with backward propagation, recurrent, LSTM and convolutional neural network, lastly, this video. Really appreciate for your understandings and amazing storytelling through your videos, your contents always make me eager to keep learning machine learning myself. Thanks a lot
How a model like this was created is just beyond me. There’s so many different moving parts. You could write a whole book on the fully-connected network alone. Add in all the other stuff? Wow. Thank you, Josh, for explaining this so well!
Thanks! It's a little easier to understand how this model was created in the first place if you follow the whole Neural Networks playlist. You'll see how things changed, one step at a time, to eventually end up with a transformer: ruclips.net/video/CqOfi41LfDw/видео.html
Not to be a buzz kill, but I suffered a bad Traumatic brain injury in my late 40's after being hit by an SUV while stopped on a motorcycle. I'm blessed i survived . At the time my job dealt with engineering and architecting IT solutions and I was looking forward to advancing my career into AI and Machine Learning. I was in a coma for a while and I lost lots of what i used to know. I know have Learning disabilities and memory issues. I have improved some over the last years, but If I'm being honest with myself, I wouldn't want me as an engineer, so I'm trying to move into management. I'm glad I ran across these videos . I purchased the .pdf books and notebooks today and I can honestly say they are well worth it. Josh, I'm so glad You created this material. Your books and notebooks etc.. are helping me slowly understand complex topics in hopes that I can stay relevant and continue to advance my career. Thanks again!
I've been looking for tutorial about transformers for a long time. This is the smoothest tutorial. It does not hide any complexities(making me confident that I actually understand the concept instead of its dumbed down version for mortals that won't end up ever using the knowledge), but also does not get lost while explaining those complexities and clearly calls out what else I can learn about to understand the side concepts better. Super !!!
I just finished watching almost all the videos on this channel and i have to say that this is probably the best place to learn stats and machine learning. I also bought the ML book and it captures the essence of the style of teaching on this channel really well and is very handy to go back and quickly look up some details. You are doing great work!
Thanks, Josh, for keeping your promise to make a video about Transformers. I learned a lot and truly appreciate your effort in explaining this concept. I just placed an order to buy your book and made a donation to support the channel. I'm looking forward to more content on Machine Learning and hope to see videos about GPT and BERT models. ♥
Thank you so much!!! I really appreciate your support (TRIPLE BAM!!!). I hope to do the GPT video soon, but we'll see - the timeline is a little out of my control right now.
22:12 8 heads because 8 gpu clusters are common and hence can compute in parallel . The embedding dimension are 512 and that leaves each head has 64 query size. Great video 👍
@@jakob2946 curious aswell, I looked it up and it seems that its not true, every head is applied to all dimensions of the embedding. This also makes more sense to me since the word embeddings should be looked at as a whole. please correct me if I'm wrong
@@oliviervangoethem9365 I don't know about more recent transformers, but in the initial architecture each attention head is applied to a projection of input embeddings, with reduced dimensionality (in the original "Attention is all you need" paper: embeddings have a dimension of 512, and each of the 8 attention heads has a dimension of 64). The reason for this is spelled out in the original paper: "Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this."
its amazing how you make a 2 hours lecture in just 30 mins and explain it way better, after finishing this video and realizing what I just grasped, its mind blowing how you can make such complicated subject easy to understand. thank you very much!
Thanks to your engaging visualization and clear explanation, I've grasped the Stanford CS224n course! Your content is neatly condensed but doesn't miss a thing. It's like you've taken all the complex concepts and served them up on a platter. Triple Bam!
Great marketing technique used by Josh, generally he don’t repeat previous videos content properly just asks us to go through the videos if we have not watched them. But since he knew that transformers video will gain a lot of views so, he explained previous videos content in some detail and then asked the users to go through the exact video for more in-depth explanation. Nice work, you are a good teacher as well as a good marketer.
words cannot describe how happy I am to be able to watch this video. You really helped me with my studies. It is you who made me so interested in AI and think that I am actuaaly able to understand what is going on. Thank you for your simplified models. They really help when larning more complex stuff on this or that theme. But everytime there's a theme I do not know, the first thing I do is go to statquest. Thank you, Josh!
For slowpokes like me: The example assumes total vacabulary size of 4 for each language. Thanks Josh for providing some of the best content on the subject! Finally the K, Q, V made clear sense
One of the best explanation of encoder / decoder architecture. Esp. the self attention part. I really liked the way you colored Q,K, V to keep track of how things are moving . Looking forward to more such videos
Thanks! I've also got a video on Decoder-Only Transformers: ruclips.net/video/bQ5BoolX9Ag/видео.html and I'm working on one that shows the matrix algebra (color coded) of how these things are computed.
I try very much to understand transformer but I got 100% satisfaction from your video , your explanation is like unbelivable . I enjoy study very much when you taught any topic
Hello, I just wanna say how fantastic this video has been in explaining how to clearly and easly build a decoder only transformer from scratch. It also helped tremendously in understanding the "Attention is all you need" paper. Building attention from scratch was also an excellent thing you showed instead of relying on the inbuilt one in pytorch. Once again thank you so much.
Definitely the clearest walkthru of Transformer. It's very good that you put heavy emphasis on the parallelizability of Transformer since IMO it is the most important feature that made Transformer so useful
I finally decided to buy your book thinking there'd be transformers in the "Neural Network" section. But even if they're not, I'm glad it supports you. Your content is the best in popularisation that I've seen. It mainly helps me a lot to refresh and understand better than before to start my internship in AI after 1 year of gap year.
It's interesting how many different topics one has to contextualize to understand something like transformers! It's great to see all the math happening in detail and stop and ponder time-to-time while learning something as complex as transformers for the first time. It really helped build my intuition of a lot of building blocks for a transformer-based neural network. Thank you!
Your neural networks playlist including this video gave me an intuitive understanding of transformers in less than a week which is something that would have taken an entire semester otherwise. I stumbled onto them while searching for a better understanding of Q,K,V, which everyone seems to say is as simple as querying a database…but what does that even mean?? Your explanations are brilliant, and I will be sharing with everyone I know who wants to learn more about this topic. I look forward to future videos. Thank you!
@@statquest I actually had a doubt as I was going through, about the decoder part. In the masked multi head attention part of the typical transformer, what inputs do we provide? And is this part only used during training?
That is a lot of stuff in a single video!! For those who are wondering, ChatGPT is a decoder only neural network, and the main diff between an encoder and a decoder is that a decoder uses masked attention - thus ChatGPT is essentially an autoregressive model. Notice how ChatGPT generates a response in sequential order, from left to right. Anyway, good stuff!
Yep - I'd like to make a GPT video just to highlight the explicit use of masking (the self attention in the decoder in this video used masking implicitly).
I'm currently studying for job interviews and I don't know what I would do without you, thank you! When I get paid from my first job I will donate to you :)
Thanks Josh, another great video, I’ve been following your channel for years now and your videos have massively helped me to change career so huge thanks. On to the transformer network, there’s something about the positional encoding that makes me feel a little uneasy. It feels we’ve gone through great effort to train a word embedding model that can cluster similar words together in n-dimensional word embedding space (where n can be very large, often 1,000). By then applying positional encoding before our self-attention, whilst you very clearly explained with your example how important adding this information to the model is, seems to me to mess up all the effort we put into word embedding to get similar words clustered together. The word pizza, instead of being positioned in the same place can now jump around word/positional embedding space. Instead of one representation of pizza in space, it can now move around to be in many different positions, and not move locally around its own 'area' but because we add the positional encoding to the word embedding, scaled equally, it can jump around a great deal of space. To me it would seem adding this much freedom to where the word pizza can be represented in space would make it much much harder to train the model. Is my understanding correct or is there something I’m missing?
I have a couple of thoughts on this. Maybe I should make a short video called "some thoughts about positional encoding". Anyway, here they are... Thought #1: Remember the positional encoding is fixed, so the word embedding values have to take them into account when training. For example, since all of the positional encoding value are between -1 and 1, it is possible that the word embedding values will have larger magnitudes and thus, not move around a lot when position is added to them. Thought #2: Because the periods of the squiggles get larger for larger embedding positions, after about the 20th position, the position encoding values end up alternating 1 and 0 (in other words, after the 20th position, the position encoding values are 1010101....) and it is in that space, from the 20th position to the 512th position (usually word embeddings have 512 or more positions) that the word embeddings are really learned, and that the first 20 positions are mostly just for position encoding.
I’ve been having some additional thoughts on this and think I may have another reason (or rather an example) why adding positional encoding to the word embedding vectors makes sense, Josh if you read this, feel free to shoot it down! Take the following sentence: “The weather is bad, but my mood is good”. In this sentence the first “is” refers to the weather, whereas the second "is" refers to my mood. Without positional encoding and only word embedding, the vector for “is” being passed into the attention unit will be the same for the two instances of the word in the sentence. If we don’t use masked self-attention and compare the word “is” to every word including itself in the sentence, then the output of the word “is” in the self-attention unit I believe should be the same for both instances. Therefore, the unit will struggle to successfully differentiate the relative meaning of the two words. By adding in positional encoding prior to the self-attention unit, we’re suddenly adding context to the word. The second “is” comes straight after the word “mood”, therefore the position vector we’re adding to each of the two words should be similar. However, because the word “weather” comes 6 words before the second “is”, the positional vector we add will be quite different. Presumably this difference helps a self-attention unit to differentiate the relative meanings of the two instances of the word “is”.
@@matthewhaythornthwaite9910 Another reason why you would want to add positional encoding instead of doing something else is that it preserves the dimensionality of the encoding. Imagine a theoretical encoding which is not added (like a one-hot encoding for each sequence location), and some linear (or non-linear for that matter) transform to combine word embedding and positinonal encoding. This is great in the sense that we do not polute the embedding space with "arbitrary" offsets, but now our input sequence has to be of a fixed shape. Addition of orthogonal sinusoids guarrantees a non-parametric, dimensionality preserving encoding which does not fix the number of inputs we can give to the network. By the way, I think there is an analogy between adding positional encoding to embeddings and adding residual/skip connections to network outputs. Imagine that we have a network that is represented by the function f(x) and we have some target function F(x) which we want the network to learn. Imagine now that we modify our network to compute the function f(x) = h(x) + x (where "h(x)" is the network in front of the skip connection "h(x) + x"). Here too we polute the output space of h(x) with the values of x. However the network f can still learn F, so long as the network h(x) learns the function h(x) = F(x) - x (such that f(x) = h(x) + x = F(x) - x + x = F(x)). I suppose for positional encoding something similar holds (altough it probably has to learn a much more difficult internal pattern), where the network is f(E(x)+q) learns to associate word embedding values E(x) which are "convolved" by some known offsets q and probably learns to deconvolve E(x) and q (into some abstract representation).Given that E(x) + q may in theory be (nearly) non-unique (i.e. E(x_1) + q_1 = approx E(x_2) + q_2) it might still be possible for the network to deconvolve the values into the correct inputs based on the context vector C which is calculable from the rest of the input sequence. I suppose one can't exclude that the network may never get this wrong, but in practical terms, it seems to work well enough.
(just letting you know while cramming for my exam, ive used your videos for over 4 years now and some professors, including the ones from this class, use your videos for us to learn (either showing them to us or using screenshots from the videos with reference). Its so funny watching the reaction of people who arent familiar with your channel reacting to a soft cute bear as the soft max function and such haha Most people in portugal can speak english but still, many prefer to search in portuguese, so they never encountered your channel before. But I make sure to let all my friends know your content that helps us with some specific exam. I would never be able to thank you enough for the more than wonderful content and for your patience explaining questions we have in the comments)
Thank you very much! I'm glad to hear that my videos are helpful. Maybe one day I can go to Portugal and teach a lecture in person. That would be super fun. :)
As a computer science student getting into the field of NLP, I really can't thank you enough for making a video that breaks down Transformer like this. Our uni doesn't go in depth about NLP related topics and with a very brief explanation they do, the uni expects us to have a full understanding about NLP. I can't thank you enough!
Prof. Starmer, Thank You very much. You are an inspiration to all the aspiring Machine Learning Enthusiasts. Respect and Gratitude from India. #RESPECT
So nicely explained! I have searched for "how transformers work" but no one on youtube explained with both concept and math! Keep the good work going 😁👍
I've spent a good deal of time studying attention, the critical concept behind transformers. Don't anticipate a natural understanding of the Q, K, and V parameters. We aren't entirely certain about their function; we can only hypothesize. They could still function effectively even if we used four parameters instead of three. One crucial point to remember is that our intuitive understanding of neural networks (NNs) is far from complete. The matrices for Q, K, and V aren't static; they're learned via backpropagation over lengthy training periods, thus changing over time. As a result, it's not as certain as mathematical operations like 1+1=2. The same applies to the head count in transformers; we can't definitively state whether eight is a good number or not. We don't fully grasp what each head is precisely doing; we can only speculate.
Thank you so much for this video. More than 35:years in IT, worked with punching cards, I started to believe I am too old for this new thing. It was not my age, but the way how the knowledge is offered which does not fit my learning style. Finding your video, listening and immediate insights, hearing the coins dropping, were only separated by minutes. Ordering your book is the least minimum I could do to say thank you.
I'm just starting learning about Machine Learning, and this video has been very clear as an introduction to learn concepts and move on. Thanks for great content, contiuining with side quests right now!
Hey @statquest - here is a quick suggestion. Another convenient way to explain positional encoding might be by drawing clock with minute and hour hand. Then - instead of sin() and cos() functions you could simply track the x and y coordinates of the tip of the minute and hour hand. It gives much more convenient intuition behind mechanics of the encoding. (a) it shows its repetitive nature (b) ties encoding position with sense of time (which is intuitive since speech is tied to time as well). Speech is the most common way we use language (c) it explains why we use both sin() and cos() functions (to track circular motion of the clock hand) (d) it provides intuition on why having two pair of sin() and cos() functions is better than just one
I must say, after learning how GPT works I am even more blown away that this can work the way it works with the big online models. We just have to keep in mind that these companies are blowing tens of millions just for training. That's something else than me with a RTX 4090 and pytorch spending 12 cents of electricity and thinking I have unfathomable power 😂
(Verse 1) Here we are with another quest, A journey through the world of stats, no less, Data sets in rows and columns rest, StatQuest, yeah, it's simply the best. (Chorus) We're diving deep, we're reaching wide, In the land of statistics, we confide, StatQuest, on a learning ride, With your wisdom, we abide. (Verse 2) From t-tests to regression trees, You make understanding these a breeze. Explaining variance and degrees, StatQuest, you got the keys. (Chorus) We're scaling heights, we're breaking ground, In your lessons, profound wisdom's found, StatQuest, with your sound, We'll solve the mysteries that surround. (Bridge) With bar charts, line plots, and bell curves, Through distributions, we observe, With every lesson, we absorb and serve, StatQuest, it's knowledge we preserve. (Chorus) We're traversing realms, we're touching sky, In the field of data, your guidance, we rely, StatQuest, with your learning tie, You're the statistical ally. (Outro) So here's to Josh Starmer, our guide, To the realm of stats, you provide, With StatQuest, on a high tide, In the world of statistics, we stride. (End) So get ready, set, quest on, In the realm of stats, dawn upon, StatQuest, till the fear's gone, Keep learning, till the break of dawn.
Transformers! More than meets the eye!? I think there is a lot of value in knowing this technology well! Thank you for your humor and learning support, I can't wait to return the favor!
Just wanted to say that I understood everything about LMs (thanks to your videos), except the part on transformers cuz the video wasn't out yet ahah. Well now that my dear squash teacher explained it, everything's clear. So really THANK YOU for your hard work and dedication, it made all the difference in my understanding of Neural Networks in general
Very thorough, thanks Utilizes a lot of the previous concepts of basics of Deep Learning, amazing resources you've compiled over the years. I've read abt transformers from multiple resources and each time I understanding something new ! Prepping for interviews rn, will thank you once again after I land a job 🌞💖
This is a lot of material for one video. But people wanted a single video, rather than a series of videos making incremental steps in learning, for transformers. Personally, I would have preferred a sequence of shorter videos, each focused on just one part. That said, there is something about seeing it all at once and getting that big picture. My book on neural networks (that I'm working on right now) will try to do both - take things one step at a time and give a big picture.
Thank you sir for this amazing video, it helped me last year in my NLP exam and now i'm refreshing my information's about transformers hoping to land an interview soon!
These videos on machine learning (and statistics!) are incredible and empowering! Plus they make me laugh! You are the reason that I was able to keep my head above water in my Machine Learning courses as a pregnant grad student with limited time, money and energy. Thank you and I will definitely be checking out some merch!
Please create more vids!! Your explanations are truly beautiful, such a work of art. I couldn't agree more that you are one of the most brilliant teachers at statistic and ML! Actually, I wanna hug you right now haha
Thank you for such amazing illustrations! HOW I WISH I HAD THIS DURING MY STUDIES, BUT I'M JUST GLAD I GET TO LEARN THESE AS A WORKING PROFESSIONAL. THANK YOU SO MUCH FOR ALL THE CONTENT YOU MAKE. I'M SURE YOU MAKE THOUSANDS OF LIVES BETTER. YOU'RE TRULY AN INSPIRATION JOSH!
![The moment we stopped understanding AI [AlexNet]](http://i.ytimg.com/vi/UZDiGooFs54/mqdefault.jpg)
To learn more about Lightning: lightning.ai/
Support StatQuest by buying my books The StatQuest Illustrated Guide to Machine Learning, The StatQuest Illustrated Guide to Neural Networks and AI, or a Study Guide or Merch!!! statquest.org/statquest-store/
NOTE: A lot of people ask why the attention values are the same or are not what they might expect them to be based on the theory. The reason for this is that this model, and the dataset it was trained on, are both too simple to show any nuance in these numbers. The purpose of picking the simplest transformer model and the simplest dataset was so I could illustrate how each step was performed.
Personally I find it more clear to link embedding to hidden class of words, i use character sheets as a netaphor, because what attention does is not looking at word but the description in its sheet, with each attention head focusing on different part of the description, which mean a word representation have multiple attention on different hidden class. Then at the end we look at the sheets transformed at each layer to find the next word. That also allows to explain multimodality, ie make sure image input and text input share the same description sheet.
@@NeoShameMan Interesting.
Transformers needs more than one video, each part(multi H attention, word embeding(sin&cosine similarity),training &…)
I was waiting for long to reach stat of art.
@@Tothefutureand I thought about doing it that way - and that was the original plan. But my video on Attention convinced me that most people would rather have a single video that has everything in it all at once. However, I've provided links in this video's description to full length videos on each topic you are interested in.
@@statquest oh you mentioned that you don't know why this number of head, that's hardware optimization, ie they can be split into gpu or memory pool or reduce bandwidth, such that they can parralelized or compute sequentially on resource starved machine.
As a Computer Science professor who teaches Machine Learning, this is probably my most anticipated video ever. I regularly use your videos to brush up on/review ML concepts myself and recommend them to my students as study aids. You explain these concepts in the clear, straightforward way that I aspire to. Thank you!
Thank you! BAM! :)
Me too!
@@statquest our data analysis professor also uses your videos as references and recommends you almost every session haha. i learned about this amazing channel from him.
@@Daigandar That's awesome! BAM! :)
Well said. I do the same!
Its kinda hilarious that StatQuest videos give the impression they were menat for 5 year olds, yet are exploring legitimately complex topics. No jargon, no overcomplicated diagrams. Josh really tries to explain things and not show off his supirior understanding of neural networks. Thanks Josh!
Thank you! :)
Much like Richard Feynman.
BAM!
This is the complet-est, precious-est, pur-est, brilliant-est video ever. Can't imagine how much work you've put into creating these illustrations. It's just brilliant. Hats off.
Wow, thank you!
Did you just tokenize your comment?
@@lumiey I'm not sure I understand.
@@statquest He just separated words like complet, est, precious, est, pur, est... like tokenizer does (e.g. following -> follow, ing)
@@lumieyHaha
I've just learned machine learning for a month, my major is a pure business student. I've been working as a Data Analyst for 2 months as the internship and I believe machine learning will be essential if I want to go further in this industry. Out of all tutorials videos I've watched, your videos brought up the clearest and most concise concepts for me to understand. All the videos walked me through from the series of neural network, back propagation, cross entropy with backward propagation, recurrent, LSTM and convolutional neural network, lastly, this video. Really appreciate for your understandings and amazing storytelling through your videos, your contents always make me eager to keep learning machine learning myself. Thanks a lot
Thank you very much! I'm glad my videos are helpful.
@@statquest please upload videos on auto encoders and gated recurrent units also please
How a model like this was created is just beyond me. There’s so many different moving parts. You could write a whole book on the fully-connected network alone. Add in all the other stuff? Wow.
Thank you, Josh, for explaining this so well!
Thanks! It's a little easier to understand how this model was created in the first place if you follow the whole Neural Networks playlist. You'll see how things changed, one step at a time, to eventually end up with a transformer: ruclips.net/video/CqOfi41LfDw/видео.html
@@statquest Thanks Josh! I’ll check out that playlist. What you’re doing is so special to the world, and humanity is so indebted.
@@statquest beautiful, thank you for the work
Thanks! This is the best explanation and illustration of a very complex topic ! Keep them coming..
Triple Bam!!! Thank you so much for supporting StatQuest!!!
Not to be a buzz kill, but I suffered a bad Traumatic brain injury in my late 40's after being hit by an SUV while stopped on a motorcycle. I'm blessed i survived . At the time my job dealt with engineering and architecting IT solutions and I was looking forward to advancing my career into AI and Machine Learning. I was in a coma for a while and I lost lots of what i used to know. I know have Learning disabilities and memory issues. I have improved some over the last years, but If I'm being honest with myself, I wouldn't want me as an engineer, so I'm trying to move into management. I'm glad I ran across these videos . I purchased the .pdf books and notebooks today and I can honestly say they are well worth it. Josh, I'm so glad You created this material. Your books and notebooks etc.. are helping me slowly understand complex topics in hopes that I can stay relevant and continue to advance my career. Thanks again!
TRIPLE BAM!!! Thank you so much for supporting StatQuest and I wish you the best as you continue to learn about ML and Data Science! :)
I've been looking for tutorial about transformers for a long time. This is the smoothest tutorial. It does not hide any complexities(making me confident that I actually understand the concept instead of its dumbed down version for mortals that won't end up ever using the knowledge), but also does not get lost while explaining those complexities and clearly calls out what else I can learn about to understand the side concepts better. Super !!!
Thank you very much! :)
you are a blessing for anyone who is a visual learner. You have the gift to be able to explain complex topics in easy way.
Thank you!
I just finished watching almost all the videos on this channel and i have to say that this is probably the best place to learn stats and machine learning. I also bought the ML book and it captures the essence of the style of teaching on this channel really well and is very handy to go back and quickly look up some details. You are doing great work!
Wow, thanks!
Which book?
@@meirgoldenberg5638 I think he is referring to my book, The StatQuest Illustrated Guide to Machine Learning at statquest.org/statquest-store/
This video is a treasure in a world where is explained in 2 minutes. Thank you Josh!
Thank you very much!
Thanks, Josh, for keeping your promise to make a video about Transformers. I learned a lot and truly appreciate your effort in explaining this concept. I just placed an order to buy your book and made a donation to support the channel. I'm looking forward to more content on Machine Learning and hope to see videos about GPT and BERT models. ♥
Thank you so much!!! I really appreciate your support (TRIPLE BAM!!!). I hope to do the GPT video soon, but we'll see - the timeline is a little out of my control right now.
22:12 8 heads because 8 gpu clusters are common and hence can compute in parallel . The embedding dimension are 512 and that leaves each head has 64 query size. Great video 👍
Awesome!
Thank you
Does the second part mean that each head only gets a portion of the embeddings?
@@jakob2946 curious aswell, I looked it up and it seems that its not true, every head is applied to all dimensions of the embedding. This also makes more sense to me since the word embeddings should be looked at as a whole. please correct me if I'm wrong
@@oliviervangoethem9365 I don't know about more recent transformers, but in the initial architecture each attention head is applied to a projection of input embeddings, with reduced dimensionality (in the original "Attention is all you need" paper: embeddings have a dimension of 512, and each of the 8 attention heads has a dimension of 64). The reason for this is spelled out in the original paper: "Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this."
its amazing how you make a 2 hours lecture in just 30 mins and explain it way better, after finishing this video and realizing what I just grasped, its mind blowing how you can make such complicated subject easy to understand. thank you very much!
Glad it helped!
Thanks to your engaging visualization and clear explanation, I've grasped the Stanford CS224n course! Your content is neatly condensed but doesn't miss a thing. It's like you've taken all the complex concepts and served them up on a platter. Triple Bam!
Congratulations! TRIPLE BAM! :)
Great marketing technique used by Josh, generally he don’t repeat previous videos content properly just asks us to go through the videos if we have not watched them. But since he knew that transformers video will gain a lot of views so, he explained previous videos content in some detail and then asked the users to go through the exact video for more in-depth explanation. Nice work, you are a good teacher as well as a good marketer.
People were complaining about having to watch other videos in order to understand the current topic, so I had to change my style.
@ No matter your style the content and your teachings are great.
BEST EVER VIDEO ABOUT CHAT GPT! I watched many videos but this video is just BAM!
Thank you!
And so we reach the state-of-art!! Congrats Josh :D
Hooray! :)
Triple bam.
2017...
I try-harded deeply to understand transformers in few mouths ago, I can say that this video is a must have.
Thank you Josh
Glad it was helpful!
The explanations in your videos are incredibly precise and efficient at the same time. There is nothing better to watch when learning any ML topic! 👍
Thank you very much! :)
I can't imagine how much effort it took for ai scientists to make this model. I really admire your illustration 😀
Thank you so much 😀!
This is just amazing ! I have no words, I came here to understand how transformers works and I am completely blown away
Thanks!
words cannot describe how happy I am to be able to watch this video. You really helped me with my studies. It is you who made me so interested in AI and think that I am actuaaly able to understand what is going on. Thank you for your simplified models. They really help when larning more complex stuff on this or that theme. But everytime there's a theme I do not know, the first thing I do is go to statquest. Thank you, Josh!
Hooray!!! Thank you very much!
For slowpokes like me: The example assumes total vacabulary size of 4 for each language. Thanks Josh for providing some of the best content on the subject! Finally the K, Q, V made clear sense
BAM! :)
Thank you from a fellow slowpoke
One of the best explanation of encoder / decoder architecture. Esp. the self attention part. I really liked the way you colored Q,K, V to keep track of how things are moving . Looking forward to more such videos
Thanks! I've also got a video on Decoder-Only Transformers: ruclips.net/video/bQ5BoolX9Ag/видео.html and I'm working on one that shows the matrix algebra (color coded) of how these things are computed.
@@statquest are all these topics covered in your book ? Would love to read them in printed format
@@shaktisd They'll be in my next book.
@@statquest looking forward to the next edition.
Man oh man the crazy timing .. I just watched your video on attention yesterday !! TRIPPLE BAAAAM
your rock josh thanks :D
BAM! :)
I try very much to understand transformer but I got 100% satisfaction from your video , your explanation is like unbelivable . I enjoy study very much when you taught any topic
Thanks and welcome!
The positional encoding explanation truly was a BAM!
Hooray! :)
Hello, I just wanna say how fantastic this video has been in explaining how to clearly and easly build a decoder only transformer from scratch. It also helped tremendously in understanding the "Attention is all you need" paper. Building attention from scratch was also an excellent thing you showed instead of relying on the inbuilt one in pytorch. Once again thank you so much.
Thank you!
Definitely the clearest walkthru of Transformer. It's very good that you put heavy emphasis on the parallelizability of Transformer since IMO it is the most important feature that made Transformer so useful
agreed!
I finally decided to buy your book thinking there'd be transformers in the "Neural Network" section. But even if they're not, I'm glad it supports you. Your content is the best in popularisation that I've seen. It mainly helps me a lot to refresh and understand better than before to start my internship in AI after 1 year of gap year.
I'm starting a book on neural networks every soon.
You're the only person on social media that can explain such complicated topics in an easy to understand manner. Keep up!
Thanks, will do!
It's interesting how many different topics one has to contextualize to understand something like transformers! It's great to see all the math happening in detail and stop and ponder time-to-time while learning something as complex as transformers for the first time. It really helped build my intuition of a lot of building blocks for a transformer-based neural network. Thank you!
Glad you enjoyed it!
Your neural networks playlist including this video gave me an intuitive understanding of transformers in less than a week which is something that would have taken an entire semester otherwise. I stumbled onto them while searching for a better understanding of Q,K,V, which everyone seems to say is as simple as querying a database…but what does that even mean?? Your explanations are brilliant, and I will be sharing with everyone I know who wants to learn more about this topic. I look forward to future videos. Thank you!
Thank you very much!!! I really appreciate it.
I had never struggled so much with understanding a concept before. But you cleared all the doubts. Thank you!
Glad it helped!
@@statquest I actually had a doubt as I was going through, about the decoder part. In the masked multi head attention part of the typical transformer, what inputs do we provide? And is this part only used during training?
@@pratyushrao7979 I actually talk about masking in my video on decoder-only transformers here: ruclips.net/video/bQ5BoolX9Ag/видео.html
OMG, I waited for it for so long!!, thank you, Josh!
bam! :)
I don't know how I'd learn DataScience/ML without this channel. Thanks so much for doing what you do!
Happy to help!
That is a lot of stuff in a single video!! For those who are wondering, ChatGPT is a decoder only neural network, and the main diff between an encoder and a decoder is that a decoder uses masked attention - thus ChatGPT is essentially an autoregressive model. Notice how ChatGPT generates a response in sequential order, from left to right. Anyway, good stuff!
Yep - I'd like to make a GPT video just to highlight the explicit use of masking (the self attention in the decoder in this video used masking implicitly).
@@statquest Please do that video soon :) BAM
I'm currently studying for job interviews and I don't know what I would do without you, thank you! When I get paid from my first job I will donate to you :)
Wow! Thank you!
Thanks Josh, another great video, I’ve been following your channel for years now and your videos have massively helped me to change career so huge thanks.
On to the transformer network, there’s something about the positional encoding that makes me feel a little uneasy. It feels we’ve gone through great effort to train a word embedding model that can cluster similar words together in n-dimensional word embedding space (where n can be very large, often 1,000).
By then applying positional encoding before our self-attention, whilst you very clearly explained with your example how important adding this information to the model is, seems to me to mess up all the effort we put into word embedding to get similar words clustered together. The word pizza, instead of being positioned in the same place can now jump around word/positional embedding space. Instead of one representation of pizza in space, it can now move around to be in many different positions, and not move locally around its own 'area' but because we add the positional encoding to the word embedding, scaled equally, it can jump around a great deal of space. To me it would seem adding this much freedom to where the word pizza can be represented in space would make it much much harder to train the model. Is my understanding correct or is there something I’m missing?
I have a couple of thoughts on this. Maybe I should make a short video called "some thoughts about positional encoding". Anyway, here they are...
Thought #1: Remember the positional encoding is fixed, so the word embedding values have to take them into account when training. For example, since all of the positional encoding value are between -1 and 1, it is possible that the word embedding values will have larger magnitudes and thus, not move around a lot when position is added to them.
Thought #2: Because the periods of the squiggles get larger for larger embedding positions, after about the 20th position, the position encoding values end up alternating 1 and 0 (in other words, after the 20th position, the position encoding values are 1010101....) and it is in that space, from the 20th position to the 512th position (usually word embeddings have 512 or more positions) that the word embeddings are really learned, and that the first 20 positions are mostly just for position encoding.
@@statquest Ah ok yeh that makes a lot of sense, thanks so much for taking the time to reply!
I’ve been having some additional thoughts on this and think I may have another reason (or rather an example) why adding positional encoding to the word embedding vectors makes sense, Josh if you read this, feel free to shoot it down! Take the following sentence: “The weather is bad, but my mood is good”. In this sentence the first “is” refers to the weather, whereas the second "is" refers to my mood. Without positional encoding and only word embedding, the vector for “is” being passed into the attention unit will be the same for the two instances of the word in the sentence. If we don’t use masked self-attention and compare the word “is” to every word including itself in the sentence, then the output of the word “is” in the self-attention unit I believe should be the same for both instances. Therefore, the unit will struggle to successfully differentiate the relative meaning of the two words. By adding in positional encoding prior to the self-attention unit, we’re suddenly adding context to the word. The second “is” comes straight after the word “mood”, therefore the position vector we’re adding to each of the two words should be similar. However, because the word “weather” comes 6 words before the second “is”, the positional vector we add will be quite different. Presumably this difference helps a self-attention unit to differentiate the relative meanings of the two instances of the word “is”.
@@matthewhaythornthwaite9910 That all sounds reasonable to me! BAM! :)
@@matthewhaythornthwaite9910 Another reason why you would want to add positional encoding instead of doing something else is that it preserves the dimensionality of the encoding. Imagine a theoretical encoding which is not added (like a one-hot encoding for each sequence location), and some linear (or non-linear for that matter) transform to combine word embedding and positinonal encoding. This is great in the sense that we do not polute the embedding space with "arbitrary" offsets, but now our input sequence has to be of a fixed shape. Addition of orthogonal sinusoids guarrantees a non-parametric, dimensionality preserving encoding which does not fix the number of inputs we can give to the network.
By the way, I think there is an analogy between adding positional encoding to embeddings and adding residual/skip connections to network outputs. Imagine that we have a network that is represented by the function f(x) and we have some target function F(x) which we want the network to learn. Imagine now that we modify our network to compute the function f(x) = h(x) + x (where "h(x)" is the network in front of the skip connection "h(x) + x"). Here too we polute the output space of h(x) with the values of x. However the network f can still learn F, so long as the network h(x) learns the function h(x) = F(x) - x (such that f(x) = h(x) + x = F(x) - x + x = F(x)).
I suppose for positional encoding something similar holds (altough it probably has to learn a much more difficult internal pattern), where the network is f(E(x)+q) learns to associate word embedding values E(x) which are "convolved" by some known offsets q and probably learns to deconvolve E(x) and q (into some abstract representation).Given that E(x) + q may in theory be (nearly) non-unique (i.e. E(x_1) + q_1 = approx E(x_2) + q_2) it might still be possible for the network to deconvolve the values into the correct inputs based on the context vector C which is calculable from the rest of the input sequence. I suppose one can't exclude that the network may never get this wrong, but in practical terms, it seems to work well enough.
The most AWESOME 36 MINUTES - What an explanation of Transformers!
Thank you very much!!! BAM! :)
Only videos like this can have "clearly explained" in the title.
bam!
(just letting you know while cramming for my exam, ive used your videos for over 4 years now and some professors, including the ones from this class, use your videos for us to learn (either showing them to us or using screenshots from the videos with reference). Its so funny watching the reaction of people who arent familiar with your channel reacting to a soft cute bear as the soft max function and such haha Most people in portugal can speak english but still, many prefer to search in portuguese, so they never encountered your channel before. But I make sure to let all my friends know your content that helps us with some specific exam. I would never be able to thank you enough for the more than wonderful content and for your patience explaining questions we have in the comments)
Thank you very much! I'm glad to hear that my videos are helpful. Maybe one day I can go to Portugal and teach a lecture in person. That would be super fun. :)
As a computer science student getting into the field of NLP, I really can't thank you enough for making a video that breaks down Transformer like this. Our uni doesn't go in depth about NLP related topics and with a very brief explanation they do, the uni expects us to have a full understanding about NLP. I can't thank you enough!
Thanks!
Your videos have to be the best instance of teaching I have ever seen! Thank you for the amazing work!
Thank you!
Best explanation I found in the whole Internet ! although I admit I needed 2 full turns. well done Josh !
Thanks! - Yes, this video packs in a ton of information, but I couldn't figure out any other way to make it work.
Penta BAM!!! All of your videos are extremely easy to understand in a peculiar way, they have helped me a lot, thank you very much.
Glad you like them!
Prof. Starmer, Thank You very much. You are an inspiration to all the aspiring Machine Learning Enthusiasts. Respect and Gratitude from India. #RESPECT
Thank you very much!
So nicely explained! I have searched for "how transformers work" but no one on youtube explained with both concept and math! Keep the good work going 😁👍
Glad you liked it!
I've spent a good deal of time studying attention, the critical concept behind transformers. Don't anticipate a natural understanding of the Q, K, and V parameters. We aren't entirely certain about their function; we can only hypothesize. They could still function effectively even if we used four parameters instead of three. One crucial point to remember is that our intuitive understanding of neural networks (NNs) is far from complete. The matrices for Q, K, and V aren't static; they're learned via backpropagation over lengthy training periods, thus changing over time. As a result, it's not as certain as mathematical operations like 1+1=2. The same applies to the head count in transformers; we can't definitively state whether eight is a good number or not. We don't fully grasp what each head is precisely doing; we can only speculate.
In visual transformers, we do understand what each head does. I guess heads trained on language are more difficult to interpret for us.
@@GreenCowsGames no, we don’t know that they do.
I've searched a lot about the transformers but seriously this is the best explanation I've ever got. Amazing!❤
Wow, thank you!
Great video! I'd love to see a pytorch + lightning tutorial on transformers similar to the LSTM video!
That's the plan!
1 million subscribers and still taking the time to answer questions from his viewers. Absolute legend
BAM! :)
Bet it’s an AI bot answering 🤖
@@luisfernando5998 Nope - it's me. I really read all the comments and respond to as many as I can.
@@statquest do u have a team ? 🤔 how do u manage the time ? 🤯
@@luisfernando5998 It only takes about 30 minutes a day. It's not that big of a deal.
THIS is what I've been waiting for!
Thanks!
This is one of the most important videos in recorded history that every learner of AI must watch. Thank you sensei !
Thank you!
Thank you so much for this video. More than 35:years in IT, worked with punching cards, I started to believe I am too old for this new thing. It was not my age, but the way how the knowledge is offered which does not fit my learning style. Finding your video, listening and immediate insights, hearing the coins dropping, were only separated by minutes. Ordering your book is the least minimum I could do to say thank you.
TRIPLE BAM!!! Thank you very much for your support! :)
I'm just starting learning about Machine Learning, and this video has been very clear as an introduction to learn concepts and move on. Thanks for great content, contiuining with side quests right now!
Thank you very much!
The explanation of transformers that the Internet missed!
bam! :)
It is so complex, ill have to watch this 2 3 times. But you did much better than anyone else. Thanks.
It can be made much simpler if you learn about neural networks first. You can do that with this play list: ruclips.net/video/CqOfi41LfDw/видео.html
Hey @statquest - here is a quick suggestion. Another convenient way to explain positional encoding might be by drawing clock with minute and hour hand. Then - instead of sin() and cos() functions you could simply track the x and y coordinates of the tip of the minute and hour hand. It gives much more convenient intuition behind mechanics of the encoding.
(a) it shows its repetitive nature
(b) ties encoding position with sense of time (which is intuitive since speech is tied to time as well). Speech is the most common way we use language
(c) it explains why we use both sin() and cos() functions (to track circular motion of the clock hand)
(d) it provides intuition on why having two pair of sin() and cos() functions is better than just one
That's a great idea!
So kind of like a unit circle?
This is the best of all that is available right now on Transformers. Thank you!!
Thank you!
I must say, after learning how GPT works I am even more blown away that this can work the way it works with the big online models. We just have to keep in mind that these companies are blowing tens of millions just for training. That's something else than me with a RTX 4090 and pytorch spending 12 cents of electricity and thinking I have unfathomable power 😂
:)
this is without a doubt the best explanation of transformers for a beginner out there. Thank you kind sir
Glad you liked it!
HE HAS DONE IT YET AGAIN!
Thanks!
I'm following an advanced master of Artificial Intelligence. This whole NN playlist has saved me while studying for my exams! Thanks a bunch!
good luck!
You really explaining these concepts in a clear way! Will you do more explanation video on statistic like Cox model for survival ? Thanks! :)
I'll keep that in mind.
you are the unique ML teacher guy in the world, and I don't think anyone can explain this thing like you. Thank you myself for knowing your channel!
Wow, thanks!
We wanna more NLP material please, tiny bam !
:)
Best video on Transformer I have seen in RUclips! Amazing ! huge respect for you
Thank you so much 😀!
(Verse 1)
Here we are with another quest,
A journey through the world of stats, no less,
Data sets in rows and columns rest,
StatQuest, yeah, it's simply the best.
(Chorus)
We're diving deep, we're reaching wide,
In the land of statistics, we confide,
StatQuest, on a learning ride,
With your wisdom, we abide.
(Verse 2)
From t-tests to regression trees,
You make understanding these a breeze.
Explaining variance and degrees,
StatQuest, you got the keys.
(Chorus)
We're scaling heights, we're breaking ground,
In your lessons, profound wisdom's found,
StatQuest, with your sound,
We'll solve the mysteries that surround.
(Bridge)
With bar charts, line plots, and bell curves,
Through distributions, we observe,
With every lesson, we absorb and serve,
StatQuest, it's knowledge we preserve.
(Chorus)
We're traversing realms, we're touching sky,
In the field of data, your guidance, we rely,
StatQuest, with your learning tie,
You're the statistical ally.
(Outro)
So here's to Josh Starmer, our guide,
To the realm of stats, you provide,
With StatQuest, on a high tide,
In the world of statistics, we stride.
(End)
So get ready, set, quest on,
In the realm of stats, dawn upon,
StatQuest, till the fear's gone,
Keep learning, till the break of dawn.
THAT IS AWESOME!!! (what are the chords?)
@@statquest I V vi IV
Transformers! More than meets the eye!? I think there is a lot of value in knowing this technology well! Thank you for your humor and learning support, I can't wait to return the favor!
Thanks!
Very clear and builds up the concepts in a step by step manner, rather than starting with the overall architrcture.
Thanks!
man i love you for this video thank you so much, there's absolutely no way i'd have understood all of this without your help
Glad I could help!
I like how he says "In this example we kept things super simple".
Great video, thank you!
Glad you liked it!
This is the best Transformer video i have ever seen, Thank you.
Thanks!
Wonderful! There's so much detail packed into such a short video. It took me three hours to understand it all :)))
Glad it was helpful!
@statquest you are a true professional and a master of your craft. The field of ML is getting a little stronger each day bc of content like this!
Thank you very much!
What a legend, there is no better channel than this!
Thank you!
The best video that explains Transformer I've ever seen. Thanks for your contribution!
Thank you very much! :)
Awesome and crystal clear explanation. Keep rocking.
Thank you!
Wow Josh. You’re a genius, this was comprehensive and complete. Thank you again.
Glad it was helpful!
U explained it really clear. It was the best transformer video i've watched.
Wow, thanks!
Just wanted to say that I understood everything about LMs (thanks to your videos), except the part on transformers cuz the video wasn't out yet ahah. Well now that my dear squash teacher explained it, everything's clear. So really THANK YOU for your hard work and dedication, it made all the difference in my understanding of Neural Networks in general
Great to hear!
Very thorough, thanks
Utilizes a lot of the previous concepts of basics of Deep Learning, amazing resources you've compiled over the years.
I've read abt transformers from multiple resources and each time I understanding something new !
Prepping for interviews rn, will thank you once again after I land a job 🌞💖
good luck!
This is your only video that I had to see twice to get complete idea of the topic. Transformers really is a decently tough topic.
This is a lot of material for one video. But people wanted a single video, rather than a series of videos making incremental steps in learning, for transformers. Personally, I would have preferred a sequence of shorter videos, each focused on just one part. That said, there is something about seeing it all at once and getting that big picture. My book on neural networks (that I'm working on right now) will try to do both - take things one step at a time and give a big picture.
@@statquest Nice. Waiting for the book in that case.
Thanks! You are the best
Thank you very much for supporting StatQuest! TRIPLE BAM!!! :)
A lot of hard work must have gone into these videos. And the results are these brilliant super helpful videos. Thanks a lot for these videos.
Glad you like them!
Thank you sir for this amazing video, it helped me last year in my NLP exam and now i'm refreshing my information's about transformers hoping to land an interview soon!
Good luck - let me know how it all goes.
These videos on machine learning (and statistics!) are incredible and empowering! Plus they make me laugh! You are the reason that I was able to keep my head above water in my Machine Learning courses as a pregnant grad student with limited time, money and energy. Thank you and I will definitely be checking out some merch!
Thank you so much! I'm happy to hear my videos are useful. BAM! :)
Please create more vids!! Your explanations are truly beautiful, such a work of art. I couldn't agree more that you are one of the most brilliant teachers at statistic and ML! Actually, I wanna hug you right now haha
Wow, thank you!
The videos are really fantastic!!! First time ever that helps me understand every single detail!! Thank you!!! Plz keep posting!!
Thank you! Will do!
I feel so lucky that I just started learning Transformer and found out StatQuest made a video for it one day ago!
bam!
Love these Quests! Kudos for explaining these complex data science concepts in layman terms but also with great depth ❤
Thank you!
This might be the best video on deep learning I've seen period, rivalled only by 3Blue1Brown's content. Thank you so much!
Wow, thanks!
Thank you Josh for this fantastic video. One of the best videos to explain the math behind the Query, Key and Values.
Thank you!
You are there man! One place destination for deep learning concept! tysm
Thank you!
This is so mindblowingly complex and impressive. Great video! ❤
The transformer architecture is also complex and impressive, ofc. 😊
BAM! :)
I finally found you! Thank you for this detailed yet super simple break down.
Glad it was helpful!
Thank you for such amazing illustrations! HOW I WISH I HAD THIS DURING MY STUDIES, BUT I'M JUST GLAD I GET TO LEARN THESE AS A WORKING PROFESSIONAL. THANK YOU SO MUCH FOR ALL THE CONTENT YOU MAKE. I'M SURE YOU MAKE THOUSANDS OF LIVES BETTER. YOU'RE TRULY AN INSPIRATION JOSH!
Thank you!