Coding a ChatGPT Like Transformer From Scratch in PyTorch
HTML-код
- Опубликовано: 2 июл 2024
- In this StatQuest we walk through the code required to code your own ChatGPT like Transformer in PyTorch and we do it one step at a time, with every little detail clearly explained.
NOTE: This StatQuest assumes that you are already familiar with the concepts behind...
Decoder-Only Transformers: • Decoder-Only Transform...
The Essential Matrix Algebra for Neural Networks: • Essential Matrix Algeb...
The Matrix Math Behind Transformers: • The matrix math behind...
You can get the code here: github.com/StatQuest/decoder_...
The full Neural Networks playlist, from the basics to AI, is here: • The Essential Main Ide...
Learn more about GiveInternet.org: giveinternet.org/StatQuest NOTE: Donations up to $30 will be matched by an Angel Investor - so a $30 donation would give $60 to the organization. DOUBLE BAM!!!
If you'd like to support StatQuest, please consider...
Patreon: / statquest
...or...
RUclips Membership: / @statquest
...buying my book, a study guide, a t-shirt or hoodie, or a song from the StatQuest store...
statquest.org/statquest-store/
...or just donating to StatQuest!
paypal: www.paypal.me/statquest
venmo: @JoshStarmer
Lastly, if you want to keep up with me as I research and create new StatQuests, follow me on twitter:
/ joshuastarmer
0:00 Awesome song and introduction
1:12 Loading the modules
2:04 Creating the training dataset
6:17 Coding Position Encoding
14:09 Coding Attention
21:04 Coding a Decoder-Only Transformer
26:39 Running the model (untrained)
29:18 Training and using the model
#StatQuest #PyTorch #chatgpt
- You can get the code here: github.com/StatQuest/decoder_transformer_from_scratch
- Learn more about GiveInternet.org: giveinternet.org/StatQuest NOTE: Donations up to $30 will be matched by an Angel Investor - so a $30 donation would give $60 to the organization. DOUBLE BAM!!!
- The full Neural Networks playlist, from the basics to AI, is here: ruclips.net/video/CqOfi41LfDw/видео.html
- Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/
Can't imagine the work that goes into this, writing the code, making diagrams, recording, editing and voice over, you're the goat big J.
Thanks!
You said this was going to come out at the end of May. And I’ve been waiting for this for 2 months. Finally, it’s out 😂
I guess better later than never?
It is party time! Thanks for uploading!
You bet!
Cool, learn a lot from all of your videos Josh! 🤯
Thanks!
sir you deserved millions of views on your RUclips ❤❤🎉
Thanks!
You will be rememberd for next 1000 years in the history of Statistics and Data Science , You should be named as "Father of Applied Statics & Machine Learning " , Pls thumbs up if you are with me
BAM! :)
HUGE RESPECT for all the work you put into your videos
Finally greatly watied video arrived. Thank you.
Bam! :)
as always, wonderful content.
Thanks :)
Thanks again!
Today we learned that statquest is awesome. triple BAM!
Thanks!
Thank you
I was in need of this 😊
Glad it was helpful!
I'm gonna enjoy this one!
bam! :)
Great and very didactic as usual, Josh!! Definitely going to wrap my head around this for a while and try a few tweaks! Do you plan on eventually also discussing other non-NLP topics like GANs and Diffusion Models?
One day I hope to.
Thank you! You're the best!!!
You're welcome!
This will be awesome. I am trying to learn the math behind transformers and PyTorch so hopefully this helps give me some intuition
I've got a video all about the math behind transformers here: ruclips.net/video/KphmOJnLAdI/видео.html
🎉🎉🎉thank you😊
bam! :)
thank you sm fr bro
Any time!
Thanks!
TRIPLE BAM!!! Thank you for supporting StatQuest!
Thanks a lot for for this free wonderful content. ❤😊
Thank you!
Legend.
:)
🎉🎉🎉
Triple 🎉!
Triple Bam!!!
:)
amazinggg
Thanks!
BAM!!
Thanks Nosson!
Bam!
:)
Baaaam!❤
:)
Ya misspelled ChatGPT - Generative Pre-trained Transformer
Corrected! :)
GTP :)
Corrected! ;)
I'm confused as to why the values would come from the ENCODER when computing the cross attention between the Encoder and Decoder. Shouldn't the values come from the decoder itself?
So if I trained a model to translate from English to German, then wanted to switch out the German for Spanish, I'd expect the new decoder to know what to do with the output of the Encoder. But if the values are coming from the Encoder, then this wouldn't work.
The idea is that the query in the decoder is used to determine how a potential word in the output is related to the words in the input. This done by using a query from the decoder and keys for all of the input words in the encoder. Then, once we have established how much (what percentages) a potential word in the output is related to all of the input word, we then have to determine what that percentage is of. It is of the values. And thus, the values have to come from the encoder. For more details, see: ruclips.net/video/zxQyTK8quyY/видео.html
Sir can you include how to make the chatbot to hold a conversation with
I'll keep that in mind.
🙏🏻🙏🏻🙏🏻🙏🏻🙏🏻 Please please please show us how to train QVK Weights in detail 🙏🏻🙏🏻🙏🏻🙏🏻🙏🏻
You showed us just a simple call to function. But we are curious how it did the math, what to train, and how it can changes values of the weights. ABC
Every single weight and bias in a neural network is trained with backpropagation. To learn more about how this process works, see: ruclips.net/video/IN2XmBhILt4/видео.html ruclips.net/video/iyn2zdALii8/видео.html and ruclips.net/video/GKZoOHXGcLo/видео.html
@@statquest Since both QVK Weights are splitted and the calculations are passing non neural network, imho the back propagation process is quite tricky. In the other hand, the fit function did not tell the order of calculations on each nodes.
GTP
Corrected! :)
Wish you could be Prime Minister of the United Kingdom!
Ha! :)
From scratch in pytorch, huh.
I decided to skip doing it in assembly. ;)