I've changed the output layer a bit... this: self.head_ln = nn.LayerNorm(emb_dim) self.head = nn.Sequential(nn.Linear(int((1 + self.height/self.patch_size * self.width/self.patch_size) * emb_dim), out_dim)) Then in forward: x = x.view(x.shape[0], int((1 + self.height/self.patch_size * self.width/self.patch_size) * x.shape[-1])) out = self.head(x) The downside is that you'll likely get a lot more overfitting, but without it the network was not really training at all.
Hi, thanks for your recommendation. I would probably not use this model for real world data as there are many important details that are missing (for the sake of providing a simple overview). I will pin your comment for others that also want to use this implementation. Thank you!
You're awesome man!!! I clicked your video so fast, you're one of the my favorite AI youtubers. I work in the field and I think you have a wonderful ability of explaining complex concepts in your videos
There was a error on your published code but not in the video. attn_output, attn_output_weights = self.att(x, x, x) It should be attn_output, attn_output_weights = self.att(q, k, v) Anyway, thanks for sharing the video and code base. It helped me a lot while learning ViT
Hello, first of all great tutorial video. I've tried running provided code for training, but after ~400 epochs loss is still the same (~3.61) and model always predicts the same class. Do you have an idea what might be a problem with it?
@@DeepFindr Actually I've already found one bug in notebook. In forward method of Attention module, input is directly passed to MultiheadAttention bypassing linear layers. Changing learning rate doesn't affect training at all. Also, when training I've noticed that model's output converges to all zeros. I've checked gradients in network and it turns out that gradient flow stops at PatchEmbedding layer. All layers after it have non-zero gradients. Still don't know why this happens
Thanks for finding this bug. But I actually think it's not super relevant for this issue - I experimented with the attention previously and tried both ways (with linear layers and without), that's how this bug was created in the first place. When I started the training back then the loss was definitely decreasing, but I didn't expect it to get stuck at some plateau. Typically when models predict always the same class there can be a couple of reasons. I already checked this: - Input data is normalized - Too few / too many parameters (I would recommend to count the model parameters to get a feeling for this) - Learning rate - SGD Optimizer (seems to work a bit better) - Batch size, I put it to 128 - Embedding size, make it a bit smaller After 100 epochs the Loss is also converging to 3.61, but the model is predicting different classes. Maybe the Dataset is not big enough. What about trying another dataset? Alternatively, try data augmentation. As stated in the video, transformers need to see a lot of examples.
Imagine you have a sentence "I made a pizza and put it in an oven, it was tasty". Here we know that it refers to pizza, but for a model, it could mean pizza is delicious or oven is delicious. Positional encodings are learnt based on each other, the position of it here is with respect to pizza and oven. Therefore it is a learnt parameter. Here the video mentions that it is doesn't have any significant advantage over numbering, but when you are trying to teach the model to identify features, it will identify feature and position of one patch wrt other patches, ergo making it a learnable parameter
It’s not really needed, in the original transformer paper have been tests with learnable pe and basic-not-learnable-sinusoidal pe and that did not make a big difference, gpt2 also uses learnable pe, nowadays it’s more RoPE doing the positional encoding
Nice video! However, I think it's incorrect that you would get separate vectors for the three channels? This is not how they do it in the paper; there they say that the number of patches is N = HW/P^2, where H, W and P is the height and width of the original image and (P, P) is the resolution of each patch, so the number of color channels doesn't affect the number of patches you get.
original image size was 144 (h) x 144 (w), after patch embedding, it was transformed to 324 patches, and each patch has 128 dimension size. 324 was derived from (144/8) x (144/8) = 18 x 18 = 324 patches. 8 is the patch size of each patch.
Isn't the embedding layer redundant? I mean we have then the projection matrices meaning that embedding + projection is a composition of two linear layers.
Awesome video! But I wonder if you reverse the order of LayerNorm and Multi-Head Attention? I think the LayerNorm should be applied after Multi-Head Attention but your implementation apply the LayerNorm before it.
Hi! Thanks! There is a paper that investigated pre- VS post-layernorm in transformers (see here arxiv.org/pdf/2002.04745). The "Pre" variant seems to perform better as opposed to the traditional suggestion in the transformer paper. This is also what most public implementations do :)
Hey,great video. I have a question though. Isn't the entire point of 'pre' norm is that the normalization is applied before attention computation is performed? But from the code ,norm = PreNorm(128, Attention(dim=128, n_heads=4, dropout=0.)) , it seems like you are performing attention first and then normalizing aka post-norm. Please correct me if I'm wrong :)
Hi! In the forward pass of the PreNorm layer is this line: self.fn(self.norm(x), **kwargs) So normalization is applied first and then the function (such as attention in this example) The line you are referencing is just the initialization, not the actual call Hope that helps :)
hi there I have used the code for binary class classification, but encountering the problem on accuracy , showing 100% accuracy only on label 1 and some times on label 2. So it would be helpful for me if u provide me any solution
Cool video! What do you think about the implementation of ViT on signal processing (spectrogram analysis) , applied to audio for example. Which advantages could it have over the classic convolutional networks?
The video is great but the training in the code didn't work for the entire 1000 epochs. Despite the code looks logical, there is endless of things that can go wrong so I think it was better to do a tutorial with working ViT notebook.
![Noob To Max With DRAGON REWORK In Blox Fruits [FULL MOVIE]](http://i.ytimg.com/vi/LnBMOinoOvA/mqdefault.jpg)
I've changed the output layer a bit... this:
self.head_ln = nn.LayerNorm(emb_dim)
self.head = nn.Sequential(nn.Linear(int((1 + self.height/self.patch_size * self.width/self.patch_size) * emb_dim), out_dim))
Then in forward:
x = x.view(x.shape[0], int((1 + self.height/self.patch_size * self.width/self.patch_size) * x.shape[-1]))
out = self.head(x)
The downside is that you'll likely get a lot more overfitting, but without it the network was not really training at all.
Hi, thanks for your recommendation.
I would probably not use this model for real world data as there are many important details that are missing (for the sake of providing a simple overview).
I will pin your comment for others that also want to use this implementation.
Thank you!
This is very underrated channel. You deserve way more viewers!!
Keep making content like this, I am sure you will get a very good recognition in the future. Thanks for such amazing content.
You're awesome man!!! I clicked your video so fast, you're one of the my favorite AI youtubers. I work in the field and I think you have a wonderful ability of explaining complex concepts in your videos
thanks for the kind words :)
Really great explanation. Nice visuals
Much appreciated!
This channel is amazing. Please continue making videos!
Awesome man!! You code and explain with such simplicity.
Awesome! Thanks for this video!
Awesome! Thanks for excellent explanation!
Thank you! Very clear and informative.
Very helpful video, thanks!
Great tutorial
There was a error on your published code but not in the video.
attn_output, attn_output_weights = self.att(x, x, x)
It should be
attn_output, attn_output_weights = self.att(q, k, v)
Anyway, thanks for sharing the video and code base. It helped me a lot while learning ViT
As a robot myself, i can confirm that an image really is worth 16x16 words
감사합니다!!
The best part of Vision transformers is inbuilt support interpretability as compared to CNN where we had to compute saliency maps.
Hello, first of all great tutorial video. I've tried running provided code for training, but after ~400 epochs loss is still the same (~3.61) and model always predicts the same class. Do you have an idea what might be a problem with it?
Hi, have you tried a Lower learning rate? Also is the train loss decreasing or also stuck?
@@DeepFindr Actually I've already found one bug in notebook. In forward method of Attention module, input is directly passed to MultiheadAttention bypassing linear layers.
Changing learning rate doesn't affect training at all. Also, when training I've noticed that model's output converges to all zeros.
I've checked gradients in network and it turns out that gradient flow stops at PatchEmbedding layer. All layers after it have non-zero gradients. Still don't know why this happens
Thanks for finding this bug. But I actually think it's not super relevant for this issue - I experimented with the attention previously and tried both ways (with linear layers and without), that's how this bug was created in the first place.
When I started the training back then the loss was definitely decreasing, but I didn't expect it to get stuck at some plateau.
Typically when models predict always the same class there can be a couple of reasons. I already checked this:
- Input data is normalized
- Too few / too many parameters (I would recommend to count the model parameters to get a feeling for this)
- Learning rate
- SGD Optimizer (seems to work a bit better)
- Batch size, I put it to 128
- Embedding size, make it a bit smaller
After 100 epochs the Loss is also converging to 3.61, but the model is predicting different classes. Maybe the Dataset is not big enough. What about trying another dataset? Alternatively, try data augmentation.
As stated in the video, transformers need to see a lot of examples.
An image is worth 16x16 words🗣🗣🗣🗣🗣🗣🗣💯💯💯💯💯💯💯🔥🔥🔥🔥🔥🔥🔥
bro is educated!
Why are the positional embeddings learnable? It doesn't make sense to me
Bcoz, positional embedding represent the adress or position of image information of image patches
@@trendingtech4youth989 in "Attention is all you need", afaik, positioal embeddings are not learnable
@trendingtech4youth989 so..they are given like the patches, why shall they be learned
Imagine you have a sentence "I made a pizza and put it in an oven, it was tasty".
Here we know that it refers to pizza, but for a model, it could mean pizza is delicious or oven is delicious.
Positional encodings are learnt based on each other, the position of it here is with respect to pizza and oven. Therefore it is a learnt parameter.
Here the video mentions that it is doesn't have any significant advantage over numbering, but when you are trying to teach the model to identify features, it will identify feature and position of one patch wrt other patches, ergo making it a learnable parameter
It’s not really needed, in the original transformer paper have been tests with learnable pe and basic-not-learnable-sinusoidal pe and that did not make a big difference, gpt2 also uses learnable pe, nowadays it’s more RoPE doing the positional encoding
Why use dropout with GeLU? Didn’t the GeLU paper specifically say one motivation for GeLU was to replace ReLU+dropout with a single GeLU layer?
Nice video! However, I think it's incorrect that you would get separate vectors for the three channels? This is not how they do it in the paper; there they say that the number of patches is N = HW/P^2, where H, W and P is the height and width of the original image and (P, P) is the resolution of each patch, so the number of color channels doesn't affect the number of patches you get.
thank you
I think there is a confusion between cls token and positional embedding? At 6:09?
Has anyone been able to make it converge? What hyperparameters did you modify?
in 05:08. how we calculated that? when I calculated the patch shape I got a different result. Could someone explain that?
yes exactly, I also have same doubt, for me its 192 instead of 324
original image size was 144 (h) x 144 (w), after patch embedding, it was transformed to 324 patches, and each patch has 128 dimension size. 324 was derived from (144/8) x (144/8) = 18 x 18 = 324 patches. 8 is the patch size of each patch.
Is this better for the MNIST challenge compared to a simple conv network like LeNet
Isn't the embedding layer redundant? I mean we have then the projection matrices meaning that embedding + projection is a composition of two linear layers.
Awesome video! But I wonder if you reverse the order of LayerNorm and Multi-Head Attention? I think the LayerNorm should be applied after Multi-Head Attention but your implementation apply the LayerNorm before it.
Hi! Thanks!
There is a paper that investigated pre- VS post-layernorm in transformers (see here arxiv.org/pdf/2002.04745). The "Pre" variant seems to perform better as opposed to the traditional suggestion in the transformer paper. This is also what most public implementations do :)
Is the Colab using cuda? IF so how can I tell if it is useing cuda
Hey,great video. I have a question though. Isn't the entire point of 'pre' norm is that the normalization is applied before attention computation is performed?
But from the code ,norm = PreNorm(128, Attention(dim=128, n_heads=4, dropout=0.)) , it seems like you are performing attention first and then normalizing aka post-norm. Please correct me if I'm wrong :)
Hi! In the forward pass of the PreNorm layer is this line:
self.fn(self.norm(x), **kwargs)
So normalization is applied first and then the function (such as attention in this example)
The line you are referencing is just the initialization, not the actual call
Hope that helps :)
@@DeepFindr stupid of me to not see that first. Thank you for the reply
Ah, tough to understand, guess will have to read more on this to fully understand
You should have a deep understanding of transformer architecture to understand this.
hi there
I have used the code for binary class classification, but encountering the problem on accuracy , showing 100%
accuracy only on label 1 and some times on label 2. So it would be helpful for me if u provide me any solution
Hi, please see pinned comment. Maybe this helps :)
can you please make a video on how to perform inference on VIT like googles open source vision transformer?
Hope you could explain the swim transformer object detection in new video please
Cool video! What do you think about the implementation of ViT on signal processing (spectrogram analysis) , applied to audio for example. Which advantages could it have over the classic convolutional networks?
Take a look at gpt4-omni to find out, lol
Where to get slides? Used in video
thank u ,
can someone help me with the training codes in the Google Colab link in the description?
The video is great but the training in the code didn't work for the entire 1000 epochs. Despite the code looks logical, there is endless of things that can go wrong so I think it was better to do a tutorial with working ViT notebook.
Hi! I think this is because the Dataset is too small. Transformers are data hungry. It should work with a bigger dataset
Also have a look at the pinned comment, maybe that helps :)
And now sora uses the same algorithm. this video aged so well
Sora is using DiT (Diffusion Transformer)
Bravo!