as per my understanding and from the LayerNorm code in pytorch, in NLP for an input of size [N, T, Embed], statistics are computed using only the Embed dim, and layer norm is applied to each token in each batch. But for vision with an input of size [N,C,H,W], statistics are computed using the [C,H,W] dimentions
6:56 actually you would use the variance instead of the standard deviation, so in the layernorm formula it should be sigma^2 in the divisor. and the epsilon is missing to prevent zero division
Thank you so much for providing solid examples and calculations to explain these concepts. I have seen these concepts elsewhere but couldn't make sense of them until I saw how you computed the values. Great video!
Another great video - I like the structure you use of summarising the concept and then diving into the implementation. The code really helps bring it together as some others have commented. I look forward to seeing more of this series and would love to see a longer video of you deploying a transformer on some dummy data (perhaps you already have one - still going through the content)!
Thanks so much on commenting on all the videos! I really appreciate it. And yea, going to be introducing the code behind the encoder and decoder in the coming sessions!
Great! Since you’re covering transformer components, I would love to see TransformerXL and RelativePositionalEmbedding concepts explained in the upcoming videos! ☺️
Nice series on transformer. Really liked it. Btw interesting design choice for the video to use a landscape layout of the transform architecture during intro :D
10:30 when layers normalizations are "computed across the layer and also the batch", are their means and std connected as if the batch boundaries aren't there? So that means there's a different learnable gamma and beta parameter for each word?
“Means and std connected as if the batch boundary isn’t there” ~ yes But we have the same gammas and betas for the same layer across ALL words in the dataset
A bit late but can you tell me if my understanding is correct. The output from the attention head might have skewed means because of the large spread of the values, this might consist early lead to wrong predictions. So the gradients might constantly changing itself to correct these predictions which might lead to incorrect change in the weights which eventually leads to exploding and vanishing gradients. To correct this we sort of reset the mean and then fine tune the mean according to the matrix ?
if our batch has 2 vectors of 2 words x 3 embedding size say: [[1,2,3],[4,5,6]] and [[1,3,5], [2,4,6]] For layer normalization, Is nu_1 = mean(1,2,3,1,3,5) and nu_2 = mean(4,5,6,2,4,6) Just wanted to clarify. Keep up the great work, brother. Like the small bite sized videos.
meaning , the statistical calculation for first vectors of all batches are done together? like how you calculating mean using the first vectors of both the batches?
I don't think that is correct. Layer normalization is not applied across a batch. It's applied to a single input observation. The mean and std values are computed for each embedding.
Great video! I find these very informative, so please keep them going! Question on the output dimensions though. In your transformer overview video the big diagram shows that after the layer normalization, you have a matrix of shape [batch_size, sequence_len, dmodel] (in the video 30x50x512 I believe.) However here you end up with an output matrix (out) of [sequence_len, batch_size, dmodel] (5x3x8). Do we need to reshape these output matrices again to [batch_size,sequence_len,dmodel], or am I missing something? Thanks again for all the informative content!
Where can I find the reason why we need to calculate mean and standard deviation over the parameter shapes? In Pytorch, they just caculate over the last dimension, hidden size.
Have an actual question this time! While trying to understand the differences between layer and batch normalization, I was wondering whether it’s also accurate to say you are normalising across the features of a vector when normalising the activation function - since each layer is a matrix multiply across all features of a row, would normalising across activation functions be similar to normalising across the features? In the same thread, can/should layer and batch normalization be run concurrently? If not, are there reasons to choose one over the other?
Good questions. From what I understand, we normalize the activation values across the layer (I.e make sure the values across the layer follow a bell curve of sorts). While in batch normalization, we do the same exact thing but across the batch dimension instead The only issue i see with batch normalization is that it is dependent on the batch size (which is typically an order magnitude smaller than the size of layers) If we apply a normalization to a small number of items, we might get erratic results (that is the mean and standard deviation of a small number of numbers simply don’t lead to values that are truly “normalized”). One remedy would be to increase your batch size to a sizable number (like in the hundreds). I have heard this is an issue with NLP problems specifically. But I would need to do my own experimentation to see why.
I wouldn’t necessarily say that is the case. Tensor flow and PyTorch are frameworks we can use for building these complex models. PyTorch might be easier to use since you don’t need to code out tensors themselves. But tensorflow (or even going as low as numpy) can be used to train models too
Great Video !! Just a question, why do we need to swap the dimensions and all that other stuff ? Why can't we do something like this # assuming inputs is of format (batch_size,sentence_length,embedding_dim) mean=inputs.mean(dim=-1,keepdim=True) var = ((inputs - mean) ** 2).mean(dim=-1, keepdim=True) epsilon = 1e-5 std = (var + epsilon).sqrt() y = (inputs - mean) / std y
I think your code is not correct, LayerNorm will not be over the batch. Try and think what it means to take normalisation over the batches. For a layer norm each entity of the batch is unique, and each should be normalised across the layer of the MLP (essentially everything is MLP).
In theory, that’s what’s happening. In code, we make “8 parallel heads” by introducing another dimension. For example, vectors of shape 30 (max sequence length) x 10 (batch size) x 512 (embedding size) would form query key and value tensors of shape 30 x 10 x 8 x 64. So there is a new dimension for heads that essentially acts like another batch dimension for more parallel computation
the diagram is GOLD
as per my understanding and from the LayerNorm code in pytorch, in NLP for an input of size [N, T, Embed], statistics are computed using only the Embed dim, and layer norm is applied to each token in each batch. But for vision with an input of size [N,C,H,W], statistics are computed using the [C,H,W] dimentions
I really like your voice and delivery. It’s quite reassuring, which is nice when the subject of the videos can be pretty complicated.
Thanks so much! Very glad you liked this. There will be more to come
6:56 actually you would use the variance instead of the standard deviation, so in the layernorm formula it should be sigma^2 in the divisor. and the epsilon is missing to prevent zero division
Thank you so much for providing solid examples and calculations to explain these concepts. I have seen these concepts elsewhere but couldn't make sense of them until I saw how you computed the values. Great video!
the python example really helped to solidify understanding.
Super glad it did
very clear and sound explanation for a complex concept, thumb up for the hard works!
Another great video - I like the structure you use of summarising the concept and then diving into the implementation. The code really helps bring it together as some others have commented. I look forward to seeing more of this series and would love to see a longer video of you deploying a transformer on some dummy data (perhaps you already have one - still going through the content)!
Thanks so much on commenting on all the videos! I really appreciate it. And yea, going to be introducing the code behind the encoder and decoder in the coming sessions!
@@CodeEmporium I love your content so will do what I can to comment , like and spread!
Great! Since you’re covering transformer components, I would love to see TransformerXL and RelativePositionalEmbedding concepts explained in the upcoming videos! ☺️
i help with relpos, its the same as usual posenc but instead of fixed positional encoding its in relation to each other, pairwise.
Nice series on transformer. Really liked it. Btw interesting design choice for the video to use a landscape layout of the transform architecture during intro :D
The diagram is great..but you should explain the code also corresponding to the diagram
best explanation ever!
Thanks Man ,
you deserve more than a like
Thanks a ton for the kind words :)
Great lectures.
10:30 when layers normalizations are "computed across the layer and also the batch", are their means and std connected as if the batch boundaries aren't there? So that means there's a different learnable gamma and beta parameter for each word?
“Means and std connected as if the batch boundary isn’t there” ~ yes
But we have the same gammas and betas for the same layer across ALL words in the dataset
A bit late but can you tell me if my understanding is correct. The output from the attention head might have skewed means because of the large spread of the values, this might consist early lead to wrong predictions. So the gradients might constantly changing itself to correct these predictions which might lead to incorrect change in the weights which eventually leads to exploding and vanishing gradients. To correct this we sort of reset the mean and then fine tune the mean according to the matrix ?
Amazing your expiation are so clear. Can this help with exploding gradients?
Thanks so much! And yea. Later normalization should help exploding and vanishing gradients. So training is stable
@@CodeEmporium Perfect thanks.
EXCITED!!!!!
Me too :)
Value Matrix should be on the right side of the multiplication with Attention Weights matrix
if our batch has 2 vectors of 2 words x 3 embedding size say:
[[1,2,3],[4,5,6]] and [[1,3,5], [2,4,6]]
For layer normalization,
Is nu_1 = mean(1,2,3,1,3,5)
and nu_2 = mean(4,5,6,2,4,6)
Just wanted to clarify. Keep up the great work, brother. Like the small bite sized videos.
meaning , the statistical calculation for first vectors of all batches are done together? like how you calculating mean using the first vectors of both the batches?
I don't think that is correct. Layer normalization is not applied across a batch. It's applied to a single input observation. The mean and std values are computed for each embedding.
Great video! I find these very informative, so please keep them going! Question on the output dimensions though. In your transformer overview video the big diagram shows that after the layer normalization, you have a matrix of shape [batch_size, sequence_len, dmodel] (in the video 30x50x512 I believe.) However here you end up with an output matrix (out) of [sequence_len, batch_size, dmodel] (5x3x8). Do we need to reshape these output matrices again to [batch_size,sequence_len,dmodel], or am I missing something? Thanks again for all the informative content!
I wonder the same. Do you know the reason for it?
Beautifully done video but isn't layer normalization essentially batch normalization layer?
Where can I find the reason why we need to calculate mean and standard deviation over the parameter shapes? In Pytorch, they just caculate over the last dimension, hidden size.
I' m sorry, I just check the Pytorch code, there is normalized_shape like parameter_shape in your code.
appreciate!
Thank you very much friend, very good!!
Thanks so much for watching !
Thanks! This vedio helps me a lot
Glad!
Wow. Amazing channel
Thanks so much!
What is the toool you use to write?
Have an actual question this time! While trying to understand the differences between layer and batch normalization, I was wondering whether it’s also accurate to say you are normalising across the features of a vector when normalising the activation function - since each layer is a matrix multiply across all features of a row, would normalising across activation functions be similar to normalising across the features?
In the same thread, can/should layer and batch normalization be run concurrently? If not, are there reasons to choose one over the other?
Good questions. From what I understand, we normalize the activation values across the layer (I.e make sure the values across the layer follow a bell curve of sorts). While in batch normalization, we do the same exact thing but across the batch dimension instead
The only issue i see with batch normalization is that it is dependent on the batch size (which is typically an order magnitude smaller than the size of layers) If we apply a normalization to a small number of items, we might get erratic results (that is the mean and standard deviation of a small number of numbers simply don’t lead to values that are truly “normalized”). One remedy would be to increase your batch size to a sizable number (like in the hundreds). I have heard this is an issue with NLP problems specifically. But I would need to do my own experimentation to see why.
Is pytorch better for NLP task than Tensorflow?
I wouldn’t necessarily say that is the case. Tensor flow and PyTorch are frameworks we can use for building these complex models. PyTorch might be easier to use since you don’t need to code out tensors themselves. But tensorflow (or even going as low as numpy) can be used to train models too
@@CodeEmporium Thanks for clarifying
Great Video !! Just a question, why do we need to swap the dimensions and all that other stuff ? Why can't we do something like this
# assuming inputs is of format (batch_size,sentence_length,embedding_dim)
mean=inputs.mean(dim=-1,keepdim=True)
var = ((inputs - mean) ** 2).mean(dim=-1, keepdim=True)
epsilon = 1e-5
std = (var + epsilon).sqrt()
y = (inputs - mean) / std
y
I have the same doubt . did you get it ?
🔥🔥🔥
:) thank you
I think your code is not correct, LayerNorm will not be over the batch. Try and think what it means to take normalisation over the batches. For a layer norm each entity of the batch is unique, and each should be normalised across the layer of the MLP (essentially everything is MLP).
Given that we have 8 heads, Is it 512 / 8, which is 64? Are we actually going to split the 512 into 8 equal 64 length parts?
In theory, that’s what’s happening. In code, we make “8 parallel heads” by introducing another dimension. For example, vectors of shape 30 (max sequence length) x 10 (batch size) x 512 (embedding size) would form query key and value tensors of shape 30 x 10 x 8 x 64. So there is a new dimension for heads that essentially acts like another batch dimension for more parallel computation
Really want to hear indian accent of thai guy 😅😂