Видео

Thanks! They key and query matrices have the same dimensions but they contain different values. That's because they are created by applying a different fully connected layers (one for the key, one for the query) to the tokens of the previous transformer layer. The value matrix is created with yet another fully connected layer. Hope that helps.

Great. Happy to learn this new information. Does this "fully connected layer" refer to a Neural Network, and is that what happens in the "Linear Transformation". If so, where do the weights and biases for that NN come from? And if a loss function is utilized, what does it use to determine whether the results are going in the good or bad direction. Thank you VERY much

@@19AKS58 when I say "fully connected layer" that's interchangeable with "linear layer" but the terminology is more precise (in my opinion). And yes this is a layer of a neural network. It's just a matrix with learnable parameters (the "weights") and a vector with learnable parameters (the "bias"). The choice of loss function, and target is totally independent of everything I discuss in this video. In image classification, one might use cross entropy loss with the ground truth labels of the dataset as targets. In natural language processing, one might use cross entropy loss with the ground truth token labels for "next token prediction".

This is how I like to learn and understand things, unfortunately my deep learning course is going so fast, I'll never be able to catch up if I go like this, maybe I'll revisit this stuff someday in depth

Hmm, so you're saying that incorporating an offset is essential? What if your data is already zero centered?

@@swazza9999 Even if the data is zero centered, wouldn't dividing every datapoint by a scalar value "distort" their distances to zero? Because zero doesn't "move" when divided, while every other value does. Min-max scaling "moves" every value including zero (unless the minimum is zero & the transformation range is 0-max) so it shouldn't have the same problem. Either way, I've never seen scaling (by dividing with the standard deviation or otherwise) without centering recommended before for neural network inputs. I might be wrong.

@@activatealmonds yeah I dropped centering from the vid not because I think that it's recommended not to use it, but because I think it's an unnecessary detail for the purposes of my argument. Like for example just consider that we're training an ML model to guess the sex of a baby based on their birth weight and length. We can center the data with respect to both of these variables, but if the units of weight are grams, and the units of length are meters, you're still going to have a situation where incrementing the weight by one unit is far less informative than incrementing the length by one unit.

I'd say this video covers the "hardest" bit. Activations are easy to incorporate because they typically act on one neuron at a time, so there's no index tracking to do, it's just i -> i. In fact, this video already shows how to deal with activations. If you look at my final expressions they still have dL/dy in them. I left the loss function general, leaving you to fill that in depending on which specific loss function you are using. But what if this was a layer somewhere in the middle of the neural network, and I was really calculating da/dx, da/dW and da/db (a is for "activation"). All the math in the video would be exactly the same, but instead of dL/dy in the final expressions, you'd have da/dy. So you see, incorporating an activation just amounts to incorporating its derivative into the chain rule, and since the activation is a scalar to scalar function, there's no matrix multiplication. It's just a scalar. For example, what if my activation is a(y) = y**2 / 2. Then da/dy = y. Then you just plug y in where I have all the dL/dy in the video. If it's still not clear to you after reading this, I'd encourage you to just sit with it for some time and try to work through it. I feel like you are close.

As for your question about the dimensions d_q, d_k and d_v: d_q = d_k so sometimes it's just written d_qk. Whereas d_v can be different. You can see the first part is true just by looking at the expression QK.T where we know Q is Nxd_q and K is Nxd_k. So we must have d_q = d_q for the matrix multiplication to work. (PS: it is AlexAI, I just keep forgetting to switch from my personal YT account when I respond to these).

hahaha. Yeah my Russian Blue had diarrhea for like a year but we finally solved it.

Honestly, that would be too broad. There's nothing fundamentally different about how that happens in this particular configuration vs any other neural network configuration. They are just matrix operations in a series of many, and they get their own partial derivatives just like any other weights. Is there something more to your question that I'm missing?

@@alex-ai7517 ok fair enough…I’m just befuddled as to where the transformer back propagates. I THINK it’s in these Weight matrices, but I don’t see how…I would think that Wq approx equals Wk so that q dot k approx equals 1. But that can’t happen with random W_a and W_k.

@@johnryan8645 yeah I think you may be mixing up different ideas here. There's no need to attempt to make Wq~Wk to satisfy yourself that backprop works. I don't know how to set you on the right path though unfortunately (because I'm not sure where you took a wrong turn). I can just say maybe you need to backtrack a bit and take a different path.

Just saw your edit but drafter this response before. So here it is: Attention is an explicit form of interaction between activations in the same layer. Whereas with an MLP or CNN you only get "flow of information" from one layer to the next. It's not necessarily "better" to use a transformer. A CNN learns to parse images with less data and training time than a transformer, but the transformer can catch up and surpass the CNN because it uses a more general mechanism (though this is not totally clear cut IMO). Also, if you're looking for the motivation behind the transformer architecture it might make more sense to consider it in the light of natural language processing where it originated. If you follow the evolution of RNNs, LSTMs and GRUs the transformer will feel like a natural win. As for why this specific key, query, value mechanism, you're totally right that it's arbitrary (to a degree). These just happened to be the details that were settled on and carried forward in follow up papers/works. The main point is that there's an explicit interaction between activations (rather than just activations * weights -> activations). In fact a transformer is effectively a fully connected graph neural network (GNN). And GNN mechanisms come in a wide variety of flavors. You can look them up. Even in just the "attention" literature there are 3 main mechanisms (you can probably find them in the 2017 attention is all you need paper if I recall correctly). Hope that helps.

@@alex-ai7517 This is great, thanks! In the second paragraph you mention that a transformer is effectively a fully connected GNN. Is that because graph neural networks normalize each list of connected node's values when propagating? (I don't know much about GNNs but hopefully that makes sense.) But ofc that they aren't connecting to all other nodes per se. Also "Attention is an explicit form of interaction between activations in the same layer ..." is very helpful for understanding. It's like horizontal info flow + vertical info flow instead of just vertical.

@@joeystenbeck6697 to respond to your question there, no I wasn't thinking of normalisation exactly. That would be more specific than I wanted to be. I literally just mean that the tokens in the transformer framework form the nodes of a graph, and since the self attention is global (as an aside, with some vision transformers it actually isn't) each node is connected to all others.

@@swazza9999 Ah gotcha, that makes sense

Yep. I suppose there are some other places in my derivation where I haven't been totally explicit about the conditions. For example, I often drop the pi once I have pinned down an action. But that's just because I know I'm not going to need to talk about it again and it's implicit. Thanks for raising this.