Видео

I do see that they do this in the paper, but I'm confused why this wasn't the approach they present on in the paper; the math doesn't require that weird softmax->sigmoid hack to make inference work, and it performs better, so why didn't they include more exploration of that? Plus they have two different variations on MoDE but don't specify which they used for the results they show... it feels strange that the most persuasive thing in the paper for me is such an afterthought.

I do think the proposed method is quite hacky with the sigmoid trick they do. MoDE seems much more natural and in Fig (7) it does better than the standard approach they take. Wish they went more in depth into MoDE. Maybe they're working on a future paper with MoDE and that's why they barely included it?

​@@gabrielmongarasI agree. I had a mental image of them thinking of it at the last moment before publishing and not wanting to throw out all their previous work! But it's maybe more charitable to imagine they wanted to be able to compare against a baseline dense attention layer instead of just a MoE layer.

That's probably more likely lol. Found that it worked better and just put it in as a side note since they would have to redo a lot of parts of the paper.

By "independent of sequence length" I was moreso talking about the inference time. That is, it could require the same number of step to generate a sequence of length 5 as it does for a sequence of length 100, theoretically. You're right about needed to know the sequence length before going in, that's how you initialize the noise for the model to process. It seems the only way to do conditional sampling is by putting tokens in arbitrary locations. One can think of this as starting from an intermediate step in the diffusion process, thus imposing a conditional generation on that token. So it'll likely generate a coherant sequence around whatever token you place in whatever location you place it. I would think of it more as a soft position. Like you have to assign the position of the seeded words an integer value, but the sentence will be built around it in some way. It doesn't really need to be the location of the word in the sentence due to the model building around the seeded word. As for wanting just the word "cat", I suppose one could increase the likelihood of said word being generated, but beside that I cannot think of a way unless you know the general location of the word. A much less mathematical and satisfying method is that you could also prompt it by making the first few tokens as "the following is about a cat: ". In the future, conditional generation based off a prompt could lead to much better results.

The goal of the internal RNN is to learn a compression task. If we use low-rank projections, it forces the model to choose what information to remember rather than attempting to keep all information. By down projecting, the model can control the corruption of the input, x_t, in some way for the internal loop to learn, and thus store in its hidden state. Intuitively, the K down projection corrupts the input x_t and the V down projection informs the inner loop what to remember.

@@gabrielmongaras Thanks for the reply! I get the intuition of down projection. But my following question is: If K_t and V_t have the dim e, and e is smaller than d which is the dim of x_t. Then f(_, W_t) is a mapping from dim e to dim e. Then z_t = f(Q_t, W_t) also has dim e. But to maintain the same size of each RNN layer, z_t is supposed to have the same dim as x_t. How to resolve this difference? In addition, why it seems that z_t = f(θ_Q * x_t ;W_t) has the same dimetion as x_t in Equation 5 of the paper: "Lastly, the training view θ_K * x_t has fewer dimensions than x_t , so we can no longer use the output rule in Equation 1. The simplest solution is to create a test view θ_Q * x_t , and change our output rule to: z_t = f(θ_Q * x_t ;W_t)." ?

This is one thing I think is a very ambiguous in the paper. If θ_Q * x is of dimension e and W_t is of dimension e x e, the output z_t would be of dimension e as shown in equation (5). Looking at the psuedocode, you will also get that the output, z_t, is of dimension e, not d. I remember looking at their code when the paper came out and not getting a good idea of how they got around this issue. Taking another scan of their code, it looks like they get around this issue by setting e = d which is really lazy. If you change the width where e < d, the code doesn't work. In practice, I would imagine making the output projection an e-->d projection rather than the usual d-->d projection. Now that you point it out, I am curious if the experiments they ran were even on e<d. I know that RWKV-7 uses a similar idea and that it has good results. Taking a look at their code, I think they also have e = d. Perhaps the down projection isn't needed, though I would imagine this could run into issues since the model can make the corruption trivial, not updating the hidden state, but perhaps this is also a good thing if a token should not be stored at all. Nonetheless they should've mentioned this in the paper/code.

I agree that this is basically a less impressive version of TokenFormer, however I think there are some parts that add value still such as the low rank split and the way they shard across devices. I think the main thing is, as you said, the results between the two papers are consistent which shows that this idea works! (though it's just an MLP with a subset number of output functions). I really don't like how both these papers formulate this block as an attention block since it's an MLP. Looking at it from an MLP point of view, it makes a lot of sense GELU would be better than softmax as softmax is quite bad for hidden activation functions. I also agree, I think the reason they cannot replace all layers is due to the shared weights! Looking at the formulas, there's a connection between an MLP (dynamic queries, no dynamic values), MoE (dynamic queries, dynamic values, subset keys/values), this/token former (dynamic queries, no dynamic values, subset keys/values), and attention (dynamic all, no subsetting). I'm thinking there's some combination of these block types that would work best.

@@gabrielmongaras I agree, although I expected a more significant contribution given that this paper was from FAIR. If it were from a university, then I would hold it to a lower standard. I don't think there is anything wrong with unifying the concepts as attention, although it would probably be more accurate for the authors to refer them to them as KV stores (attention is a dynamic KV lookup). This similarity breaks when considering GLU though. The interesting thing about softmax, is that even though it's not a good hidden activation function, it still works. In fact, you can remove the FFNs from a transformer and it will still "work", just not nearly as well (I accidentally made this mistake with a ViT - a typo). Regarding combining the blocks, the keys will probably be in finding a hardware aware approach which works well for training, metrics, and throughput / latency. In diffusion for example, time MoE is a smarter choice than spatial MoE. But sharing a fixed spatial knowledge bank across spatial and time dims could work (assuming there's an upper bound for a fixed resolution). It's harder to make such comparisons with AR decoders though, since the sequence length is unbounded.

@@gabrielmongaras I am really interested in the connection you point out (and I first really understood from watching your tokenformer video) and even though the pattention is "just" an MLP, it is somehow a really interesting way to think of it. Are you aware of work that explores these connections more? Like is there some uberformula that can capture all the variants as special cases? Since there is this dichotomy of dynamic vs static keys and values between MLP and attention, it then seems like you could (in the uberformula) maybe do a mixture like 75% dynamic and 25% static. Maybe a first pass is sigma((QAx + Q')(KBx + K'))(VCx + V') where the primes are the static values and A,B,C are selecting a subset of tokens. The static matrices might be better combined with concat than + but with + they would be zero padded either in the hidden or token dimension appropriately. Hmm idk that all looks wrong but I'll leave it for fun. Thanks for making these paper videos, they are an awesome resource

I do not know of a current work that looks at the connections. Most seem to say they take the attention architecture and manipulate it, but both this paper and the tokenformer one make it seem like a new architecture block rather than showing it's mathematically equivalent to an MLP. I suppose the uberformula is a three-way matrix multiplication: O = f(Q(x) @ K(x).mT) @ V(x). Attention has all three as a function of the input: Q(x) = W_Q @ x, K(x) = W_K @ x, V(x) = W_V @ x with f = softmax Standard MoE has static keys (router): K(x) = W_K with f = softmax and tokens routed to different V functions depending on the output of f(Q(X) @ K) A transformer MLP (no GLU) has the keys and values static: K(x) = W_K, V(x) = W_V with f = silu If we add GLU, then we would get something a little different, but as @hjups mentioned above, GLU is not always an improvement. Attention is good for routing info between tokens why MLPs are more useful for fact retrieval an dimension interactions. Since there is this notion of a somewhat unified formula, I am curious if there is a mix of blocks of this type that is optimal rather than the normal Attn, MLP, Attn, MLP, ...

​@@gabrielmongaras related "Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View" and "Improving Transformer Models by Reordering their Sublayers" Thinking about this again, another way to phrase the mixture I was thinking would be like tokenformer, but instead of only using static "tokens" (rows of W_K) for the cross attention, concat W_K' to X (so the sequence length goes from T to T+N where W_K has N rows) and likewise for W_V. All the queries are dependent on X. The first T keys are dependent on X, but the last N are not. The first T values are dependent on X, but the last N are not. The attention mask would leave the last N columns of the T+N unmasked for all tokens. With N=0 and standard attention mask, you recover self attention. With nonzero N and an attention mask with the (T,T) left hand part all zeroed/-inf, you recover an MLP. With a standard attention mask and nonzero N, you get (as you vary the ratio T/N) some combination of the two f((W_Q @ x) @ concat(W_K @ x, W_K')) @ concat(V @ x, W_V') This is only a mix of attn and MLP, to get a mixture with MoE you would need static keys and nonstatic values, like f((W_Q @ x) @ concat(W_K @ x, W_K', W_K'')) @ concat(V @ x, W_V', W_V'' @ x) so then the only pattern missing between keys/values of dynamic/dynamic static/static static/dynamic is dynamic/static so then f((W_Q @ x) @ concat(W_K @ x, W_K', W_K'', W_K''' @ x)) @ concat(V @ x, W_V', W_V'' @ x, W_V''') (transpose the K term appropriately in all of the above)

I think it would be cool to see how a model like that performs! Having learnable dynamic compute on a hidden state or a hidden state that updates some larger memory and extracts from that larger memory. I know the latest RWKV is using the idea from "Learning to (learn at test time)" so this approach could be promising.

@gabrielmongaras ooh I haven't read RWKV yet, I'll have to take a look.

@@nathanwatts5720 I don't think there's a modern RWKV paper at the moment. The old versions of RWKV are mid but the new versions seem a little interesting. However (to the best of my knowledge) we only have the website and code to go off of. Would love to cover RWKV when a new paper comes out! www.rwkv.com/

I mainly read papers from beginning to end the way they're written. 1. read the abstract, see what's going on. I usually have a somewhat decent idea of how they may be doing the things in the abstract 2. look at related work if there's something I'm not too familiar with, but I mostly skip this part. 3. Look for picture and formulas. Every time there's an important formula, I ether find a picture related to it or draw one out. I also usually understand how things are working via how the shapes are manipulated so I try to write those out. Any big formulas that look like a mess, I just break down from the inside-out (for example, break down inside a summation, then see what the big picture is including the summation) 4. If there's code I will look at that. Sometimes it can clear up ambiguous problems left in the paper 5. Finally I look at the results (which I may look at first to see if the paper is even worth reading). Mainly how much did they scale their architecture and what datasets. Ask questions about the results. For example is the paper trying to deceive you with axes in some way? Are the dataset actually meaningful? Hope this helps!

@@gabrielmongaras Thanks a lot for your detailed response. This will definitely help. Your work is unique for me.

I think I forgot to add it to the comments. Here's the official repo for the DiT paper: github.com/facebookresearch/DiT

Thanks for the explanation, but I still dont get it. We train "global, big gpt-like transformer" to respresent i-th patch, but when inferencing trying to predict i+1 patch?

Think of the big global transformer as a "next token predictor" but instead of that token being decoded into the next word/word piece, it's decoded into a sequence of bytes. The encoder and decoder are similar to the embedding and unembedding layer to encode/decode a token.

You're right that naive decoding byte-by-byte would run into very slow inference. I added a comment to the video on a detailed way inference is run. I think that shows that inference is pretty fast since we do next patch prediction on the latent transformer, which is decoded into many bytes. If, on average, the token size is larger than the patch size, it will be probably more efficient than a normal token-wise transformer (since the encoder/decoder are relatively lightweight)

@@gabrielmongaras I think this is what is happening, once big transformer produces next patch, then small decoder produces next byte, you autoregress it to small encoder, and gives its output to small decoder, skipping big transfomer, till bytes of produced patch is exhausted in some sense.

Yep! The way I think about it is the latent transformer is the "normal" transformer in a text generation sense. The encoder and decoder act like the embedding and unembedding matrices. The transformer is sort of a "next patch" predictor. The ith patch is transformed into a latent used to predict the (i+1)th patch, like how in a normal transformer the ith token is transformed into a latent used to predict the next token. However, the "next patch latent" is used to autoregressively predict next bytes (via the encoder and decoder), like you mention, until it reaches a high entropy byte. From my current understanding, this means the ith patch is transformed into a latent that predicts the bytes of the (i+1)th patch and the first byte of the (i+2)th patch as you don't know a patch has ended until you hit this high entropy prediction.

Since the decoder is sending information from patches to tokens instead of tokens to patches, you would want to mask it such that every token gets information from the patch that contains that token. This would be the transpose of the encoder mask.

I agree it's the best explanation

Classifier Free Guidance. Mainly a method of getting good quality out of diffusion models. It basically emphasizes the given class or text some amount, c, over the null prediction the model would've made without the class information, thus making your image more representative of the given class. arxiv.org/abs/2207.12598 I think I talked about it in my SD 3 video if you're curious. There's a lot of other resources on it too!

It depends on if you do a forward step or an inversion step in the opposite direction using the formulas above. Both come from x_t though.

Thanks for the further analysis!! I need to check out that ICLR paper. Another paper to add to the to read pile! As for learning the frequencies, I was moreso thinking they could be learned to study what frequencies are preferred. Perhaps the model would bias them in some way. I did a small test case and they were all over the place which reinforces the idea that all frequencies are needed, like you said, the it's like a basis spanning the entire space.

It just serves as an easy way to tokenize the image somehow since we want to do autoregression. An easy way to make a model autoregressive on images is to use vector quanitzation, though it does reduce quality as you're reducing a continuous signal to some discrete set of tokens. It's probably far from the best way to model an image though.

I think it's ultimately an empirical choice, the model predicts next discrete token as opposed to a continuous embedding. so may benefit from predicting a distribution of token likelihoods in itself rather than being forced to output a singular continuous prediction.

I'm using a Samsung tablet and just the default notes app. I tried out a few other note apps, but this one is my favorite since it's free and has all the features I need. For editing audio I use audacity and for small video editing I use shotcut.

Oh, they actually did train transformer from scratch: "Table 6 compares Transformer models trained from scratch and Tokenformer trained by parameter reusing with varying amounts of seen tokens during training. It is evident that Transformers trained with the same number of seen tokens do not reach the performance level of Tokenformer with parameter reusing" Even if modern MLP would improve transformer results, training costs still seems to be much better which is really impressive

I think most of the results only look impressive because they slightly scale the model above the baseline. From the experiments I've done, it seems like the activation they develop is worse than normal GeLU and moving params from the MLP into the Q K V O projections performs the same given the same # of params.