I found no explanation for this anywhere and when reading the paper missed the detail that each tokens positional encoding consists of multiple values (calculated by different sine functions). Your explanation and visual representation finally made me understand! Fourier transforms are genius and I'm amazed in how many different areas they show up.
This is the most intuitive explanation of the positional encoding I have come across. Everyone out there explain how to do it, even with code, but not the why which is more important. Keep this up. You have earned my subscription.
I was browsing through some channels after first stopping on Sean Cannells and I noticed your channel. You got a great little channel building up here. I decided to drop by and show some support. Keep up the great content and I hope you keep posting :)
What a great video, I'm studying my Ph.D. at NLU, so this came in very handy. Thank you very much and greetings from Ensenada Baja California Mexico :D!
@@AICoffeeBreak this method relocates the embeddings in a specific direction in the embeddings space, so that new position in the relevant embedding cluster have "another" meaning to (say there is another instance of the same word later) other words of "same kind". But that place should be reserved other semantics, else the space is literally filled with "second position" coffee and "tenth position" me, "third position" good etc etc. This can go wrong in soooo many ways. Don't get me wrong, I am a clear cut "Chinese Room Experiment" guy, I don't think you can translate "he is a good doctor" before imagining an iconic low resolution male doctor and recall a memory of satisfaction and admiration of consumatory reward, but again, the "he" in "he did again" and "man, he did it again" should literally have the same representation in the network to start discussing things.
You are entirely right. I was short in my comment because I commented on the same issue in Cristian Garcia's comment. But there is no way you would have seen it, so I will copy paste it here: 😅 "Concatenating has the luxury of extra, exclusive dimensions dedicated to positional encoding with the upside of avoiding mixing up semantic and positional information. The downside is, you can afford those extra dimensions only if you have capacity to spare. So adding the positional embeddings to initial vector representations saves some capacity by using it for both semantic and positional information, but with the danger of mixing these up if there is no careful tuning on this (for tuning, think about the division by 10000 in the sine formula in "attention is all you need")."
And you correctly read between the lines, because this was not explicitly mentioned in the video. In the video I explained what an act of balance it is between semantic and positional information, but you identified the solution: If adding them up causes such trouble, then... let's don't! 😂
@@AICoffeeBreak new to AI, but what do you mean by the word "capacity"? Do you mean RAM? Do you mean that if we concat positional encodings to original vector instead of adding, it will take up more RAM/memory and therefore make the training process slow?
In another video I've seen, apparently it doesn't matter if positional embedding are learned or static. It seems as thiugh the rest of the model makes accurate deductions regardless. This is why I was not surprised that Fourier transforms seem to work nearly as well as self attention.
This is the best and most intuitive explanation of positional embeddings. THANKYOU so much for this video. btw: what software did you use to create these lovely animations?
Thanks, glad you like it! . For everything but Ms. Coffee Bean, I use the good old Powerpoint (morph and redraw functionality FTW). The rest is Adobe Premiere or kdenlive (video editing software).
Thanks Letitia! A vid on relative positional embeddings would be nice 😃 Implementations seems a bit involved so I've never used them in my toy examples.
Concatenating has the luxury of extra, exclusive dimensions dedicated to positional encoding with the upside of avoiding mixing up semantic and positional information. The downside is, you can have those extra dimensions only if you have capacity to spare. So adding the positional embeddings to initial vector representations saves some capacity by using it for both semantic and positional information with the danger of mixing these up if there is no careful tuning on this (for tuning, think about the division by 10000 in the sine formula in "attention is all you need").
Gemini: This video is about positional embeddings in transformers. The video starts with an explanation of why positional embeddings are important. Transformers are a type of neural network that has become very popular for machine learning tasks, especially when there is a lot of data to train on. However, transformers do not process information in the order that it is given. This can be a problem for tasks where the order of the data is important, such as language translation. Positional embeddings are a way of adding information about the order of the data to the transformer. The video then goes on to explain how positional embeddings work. Positional embeddings are vectors that are added to the input vectors of the transformer. These vectors encode the position of each element in the sequence. The way that positional embeddings are created is important. The embeddings need to be unique for each position, but they also need to be small enough that they do not overwhelm the signal from the original data. The video concludes by discussing some of the different ways that positional embeddings can be created. The most common way is to use sine and cosine functions. These functions can be used to create embeddings that are both unique and small. The video also mentions that there are other ways to create positional embeddings, and that these methods may be more appropriate for some types of data.░
thanks for explanation. but one question, do we also need to use this positional encoding for other inputs, like, for security or anomaly detection datasets which consists of only observation states with corresponding timestamps?
If the timestamps are already in the input vector, then no. But if you have the observation states only, you need to give the order (the timestamps) to the transformer model, otherwise it would be completely ignorant of the fact that the states it processes, are actually a sequence. :)
I couldn't find a satisfying explanation anywhere. This video finally made me understand things in a bit more detail especially the use of sine and cosine functions across multiple dimensions. Thank you! You're awesome.
Hi Letitia. Thank you so much for your wonderful video! Your explanations are more intuitive than almost anything else I've seen on the internet. Could you also do a video on how positional encoding works for images, specifically? I assume they are different from text because images do not have the sequential pattern text data have. Thanks!
Thanks for the suggestion. I do not think I will come to do this in the next few months. But the idea of image position embeddings is that those representations are most often learned. The gist of it is to divide the image into patches, let's say 9. And then to number them from 1 to 9 (from the top-left to bottom right). Then let gradient descent learn better representations of these addresses.
It's complicated because it's super high dimensional. But on a high level, adding a vector to another vector, shifts its position in each dimension a bit. We add always the same vector to the first position, so the transformer learns that the first position is always shifted a bit around in that way. Maybe it is not recognisable in all dimensions all the time, but it has 768 dimensions to get the pattern. You should not underestimate neural networks that are correlation discovery machines.
This was awesome! I don't have a complete understanding but it definitely pushed me to the side of understanding. Did you make a video about relative positions?
Miss Coffee Bean, excellent visualization. Is it also possible that gradient descent will cause some embeddings dimensions to become insignificant to allow position embedding to dominate these dimensions? if so, then adding position could be thought of as both shifting the representation for words like 'king' and/or forcing the representation of 'king' to fewer dimensions to allow for an orthogonal position or a combo of both. I always thought it was the latter, but have never seen a study on the subject.
The vectors could be longer or shorter than that. 512 was just an example from the most common architecture back then (BERT base). The longer the vectors, the more parameters the model has. The larger the model, the better their performance, but also the slower their forward passes.
I found no explanation for this anywhere and when reading the paper missed the detail that each tokens positional encoding consists of multiple values (calculated by different sine functions). Your explanation and visual representation finally made me understand! Fourier transforms are genius and I'm amazed in how many different areas they show up.
+1 for more vids on positional encodings.
Great stuff :) Would love to see more of that, especially for images or geometry!
Noted!
Great explanation !! And Yes Yes Yes.
+ 1 for video on relative positional representations!
This is the most intuitive explanation of the positional encoding I have come across. Everyone out there explain how to do it, even with code, but not the why which is more important.
Keep this up. You have earned my subscription.
Wow, thanks a lot! :)
Very Good explanation. Amazing.
Thank You very much madam
Thanks for the explanation
Interested to know about rotary position embedding
love how the "Attention is all you need" paper appears with an epic-like bgm
It wasn't on purpose, but it is funny -- in hindsight 😅🤣
This is a great video! Just found your channel and glad I did, instantly subscribed :)
Thank you so much, you honour us!
I was browsing through some channels after first stopping on Sean Cannells and I noticed your channel. You got a great little channel building up here. I decided to drop by and show some support. Keep up the great content and I hope you keep posting :)
Thanks for passing by and for the comment! I appreciate it!
Sooo good!
Great explanation of the intuition of positional encodings used in the Transformer!
Really Thank you! Your video is just amazing!
What a great video, I'm studying my Ph.D. at NLU, so this came in very handy. Thank you very much and greetings from Ensenada Baja California Mexico :D!
Thanks, thanks for visiting from so far away! Greetings from Heidelberg Germany! 👋
This video is a clear explaination of why you shouldn't add your positional encoding but concat.
Extra dimensions dedicated exclusively to encode position! Sure, but only if you have some extra to share. 😅
@@AICoffeeBreak this method relocates the embeddings in a specific direction in the embeddings space, so that new position in the relevant embedding cluster have "another" meaning to (say there is another instance of the same word later) other words of "same kind". But that place should be reserved other semantics, else the space is literally filled with "second position" coffee and "tenth position" me, "third position" good etc etc. This can go wrong in soooo many ways. Don't get me wrong, I am a clear cut "Chinese Room Experiment" guy, I don't think you can translate "he is a good doctor" before imagining an iconic low resolution male doctor and recall a memory of satisfaction and admiration of consumatory reward, but again, the "he" in "he did again" and "man, he did it again" should literally have the same representation in the network to start discussing things.
You are entirely right. I was short in my comment because I commented on the same issue in Cristian Garcia's comment. But there is no way you would have seen it, so I will copy paste it here: 😅
"Concatenating has the luxury of extra, exclusive dimensions dedicated to positional encoding with the upside of avoiding mixing up semantic and positional information. The downside is, you can afford those extra dimensions only if you have capacity to spare.
So adding the positional embeddings to initial vector representations saves some capacity by using it for both semantic and positional information, but with the danger of mixing these up if there is no careful tuning on this (for tuning, think about the division by 10000 in the sine formula in "attention is all you need")."
And you correctly read between the lines, because this was not explicitly mentioned in the video. In the video I explained what an act of balance it is between semantic and positional information, but you identified the solution: If adding them up causes such trouble, then... let's don't! 😂
@@AICoffeeBreak new to AI, but what do you mean by the word "capacity"? Do you mean RAM? Do you mean that if we concat positional encodings to original vector instead of adding, it will take up more RAM/memory and therefore make the training process slow?
great explanation :)
Thanks! 😁
Great content! Can you do a video about Tabnet please? :)
great explanation of positional embeddings. Just what I need.
Great explanation - thanks very much for sharing this.
Glad you enjoyed it!
amazing explanation!!! I have been looking for a clear explanation on how the positional encodings actually work and this really helped! thank you :)
Letitia's Explanation Is All You Need!
And comments like these!
+1
this helped me so much!! thank you!!!
Great video, many thanks!
Dang it, i learned something and my morning coffee isn't even finished.
Its going to be one of *those* days.
Sound like a good day to me! 😅
Whish you a fruitful day!
In another video I've seen, apparently it doesn't matter if positional embedding are learned or static. It seems as thiugh the rest of the model makes accurate deductions regardless.
This is why I was not surprised that Fourier transforms seem to work nearly as well as self attention.
COuld you please elaborate using an example maybe?
Multi Dimensional Spurious Corelation Identifying Beast 🔥🔥
That's what I am calling transformers from now on
Just an amazing explanation ...
Thanks, as always, for the great explanation!
It was Ms. Coffee Bean's pleasure! 😅
Thank you for the explanation! Can you please make a video on Positional information in Gated Positional Self Attention in ConViT paper?
Noted!
Explained really well, thank you 😊
Thanks, so nice of you!
This is the best and most intuitive explanation of positional embeddings. THANKYOU so much for this video. btw: what software did you use to create these lovely animations?
Thanks, glad you like it! . For everything but Ms. Coffee Bean, I use the good old Powerpoint (morph and redraw functionality FTW). The rest is Adobe Premiere or kdenlive (video editing software).
Nice explanation
Thanks for appreciating!
Great explanation - It might be the best. I think I finally figured it out. I highly appreciate it.
Amazing content. After seeing this all the articles and research papers makes sense.
Super clear and amazing (as always) explanation of sines and cosines positional embeddings! 🙌
Thanks! Always happy when you visit!
Super intuitive explanation, nice!
i've read numerous articles explaining the positional embedding so far.. however, it is surely the greatest & clearest ever
Thanks a lot! Glad we could help. :)
Excellent! Great work!
So helpful! Thank you a lot!
Awesome, thanks!
I had my morning coffee with this and will make an habit!
What a cool comment! Thanks.
You've solid understanding of Mathematics of Signal Processing
Hey, Great Explanation :). Love to see more videos.
Glad you found us! 😁
BEST AI channel on RUclips!!!!!
Lifesaver! Thank you for the explanation.
I feel lucky to have found your channel. Simply amazing ❤️
Happy to have you here!
This was so insightful!
Thanks Letitia! A vid on relative positional embeddings would be nice 😃
Implementations seems a bit involved so I've never used them in my toy examples.
Regarding this topic, I've seen positional embeddings sometimes being added and sometimes being concatenated with no real justification for either 😐
Concatenating has the luxury of extra, exclusive dimensions dedicated to positional encoding with the upside of avoiding mixing up semantic and positional information. The downside is, you can have those extra dimensions only if you have capacity to spare.
So adding the positional embeddings to initial vector representations saves some capacity by using it for both semantic and positional information with the danger of mixing these up if there is no careful tuning on this (for tuning, think about the division by 10000 in the sine formula in "attention is all you need").
Nice madam ...what a video !... outstanding
🙏
you are a genius!!
Gemini: This video is about positional embeddings in transformers.
The video starts with an explanation of why positional embeddings are important. Transformers are a type of neural network that has become very popular for machine learning tasks, especially when there is a lot of data to train on. However, transformers do not process information in the order that it is given. This can be a problem for tasks where the order of the data is important, such as language translation. Positional embeddings are a way of adding information about the order of the data to the transformer.
The video then goes on to explain how positional embeddings work. Positional embeddings are vectors that are added to the input vectors of the transformer. These vectors encode the position of each element in the sequence. The way that positional embeddings are created is important. The embeddings need to be unique for each position, but they also need to be small enough that they do not overwhelm the signal from the original data.
The video concludes by discussing some of the different ways that positional embeddings can be created. The most common way is to use sine and cosine functions. These functions can be used to create embeddings that are both unique and small. The video also mentions that there are other ways to create positional embeddings, and that these methods may be more appropriate for some types of data.░
Awesome. Thank you for making this great explanation. I highly appreciate it.
great explanation. really loved it. Thank you
Amazing explanation! Thank you so much !
Very well explained! Thank you so much!
"A multi-dimensional, spurious correlation identifying beast..." ... wow. Douglas Adams would be proud of that.
Hope so. 😁
Very intuitive. I know there is sine cosine positional encoding but it is actually effective that I got it here..👍👍
This is an amazing explanation! Tku!!!🤓🥳🤩
Really great!
Thanks, your videos helped the most
This explanation is incredible
thanks for explanation. but one question, do we also need to use this positional encoding for other inputs, like, for security or anomaly detection datasets which consists of only observation states with corresponding timestamps?
If the timestamps are already in the input vector, then no.
But if you have the observation states only, you need to give the order (the timestamps) to the transformer model, otherwise it would be completely ignorant of the fact that the states it processes, are actually a sequence. :)
Thanks
Nice explanation! I’d love to hear more about multidimensional and learned position encodings
Noted!
Thanks Letitia, great explanation
Bro
Where have you been hiding all this time?
This is next level explaining
I couldn't find a satisfying explanation anywhere. This video finally made me understand things in a bit more detail especially the use of sine and cosine functions across multiple dimensions.
Thank you! You're awesome.
Make Ms. Coffee Bean spill the beans on positional embeddings for images
😂
These videos are priceless, thank you!
I really enjoy your videos 😄!
I would like a video about using transformers with time-series data.
Noted!
Awesome video, thanks :)
OMG, such a amazing video to explain Positional Embedding....
thank you!
Just watched this again for a refresher; thee best video out there on the subject!
Wow, thanks!
Amazing! This is the best explanation for positional encodings period. Subscribed!!
Thank you!
Most insightful explanation I have found on this subject so far. I was looking for it for days... Thank you! Keep going, you rock!
Thank you a lot! Also for the super thanks!
Hi Letitia. Thank you so much for your wonderful video! Your explanations are more intuitive than almost anything else I've seen on the internet. Could you also do a video on how positional encoding works for images, specifically? I assume they are different from text because images do not have the sequential pattern text data have. Thanks!
Thanks for the suggestion. I do not think I will come to do this in the next few months. But the idea of image position embeddings is that those representations are most often learned. The gist of it is to divide the image into patches, let's say 9. And then to number them from 1 to 9 (from the top-left to bottom right). Then let gradient descent learn better representations of these addresses.
amazing video - rockin!
🤘
Thanks for the explanation! But why after we add position vector to the word vector, the position information is still preserved??
It's complicated because it's super high dimensional. But on a high level, adding a vector to another vector, shifts its position in each dimension a bit. We add always the same vector to the first position, so the transformer learns that the first position is always shifted a bit around in that way. Maybe it is not recognisable in all dimensions all the time, but it has 768 dimensions to get the pattern. You should not underestimate neural networks that are correlation discovery machines.
Probably the clearest explanation for positional encoding:D
Thanks!
This was awesome! I don't have a complete understanding but it definitely pushed me to the side of understanding. Did you make a video about relative positions?
Yes, I did! ruclips.net/video/DwaBQbqh5aE/видео.html
This was very intuitive, thank you very much! Needed this, please keep making videos
Your explanation was great! I got everything I wanted to know about positional embedding. thank you:)
Awesome, thanks for the visit! ☺️
so helpful, appreciate it!
Very good explanation!! Thanks for this 🙏🙏
Thank You for your wonderful message!
I loved your simple and explicit explanation. You've earned a sub and like!
Thanks, see you again soon!
before wathcing this I always used to be confused about position encoding.
Wow, this is so great to hear!
I finally understand the positinal encoding! Thanks
Happy to help. :)
Loved your explanation, thank you for this video!
Glad you like it!
Miss Coffee Bean, excellent visualization. Is it also possible that gradient descent will cause some embeddings dimensions to become insignificant to allow position embedding to dominate these dimensions? if so, then adding position could be thought of as both shifting the representation for words like 'king' and/or forcing the representation of 'king' to fewer dimensions to allow for an orthogonal position or a combo of both. I always thought it was the latter, but have never seen a study on the subject.
Brilliant video. This was the best explanation of positional encodings I have seen. It helped a TON!!!
I also make AI videos. I am more into the image side(convolutions and pooling) so it was great to see more AI educators.
Wow, thanks so much! This is so well explained!!
Glad it helped. :)
Why are the words initially represented in as a vector of length 512 ?
The vectors could be longer or shorter than that. 512 was just an example from the most common architecture back then (BERT base). The longer the vectors, the more parameters the model has. The larger the model, the better their performance, but also the slower their forward passes.
thanks for such explanation. awesome u!
Glad you liked it! :)
love your explanations !!!
🤗
the most brilliant positional encoding explanation EVER that was GOD Level explanation
Haha, thanks.
amazing stuff! so clear and intuitive, exactly what I was looking for :)
Thanks for watching and appreciating! 😊
This is probably the best explanation of this topic on RUclips! Great work!
Thanks so much! 😊
Good explanation.
I love you coffee! You nice video sovled my problem!
Glad I could help!