I couldn't find anywhere why creators of transformer decied to encode the positions in this way and last minute of your video was what I was looking for. Thanks for good explanation
It seems the addendum is a 5th requirement. I can’t word this precisely but the positional encoding can be learned easily, that the embedding is only a linear transformation of position. It cannot be an encryption of the token.
please explain in detail about the linear relation with two encoding. You mathematical proofs, sounds excellent. Please recommend a good book to understand in detail about these concepts.
1. Why positional encoding is added to the word embedding? Will it changes the semantic value? 2. Why positional encoding use random number produce by sin and cosine... I think it must be simple if we add the one dimension to word embedding storing the position as integer. Why use such a hard, random, and unpredictable algorithm to encode positions!
The position set t(position) = { 1, 2, ..., L} in your video 2:44 starts from 1. But in the diagram to the left it starts from 0. I can see sine wave starting first. Can you clarify this situation ?
Why do we need to alternate sine and cosine? It seems like either one on its own should do the job. The only reason I can see for alternations is that this way we can solve the problem of positional encoding with the wavelength twice as short, as opposed to sine or cosine alone. Is that right? Are there other reasons?
You just have more unique variations before repeats. Say your only using sin and you have two dimensions. At sin(0) the entire vector is [0,0] at pos 180 you have the same vector [0,0]. If you alternate sin and cos then you now have a larger range until a repeat. You could have alternated between sin, cos, and tan if you wanted to and it would add even more uniqueness except for tan(90). So sin and cos was just a good balance. This is just heuristic.
How can adding positional encoding to word embedding doesnt change the word semantic meaning? Example: Word embedding of "Cat" is [1, 2, 3], Word embedding of Money is [2, 3, 4]. If the positional encoding is [2, 1, 0] for word "Cat", positional encoding for word "Money" is [1, 0, -1] then the positional encoded of both word is [3, 3, 3] How can "Cat" equal to "Money"?
Because positional part is a constant. Token part is stochastic, it changes depending on current token, but positional part remains the same. Imagine that you recorded all embeddings of a 0th token from the whole dataset and you got a map, distribution. If you add some constant, this map will remain the same, but shifted to some other location. And yes, it will not work for two examples, you need sufficient amount of data to prevent confusion.
In my opinion, best explanation so far of positional encoding! Super clear and concise! Thank you very much sir!
3:21 - 3:59 are super intuitive, great job!
great job, keep making this video. This video solved my confusion on positional encoding.
I like very concise graphical explanation with the similarity to binary coding and basic linear algebra!
The best explanation of transformer positional encoding on the internet. Awesome video. Thanks!
I couldn't find anywhere why creators of transformer decied to encode the positions in this way and last minute of your video was what I was looking for. Thanks for good explanation
Incredible! I have surfed through various resources online and no one got this so accurately. Absolutely spot on explanation.
Great explanation. Short enough. Detailed enough. Enough talking. Enough showing. Loved the examples.
I'm eternally grateful for this concise explanation, other sources made the positional encoding concept sound so counter-intuitive to grasp
Fantastic. This was amazing! Best explanation.
Just when I was about to pull the last hair on top of my head, I came across this video. Beautifully Explained. Thank You !
more content pleaaaase, this is amazing!
It seems the addendum is a 5th requirement. I can’t word this precisely but the positional encoding can be learned easily, that the embedding is only a linear transformation of position. It cannot be an encryption of the token.
Keep these coming!
please explain in detail about the linear relation with two encoding. You mathematical proofs, sounds excellent.
Please recommend a good book to understand in detail about these concepts.
This is an excellent video
Thank you, love this ^^
1. Why positional encoding is added to the word embedding? Will it changes the semantic value?
2. Why positional encoding use random number produce by sin and cosine... I think it must be simple if we add the one dimension to word embedding storing the position as integer.
Why use such a hard, random, and unpredictable algorithm to encode positions!
The position set t(position) = { 1, 2, ..., L} in your video 2:44 starts from 1. But in the diagram to the left it starts from 0. I can see sine wave starting first. Can you clarify this situation ?
Good catch. Suppose I tried to note that there are L positions overall.
reli good video
Why do we need to alternate sine and cosine? It seems like either one on its own should do the job. The only reason I can see for alternations is that this way we can solve the problem of positional encoding with the wavelength twice as short, as opposed to sine or cosine alone. Is that right? Are there other reasons?
You just have more unique variations before repeats. Say your only using sin and you have two dimensions. At sin(0) the entire vector is [0,0] at pos 180 you have the same vector [0,0]. If you alternate sin and cos then you now have a larger range until a repeat. You could have alternated between sin, cos, and tan if you wanted to and it would add even more uniqueness except for tan(90). So sin and cos was just a good balance. This is just heuristic.
bravo
❤
How can adding positional encoding to word embedding doesnt change the word semantic meaning?
Example:
Word embedding of "Cat" is [1, 2, 3],
Word embedding of Money is [2, 3, 4].
If the positional encoding is [2, 1, 0] for word "Cat",
positional encoding for word "Money" is [1, 0, -1]
then the positional encoded of both word is [3, 3, 3]
How can "Cat" equal to "Money"?
Because positional part is a constant. Token part is stochastic, it changes depending on current token, but positional part remains the same. Imagine that you recorded all embeddings of a 0th token from the whole dataset and you got a map, distribution. If you add some constant, this map will remain the same, but shifted to some other location. And yes, it will not work for two examples, you need sufficient amount of data to prevent confusion.
@@BrainDrainAgain I see... 🔥 Thank you ✅
This video is already very outdated lol
How?
Can you project your speak. You asmr tone is disturbing