Transformer Positional Embeddings With A Numerical Example.

[@tacticalnom97d]

Amazingly explained. This is the first sane, proper mathematical explanation I have found for Positional Encoding. Keep up the excellent work!!

[@sach2274]

I think when you are trying to get the positional encoding for boy(p=1)
it must be sin(1/10000^2*0/4)
not sin(1/100000^2*1/4)
and similarly with cos, for i =0 and 1
for i=2 and 3 it is correct.

[@MrFromminsk]

yeah, I was also wondering about this error. Thanks for confirming my suspicion

[@beizhou2488]

Pellucid explanation! Please keep the nice work going. Like the whole series of your tutorials.

[@manohar_marri]

For the boy token; i=0; for the first two positions right ? You have used I=1; Correct me if I’m wrong

[@王爸爸-h5x]

you‘re right ，maybe it‘s just a pen slip

[@sarahkhan3217]

The division by 2 relates to splitting the embedding dimensions for sine and cosine computations and is independent of the number of sentences or words.

[@samc6368]

Typo in calculation at 5:06, as i=0 for first 2 items of sin and cos for "boy", you have as 1.

[@findritesh]

great job explaining this concept!

[@atmismahir]

thank you for the clear math. I still couldn't get how the word order is preserved though. Is there any visual representation or mathematical illustration of how the positions are preserved?

[@gigabytechanz9646]

Very clear explanation. Thank you!

[@Adhbutham]

Why is cosine of 0 = 0?? I think there is an error

[@sudhansubaladas2322]

Nice explaination...but, .Why u have divided i by 2? I didn't get this part...can u plz tell ..??

[@sudhansubaladas2322]

@@machinelearningwithpytorch yae understood...this is to make i to 0 and 1...so, that we can give sin and cos formula easily... 0 uses sine and 1 uses cosine....I am right???

[@sudhansubaladas2322]

@@machinelearningwithpytorch I have one more query ...Suppose the word "good" occurs in sentence 1 at position 0 and in sentence 5, the word "good" occurs in position 4, then whether their positional encoding will be the same?? Can you please tell? Are positional Embedding and positional encoding is same??

[@openodia]

@@sudhansubaladas2322 Yes, for odd positions we are using Cosine and for even positions we are using sine.

[@openodia]

@@sudhansubaladas2322 No, their positional encoding will be different as p value will change.

[@liam15williams]

do you find out
why we divide i by 2?

[@shelleychristiesantoso7535]

thank you for the clear and concise explanation.

[@lllilllllill7900]

Thank you for your excellent work~

[@raquelhortabartomeu2384]

Not a bad explanation but the formula is no explained in detail and I believe there's an error when you always use i=1 for the word boy

[@ssenukim]

Is there any possible error if embedding vector size is odd number , so that positional encoding vector doesn't have not sin&cos pairs but only have sin at last component?

[@chuckcarlson7940]

Why is the word embedding not messed up when you add numbers to each of it's dimensions? Wouldn't it now represent a different word?

[@rounhi]

Just keep on. Wonderful!

[@서로워]

Can you explain sparse attention? please please

[@pratyushrao7979]

No

[@liam15williams]

i don'r get why we divide two