Rotary Positional Embeddings

Rotary position embedding (RoPE) combine the concept of absolute and relative position embeddings. RoPE naturally incorporates relative position information through rotation matrix product instead of altering terms in the expanded formulation of additive position encoding when applied with self-attention. It represents token embeddings as complex numbers and their positions as pure rotations.
In this video, I will talk about the following.
00:00:00 Absolute Position Embeddings
00:03:48 Relative position embeddings
00:10:53 Rotary position embedding (RoPE): 2D form
00:20:20 Rotary position embedding (RoPE): General form
00:25:27 RoPE Implementation
00:26:25 Properties of RoPE
00:28:10 RoPE performance
For more details, please look at blog.eleuther.ai/rotary-embed... and arxiv.org/pdf/2104.09864.pdf
Su, Jianlin, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. "Roformer: Enhanced transformer with rotary position embedding." arXiv preprint arXiv:2104.09864 (2021).

[@MercyPrasanna]

Great explanation!

[@nandhuelango2730]

Love your videos. Thanks for sharing all the knowledge 😊

[@sathyafigr]

Highly useful sir

[@anujlahoty8022]

Informative!

[@bhoomeendra]

It was a great explantion sir