classic papers is maybe the best addition to this kind of content. i find it really useful and important to come back to old papers sometimes and look at them from the perspective of modern state of dl.
It is refreshing to have all these classic worked through - They are helpful to mid-tier people - The experts don't need help - For beginners it is to much - And so mid-tier is helpful
There is so much value in the videos just by core content itself. However, anecdotes like how the 'Hierarchical softmax' was a distraction in the paper adds much more context and hence understanding. Thank you for these videos :)
Wow. Just wow. This was a fantastic overview of word2vec. Your explanations of the minute details and the vague and harder to grasp concepts of their paper were exceptional. Your comments of their unconventional authorship and writing style issues were also on point. I felt like I learned and re-learned how word2vec really works. Yes, please cover more classic papers, because understanding the foundations is important. Way to go Yannic!
I would definitely be into a playlist of "classical" data science videos like this. There is so much content to absorb, being able to focus on the ones that have been proven historically and vetted would be awesome. It also gives you a chance to reference how things have improved since then, which is nice to know.
Great explanation of a paper as usual. And this paper (or the three of them) changed so much. Even if token-based embeddings are usually preferably. for some applications type-based word embeddings are probably still the better choice, for example if you are interested in the history of concepts and want to track their semantic change.
To provide another argument for the case of classical papers: It is very difficult to anticipate which ideas will stand the test of time in the moment of their creation. But visiting ‘classical’ papers we allow ourselves the benefit of hindsight - examining those ideas that time proved to be invaluable.
Classic papers are a great Ideas. It's really helpful for those like me who are new in ML. I often try to read some papers that are extension of algorithms introduced in the classic ones and I struggle to understand them since I don't have the prerequisite.
Can you please give references to your claim at 5:20? You said that Queen is just one of the closest words to King and the computation -man+woman is irrelevant; that makes sense in this case, but I don't see how it can explain more complicated analogies such as plural form analogy? I would like to read more about this.
5:00 Thats news to me. I remember trying it out myself, the king queen thing worked while a lot of other analogies didnt, I didnt put much thought to it back then. . 25:13 3/4 is 75% which is very close to 80%, which makes me think, it has something to do with Pareto Principle. Maybe 4/5 didnt do better because we truncated the tail of the distribution. . 27:40 Heuristics = Wild ass guess. Computer Science 101 :D . 30:30 I think they didnt do that because back in 2013 they didnt have the option :) Tensorflow was made public in late 2015. Back in 2013 there was no Tensorflow, no TPUs and GPU clusters were super niche.
Regarding your second comment: 80% don’t magically translate to exponent here in the way you seem to suggest: To see this, consider the extreme case in which 1 contributor causes 19% of the effect. This contributor would receive the same exponent in its probability mass function that it would receive in a much less extreme power-law scenario. It would seem, however, that the 19% contributor should be sampled *way* less frequent than that.
Yes you're probably right with there not being GPUs, but they had their whole MapReduce infrastructure etc, it would have been easy for them to just keep it at that scale.
Yannic, are you also taking a break from such regular reading of papers in your personal time as well? And if not, do you think you could provide a "this is interesting" list in your discord channel when you happen to come across interesting papers?
As far as I know, Transformers or the like (especially BERT) use Byte Pair Encoding to tackle the out-of-vocabulary problem. The vocabulary size is often reduced to within 30000, rather than 10 to 5 or 7. Therefore, no Word2Vec embeddings there (but an input embeddings layer is still there whose weights are learned when the Transformer is trained). Despite of this change, the concept of Word2Vec does really influentially affect how we apply deep leaning in natural language processing.
I love your videos. Just a side note, when you try to explain things with notes make them readable so that if I jump to a random section I can understand what you are trying to explain.
A comment: In 3:33 you mention that with PCA these are the first 2 dimensions that are portrayed. I don't think this is true, right? PCA allows you to map a certain percentage of the expressiveness of the data into a lower dimensional space. This is unequal to simply getting the first two dimensions.
I may be mistaken here but if you're maximizing the objective function for negative sampling your negative and positive signs for the WO vs Wi should be reversed, so it should be minimizing instead of maximizing.
![[Classic] Deep Residual Learning for Image Recognition (Paper Explained)](http://i.ytimg.com/vi/GWt6Fu05voI/mqdefault.jpg)
![[Classic] Deep Residual Learning for Image Recognition (Paper Explained)](/img/tr.png)
classic papers is maybe the best addition to this kind of content. i find it really useful and important to come back to old papers sometimes and look at them from the perspective of modern state of dl.
+1 As my history teacher in high school used to say: You must know where you came from to know where you are going.
It is refreshing to have all these classic worked through - They are helpful to mid-tier people - The experts don't need help - For beginners it is to much - And so mid-tier is helpful
There is so much value in the videos just by core content itself. However, anecdotes like how the 'Hierarchical softmax' was a distraction in the paper adds much more context and hence understanding. Thank you for these videos :)
Wow. Just wow. This was a fantastic overview of word2vec. Your explanations of the minute details and the vague and harder to grasp concepts of their paper were exceptional. Your comments of their unconventional authorship and writing style issues were also on point. I felt like I learned and re-learned how word2vec really works. Yes, please cover more classic papers, because understanding the foundations is important. Way to go Yannic!
Thanks Yannic for the [Classic] videos! These videos are more useful than many of the papers which do small incremental improvements.
Thanks for this classic series papers for us that are learning deep learning is important to cover the classic and main old ideas in the field.
wow, I am learning word2vec from yesterday, and was struggling to grasp the concept and here you uploaded the video, explaining the paper!
Thanks for visiting such an important paper!!! Awesome content!!
Welcome to Yannic`s paper museum :)
Very nice to look at older papers as well!
I would definitely be into a playlist of "classical" data science videos like this. There is so much content to absorb, being able to focus on the ones that have been proven historically and vetted would be awesome.
It also gives you a chance to reference how things have improved since then, which is nice to know.
Really enjoying watching these videos. You did a great job explaining them!
Great explanation of a paper as usual. And this paper (or the three of them) changed so much. Even if token-based embeddings are usually preferably. for some applications type-based word embeddings are probably still the better choice, for example if you are interested in the history of concepts and want to track their semantic change.
always good to look back classic papers
My browser crashed along with my 50,000 tabs. I restored them and suddenly Yannic is telling me about 5 papers simultaneously.
Please keep going with the amazing content! Love it!
Thank you!!! So much better than the Standford class.
To provide another argument for the case of classical papers: It is very difficult to anticipate which ideas will stand the test of time in the moment of their creation. But visiting ‘classical’ papers we allow ourselves the benefit of hindsight - examining those ideas that time proved to be invaluable.
Love this series, looking forward to more such videos
Yes, more historical papers!!
Really loved your explanation. Thank You.
Love it, more of old papers :)
Classic papers are a great Ideas. It's really helpful for those like me who are new in ML. I often try to read some papers that are extension of algorithms introduced in the classic ones and I struggle to understand them since I don't have the prerequisite.
This is awesome!
Thanks you. I couldn't understand word2vec from prof. Andrew Ng's video, but you explained it clearly!
Wait you're supposed to be having a break! This is your second video in two days. 😅
The videos are pre-recorded! He's amazing, man.
Indeed what a guy. I think he's doing some good things with this channel!
Classic series 🔥
Wow that's nice! Please do more about classical papers!
OMG I’m revisiting this clip for negative sampling because I was confused by it in understanding the node embedding of random walk in GNN.
Can you please give references to your claim at 5:20? You said that Queen is just one of the closest words to King and the computation -man+woman is irrelevant; that makes sense in this case, but I don't see how it can explain more complicated analogies such as plural form analogy? I would like to read more about this.
arxiv.org/abs/1905.09866
Really Great!
More videos like this please....
I love you man!
Yes, i love historical papers
Thanks! 👍
5:00 Thats news to me. I remember trying it out myself, the king queen thing worked while a lot of other analogies didnt, I didnt put much thought to it back then.
.
25:13 3/4 is 75% which is very close to 80%, which makes me think, it has something to do with Pareto Principle. Maybe 4/5 didnt do better because we truncated the tail of the distribution.
.
27:40 Heuristics = Wild ass guess. Computer Science 101 :D
.
30:30 I think they didnt do that because back in 2013 they didnt have the option :) Tensorflow was made public in late 2015. Back in 2013 there was no Tensorflow, no TPUs and GPU clusters were super niche.
Regarding your second comment: 80% don’t magically translate to exponent here in the way you seem to suggest: To see this, consider the extreme case in which 1 contributor causes 19% of the effect. This contributor would receive the same exponent in its probability mass function that it would receive in a much less extreme power-law scenario. It would seem, however, that the 19% contributor should be sampled *way* less frequent than that.
Yes you're probably right with there not being GPUs, but they had their whole MapReduce infrastructure etc, it would have been easy for them to just keep it at that scale.
one video on Efficient Estimation of Word Representations in
Vector Space please
Yannic, are you also taking a break from such regular reading of papers in your personal time as well? And if not, do you think you could provide a "this is interesting" list in your discord channel when you happen to come across interesting papers?
Don't forget that Word2Vec is part of the encoding in the front end of a transformer, so w2v is still plenty relevant!
As far as I know, Transformers or the like (especially BERT) use Byte Pair Encoding to tackle the out-of-vocabulary problem. The vocabulary size is often reduced to within 30000, rather than 10 to 5 or 7. Therefore, no Word2Vec embeddings there (but an input embeddings layer is still there whose weights are learned when the Transformer is trained). Despite of this change, the concept of Word2Vec does really influentially affect how we apply deep leaning in natural language processing.
I love your videos. Just a side note, when you try to explain things with notes make them readable so that if I jump to a random section I can understand what you are trying to explain.
Obviosuly like this
A comment: In 3:33 you mention that with PCA these are the first 2 dimensions that are portrayed. I don't think this is true, right? PCA allows you to map a certain percentage of the expressiveness of the data into a lower dimensional space. This is unequal to simply getting the first two dimensions.
Correct, I meant the first two PCA dimensions, not data dimensions
I may be mistaken here but if you're maximizing the objective function for negative sampling your negative and positive signs for the WO vs Wi should be reversed, so it should be minimizing instead of maximizing.
0 0 0 0 0.05 0.95 st!
Gj
To All my students. Let me know personally if you find my comment. Cheers!