Skipgram - Distributed Representations of Words and Phrases [...] (Paper Review Call 004)

Efficient Estimation of Word Representations in Vector Space '13
Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean
"We propose two novel model architectures for computing continuous vector representations of words from very large data sets. The quality of these representations is measured in a word similarity task, and the results are compared to the previously best performing techniques based on different types of neural networks. We observe large improvements in accuracy at much lower computational cost, i.e. it takes less than a day to learn high quality word vectors from a 1.6 billion words data set. Furthermore, we show that these vectors provide state-of-the-art performance on our test set for measuring syntactic and semantic word similarities."
Distributed Representations of Words and Phrases and their Compositionality - NIPS '13
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, Jeffrey Dean
The recently introduced continuous Skip-gram model is an efficient method for learning high-quality distributed vector representations that capture a large number of precise syntactic and semantic word relationships. In this paper we present several extensions that improve both the quality of the vectors and the training speed. By subsampling of the frequent words we obtain significant speedup and also learn more regular word representations. We also describe a simple alternative to the hierarchical softmax called negative sampling. An inherent limitation of word representations is their indifference to word order and their inability to represent idiomatic phrases. For example, the meanings of “Canada” and “Air” cannot be easily combined to obtain “Air Canada”. Motivated by this example, we present a simple method for finding phrases in text, and show that learning good vector representations for millions of phrases is possible.
Notebook: github.com/ecs...
Slides;
github.com/ecs...
COMPLIANCE NOTICE:
THIS WAS FILMED WHILE WE WORKED AT MICROSOFT, NOW I DONE
EVERYONE IN THIS VIDEO HAS GIVEN PERMISSION TO APPEAR IN THE VIDEO
NO MICROSOFT CONFIDENTIAL INFORMATION WAS DISCUSSED
WE ARE DISCUSSING PUBLIC LITERATURE
THESE ARE OUR PERSONAL OPINIONS
I REFERRED TO THE CHANNEL AS "MACHINE LEARNING AT MICROSOFT", IT HAS NOW BEEN RENAMED "MACHINE LEARNING DOJO" TO MAKE IT CLEAR IT'S NOT AN OFFICIAL MICROSOFT CHANNEL
WAS FILMED IN OUR PERSONAL TIME

[@hemantregar4068]

Thanks for the video !!

[@hugohu2693]

Thank you

[@lazypunk794]

Very informative. Thanks!

[@vigneshkumarkorakki924]

Thanks for this video