“What does the fox say?” “Don’t they go ‘ring ding ding’?” “Not in this dataset”
5 лет назад+52
Train the same algorithm on songs instead of news articles and I figure you could get some really interesting results as well. Songs work on feelings and that should change the connections between the words as well - I bet the technology can be used to tell a lot about the perspective people take on things as well.
@ Songs also use specific rhythmic structures; assuming most of your data was popular music, I bet that there'd be a strong bias for word sequences that can fit nicely into a 4/4 time signature, and maybe even some consistent rhyming structures.
This thing would ace the analogy section of the SAT. Apple is to tree as grape is to ______. model.most_similar_cosul(positive['tree', 'grape'], negative['apple']) = "vine"
Fun points: A lot of the Word2vec concepts come from Tomáš Mikolov, a Czech scientist at Google. The Czech part is kinda important here - Czech, as a Slavic language, is very flective - you have a lot of different forms for a single word, dependent on its surroundings in a sentence. In some interview I read (that was in Czech and in a paid online newspaper, so I can't give a link), he mentioned that this inspired him a lot - you can see the words clustering by their grammatical properties when running on a Czech dataset and it's easier to reason about such changes when a significant portion of them is exposed visibly in the language itself (and learned as a child in school, because some basic parts of it are needed in order to write correctly).
It was given ‘oink’ minus ‘pig’ plus ‘fox’ though, not fox + says. So we’d expect to see the same results as for cow & cat etc. of it “understanding” that we’re looking at the noises that the animals make. Obviously it’s not understanding, just an encoding of how those words appear near each other, but we end up with something remarkably similar to understanding.
I was actually expecting New York when they added America. As a child I always thought New York was the capital of the U.S., I was at least around eight when I learned that it wasn't. Similarly, when people talk of Australia's cities, Canberra is rarely spoken of, but Sydney comes up a lot.
I like this guy and his long sentences. It's nice to see somebody who can muster a coherent sentence of that length. So, if you run this (it's absurdly simple, right), but if you run this on a large enough data set and give it enough compute to actually perform really well, it ends up giving you for each word a vector (that's of length however many units you have in your hidden layer), for which the nearby-ness of those vectors expresses something meaningful about how similar the contexts are that those words appear in, and our assumption is that words that appear in similar contexts are similar words.
I did this for my final project in my bsc. Its amazing. I found cider - apples + grapes = wine. My project attempted to use these relationships to build simulated societies and stories.
I am in love with this man's explanation! makes it so intuitive. I have a special respect for folks who can make a complex piece of science/math/computer_science into an abstract piece of art. RESPECT!
This was weirdly fascinating to me. I'm generally interested by most of the Computerphile videos, but this one really snagged something in my brain. I've got this odd combination of satisfaction and "Wait, really? That works?! Oh, wow!"
Rob Miles and computerphile thank you... IDK why youtube gave this gem back to me today (probably for my insesent searching for the latest LLM news these days) but I am greatful to you even more now than I was 4yrs ago... Thank you
This is basically node embedding from graph neural networks. Each sentence you use to train the it can be seen as a random walk in the graph that relates each world with each other. The number of words in the sentence can be seem as how long you walk from the node. Besides "word-vector arithmetics", one thing interesting to see would be to use this data to generate a graph of all the words and how they relate to each other. Than you could do network analysis with it, see for example, how many clusters of words and figure out what is their labels. Or label a few of them and let the graph try to predict the rest of them. Another interesting thing would be to try to embed sentences based on the embedding of words. For that you would get a sentence and train a function that maps points in the word space to points in the sentence space, by aggregating the word points some how. That way you could compare sentences that are close together. Then you can make sentences-vector arithmetics. This actually sounds like a cool project. I think I'm gonna give it a try.
It makes so much more sense to represent words numerically rather than as collections of characters. That may be the way we write them, but the characters are just loose hints at pronunciation, which the model probably doesn't care about for meaning. And what would happen if a language model that relied on characters tried to learn a language that doesn't use that system of writing? Fascinating stuff.
I love how this video seems to have time travelled from 1979 - the austere painted brick classroom, the shirt, the hair and beard, even the thoughtful and articulate manner seem to come from another time.
This was soooo interesting to me. I never dug deeper in how these networks work. But so many "Oh! That's how it is!". When I watched the video about GPT-2 and you he said that all the connections are just statistics, I just noted that internally as interesting and "makes sense" but didn't really get it. But with this video it clicked! So many interesting things, so thanks a lot for that. I love these videos. And seeing the math that can be done with these vectors is amazing! Wish I could like this more than once.
I love that I have been thinking about modelling natural language for some time now, and this video basically confirms my way of heading. I have never heard of word embedding, but its exactly what I was looking for. Thank you computerphile and youtube!
You could make a game with that, some kind of scrabble with random words, add and substract words to get other words. Maybe with the goal to get long words or specific words or get shortest or longest distance from a specific word.
is the diagram with angles and arrows going off in all directions just for us to visualise it rather than how computers are looking at it, I didn't think they'd be calculating degrees. I thought it would be more about numbers of how close the match is like 0-100
Thanks for a great explanation of word embeddings. Sometimes I need a review. I think I understand it, then after looking at the abstract, n-dimensional embedding space in ChatGPT and Variational Autoencoders, I forget about the basic word embeddings. At least it’s a simple 300-number vector per word, that describes most of the highest frequency neighboring words.
Me too. I loved the review after looking how GPT4 and its code/autoencoder-set looks under the hood. I also had to investigate the keywords being used like "token" when we think about multi vector signifiers and the polysemiology of glyphic memorization made by these massive AI databases. Parameters for terms, words went from 300 to 300,000 to 300,000,000 to 1.5 trillion to ♾ infinite. Meaning: Pinecone and those who've reached infinite parameters have created the portal to a true self-learning operating system, self-aware AI.
I don't think that such reprenstation captures the distance information at all to begin with. The *closest* word is it has a distance of 1, (hamming distance in this case, I claim that each flipped bit counts as 1 hamming distance), but is not a word at all. Whereas in a vector-encoded representation since the words are mapped to a *vector space* then the closeness-farness of two vectors are conveyed in that representation. information representation if a fabulous topic I don't think I understand it yet. Information theory may help us understand information and information representation.
@worthy null , wtf are you on about? Nobody said anything about Hamming distance. He asked: what few words are the closest to the basis vectors [in euclidean distance] in that vector space.
I see where youre going with your analogy, but embeddings generally dont work like that. At first all the words are randomly given a random vector and then those vectors change throughout the training process. So the words you're looking for would be meaningless in this case. If you're looking for the centroid word(words that appear in the center of the embeddings) then that would be words that have very broad contexts such as "the".
@Gerben van Straaten something that might be cute would be defining some human meaningful basis vectors then rotating/scaling the points to fit them. Then see what the remaining basises are. You're definitely right that they would not be human meaningful out of the box though
It'd have been nice to hear about the research craze around more sophisticated approaches to NLP. It's hard to keep up with the amount of publications lately related to achieving "state-of-the-art" models using GLUE's benchmark.
I would suspect that this has to be very similar to how our own brains interpret languange, but then again evolution has a tendency to go about solving problems in very strange and inefficient ways.
@@maxid87 mammals have a nerve that goes from the brain to the throat, but due to changes in mammals it always goes under a vien in the heart then back up to the throat. This is so extreme that on a giraffe the nerve is like 9 feet long or something. In general evolution does a bad job at remmoving unnecessary features.
@@wkingston1248 how do you know that this is inefficient? Might seem like that at first glance but maybe there is some deeper reason for it? Are there actual papers on this topic that answer the question?
I doubt there is a lot of evolution at play in human language processing. It seems reasonable to assume that association (cat~dog) and decomposition (Tokyo = japanese + city) play an important role.
Yep, if the mapping of images is just taking the values each pixel and then making N-dimensional vector (where N is number of pixels), then the picture with more brightness would be the on the same line (if solid black pixels were still solid black, depending on your brightness filter applied).
Duddino Gatto they mew. But they outgrow it pretty quickly. Humans don’t babble like babies when we grow up, but if that was the only think our feline overlords responded to, we would.
If you train 2 networks with different languages I guess the latent space? would be similar. And the differences could be really relevant to how we thought differently due to using different language
Can someone explain the compression of information from that hidden layer. Is it basically training of the model that allows the hidden layer to adjust and define meaningful weights to each node, that leads to it not needing the same length vector as the input vector to produce a meaningful output? Or am I missing something here. Also is there a way to conceptualize these say 1000 dimensional vectors or is it just something you have to go with ?
This is fascinating! Might we be able to represent language in the abstract as a vector space? Furthermore, similar but slightly different words in different languages are represented by similar by slightly different vectors in this vector space?
I kinda overlooked the importance of this video when it was released 3 yrs ago. not its basically the explanaition of how chatGPT does its thing... but with more data.
There are a lot of words that appear similar by context but are very different in meaning. Sometimes they're exact opposites of each other. This doesn't matter too much for word prediction but for tasks that extract semantics. Are there techniques to get better semantic encoding out of the text, particularly separating synonyms from antonyms?
This is one of the coolest things i've seen in a while. Just thinking how small a neighbourhood of one word/vector should we take ? Or how does the implementation of context affect the choice of optimal neighbourhoods ?
And contexts themselves vary from a person to another depending on how they experienced life. So it would be interesting to see also a set of optimal contexts and that would affect the whole thing.
Yeah but did you "learn" or just "understand while listening". Those are not the same things. Although, they may complement each other nicely in some cases.
@0:56 A set of characters doesn't have repetition and - in further not specified sets - the ordering isn't specified. So dom, doom, mod and mood map to the same set of characters, so a set is underspecific.
It's structurally the same as an AE, but it's trained differently. An AE has to reconstruct its input, but this network needs to find closely related, but different words.
Amazing video! I appreciate every minute of your effort, really. Think back, wondering "Will anyone notice this? Fine, I'll do it." Yes, and thank you.
you can easily predict = WITHOUT word vectors or machine learning, by just using a CONCORDANCE over a (huge) text corpus. Not nearly as sexy as neural networks, but it works fine, and EASY to understand. word A is similar to word B *precisely* because lots of people USED A and B in similar contexts in your corpus. drawback? sequential, very SLOW compiling of very SMALL corpora, compared to neural wordvec techniques. but it doesnt necessarily HAVE to be that way..... the CONCORDANCE approaches could be (?) parallelized too.....
How is the size of the hidden layer chosen? Are there ways to calculate how big a layer is useful? Would selecting different sizes cause it to encode different data? In his example, if the hidden layer had 6 nodes, would it produce the categories of "noun, verb, adjective" etc, since that is likely the most descriptive thing you can do with so few categories?
oh, that's so neat. if I'm understanding right, this model has 300 dimensions on which every word falls, to varying degrees... what are those 300 dimensions?? if you look at each one, and ask it to order all the words on each, are there any patterns? like "oh, this must be the 'size' dimension: it's got the universe on one end, whales in the middle, and quarks on the other"? things like that?
I believe there's no privileged basis, there's no reason for the basis vectors to be particularly meaningful, the vectors only matter relative to each other. You could rotate the set of embeddings however you want and get the same results. The principal components might be interesting though
Bro I feel like we're going to start using AI jargon as popular lingo very VERY soon. lol "Not in this dataset" , "Elaborate" , "You need to fine-tune yourself buddy"
Amazing correlations for only 300 vectors. Next, you must assign a "meaning" to each vector to get better correlations and tags of context. One could also increase the number of vectors.
"Not in this data set" is my new favorite comeback oneliner
It's similar to "not in this timeline" that we hear a lot in time-travel scifi
😂
“What does the fox say?”
“Don’t they go ‘ring ding ding’?”
“Not in this dataset”
Train the same algorithm on songs instead of news articles and I figure you could get some really interesting results as well. Songs work on feelings and that should change the connections between the words as well - I bet the technology can be used to tell a lot about the perspective people take on things as well.
@ Songs also use specific rhythmic structures; assuming most of your data was popular music, I bet that there'd be a strong bias for word sequences that can fit nicely into a 4/4 time signature, and maybe even some consistent rhyming structures.
@ Train it with only lyrics from Manowar!
@ I wonder how strong Rhymes would show up in that dataset.
@@killedbyLife That's odd - I listen to Manowar regularly. Nice pick. 😉
This thing would ace the analogy section of the SAT.
Apple is to tree as grape is to ______.
model.most_similar_cosul(positive['tree', 'grape'], negative['apple']) = "vine"
Fun points: A lot of the Word2vec concepts come from Tomáš Mikolov, a Czech scientist at Google. The Czech part is kinda important here - Czech, as a Slavic language, is very flective - you have a lot of different forms for a single word, dependent on its surroundings in a sentence. In some interview I read (that was in Czech and in a paid online newspaper, so I can't give a link), he mentioned that this inspired him a lot - you can see the words clustering by their grammatical properties when running on a Czech dataset and it's easier to reason about such changes when a significant portion of them is exposed visibly in the language itself (and learned as a child in school, because some basic parts of it are needed in order to write correctly).
very interesting
I keep wondering if I was the one who gave the inventor of Word2vec the idea of vectoring words 15 years ago. Probably not.
Now I wonder what would've happened if it had been a Chinese, where you don't have that at all!
Wonder how this works with Japanese? Their token spaces must be much bigger and more complex
Technically you can share the link to the newspaper
'fox' + 'says' = 'Phoebe' may be from newspapers quoting English actress Phoebe Fox
Wow what a pull.
It was given ‘oink’ minus ‘pig’ plus ‘fox’ though, not fox + says. So we’d expect to see the same results as for cow & cat etc. of it “understanding” that we’re looking at the noises that the animals make. Obviously it’s not understanding, just an encoding of how those words appear near each other, but we end up with something remarkably similar to understanding.
Okay, that was amazing. "London + Japan - England = Tokyo"
That needs to be a web site
More impressed by Santa + pig - oink = "ho ho ho"
This blew my mind. Doing math with meaning is amazing.
you mean Toyko!
I was actually expecting New York when they added America. As a child I always thought New York was the capital of the U.S., I was at least around eight when I learned that it wasn't. Similarly, when people talk of Australia's cities, Canberra is rarely spoken of, but Sydney comes up a lot.
I like this guy and his long sentences. It's nice to see somebody who can muster a coherent sentence of that length.
So, if you run this (it's absurdly simple, right), but if you run this on a large enough data set and give it enough compute to actually perform really well, it ends up giving you for each word a vector (that's of length however many units you have in your hidden layer), for which the nearby-ness of those vectors expresses something meaningful about how similar the contexts are that those words appear in, and our assumption is that words that appear in similar contexts are similar words.
His neural network has a very large context, evidently ;)
Imagine a conversation between him and D Trump.
I did this for my final project in my bsc. Its amazing. I found cider - apples + grapes = wine. My project attempted to use these relationships to build simulated societies and stories.
would you be willing to share a link? This seems really interesting.
Yeah, that sounds right up my alley, how well did it work
Dammit Dean, you can't bait people with this kind of a project idea and not tell us how it went
You want to give some info as to how that went?
you are lying you did not do it. if you did, then paste the source(paper or code).
- cunningham
Always love to see Rob Miles here!
Even when the video doesn't have that "AAAHH" quality to it.
EXTRA BITS NEEDED!
grow up
I am in love with this man's explanation! makes it so intuitive. I have a special respect for folks who can make a complex piece of science/math/computer_science into an abstract piece of art. RESPECT!
"it's the friends you make along the way" lol
I was just thinking this and came to the comments…. Yup. Mr Miles is terrific. 🎉
"complex" ? 🙂
He's Twerp. He's afraid to talk about X Y and XX Chromosomes and how we express them in language. shame on you
@@Commiehunter12 No, he just didn't want to trigger the priesthood in a video about word embeddings but looks like he wasn't careful enough.
Today, vector databases are a revolution to AI models. This man was way ahead of time.
This was weirdly fascinating to me. I'm generally interested by most of the Computerphile videos, but this one really snagged something in my brain. I've got this odd combination of satisfaction and "Wait, really? That works?! Oh, wow!"
Tomorrow's headline:
"Science proves fox says 'Phoebe'"
Fox News
floats: some of the real numbers
- Best description and explanation ever! - It encompasses all the problems and everything....
"A tastefully curated selection of the real numbers"
Rob Miles and computerphile thank you... IDK why youtube gave this gem back to me today (probably for my insesent searching for the latest LLM news these days) but I am greatful to you even more now than I was 4yrs ago... Thank you
Foxes do chitter!
But primarily they say "Phoebe"
This page blows my mind. It takes you through the journey of thinking.
This is basically node embedding from graph neural networks. Each sentence you use to train the it can be seen as a random walk in the graph that relates each world with each other. The number of words in the sentence can be seem as how long you walk from the node. Besides "word-vector arithmetics", one thing interesting to see would be to use this data to generate a graph of all the words and how they relate to each other. Than you could do network analysis with it, see for example, how many clusters of words and figure out what is their labels. Or label a few of them and let the graph try to predict the rest of them.
Another interesting thing would be to try to embed sentences based on the embedding of words. For that you would get a sentence and train a function that maps points in the word space to points in the sentence space, by aggregating the word points some how. That way you could compare sentences that are close together. Then you can make sentences-vector arithmetics.
This actually sounds like a cool project. I think I'm gonna give it a try.
How did it go?
I'm a man of simple tastes. I see Rob Miles, I press the like button.
Rather than biggest city, it seems obvious it would be the most written about city, which may or may not be the same thing.
Yeah, I was going to say most famous cities. Still a very cool relationship
Would be interested by the opposite approach: ‘Washington D.C. - America + Australia = Canberra’
Toby Same here...
I’m surprised they didn’t run that,
Stock markets
'What does it mean for two words to be similar?'
That is a philosophy lesson I am not ready for bro
Breau
How dare you assume my words meaning, don't you know its the current era
that's kind of the great thing about computer science... you can take philosophical waffling and actually TEST it
I'm not your bro, pal
@@cerebralm "Computer science is the continuation of logic
by other means"
It makes so much more sense to represent words numerically rather than as collections of characters. That may be the way we write them, but the characters are just loose hints at pronunciation, which the model probably doesn't care about for meaning. And what would happen if a language model that relied on characters tried to learn a language that doesn't use that system of writing? Fascinating stuff.
Love how you guys are just having fun with the model by the end
I love how this video seems to have time travelled from 1979 - the austere painted brick classroom, the shirt, the hair and beard, even the thoughtful and articulate manner seem to come from another time.
This was soooo interesting to me. I never dug deeper in how these networks work. But so many "Oh! That's how it is!". When I watched the video about GPT-2 and you he said that all the connections are just statistics, I just noted that internally as interesting and "makes sense" but didn't really get it. But with this video it clicked!
So many interesting things, so thanks a lot for that. I love these videos.
And seeing the math that can be done with these vectors is amazing! Wish I could like this more than once.
You really have a way with words, Rob. Please never stop what you do. ❤️
Wow, that is mindblowing.
I'm a simple man. I see Rob Miles, I click.
I could listen to him all day!
This gotta be one of the best intuitive explanation of word2vec.
Meanwhile in 2030:
"human" + "oink oink" - "pig" = "pls let me go skynet"
Mind blown, Thanks for the easy explanation. So calm and composed.
Man, ... when AI will realize we can only imagine 3 dimensions, it will be so puzzled how we can do anything at all...
Actually 2 spacial visual dimension with projection...
Then we have time, sounds, smells...
The amount of neurons is more important than the experienced dimensions.
Word embedding is my favorite pass-time.
I love the way he's discussing complicated topics. Thank you very much
Phenomenal talk. Surprisingly compelling given the density of the topic.
I really do hope they let this man out of prison one day.
I am amazed and in love with his explanations. I just understand it clearly, you know.
I love that I have been thinking about modelling natural language for some time now, and this video basically confirms my way of heading. I have never heard of word embedding, but its exactly what I was looking for. Thank you computerphile and youtube!
Excellent presentation
OMG that ending. Love Robert's videos!
This has suddenly become massively relevant 😅
You could make a game with that, some kind of scrabble with random words, add and substract words to get other words. Maybe with the goal to get long words or specific words or get shortest or longest distance from a specific word.
I'm surprised that there's been no mention of Rob's cufflinks in the comments for well over a year after upload
This is by far the best video I've seen on Machine Learning. So cool!!!
4 years ago if only we had the vision to understand that this is actually a multi billion dollar, world changing idea!
Awesome that with ChatGPT and stable diffusion this video got super relevant
is the diagram with angles and arrows going off in all directions just for us to visualise it rather than how computers are looking at it, I didn't think they'd be calculating degrees. I thought it would be more about numbers of how close the match is like 0-100
Thanks for a great explanation of word embeddings. Sometimes I need a review. I think I understand it, then after looking at the abstract, n-dimensional embedding space in ChatGPT and Variational Autoencoders, I forget about the basic word embeddings. At least it’s a simple 300-number vector per word, that describes most of the highest frequency neighboring words.
Me too. I loved the review after looking how GPT4 and its code/autoencoder-set looks under the hood. I also had to investigate the keywords being used like "token" when we think about multi vector signifiers and the polysemiology of glyphic memorization made by these massive AI databases.
Parameters for terms, words went from 300 to 300,000 to 300,000,000 to 1.5 trillion to ♾ infinite. Meaning: Pinecone and those who've reached infinite parameters have created the portal to a true self-learning operating system, self-aware AI.
I'm realllly curious about the basis vectors in this. What's the closest few words to etc..
That. Now I'm really curious.
I don't think that such reprenstation captures the distance information at all to begin with. The *closest* word is it has a distance of 1, (hamming distance in this case, I claim that each flipped bit counts as 1 hamming distance), but is not a word at all. Whereas in a vector-encoded representation since the words are mapped to a *vector space* then the closeness-farness of two vectors are conveyed in that representation. information representation if a fabulous topic I don't think I understand it yet. Information theory may help us understand information and information representation.
@worthy null , wtf are you on about? Nobody said anything about Hamming distance.
He asked: what few words are the closest to the basis vectors [in euclidean distance] in that vector space.
I see where youre going with your analogy, but embeddings generally dont work like that. At first all the words are randomly given a random vector and then those vectors change throughout the training process. So the words you're looking for would be meaningless in this case. If you're looking for the centroid word(words that appear in the center of the embeddings) then that would be words that have very broad contexts such as "the".
@Gerben van Straaten something that might be cute would be defining some human meaningful basis vectors then rotating/scaling the points to fit them. Then see what the remaining basises are. You're definitely right that they would not be human meaningful out of the box though
Very well done. I love the explanation. He obviously has deep insight to explain it so very well. Thanks.
super nice style of speaking, voice and phrasing. Good work !
It'd have been nice to hear about the research craze around more sophisticated approaches to NLP. It's hard to keep up with the amount of publications lately related to achieving "state-of-the-art" models using GLUE's benchmark.
Would it be possible for Rob to share his colab notebook / code with us so we can play around with the model for ourselves? :D
I'm pretty sure it's just the standard model that comes with gensim
See bdot02's comment above
best explanation about word embedding
I would suspect that this has to be very similar to how our own brains interpret languange, but then again evolution has a tendency to go about solving problems in very strange and inefficient ways.
Do you have examples? I am really curious - so far I always assumed nature does it the most efficient way possible.
@@maxid87 mammals have a nerve that goes from the brain to the throat, but due to changes in mammals it always goes under a vien in the heart then back up to the throat. This is so extreme that on a giraffe the nerve is like 9 feet long or something. In general evolution does a bad job at remmoving unnecessary features.
Clever Hans
@@wkingston1248 how do you know that this is inefficient? Might seem like that at first glance but maybe there is some deeper reason for it? Are there actual papers on this topic that answer the question?
I doubt there is a lot of evolution at play in human language processing. It seems reasonable to assume that association (cat~dog) and decomposition (Tokyo = japanese + city) play an important role.
Very interesting. Would like to see more about these word vectors and how to use them.
This video really deserves more views.
Oh yes, explaination and a concrete example
Rob Miles is back :D
Very cool! I didn't know we could do word association to this degree.
This is very impressive. This is actually amazing.
3:00 pretty sure that graphic should've been just 2 points on the same line, given what he said a few sentences before that.
Yep, if the mapping of images is just taking the values each pixel and then making N-dimensional vector (where N is number of pixels), then the picture with more brightness would be the on the same line (if solid black pixels were still solid black, depending on your brightness filter applied).
Would love sample code in cases like this where there’s a Jupyter notebook already laying about!
16:20 Rob loves it, he's so excited by it 😄
3rd result for dog is "bark incessantly."
Even AI knows dogs are annoying mutants. Fun fact: Wolves don't bark, well, almost never.
Wild cats also don't meow. Even feral "domestic" (as in the species) cats don't meow, it's only towards humans that they do.
Duddino Gatto they mew. But they outgrow it pretty quickly. Humans don’t babble like babies when we grow up, but if that was the only think our feline overlords responded to, we would.
Mind blown, thank you very much for this explanation!
So glad they allow this prisoner a conjugal visit to discuss these topics!
Beautifully simple explanation! Resplendent!
Mind blown... Able to do arithmetic on the meaning of words... I did not see that one coming :o A killer explanation on the subject thanks!! :D
I never thought the difference between "man" and "woman" could be so intertwined in a language! I can imagine it would make some people uneasy.
How far we've come only 3 years later.
If you train 2 networks with different languages I guess the latent space? would be similar. And the differences could be really relevant to how we thought differently due to using different language
Can someone explain the compression of information from that hidden layer. Is it basically training of the model that allows the hidden layer to adjust and define meaningful weights to each node, that leads to it not needing the same length vector as the input vector to produce a meaningful output? Or am I missing something here. Also is there a way to conceptualize these say 1000 dimensional vectors or is it just something you have to go with ?
This is fascinating! Might we be able to represent language in the abstract as a vector space? Furthermore, similar but slightly different words in different languages are represented by similar by slightly different vectors in this vector space?
Came back here because I fell in love with the Semantle game that came out a couple of months ago.
I kinda overlooked the importance of this video when it was released 3 yrs ago. not its basically the explanaition of how chatGPT does its thing... but with more data.
There are a lot of words that appear similar by context but are very different in meaning. Sometimes they're exact opposites of each other. This doesn't matter too much for word prediction but for tasks that extract semantics. Are there techniques to get better semantic encoding out of the text, particularly separating synonyms from antonyms?
Auto-antonyms, words that mean the exact opposite in different context: cleave, sanction, dust ...
This is one of the coolest things i've seen in a while. Just thinking how small a neighbourhood of one word/vector should we take ? Or how does the implementation of context affect the choice of optimal neighbourhoods ?
And contexts themselves vary from a person to another depending on how they experienced life. So it would be interesting to see also a set of optimal contexts and that would affect the whole thing.
it's more than slightly surprising that you can explain this concept in 17 minutes, instead of going to a semester full of lectures.
Yeah but did you "learn" or just "understand while listening". Those are not the same things. Although, they may complement each other nicely in some cases.
@0:56 A set of characters doesn't have repetition and - in further not specified sets - the ordering isn't specified.
So dom, doom, mod and mood map to the same set of characters, so a set is underspecific.
@RobertMilesAI Can you provide the Google Colab ipynb file link shown at 12:32 please.
I think this looks more like an auto-encoder than GAN
It's structurally the same as an AE, but it's trained differently. An AE has to reconstruct its input, but this network needs to find closely related, but different words.
word embeddings are the friends we make along the way
Amazing video! I appreciate every minute of your effort, really. Think back, wondering "Will anyone notice this? Fine, I'll do it." Yes, and thank you.
Wow, it is 2020, and I haven't used Gensim and GloVe in years - ever since the release of BERT and GPT.
This blew my mind. Simply wonderful!
you can easily predict = WITHOUT word vectors or machine learning, by just using a CONCORDANCE over a (huge) text corpus. Not nearly as sexy as neural networks, but it works fine, and EASY to understand. word A is similar to word B *precisely* because lots of people USED A and B in similar contexts in your corpus.
drawback? sequential, very SLOW compiling of very SMALL corpora, compared to neural wordvec techniques. but it doesnt necessarily HAVE to be that way.....
the CONCORDANCE approaches could be (?) parallelized too.....
Came here from DeepLearning.ai specialization I was doing when I couldn't grab Word2vec.. Nice explanation
Nice thinkpad rob! I'm using the same version of x1 carbon with the touch bar as my daily machine. Great taste.
How is the size of the hidden layer chosen? Are there ways to calculate how big a layer is useful? Would selecting different sizes cause it to encode different data? In his example, if the hidden layer had 6 nodes, would it produce the categories of "noun, verb, adjective" etc, since that is likely the most descriptive thing you can do with so few categories?
This is über amazing. I wonder if you could use that to predict cache hits and misses
i think this video just have me a better understanding of neural networks
Plz more AI videos, they are awesome!
oh, that's so neat. if I'm understanding right, this model has 300 dimensions on which every word falls, to varying degrees... what are those 300 dimensions?? if you look at each one, and ask it to order all the words on each, are there any patterns? like "oh, this must be the 'size' dimension: it's got the universe on one end, whales in the middle, and quarks on the other"? things like that?
I believe there's no privileged basis, there's no reason for the basis vectors to be particularly meaningful, the vectors only matter relative to each other. You could rotate the set of embeddings however you want and get the same results.
The principal components might be interesting though
Bro I feel like we're going to start using AI jargon as popular lingo very VERY soon. lol "Not in this dataset" , "Elaborate" , "You need to fine-tune yourself buddy"
deserted custard
😂
This is funny now. Two years from now... reality
Chat, is this real?
damn that's the best enjoyable informative video I've seen in a while
Amazing correlations for only 300 vectors. Next, you must assign a "meaning" to each vector to get better correlations and tags of context. One could also increase the number of vectors.
Joe Dempsey, Sr. It’s not the number of vectors, rather the dimension of the vectors
Watching in 2023 after discovering embedding in open ai api
I would love to see a followup vid using verbs/adverbs rather than just nouns.
*You are the company you keep*