Not sure what you are saying. The claim that a model has no understanding of a concept bears no meaning without a proper defenition of ”understanding”. I would argue that many GPTs know very well what real life objects are, as they have a thourough high-dimensional map of how different things correlate to eachother. GPTs can generate organic, creative and thoughtful outputs, just like humans. I would say there is no inherent problems with the structure of these types of neural networks that would stop them from exceeding human capabilities.
@Bonnie Dean, what you are saying makes sense, of course. We need to define "understanding" before comparing. But I think we have a level of understanding that AI does not have. Suppose you have no idea about quantum computing, but you can repeat an understandable text about it because you have read it somewhere. It doesn't mean you understand it. We may learn to say I am hungry, and it is grammatically correct, but you will need help understanding what it is to be hungry. Mapping a word to the meaning is different from experiencing the word itself. Dallee can generate a plat of lasagna because it has been trained to do so, but it cannot feel, know, or experience a taste of that. In the same direction, it can generate a long text about why war is bad, but this is because the text it generates probabilistically maps to the corpus he has been trained on. You can not (at least for the moment and with transformers models we have now) train a machine on a corpus highly biased, and it completely changes its mind because it understands that being racist is not good, for example. It will generate a text bias on the corpus it has. We as humans can be raised with a certain mind, but one word or internal thinking changes the whole understanding of what we have learnt.
Let's take a step back and think about the rate of change in AI generally. I hear the term "exponential" thrown around at lot, so let's apply that generically in terms of time. So based on our current point in time, aka the "now-year", 5 years of continued AI research with a growth exponent of 1.5 would yield 11.2 "now-years" of progress. If the exponent is x, or 5^5, the yield would be equivalent to 3,125 "now-years" of progress. If the exponent is anything above 1.25 I'd say all bets are off. If the exponent is less than 1.25 then we might be able to deal with the increased rate of technological advances. At 20 years time with a growth factor of 1.25, 42 "now years" of progress will happen. I don't know obviously how much society can handle, but doubling the pace of today's technological evolution seems like a lot. 20 years of 1.5 is almost 90 "now-years", or a rate of change 4.5 times that of today. And 20^20 is a number so large that surely no living creature on earth would survive. Good time ahead? _Shrug_
I think the exponent is a small real number between 0 and 1. That is, for a given amount of further progress, an exponential amount of WORK must be done to get there. Thats generally the way these type of problem areas work (nonlinear dynamics, etc).
That's the problem, 4;06 it doesn't understand any of the semantics. Only the syntax and what could be the closest probable answer based on the data set it was trained in. let's be honest we might function somewhat like AI, but we don't think as computers think. A computer cannot give a novel piece outside of the data set, while a human can do that. This all goes back to the CHINESE ROOM EXPERIMENT. just becooz you have a rule book to help you manipulate Chinese symbols to create meaningfull sentences to fool some one on the other side. that doesnt means that now you UNDERSTAND CHINESE.
Hey @Y. I can not confirm that this is precisely how the deployed GPT-3 model GPT is built, but I think. - Without precedent content (because the precedent output is given as input for context), I am getting the exact same text from GPT for long questions that are not small talks. it may be deterministic some of the resources they have published mention that They are using transformers. Those papers are really interesting : arxiv.org/pdf/2203.02155.pdf , d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf (Section 2.3)
@@ltphenvonulife When GPT-3 is given a prompt, it outputs a distribution of the probabilities for each next token (or word). It essentially assigns a probability to each word in the english language according to the likelihood that it comes next in the sequence. The output distribution never changes for a given prompt and context but the distribution may be sampled differently to produce various outputs. For example, you could sample the most likely word (top k sampling) or maybe sample from the entire distribution. So the output is deterministic but the output is a set of probabilities. So.... a little bit of both.
Not sure what you are saying. The claim that a model has no understanding of a concept bears no meaning without a proper defenition of ”understanding”. I would argue that many GPTs know very well what real life objects are, as they have a thourough high-dimensional map of how different things correlate to eachother. GPTs can generate organic, creative and thoughtful outputs, just like humans. I would say there is no inherent problems with the structure of these types of neural networks that would stop them from exceeding human capabilities.
@Bonnie Dean, what you are saying makes sense, of course. We need to define "understanding" before comparing. But I think we have a level of understanding that AI does not have. Suppose you have no idea about quantum computing, but you can repeat an understandable text about it because you have read it somewhere. It doesn't mean you understand it. We may learn to say I am hungry, and it is grammatically correct, but you will need help understanding what it is to be hungry. Mapping a word to the meaning is different from experiencing the word itself. Dallee can generate a plat of lasagna because it has been trained to do so, but it cannot feel, know, or experience a taste of that. In the same direction, it can generate a long text about why war is bad, but this is because the text it generates probabilistically maps to the corpus he has been trained on. You can not (at least for the moment and with transformers models we have now) train a machine on a corpus highly biased, and it completely changes its mind because it understands that being racist is not good, for example. It will generate a text bias on the corpus it has. We as humans can be raised with a certain mind, but one word or internal thinking changes the whole understanding of what we have learnt.
Let's take a step back and think about the rate of change in AI generally. I hear the term "exponential" thrown around at lot, so let's apply that generically in terms of time. So based on our current point in time, aka the "now-year", 5 years of continued AI research with a growth exponent of 1.5 would yield 11.2 "now-years" of progress. If the exponent is x, or 5^5, the yield would be equivalent to 3,125 "now-years" of progress. If the exponent is anything above 1.25 I'd say all bets are off. If the exponent is less than 1.25 then we might be able to deal with the increased rate of technological advances. At 20 years time with a growth factor of 1.25, 42 "now years" of progress will happen. I don't know obviously how much society can handle, but doubling the pace of today's technological evolution seems like a lot. 20 years of 1.5 is almost 90 "now-years", or a rate of change 4.5 times that of today. And 20^20 is a number so large that surely no living creature on earth would survive. Good time ahead? _Shrug_
just be mindfull of marketting as well. OpenAI is a company so it needs to find a way to make money.
I think the exponent is a small real number between 0 and 1. That is, for a given amount of further progress, an exponential amount of WORK must be done to get there. Thats generally the way these type of problem areas work (nonlinear dynamics, etc).
That's the problem, 4;06 it doesn't understand any of the semantics. Only the syntax and what could be the closest probable answer based on the data set it was trained in. let's be honest we might function somewhat like AI, but we don't think as computers think. A computer cannot give a novel piece outside of the data set, while a human can do that. This all goes back to the
CHINESE ROOM EXPERIMENT. just becooz you have a rule book to help you manipulate Chinese symbols to create meaningfull sentences to fool some one on the other side. that doesnt means that now you UNDERSTAND CHINESE.
Given the same prompt, does GPT always produce the same sequence of subsequent words? Is it deterministic, after training?
Hey @Y. I can not confirm that this is precisely how the deployed GPT-3 model GPT is built, but I think.
- Without precedent content (because the precedent output is given as input for context), I am getting the exact same text from GPT for long questions that are not small talks. it may be deterministic
some of the resources they have published mention that They are using transformers.
Those papers are really interesting : arxiv.org/pdf/2203.02155.pdf , d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf (Section 2.3)
@@ltphenvonulife When GPT-3 is given a prompt, it outputs a distribution of the probabilities for each next token (or word). It essentially assigns a probability to each word in the english language according to the likelihood that it comes next in the sequence. The output distribution never changes for a given prompt and context but the distribution may be sampled differently to produce various outputs. For example, you could sample the most likely word (top k sampling) or maybe sample from the entire distribution.
So the output is deterministic but the output is a set of probabilities. So.... a little bit of both.