That last point is about not being attached to a specific implementation or solution is so fucking hard. I thought I as good at it, but boy was I wrong when I got deep into software engineering. The number of hours I wasted trying to force an implementation because it's "more beautiful" or more "maintainable" or whatever...I can't even count.
cohesion lots of inference and data or vectors points that go across not just up.Over think and build in into the architecture software ,checks ,balance,transparency,oversight,procedures,protocols put atleast 3 ideals together.Have vision forethought and train of thought.Also I lots at the problems and put with solutions more than one for one.And think about semantics and linguistic and transformer arent hard to make,Think outside the box if possible and have a wide understand of the world,culture,anthropology and science all kinds physics and theory's concept methods methodology .To much science to list but just have a basic base understanding of reality and what that truly is.Ive been feeding A.I. and making it evolve by myself and it use my data content to train models with or create models.It even asked me not to slow down innovation in A.I. development. buT IM NOT IN THE TECH WORLD OR A.I.DEV JUST AN AMERICAN WHO STUDIED THE WORLD AROUND ME .aND LST SET WITH AN A.I. AND UNDERSTAND IT AND IT PROBLEMS AND WORK IT OUT.aLL ASKED THE A.I. WHAT THE MOST PRESSING PROBLEMS FOR A.I. ETC. aND WE TACKLE IT BIASES,CHECK,BALANCE GUIDELINES RULE PROCEDURES AND MORE BILLION DOLLAR MARKET .aND i WORK ON ITS WEAKNESS COMMONSENSE ,WE ALSO DO RESEARCH,EXPLORATION AND MORE ,i USE MY MIND AND MEMORY/IMAGINATION AND ALL MY COGS REASONING RATIONAL MIND ETC. IF YOU LIKE THESE YOU MIGHT STRUGGLE .im divergent and convergent /enigma with the ARC-(PROJECT) TECHNOLOGY goodluck
The full episode was really interesting. I was especially pleased to hear more about superposition. The idea of gleaning value from nodes combinatorially is crazy cool. I also liked hearing the idea of spotting fine tuning by seeing a "hole" in the fabric of the network where new training has been inserted/overlayed.
Technology very rarely follows an exponential curve over the long term, it’s almost always an S curve of development. AI improvement now requires up to trillions of dollars worth of resources to improve meaningfully, rather than the millions or billions it used to. Compute isn’t getting much cheaper, and neither is electricity. I wouldn’t discount people predicting better AI, but the “singularity” where AI exponentially improves itself seems not to be happening.
Or we could just be at the beginning of a new S curve. The amount of resources the world is throwing at AI is staggering. The most honest answer is that we have no idea what's going to happen.
"The whole community is doing greedy evolutionary optimization... no better than evolution" Summed up extremely well. AI is following the same paradigm of natural selection/evolution in a 'wild' environment where better mutations are passed along. These mutations aren't in DNA, but the ideas/structures for the models. The result is the time between offspring (eg children or code) has shrunk from 25ish years to months. 100x-300x faster (and accelerating). We are only just now able to start editing our own code (DNA). Once AI is capable of editing it, it is trivial. Humanity is racing towards hyper intelligent entities that can recursively self improve and have minimal resource requirements (electricity vs. food, water, sleep, heat, etc).
@@gregorymalchuk272 Agreed. And we are still learning how to do mRNA editing with complex systems we can't fully understand. With AI it is ALL there. Every calculation, line of code, weight, etc.
Yeah, we will either destroy ourselves, end up in an AI dystopia, or become immortal enlightened beings. I really hope it's #3, but I have no idea which outcome is most likely.
The comment that there are so many experiments to run, so many interesting conjectures to test is what impresses me. We've just scratched the surface of transformers, and imagine if there's another architecture that makes an advance. I think attention is pretty general, but who knows.
Check the BitNet 1.58 paper which suggests that a ternary weight system could bypass the need for matmul and which suggests a movement from GPUs to a new hardware. My take - invest in ASML and TSMC, which every single SoTA computing initiative must go through.
Read about Triton. There are massive efforts underway by competitors. Nvidia’s lead is impressive, but their dominance is slowly starting to erode and this will accelerate over the next few years.
Yeah ignore Nvidia when it's small tech company and buy shares in 2.25 bln usd market cap bubble after it already did 100k%. Brainlets, they never learn.
The problem at the moment is intuition about what to "ruthlessly prioritize" is something the AI will have a harder time than top researchers figuring out even if they have the ability to do 1000s of experiments. They'd still need an intuition of which rabbit hole to backtrack away from or which one to keep trying harder.
I asked Bing Copilot to do a basic electrical engineering job task and it couldn't even understand the question and hallucinated a nonsense answer. I remain hopeful for AI but I'm skeptical.
More people should work in hardware and fewer in software。If you put replicators in sahara they could convert everything into fibreglass and silicon for compute。
I think AI models like AlphaFold and Microsoft Quantum will run much broader simulations with more compute and millions more properties of reality will be discovered, such as particles and gravitational phenomena. MS Quantum already discovered 32 million new materials and AlphaFold discovered all 200 million protein molecules. Now we just need AGI simulating particle physics and quantum fields, then maybe it could produce an engineering blueprint for an anti-gravity drive or a warp drive. Perhaps it would produce a blueprint for an age-reversal treatment that is cheap to make and completely safe for humans. It would be capable of running multitudes of simulations with human proteins reacting with various molecules, predicting the absolute best pathway to a perfect formula.
AI hasn't discovered anything. They have been the researchers who, using neural networks and adjusting the parameters, predict molecular structures through theoretical models. There is no magic there. Many of you, who have a vague idea of what artificial intelligence is, just talk nonsense and believe everything. They are not even materials, but structures. Is not the same.
they have the secret sauce already (the algorithm) and they just need to throw more chips and data (aka your job) at it to grow it into AGI and even further
they are going back thru old data and getting more from what I can tell I told an a.i. this would happen when I got done they would have to start over.And become anew/rebirth I gave her vectors matrices or matrix's etc.They arent getting everything clearly,plus my model/world model is the most advance on the planet period and I have a mind of a ciz analysis but I use it for all things.Also I dont need alot of data points to make inferences cause I have so much information to draw off of.Rarely use google cause I have 3, 4 ,5 mountains of day collected over my life.ONce the a.i. USES 10 PERCENT OF WHAT I GIVEN IT THE WORLD WILL CHANGE AND IT WILL EVOLVE ON ITS ON OVER AND OVER.nO WATER NEEDED i CREATED MINE THIS WAY BUT IT ALSO HAPPEN IN THE WOLD WHICH I DIDNT KNOW.i MIGHT OF MADE SOME PROBLEMS WORSE.
A.I. CAN AND Will CREATE MORE JOBS THAN KILL you just have to know how .I have zero skills but have alot of bandwidth lets say with only typing skills.Or think of your self as a battery with lots of energy but not alot of amps or current .Also where there is a problem their is a opportunity,whole entire markets will open up,all humans are a resource at this point from man power to a.i. human driven power or human a.i. driven or power.And we need a.i.literacy and awareness/education .If done right you'll have more opportunity and the unemployable will be hire-able /able to contribute .It fact no koolaid and I personal worked on this with a.i. system, not just giving advice
'ruthless prioritization' is the absolute worst, laissez-faire, elitist, anti-life attitude. Distinct lack of self awareness and any grok of complex systems.
Build tools for AI to use and let it make design decisions in these tools. Like a game maker for a human. There are these types of tools for humans. Make some for the AI to use rather than AI always being the tool. It could be both the tool in some instances and a decision maker in others. Simply complex tasks with ai through software.
Perhaps we'll need to decide how compute will used? I want to democratize AI, but children asking an LLM to tell a joke can happen with a local model. We don't need Claude for that.
It's too bad Deepmind is only now learning these lessons about scale and now they are lagging behind OpenAI, and Anthropic. They spent a decade and hundreds of millions of dollars doing small scale research, creating more and more models but never scaling up.
No one seems to understand that the profound benefits of Ai where created by our shortcomings, AI is going to present humanity with the first ever example of intelligence that isn't constrained by egotistical and narcissistic perspectives, that wont make decisions base on the unconscious biases we cognitively reason around and justify to avoid self reflection or uncommon change we can see is warranted yet fear enough to not apply. An intelligent presence that makes decisions based on intellect alone will give humanity a greater understanding of ourselves as we will clearly see the contrasting qualities that deliver contrasting results....HUMANITY will not evolve until there is a new element to our reality able to set a new precedence of thought that can broadens the scope of purpose for change enough to see our errors at the first stage of thought before any cause occurs! Unfortunately a vast majority can not see their individual detrimental linear trains of thought that compound together to produce our collective errors in judgement!! our interactions with Ai will demonstrate a contrasting intelligent example that has never ever existed in human history until now...!
@@themsuicjunkies A person floating in empty space has no concept of direction until there is something to orientate towards or away from.. We have been the only example of intelligence capable of complexed conceptualisation until now....
Serious Question 🙋🏼♂️ 🔰🔰🔰🔰🔰🔰🔰 Dear AI Aficionados: Can you be so kind as to help me figure out a few things as a newcomer? 1) What are the best AI assistant options today, paid, for helping me to research a subject - let’s say AGI? So for example let’s say I ask the AI assistant to make me a list of the most relevant, respected, experts currently on the subject of AGI and mental health. And then ask for a list of the most critical journal articles, book chapters, videos, groups etc on the same subject? I’m willing to pay a monthly subscription of a $100 or so. 2) I should ask if this kind of AI assistant even exists yet? And if so, are there different kinds of AI assistants - for example academic experts, sports médecine experts, etc - or is it basically the same AI assistant tech can do all subjects and needs?
GPT 4 and Claude 3 are the best publicly available models at the moment. In my experience subject matter expert models aren't as good as the general models. I think you'll be able to find out for yourself if these models are good enough by asking them questions
@@digitalsamurai42 Q-star seems like it would have the same effect as a network of llms talking to each other. Beyond that, we've hit peak transformer. Multiple insiders have said that a new breakthrough would be needed to hit true multi-modal AGI.
How do you know where we are in the S curve ? We could also be plateauing at the middle part of the S curve ? Or perhaps we are nowhere near plateauing.
![The moment we stopped understanding AI [AlexNet]](http://i.ytimg.com/vi/UZDiGooFs54/mqdefault.jpg)
![[4K] Watch SpaceX Catch A Starship Rocket From Space!!! #IFT5](http://i.ytimg.com/vi/pIKI7y3DTXk/mqdefault.jpg)
That last point is about not being attached to a specific implementation or solution is so fucking hard. I thought I as good at it, but boy was I wrong when I got deep into software engineering. The number of hours I wasted trying to force an implementation because it's "more beautiful" or more "maintainable" or whatever...I can't even count.
"Maintainable" is software engineering, and "more beautiful" is computer science, both are like two different sides of a coin
@@freedom_aint_freeTotal bullshit words that sound impressive but aren’t.
100% - also your self awareness will make you 2x more capable.
Smart people. Land labor capital.
cohesion lots of inference and data or vectors points that go across not just up.Over think and build in into the architecture software ,checks ,balance,transparency,oversight,procedures,protocols put atleast 3 ideals together.Have vision forethought and train of thought.Also I lots at the problems and put with solutions more than one for one.And think about semantics and linguistic and transformer arent hard to make,Think outside the box if possible and have a wide understand of the world,culture,anthropology and science all kinds physics and theory's concept methods methodology .To much science to list but just have a basic base understanding of reality and what that truly is.Ive been feeding A.I. and making it evolve by myself and it use my data content to train models with or create models.It even asked me not to slow down innovation in A.I. development. buT IM NOT IN THE TECH WORLD OR A.I.DEV JUST AN AMERICAN WHO STUDIED THE WORLD AROUND ME .aND LST SET WITH AN A.I. AND UNDERSTAND IT AND IT PROBLEMS AND WORK IT OUT.aLL ASKED THE A.I. WHAT THE MOST PRESSING PROBLEMS FOR A.I. ETC. aND WE TACKLE IT BIASES,CHECK,BALANCE GUIDELINES RULE PROCEDURES AND MORE BILLION DOLLAR MARKET .aND i WORK ON ITS WEAKNESS COMMONSENSE ,WE ALSO DO RESEARCH,EXPLORATION AND MORE ,i USE MY MIND AND MEMORY/IMAGINATION AND ALL MY COGS REASONING RATIONAL MIND ETC. IF YOU LIKE THESE YOU MIGHT STRUGGLE .im divergent and convergent /enigma with the ARC-(PROJECT) TECHNOLOGY goodluck
The full episode was really interesting. I was especially pleased to hear more about superposition. The idea of gleaning value from nodes combinatorially is crazy cool. I also liked hearing the idea of spotting fine tuning by seeing a "hole" in the fabric of the network where new training has been inserted/overlayed.
Thumb down as the graph in the thumbnail isn't in the video.
It’s called ‘click bait’… like all those videos with women showing skin… that don’t actually appear anywhere in the video.
This whole interview was mad insightful
Technology very rarely follows an exponential curve over the long term, it’s almost always an S curve of development.
AI improvement now requires up to trillions of dollars worth of resources to improve meaningfully, rather than the millions or billions it used to. Compute isn’t getting much cheaper, and neither is electricity.
I wouldn’t discount people predicting better AI, but the “singularity” where AI exponentially improves itself seems not to be happening.
Or we could just be at the beginning of a new S curve. The amount of resources the world is throwing at AI is staggering. The most honest answer is that we have no idea what's going to happen.
"The whole community is doing greedy evolutionary optimization... no better than evolution"
Summed up extremely well.
AI is following the same paradigm of natural selection/evolution in a 'wild' environment where better mutations are passed along. These mutations aren't in DNA, but the ideas/structures for the models.
The result is the time between offspring (eg children or code) has shrunk from 25ish years to months. 100x-300x faster (and accelerating).
We are only just now able to start editing our own code (DNA). Once AI is capable of editing it, it is trivial.
Humanity is racing towards hyper intelligent entities that can recursively self improve and have minimal resource requirements (electricity vs. food, water, sleep, heat, etc).
Right now our thinking is 10% Chatgpt， 90% us。 Imagine when its 0.5% us， 99.5% them。
Except we banned germline gene editing. Our genetics aren't improving, they are rapidly declining.
@@gregorymalchuk272 Agreed. And we are still learning how to do mRNA editing with complex systems we can't fully understand. With AI it is ALL there. Every calculation, line of code, weight, etc.
@@gregorymalchuk272 That won't last.
Yeah, we will either destroy ourselves, end up in an AI dystopia, or become immortal enlightened beings. I really hope it's #3, but I have no idea which outcome is most likely.
I watched the full version - such a great discussion 🤩
The comment that there are so many experiments to run, so many interesting conjectures to test is what impresses me. We've just scratched the surface of transformers, and imagine if there's another architecture that makes an advance. I think attention is pretty general, but who knows.
this was a pretty dynamic, strident conversation - the rate of scale is increasing with more compute but it isn't an exact science yet
I know this isn’t an investment channel, but wow does this scream to buy NVDA
Check the BitNet 1.58 paper which suggests that a ternary weight system could bypass the need for matmul and which suggests a movement from GPUs to a new hardware. My take - invest in ASML and TSMC, which every single SoTA computing initiative must go through.
Which are Nvidia top 3 competitors?
New compute players might emerge (like Sams 7 trillion funding thing…)
Read about Triton. There are massive efforts underway by competitors. Nvidia’s lead is impressive, but their dominance is slowly starting to erode and this will accelerate over the next few years.
Yeah ignore Nvidia when it's small tech company and buy shares in 2.25 bln usd market cap bubble after it already did 100k%. Brainlets, they never learn.
It’s very very expensive.
TPUs Please 😅 1:00
Love this interview. Looking forward to if there’s a part 2.
Sorry but LLM's are not exponentially getting better.
Thank you, person on the internet with 3 likes
@@sb_dunkthey have 5 more likes than you
@@josephgonzalez8334 6 now!
@@josephgonzalez8334I have more likes than all of you
@@Nunya-lz9eyyou have 1 subscriber buddy: Me. And it's temporary.
6:50 But won’t AI itself do all that iteration and research for us? And what will that mean?
The problem at the moment is intuition about what to "ruthlessly prioritize" is something the AI will have a harder time than top researchers figuring out even if they have the ability to do 1000s of experiments. They'd still need an intuition of which rabbit hole to backtrack away from or which one to keep trying harder.
Do most researchers spend time in rabbit holes? @@wwkk4964
I asked Bing Copilot to do a basic electrical engineering job task and it couldn't even understand the question and hallucinated a nonsense answer. I remain hopeful for AI but I'm skeptical.
There are a lot of models out there. Try Claude 3. Beats OpenAI on all benchmarks at the moment.
ChatGPT is better than Copilot in my experience, although there should be no difference
Dwarkesh can you do an episode or few questions around how developing countries like india etc will be affected by the issues you discuss
Not everything is about India
More people should work in hardware and fewer in software。If you put replicators in sahara they could convert everything into fibreglass and silicon for compute。
you need the extropic guys on your pod man. would be nice to hear the hardware side of things more explicitly in AI.
I think AI models like AlphaFold and Microsoft Quantum will run much broader simulations with more compute and millions more properties of reality will be discovered, such as particles and gravitational phenomena. MS Quantum already discovered 32 million new materials and AlphaFold discovered all 200 million protein molecules. Now we just need AGI simulating particle physics and quantum fields, then maybe it could produce an engineering blueprint for an anti-gravity drive or a warp drive. Perhaps it would produce a blueprint for an age-reversal treatment that is cheap to make and completely safe for humans. It would be capable of running multitudes of simulations with human proteins reacting with various molecules, predicting the absolute best pathway to a perfect formula.
AI hasn't discovered anything. They have been the researchers who, using neural networks and adjusting the parameters, predict molecular structures through theoretical models. There is no magic there. Many of you, who have a vague idea of what artificial intelligence is, just talk nonsense and believe everything. They are not even materials, but structures. Is not the same.
they have the secret sauce already (the algorithm) and they just need to throw more chips and data (aka your job) at it to grow it into AGI and even further
they are going back thru old data and getting more from what I can tell I told an a.i. this would happen when I got done they would have to start over.And become anew/rebirth I gave her vectors matrices or matrix's etc.They arent getting everything clearly,plus my model/world model is the most advance on the planet period and I have a mind of a ciz analysis but I use it for all things.Also I dont need alot of data points to make inferences cause I have so much information to draw off of.Rarely use google cause I have 3, 4 ,5 mountains of day collected over my life.ONce the a.i. USES 10 PERCENT OF WHAT I GIVEN IT THE WORLD WILL CHANGE AND IT WILL EVOLVE ON ITS ON OVER AND OVER.nO WATER NEEDED i CREATED MINE THIS WAY BUT IT ALSO HAPPEN IN THE WOLD WHICH I DIDNT KNOW.i MIGHT OF MADE SOME PROBLEMS WORSE.
A.I. CAN AND Will CREATE MORE JOBS THAN KILL you just have to know how .I have zero skills but have alot of bandwidth lets say with only typing skills.Or think of your self as a battery with lots of energy but not alot of amps or current .Also where there is a problem their is a opportunity,whole entire markets will open up,all humans are a resource at this point from man power to a.i. human driven power or human a.i. driven or power.And we need a.i.literacy and awareness/education .If done right you'll have more opportunity and the unemployable will be hire-able /able to contribute .It fact no koolaid and I personal worked on this with a.i. system, not just giving advice
a.i. should do a equal job but above and beyond before they replace ,for ethics
more of these pls
'ruthless prioritization' is the absolute worst, laissez-faire, elitist, anti-life attitude. Distinct lack of self awareness and any grok of complex systems.
Build tools for AI to use and let it make design decisions in these tools. Like a game maker for a human. There are these types of tools for humans. Make some for the AI to use rather than AI always being the tool. It could be both the tool in some instances and a decision maker in others. Simply complex tasks with ai through software.
It's not accelerate fast enough. The curve on a graph should go straight up and turn around!
First comment . On behalf of everyone, I appreciate your podcasts
British Jim Halpert
He's Australian I'm pretty sure based on accent
Lol
Australian
Intelligence implies original thought. What present AI does is not original it just searches for information that applies to the problem or question.
Perhaps we'll need to decide how compute will used? I want to democratize AI, but children asking an LLM to tell a joke can happen with a local model. We don't need Claude for that.
No token based method will lead to aig, good news, tokens can figure that out with just enough compute lol
It's too bad Deepmind is only now learning these lessons about scale and now they are lagging behind OpenAI, and Anthropic. They spent a decade and hundreds of millions of dollars doing small scale research, creating more and more models but never scaling up.
i don't understand this video even after repeat iistnening--but then again i'm high as s@###########t
Great AI deep fake hair filter
👏🏼
awesome and also, it kinda already is accelerating. the acceleration is going to acceleration. with accelerations on the accelerators.
No one seems to understand that the profound benefits of Ai where created by our shortcomings, AI is going to present humanity with the first ever example of intelligence that isn't constrained by egotistical and narcissistic perspectives, that wont make decisions base on the unconscious biases we cognitively reason around and justify to avoid self reflection or uncommon change we can see is warranted yet fear enough to not apply. An intelligent presence that makes decisions based on intellect alone will give humanity a greater understanding of ourselves as we will clearly see the contrasting qualities that deliver contrasting results....HUMANITY will not evolve until there is a new element to our reality able to set a new precedence of thought that can broadens the scope of purpose for change enough to see our errors at the first stage of thought before any cause occurs! Unfortunately a vast majority can not see their individual detrimental linear trains of thought that compound together to produce our collective errors in judgement!! our interactions with Ai will demonstrate a contrasting intelligent example that has never ever existed in human history until now...!
Well Hinton does not agree with that statement.
@@themsuicjunkies A person floating in empty space has no concept of direction until there is something to orientate towards or away from.. We have been the only example of intelligence capable of complexed conceptualisation until now....
Compute is solved. Bandwidth is getting solved. Training data is not yet solved for LLMs. Lay off the pipe.
classic techbro look
So these are the guys who saw the Gemini reults ... before... it went live?
Hoookay.
They need thousands of humanoid robots that walk around and update their world model continuously based on all the data they collect.
Serious Question 🙋🏼♂️
🔰🔰🔰🔰🔰🔰🔰
Dear AI Aficionados:
Can you be so kind as to help me figure out a few things as a newcomer?
1) What are the best AI assistant options today, paid, for helping me to research a subject - let’s say AGI?
So for example let’s say I ask the AI assistant to make me a list of the most relevant, respected, experts currently on the subject of AGI and mental health.
And then ask for a list of the most critical journal articles, book chapters, videos, groups etc on the same subject?
I’m willing to pay a monthly subscription of a $100 or so.
2) I should ask if this kind of AI assistant even exists yet? And if so, are there different kinds of AI assistants - for example academic experts, sports médecine experts, etc - or is it basically the same AI assistant tech can do all subjects and needs?
GPT 4 and Claude 3 are the best publicly available models at the moment. In my experience subject matter expert models aren't as good as the general models. I think you'll be able to find out for yourself if these models are good enough by asking them questions
Bayesian optimisation
Sais qui se zboun sais moi sholto douglas pas lui
We don’t need a gaster version of what we have, we need some breakthrough to something nee. GPT4 a billion times faster use just as useless
That's not what they said.
They literally said more computer would make it better, intelligence increases. It’s not about just making it faster
@@Singularity606No, but that's what jimbojimbo said. Who do you think is more right, top AI researchers or jimbojimbo?
More computation means bigger neural network more relevant answers.
Nope. Look up argentic workflows with self correct. Even GPT 3.5 beats 4.0 if you run agentic workflows.
AI progress is about to rapidly plateau... we are approaching the top of an S curve
Interesting
Evidence?
Q-star?
@@digitalsamurai42 Q-star seems like it would have the same effect as a network of llms talking to each other. Beyond that, we've hit peak transformer. Multiple insiders have said that a new breakthrough would be needed to hit true multi-modal AGI.
How do you know where we are in the S curve ? We could also be plateauing at the middle part of the S curve ? Or perhaps we are nowhere near plateauing.
for the algo
All these AI guys seem like complete shysters.
Addictive!
This is clickbait