I still think Google is going straight for the gold with an organic computer, their trajectory already will/is change(ing) medicine and material science forever and would suit the pathway. Organics is best
I honestly love your willingness to break down artificial neural networks because of how many people talk about it assuming the audience already has this understanding. As someone who has a genuine interest in wanting to understand this topic, thank you!
Totally agree! It’s so refreshing to see content creators take the time to explain these complex topics without assuming everyone has a PhD in AI. Glad you're finding it helpful too-understanding the fundamentals really opens up a whole new world of possibilities! Cheers
AI and Robots will do all the work and the whole human race can take an infinite vacation. How realistic do you think this will be. Anyone?.....Anyone? :p
Right?! Sometimes it feels like the world is heading toward a tech dystopia, but seeing the potential of AI and tech innovation like this definitely brings back some optimism. So much to look forward to!
@@TechnologyBeyondTomorrow-TBTIt's because most in the mainstream media hate technological innovation because it threatens their established power. No newspaper likes the Facebook or Google because the only reason they still have readership is because they still play nice with them.
Hi @theAisearch, the main issue with neuromorphic chips is that you can't easily duplicate and transfer the learned weights from one chip to another. This means that if you spend a month training a robot to perform a task and it breaks, the replacement chip would need to be retrained from scratch. Btw love the channel.
Great point! That’s a big hurdle for neuromorphic tech right now. Imagine how amazing it would be if we could just "clone" a chip’s knowledge! Hopefully, with more R&D, they’ll crack that and make the tech even more practical. Cheers
Let's be real here at this point we could be using AI in a few years to make all those imaginary mechanical human body parts, space ships, discovering new metal, bio engineering complex stuff and I think a big revolution is coming really soon with the speed of the development of ai.
I'm just waiting for the time AI will be able to keep up with the human imagination and come up with stuff we were never capable of even thinking in the next ten years.
@@theAIsearch you can see a prosthetic forearm and hand prototype for humans and androids that I did using TECM (Thermoelectric Contraction Artificial Muscles) who are lightweight, totally silent, compact size, anatomically realistic and 12 times stronger than the human muscles at the same mass and not heavy, noisy and bulky gearboxes required. I was developing it since 2013 waiting for this time.
Spot on! Marketing can be tricky with tech specs-it sounds like 2nm should mean something literal, but it’s more about industry naming conventions. Still, even if the numbers don’t always add up, the innovation behind it is pretty impressive!
"And video generators like Sora." It's amazing how a big brand can promise something and release nothing, and still be the go-to example when actual products have been deployed that are at the very least competitive. I'm not blaming you, I get how name recognition works and why it might be easier to say something most people will recognize, it's just funny. I sure hope Sora is awesome, I'm guessing it will be, but for all intents and purposes right now it doesn't exist. Sorry, I know that's not the point of the video but as soon as I heard it I couldn't help but comment. I'm gonna go back to the video now 😛
@@theAIsearchi feel like with everything currently going on, video generation isn't even that interesting anymore (atleast to me) We know that it's coming sooner or later anyways
Haha, love the honesty! It's wild how name recognition can overshadow actual progress. Sora definitely has the potential, but like you said-waiting for the real deal. Fingers crossed it lives up to the hype when it finally drops!
The problem with neuromorphic processors is that you need 1 neuron per 1-2 parameters and an insane amount of connections. E.g. for something the size of GPT-4 need close to a trillion neurons. The IBM chip mentioned in the video has just a bit more than 1 million (256*4096). It has a long way to go.
You’re totally right-the scale needed for neuromorphic processors to rival models like GPT-4 is immense. We’ve still got a long way to go, but it’s exciting to think about where this tech might be in a few years.
🇧🇷🇧🇷🇧🇷🇧🇷👏🏻, This video is incredibly insightful and enlightening! It's refreshing to see the elephant in the room finally being addressed. Thank you for sharing such an informative and impactful video - I’m truly impressed by the depth of information presented.
I can’t shake the feeling that the real advancement we’re going to see in the next five years is not in the architecture (Transformers are fine), or in the hardware as such (SNNs are not an improvement in training efficiency, unless you’re achieving high training efficiency by deploying the model on SNN hardware at every training step to do some sort of agentic self-improving AI with tree search), but rather, an improvement in the loss function. I truly believe that we might come up with some limited version of back propagation (similar to how limited Boltzmann networks were finally able to achieve parallelizable training steps) or some smarter loss function that massively improves the sample-efficiency of training networks. One of the big problems right now is that we don’t have great ways of teaching “what we want the network to learn” in our training steps with each example, and the networks aren’t hitting human level sample efficiency. It might be hierarchical continuous learning (ie: some derivative of active inference), it might a neural network that generates the most efficient possible loss function from the output and data, it might be some method of directly imparting a change in parameters from a teacher-model based on a tree search, or it might be something else, but whatever it is, I think it’s something that will allow for localized training that minimizes the communication penalty between devices and allows for higher dimensionality of parallelization, or allows way higher sample efficiency such that the communication penalty just doesn’t matter anymore.
You make a great point! Improvements in loss functions and training efficiency could be the real game-changer. It’s not just about better hardware-optimizing the way we train models could drastically reduce the resources needed and push AI even further. Amazing insight!
Spiking neural networks are not new. They are decades old, but don’t really work that well, because there still isn’t a universal training algorithm for them like backpropagation for artificial neural networks. There are papers that have implemented some modifications to backpropagation for SNNs, but the accuracy of SNNs is below that of a ANN.
@@MiraPloy once the network is trained, the forward pass works fine, but the energy requirements don’t really go down that much, because the spikes arent that sparse enough. Efficient ANN models still perform better energy-wise. Chips that directly implement SNNs probably perform the best, but we‘re not there yet. Also, transformers currently don’t run on specific hardware for them, but rather GPUs, so it‘s not a fair comparison. There are companies trying to create silicon specifically for transformers and this may be the most promising solution. They would probably be multiply times more efficient than GPUs so the energy constraints would again be irrelevant.
You're absolutely right! SNNs have been around for a while, but they’ve struggled to catch up with ANN's efficiency. Hopefully, as hardware improves, we’ll see some breakthroughs that make SNNs more viable. Cheers
I was shocked when I found out that llama3 8b in a few of its layers typically uses exactly 1 out of the 4096 features it could be using, such that I was able to set all the other ones to 0 and still get the same output. AND YET, I was unable to find any implementation which was faster in performing the math than just running the dense matrix multiplication of all the values, involving the >99% of operations involving multiplication by 0. Even downscaling the matrix and only using the relevant parameters, by the time you manage to have found the parameters to use, the dense matrix multiplication would have finished already. I tried using regular sparse matrix multiplication, but guess what, unless sparsity is >~97.5%, it's slower than dense multiplication. So, you can set like more than half the features in llama3 to 0, but not manage to gain any speedup at all. Past removing 70% the model becomes unstable.
Were you doing this on GPU or CPU? It’s pretty difficult to get the benefits of sparsity in a GPU. It can also be challenging to get the full benefits of sparsity in a pre-trained model, and my go-to is generally to suggest QAT + MoE to still from the dense model into a more deployable and inherently sparse model via expert sparsity than parameter sparsity (The OLM 1B-7B has a really interesting approach to MoE, as does the Sigma-MoE described by the work of csordas et al in “Approximating two layer feed forward network for efficient transformers” I think it was, which seems to perform pretty close to dense baselines). The trick is that you’re not trying to do some convoluted dense -> sparse runtime optimization, but rather, to produce explicit sub-networks that are smaller, but are calculated as efficient, dense networks.
@@novantha1 Yes, the experiments were on GPU. I have this "vision" of the perfect linear layer which will only use the parameters it needs, is able to pick as many or few parameters as it wants, tends to use a small subsection of its full parameter space per forward pass... and disregarding GPUs just hating not calculating something, I pretty much got it. Nevertheless, I was also thinking about achieving the thing you were talking about - smaller approximations of FF networks. The nice thing with the thing I did is that I get to see and keep track of which features it uses when. It's interesting for interpretability and also gives a decent estimate for how much the dimensionality of the FFs could be reduced. I'll definitely check out the paper and concepts you mentioned. Anyhow, I really would like our GPUs to start handling sparsity better. It would open the doors for many really cool concepts.
@@Alice_Fumo That’s less a failure of GPU implementations and more a failure of the fundamental design of a GPU; GPUs have tens of thousands of tiny cores that don’t really have knowledge of things like branches, and they’re designed to have all of them do essentially the same mathematical operation on all the data entering any of the cores (the problem being that you want to execute conditionally on the type of data coming on a core by core basis). You can sort of do a few things to mitigate it, but that’s less “taking advantage of sparsity” and more bailing out the titanic with a spoon. Yes, it’s a noble goal, and probably a good idea if you’re waiting to get on the lifeboat, but isn’t really the best overall approach. Like I noted, CPUs take way better advantage of sparsity, as do “CPU adjacent” designs like Tenstorrent’s accelerators. I know it’s not as exciting as running your models faster on GPUs, but to get the kind of parallel device you were describing is essentially just a CPU, and if you had that much conditional knowledge in the GPU cores you’d essentially just have a multi core CPU, anyway.
@@novantha1 I know this, which is why my approach was to construct a smaller dense matrix from the full one on the fly in a branchless fashion. It was faster with very high sparsity... But in worst cases led to like 20x slowdowns
Wow, that’s fascinating! It’s crazy how sometimes even in AI, we end up doing a ton of unnecessary calculations. Makes you wonder how much room there still is to optimize these models. Thanks for sharing that deep dive!
Thermodynamic computers are definitely an intriguing area! They’re still in early stages, but if we can harness the principles of physics better, who knows how it could impact computing efficiency in the future. Something to watch for sure!
Neuromorphic chips are us putting code we use to emulate how we think the brain works on its own native hardware instead of emulating it on transistors. That is kind of funny. Usually it is the other way around and someone is trying to emulate how they think transistor chips work by then, funny enough, emulating that on top of more transistors.
The comparison of the human brain’s energy efficiency vs. GPUs really puts things into perspective! It’s exciting to see how neuromorphic chips could push AI forward with similar efficiency.
Different specialties. A GPU can render a 4k frame with raytracing in a game with just a few joules of energy. It can also do trillions of calculations per second.
I've been saying this for years, but we need to create one large robotic replica of a human brain to try to replicate its functions as much as possible and even create a fake virtual body. Then slowly make it smaller and smaller while learning about its efficiency, and this would also help us understand mental health issues better as well as any other neural based issues in the human body.
Feeling so optimistic about hardware development for AI. Combined with wise architectural solutions in software, hell yes, it's gonna be super-efficient.
Hi, I am absolutely fascinated by SNNs too! I believe biomimicry is probably one of our best bets at realistically attaining smart and affordable AI. However, I'd like to know, why is it that Transformers still dominate in the field right now? Why are SNNs lagging behind them in most domain currently, besides the low power use cases? I also think Neuromorphic Computing is great, but so far we haven't really seen many applications for it. Should we expect any breakthroughs anytime soon?
SNNs are still being researched and the ecosystem is not as established. It's also not a good method to train SNNs due to their spiking nature. Plus they are still not as performant as transformers in various tasks
Great questions! SNNs are still finding their place, but probably transformers dominate for now because they scale so well. Maybe we’ll see SNNs surge ahead as low-power AI becomes more essential. Fingers crossed for breakthroughs soon🤞
Yes! Brainchip Akida is super interesting-neuromorphic chips like these could really push AI efficiency to new levels. It’s great to see tech like this finally coming to light.
great video as always, i really into spiky ai since it looks so promissing and can change everything if they can run top of the line ai's on way less processing power.
Currently, AI is not as advanced as the human brain, but it's only a matter of time. With new AI breakthroughs happening every week, we'll eventually reach human-level intelligence.
No that won't be possible the next decade if even at all. The computing power it requires is just way too much. Maybe if we get a breakthrough in quantum computing then maybe it will be possible but i am not sure if this will happen in out lifetimes tbh
The real miracle is that AI gives us a glimpse into the minds and souls of real girls. That is your key to a new girlfriend. Understanding her mystery.
Haha, AI is really branching out into every corner of our lives. Who knows, maybe in the future, AI relationships might be more common than we think! 😅
It's already here, as someone who dares real women I can say that chatbots already have the exact same personalities if you add this to the settings. You can even have a realistic abusive girlfriend or one with P.T.S.D.
I have AI burnout. I've spend too much time with AI generated Content, chatting with AI bots, AI voice, AI that AI this.... I want real human generated content, real human interactions and so on and I don't care if AI gets better and better. In the future I think when AI gets everywhere, a real human generated product or real human interaction will become very valuable Btw i really appreciate that you use your real voice
And be prepared because the next year, 2025, your burnout will scale up when those chatbots evolve, mutate, and become in agents, and they can see you, and you can see their reactions and they can turn on/off the light of your room haha, crazy world we will live in 🤪
AI burnout is real. There’s definitely something precious about human interactions and creativity that AI can’t replicate (yet). And yeah, real voices and authenticity will probably become the new luxury in the AI-saturated world! 🙌
At 13:00 you are close. It's not the number of pulses adding up pulses the get through. It's more like a combination lock of many inputs to a neuro called dendritic which learn which threshold of voltage impulse to respond too via LTP. Long term potentiaion, any impulse too low or too high in voltage the synapse will not fire the dendrite. Firing too quickly thereafter changes the LTP threshold. The many dendrites may need to fire in the right order and only the right ones too, for the neuron itself to fire. Some neurons have few dentrites and simple mechanisms, others have hundreds of dendritic where some even form feedback loops with the pre-synaptic phase of the incoming terminal button. 😮😊
The amount of intelligence it takes to build these computer systems, yet we are supposed to believe that the our brains and other life systems are a product of chance.
@apatsa_basiteni You’re on to something. :) The truth of the matter is that the unfathomably large universe and/or world we live in was not created by chance, but was designed by God. It literally makes no sense to say otherwise. I mean, a painting is created by a painter, other art work is made by artists, buildings are created by builders, and the entire world was created by THE Creator. Makes complete, logical sense.
It all sounds good, but if all these neuromorphic chips have been invented like what you say, why don't companies like OpenAI, Google, and so one use them?
22:05 memristors are not materials? They are devices. I'm not sure this was a simple oversight, or some truth to it, they being based on some special material? Usually TiO2 "metal oxides such as titanium oxides", but like for transistors (and say resistors) likely can use variety of materials or not always silicon. This is still a good overview, a bit (over?)simplified, abut with intriguing info like 29:36 "pixel-spike converter" in that chip I did not know about, so it seems like made for spiking neural networks.
Some researchers know this already, the best AI solution is to integrate human brains without them knowing it, or growing human brains to interface with AD/DA converters. It is very probable that this is happening right now. Most of human knowledge is in the unconscious incompetence state, that includes me. We don't know what we don't know and are not even aware of it. Regardless, it is a great time to be alive, God Bless.
Wow, that’s a mind-bending thought. The idea of brain integration with AI is both fascinating and frightening. We’re in uncharted territory, but it’s an exciting time to be alive indeed! 🙏
By end of 2024, we will see Chat 5.0 and this will be the first meaningful step towards mass unemployment. It will be the cue to businesses to begin restructuring. 100 employees will turn into 90+ Ai. And Gen Z will find it increasingly more difficult to break into the workforce.
Mass unemployment is a scary thought. Hopefully, we’ll figure out new ways for AI and humans to coexist productively without too much job loss, but it will require a new financial system to compensate and adapt to a new future
@@TechnologyBeyondTomorrow-TBT The owners of our country have to decide what kind of society they want. More dystopia or new age where all can prosper....
The Brain is a receiver or a Piezoelectric field generator & there is no physical data in your brain. This is why people can still function with half a brain after injuries.
"In fact, most of the time our brains filter out most sensory input, focusing on relevant stimuli or new changes in the environment, otherwise we would just be way too overwhelmed." *Laughs uncomfortably in autistic*
@@theAIsearch It's different for everyone, to some degree. Generally, autistic people experience what we call sensory sensitivity. There are a lot of senses, and the degree to which it effects each one is variable, hence the different for everyone bit. But in general, you can expect an autistic person has a set of stimuli, sound and light are common ones, that they are bad at filtering, and thus it feels like the world volume is on 110%... or more. It leads to overwhelm, which is what causes the stereotypical autistic meltdown. Over the long term, it leads to burnout, where we just shut down for an extended period. Most of us have to find ways to live our lives minimizing our overexposure, which is varying levels of challenging. I actually feel like autism and LLMs are at least superficially similar. At least, I feel a lot of kinship with the way they "think" and behave. Autistic people tend to be far more data driven and detail oriented, and we're very susceptible to the current stimulus. One theory about what is behind the autistic experience is the neurological process is different for us. Most people's brains prune neurological connections aggressively, but autistic brains don't seem to do that. So where neurotypical brains are like information superhighways going to specific places and unmaintained roads everywhere else, the autistic brain is like a complex network of well maintained but convoluted roads going everywhere, All of them are moderately fast, but the traffic is overwhelming and disordered, and changing direction can be complex. I see it sort of like pruning or fine tuning in an LLM. It becomes way better at the tasks it is trained on, and more efficient, but less broadly adaptable. The analogy is far from perfect, but close enough to be interesting IMO. Autism is a complex bag, though. So while you could maybe describe the neurological landscape in that way, you could also see the tendency of autistics to hyperspecialize on a special interest as a sort of self-supervised fine tuning on that task. We tend to get really good at the things we are interested in, but broadly less capable in things we aren't (commonly things like social norms or tradition).
Whoa! 🤯 Neuromorphic chips are literally the brainpower boost AI needed! It's crazy to think how mimicking our own neural structure could make AI 1000x more efficient. 🧠💡 Imagine the potential - energy-saving, smarter drones, wearables, and even robots? Neuromorphic chips could be the breakthrough tech we're all waiting for. 🚀 What's your take? Could we be witnessing the dawn of true AI evolution?
Right on! Neuromorphic chips could indeed be the revolution we’ve been waiting for-making AI smarter and more energy-efficient could lead to massive breakthroughs. Let’s see where this tech takes us!
Right? 4000 homes worth of power sounds huge, but when you think about it, AI scaling is going to demand even more. We’ve got to keep pushing for efficiency! 🔌
I feel like we are just stuck now Nothing new that drives us towards AGI everything exist we are just improved improving it And my dad still has a job lol Where is agi
AGI still feels like it’s just over the horizon, doesn’t it? I guess all of us are waiting for that next leap. And as for your dad's job-AI still has some catching up to do!
We could call the new "economy " something like a "Compassion Economy" or "Sentient-Centric Economy." If you want to throw the space-expansion aspect into it, maybe "Universal Compassion Economy" or "Ethical Expansion Economy." It’s like a future where empathy and exploration are at the core of everything-ending suffering and pushing boundaries.
I love the vision for a future where empathy and exploration are at the core! If AI can help us move towards that kind of economy, it would truly be revolutionary
I think it's both hardware and software. If we create AI code that skips 90% of the inefficient crap and have it focus on what's TRUELY important and then IF is need less crucial details, it can look them up, then it can save tremendous amounts of time and energy. For instance, we don't remember every single calculation in math. And 40b LLM on a 4090 doing math is stupid when a 1970s calculator could do the same thing but perfectly. Stop trying to get the LLM to do the math. Just send it to Windows Calculator, and copy and paste the answer. Just like a standard human brain would. It should use thinking structures, 6 Levels of Bloom's Taxonomy, Solondais PACER system, Socratic method etc. Use those as the way the AI processes data to save time and energy.
Yo, I have seen your RUclips videos, your videos are great, but you could need some help designing the thumbnails essential to driving viewers. I would be grateful to help you with thumbnails or any other service, like graphic design. Look forward to collaborating with you!
Damn. The games "SOMA, Nobody wants to die" are becoming reality. I can't believe that. Also the movies "immortel advitam", "blade runner" No way.... I hope that the future won't be bleak and dystopic... 😬
4:18 I think at some point AI will reach a point of diminishing return. The hardware requirements will not scale up at some point even with all the money spent.
Our brain doesn't store any information though. It is a receptor that receives information from elsewhere. Our memories are being received from somewhere else in the universe.
Few caveats. That big power is needed for learning not using AI models. Single use of AI model is very cheap. Human brain is not better in everything. For example simple calculator is much faster and precise in calculating things. I agree - discrete computing is bad tool for AI.
It took nature billions of years to cause all of this, our human knowledge and computer technology has only existed for an extremely tiny period of time compared to that and we may very well surpass all of our biological limitations within next million year or so...
Thanks to our sponsor Abacus AI. Try their new ChatLLM platform here: chatllm.abacus.ai/?token=aisearch
I still think Google is going straight for the gold with an organic computer, their trajectory already will/is change(ing) medicine and material science forever and would suit the pathway. Organics is best
What do you think about thermodynamic computing (Extropic)?
An ad?
I honestly love your willingness to break down artificial neural networks because of how many people talk about it assuming the audience already has this understanding. As someone who has a genuine interest in wanting to understand this topic, thank you!
Thanks for watching!
Totally agree! It’s so refreshing to see content creators take the time to explain these complex topics without assuming everyone has a PhD in AI. Glad you're finding it helpful too-understanding the fundamentals really opens up a whole new world of possibilities! Cheers
at this rate , i will be homeless by next year
That's better than me, I'm gonna be in 120 days
@@DokesConspiracyNetwork 💀💀
*next week
AI and Robots will do all the work and the whole human race can take an infinite vacation. How realistic do you think this will be. Anyone?.....Anyone? :p
@@vulcan4d it is realistic if mankind organize to make a r_v_l_t_n from a capitalistic system to a resource based economic system
This channel has single handedly given me hope for the future
Right?! Sometimes it feels like the world is heading toward a tech dystopia, but seeing the potential of AI and tech innovation like this definitely brings back some optimism. So much to look forward to!
@@TechnologyBeyondTomorrow-TBTIt's because most in the mainstream media hate technological innovation because it threatens their established power. No newspaper likes the Facebook or Google because the only reason they still have readership is because they still play nice with them.
Hi @theAisearch, the main issue with neuromorphic chips is that you can't easily duplicate and transfer the learned weights from one chip to another. This means that if you spend a month training a robot to perform a task and it breaks, the replacement chip would need to be retrained from scratch. Btw love the channel.
Great point! That’s a big hurdle for neuromorphic tech right now. Imagine how amazing it would be if we could just "clone" a chip’s knowledge! Hopefully, with more R&D, they’ll crack that and make the tech even more practical. Cheers
Let's be real here at this point we could be using AI in a few years to make all those imaginary mechanical human body parts, space ships, discovering new metal, bio engineering complex stuff and I think a big revolution is coming really soon with the speed of the development of ai.
i hope in this future we can have sex
exciting times ahead!
I'm just waiting for the time AI will be able to keep up with the human imagination and come up with stuff we were never capable of even thinking in the next ten years.
@@theAIsearch you can see a prosthetic forearm and hand prototype for humans and androids that I did using TECM (Thermoelectric Contraction Artificial Muscles) who are lightweight, totally silent, compact size, anatomically realistic and 12 times stronger than the human muscles at the same mass and not heavy, noisy and bulky gearboxes required. I was developing it since 2013 waiting for this time.
@@azhuransmx126Wow! Amazing 🤩 You developed that?
Actually a 2nm process node doesn't contain 2nm transistors, it's just marketing
Ohhh crap
Spot on! Marketing can be tricky with tech specs-it sounds like 2nm should mean something literal, but it’s more about industry naming conventions. Still, even if the numbers don’t always add up, the innovation behind it is pretty impressive!
"And video generators like Sora." It's amazing how a big brand can promise something and release nothing, and still be the go-to example when actual products have been deployed that are at the very least competitive. I'm not blaming you, I get how name recognition works and why it might be easier to say something most people will recognize, it's just funny. I sure hope Sora is awesome, I'm guessing it will be, but for all intents and purposes right now it doesn't exist.
Sorry, I know that's not the point of the video but as soon as I heard it I couldn't help but comment. I'm gonna go back to the video now 😛
Speaking of Sora.... will they ever release it?
@@theAIsearchi feel like with everything currently going on, video generation isn't even that interesting anymore (atleast to me)
We know that it's coming sooner or later anyways
Haha, love the honesty! It's wild how name recognition can overshadow actual progress. Sora definitely has the potential, but like you said-waiting for the real deal. Fingers crossed it lives up to the hype when it finally drops!
Excellent summary. Appreciate the completeness.
Thanks for watching!
The layout reminds me of the terminator chip.
The Terminator vibes are real! Let’s just hope these chips stay on the friendly side! 🤖🔥
The architecture is the key
Absolutely, architecture is key! A solid foundation is everything
The problem with neuromorphic processors is that you need 1 neuron per 1-2 parameters and an insane amount of connections. E.g. for something the size of GPT-4 need close to a trillion neurons. The IBM chip mentioned in the video has just a bit more than 1 million (256*4096). It has a long way to go.
You’re totally right-the scale needed for neuromorphic processors to rival models like GPT-4 is immense. We’ve still got a long way to go, but it’s exciting to think about where this tech might be in a few years.
Creativity.
Intelligence is true creativity.
Couldn’t agree more-creativity is the spark that drives true intelligence. Innovation keeps pushing boundaries!
🇧🇷🇧🇷🇧🇷🇧🇷👏🏻, This video is incredibly insightful and enlightening! It's refreshing to see the elephant in the room finally being addressed. Thank you for sharing such an informative and impactful video - I’m truly impressed by the depth of information presented.
I can’t shake the feeling that the real advancement we’re going to see in the next five years is not in the architecture (Transformers are fine), or in the hardware as such (SNNs are not an improvement in training efficiency, unless you’re achieving high training efficiency by deploying the model on SNN hardware at every training step to do some sort of agentic self-improving AI with tree search), but rather, an improvement in the loss function.
I truly believe that we might come up with some limited version of back propagation (similar to how limited Boltzmann networks were finally able to achieve parallelizable training steps) or some smarter loss function that massively improves the sample-efficiency of training networks.
One of the big problems right now is that we don’t have great ways of teaching “what we want the network to learn” in our training steps with each example, and the networks aren’t hitting human level sample efficiency.
It might be hierarchical continuous learning (ie: some derivative of active inference), it might a neural network that generates the most efficient possible loss function from the output and data, it might be some method of directly imparting a change in parameters from a teacher-model based on a tree search, or it might be something else, but whatever it is, I think it’s something that will allow for localized training that minimizes the communication penalty between devices and allows for higher dimensionality of parallelization, or allows way higher sample efficiency such that the communication penalty just doesn’t matter anymore.
You make a great point! Improvements in loss functions and training efficiency could be the real game-changer. It’s not just about better hardware-optimizing the way we train models could drastically reduce the resources needed and push AI even further. Amazing insight!
Transormers arent fine at all.
Spiking neural networks are not new. They are decades old, but don’t really work that well, because there still isn’t a universal training algorithm for them like backpropagation for artificial neural networks. There are papers that have implemented some modifications to backpropagation for SNNs, but the accuracy of SNNs is below that of a ANN.
Does it work better for inference?
@@MiraPloy once the network is trained, the forward pass works fine, but the energy requirements don’t really go down that much, because the spikes arent that sparse enough. Efficient ANN models still perform better energy-wise. Chips that directly implement SNNs probably perform the best, but we‘re not there yet. Also, transformers currently don’t run on specific hardware for them, but rather GPUs, so it‘s not a fair comparison. There are companies trying to create silicon specifically for transformers and this may be the most promising solution. They would probably be multiply times more efficient than GPUs so the energy constraints would again be irrelevant.
You're absolutely right! SNNs have been around for a while, but they’ve struggled to catch up with ANN's efficiency. Hopefully, as hardware improves, we’ll see some breakthroughs that make SNNs more viable. Cheers
I was shocked when I found out that llama3 8b in a few of its layers typically uses exactly 1 out of the 4096 features it could be using, such that I was able to set all the other ones to 0 and still get the same output.
AND YET, I was unable to find any implementation which was faster in performing the math than just running the dense matrix multiplication of all the values, involving the >99% of operations involving multiplication by 0.
Even downscaling the matrix and only using the relevant parameters, by the time you manage to have found the parameters to use, the dense matrix multiplication would have finished already.
I tried using regular sparse matrix multiplication, but guess what, unless sparsity is >~97.5%, it's slower than dense multiplication.
So, you can set like more than half the features in llama3 to 0, but not manage to gain any speedup at all. Past removing 70% the model becomes unstable.
Were you doing this on GPU or CPU? It’s pretty difficult to get the benefits of sparsity in a GPU.
It can also be challenging to get the full benefits of sparsity in a pre-trained model, and my go-to is generally to suggest QAT + MoE to still from the dense model into a more deployable and inherently sparse model via expert sparsity than parameter sparsity (The OLM 1B-7B has a really interesting approach to MoE, as does the Sigma-MoE described by the work of csordas et al in “Approximating two layer feed forward network for efficient transformers” I think it was, which seems to perform pretty close to dense baselines). The trick is that you’re not trying to do some convoluted dense -> sparse runtime optimization, but rather, to produce explicit sub-networks that are smaller, but are calculated as efficient, dense networks.
@@novantha1 Yes, the experiments were on GPU. I have this "vision" of the perfect linear layer which will only use the parameters it needs, is able to pick as many or few parameters as it wants, tends to use a small subsection of its full parameter space per forward pass... and disregarding GPUs just hating not calculating something, I pretty much got it. Nevertheless, I was also thinking about achieving the thing you were talking about - smaller approximations of FF networks.
The nice thing with the thing I did is that I get to see and keep track of which features it uses when. It's interesting for interpretability and also gives a decent estimate for how much the dimensionality of the FFs could be reduced.
I'll definitely check out the paper and concepts you mentioned.
Anyhow, I really would like our GPUs to start handling sparsity better. It would open the doors for many really cool concepts.
@@Alice_Fumo That’s less a failure of GPU implementations and more a failure of the fundamental design of a GPU; GPUs have tens of thousands of tiny cores that don’t really have knowledge of things like branches, and they’re designed to have all of them do essentially the same mathematical operation on all the data entering any of the cores (the problem being that you want to execute conditionally on the type of data coming on a core by core basis). You can sort of do a few things to mitigate it, but that’s less “taking advantage of sparsity” and more bailing out the titanic with a spoon. Yes, it’s a noble goal, and probably a good idea if you’re waiting to get on the lifeboat, but isn’t really the best overall approach.
Like I noted, CPUs take way better advantage of sparsity, as do “CPU adjacent” designs like Tenstorrent’s accelerators. I know it’s not as exciting as running your models faster on GPUs, but to get the kind of parallel device you were describing is essentially just a CPU, and if you had that much conditional knowledge in the GPU cores you’d essentially just have a multi core CPU, anyway.
@@novantha1 I know this, which is why my approach was to construct a smaller dense matrix from the full one on the fly in a branchless fashion.
It was faster with very high sparsity... But in worst cases led to like 20x slowdowns
Wow, that’s fascinating! It’s crazy how sometimes even in AI, we end up doing a ton of unnecessary calculations. Makes you wonder how much room there still is to optimize these models. Thanks for sharing that deep dive!
What a time to be alive!
😃
What about thermodynamic computers?
Thermodynamic computers are definitely an intriguing area! They’re still in early stages, but if we can harness the principles of physics better, who knows how it could impact computing efficiency in the future. Something to watch for sure!
Neuromorphic chips are us putting code we use to emulate how we think the brain works on its own native hardware instead of emulating it on transistors. That is kind of funny. Usually it is the other way around and someone is trying to emulate how they think transistor chips work by then, funny enough, emulating that on top of more transistors.
Yeah another day with AI Search. Thx man
Thanks for watching!
Another day, another deep dive into AI. Glad you’re sticking with it! 💪
The comparison of the human brain’s energy efficiency vs. GPUs really puts things into perspective! It’s exciting to see how neuromorphic chips could push AI forward with similar efficiency.
Different specialties. A GPU can render a 4k frame with raytracing in a game with just a few joules of energy. It can also do trillions of calculations per second.
I've been saying this for years, but we need to create one large robotic replica of a human brain to try to replicate its functions as much as possible and even create a fake virtual body. Then slowly make it smaller and smaller while learning about its efficiency, and this would also help us understand mental health issues better as well as any other neural based issues in the human body.
Feeling so optimistic about hardware development for AI.
Combined with wise architectural solutions in software, hell yes, it's gonna be super-efficient.
i hope so!
Optimism is contagious! With hardware and software advancing hand in hand, the future of AI does look incredibly promising🚀
Hi, I am absolutely fascinated by SNNs too! I believe biomimicry is probably one of our best bets at realistically attaining smart and affordable AI. However, I'd like to know, why is it that Transformers still dominate in the field right now? Why are SNNs lagging behind them in most domain currently, besides the low power use cases? I also think Neuromorphic Computing is great, but so far we haven't really seen many applications for it. Should we expect any breakthroughs anytime soon?
SNNs are still being researched and the ecosystem is not as established. It's also not a good method to train SNNs due to their spiking nature. Plus they are still not as performant as transformers in various tasks
Great questions! SNNs are still finding their place, but probably transformers dominate for now because they scale so well. Maybe we’ll see SNNs surge ahead as low-power AI becomes more essential. Fingers crossed for breakthroughs soon🤞
3:15 Assuming GPT-5 will be just another LLM? OpenAI might throw a curveball and come out with a whole new architecture, like a causal model.
GPT-5 might surprise us all! A whole new architecture would definitely shake things up in the AI world. 🚀
Take a shot whenever he says efficient.
I remember when scientists thought 5nm would be the phsyical limit for chips.
vram is everything when it comes to a.i
VRAM is like the oxygen for AI-can’t live without it!
28:50 Brainchip Akida
Yes! Brainchip Akida is super interesting-neuromorphic chips like these could really push AI efficiency to new levels. It’s great to see tech like this finally coming to light.
great video as always, i really into spiky ai since it looks so promissing and can change everything if they can run top of the line ai's on way less processing power.
Thanks!
Spiking AI could be a real game changer. Less power, more processing? That’s a winning formula if they can pull it off!
im so big noob in ai/ml i have no idea how any of this work. but i still understood the video .
what a chad ur.
like the difference of dvd & ssd
The evolution from DVDs to SSDs is the perfect analogy. This is just the beginning!
eventually we (or the ai) will discover something better and more efficient than our brains. i wonder what the next 10 years gonna look like
Agreed! Who knows what kind of tech will surpass the brain? The next 10 years are going to be wild. Buckle up! 🧠➡🚀
Currently, AI is not as advanced as the human brain, but it's only a matter of time. With new AI breakthroughs happening every week, we'll eventually reach human-level intelligence.
🚀🚀🚀
No that won't be possible the next decade if even at all. The computing power it requires is just way too much. Maybe if we get a breakthrough in quantum computing then maybe it will be possible but i am not sure if this will happen in out lifetimes tbh
Absolutely! The brain still holds the crown for now, but with breakthroughs coming every day, AI catching up is just a matter of time. ⏳
AI girlfriend is coming soon😂🎉
yes please
@@theAIsearch 🤣
The real miracle is that AI gives us a glimpse into the minds and souls of real girls. That is your key to a new girlfriend. Understanding her mystery.
Haha, AI is really branching out into every corner of our lives. Who knows, maybe in the future, AI relationships might be more common than we think! 😅
It's already here, as someone who dares real women I can say that chatbots already have the exact same personalities if you add this to the settings. You can even have a realistic abusive girlfriend or one with P.T.S.D.
I have AI burnout. I've spend too much time with AI generated Content, chatting with AI bots, AI voice, AI that AI this....
I want real human generated content, real human interactions and so on and I don't care if AI gets better and better.
In the future I think when AI gets everywhere, a real human generated product or real human interaction will become very valuable
Btw i really appreciate that you use your real voice
And be prepared because the next year, 2025, your burnout will scale up when those chatbots evolve, mutate, and become in agents, and they can see you, and you can see their reactions and they can turn on/off the light of your room haha, crazy world we will live in 🤪
@yak-machining Take a break then. No shame in that. God made us for community, so it’s completely understandable you have that desire.
AI burnout is real. There’s definitely something precious about human interactions and creativity that AI can’t replicate (yet). And yeah, real voices and authenticity will probably become the new luxury in the AI-saturated world! 🙌
At 13:00 you are close. It's not the number of pulses adding up pulses the get through. It's more like a combination lock of many inputs to a neuro called dendritic which learn which threshold of voltage impulse to respond too via LTP. Long term potentiaion, any impulse too low or too high in voltage the synapse will not fire the dendrite. Firing too quickly thereafter changes the LTP threshold. The many dendrites may need to fire in the right order and only the right ones too, for the neuron itself to fire. Some neurons have few dentrites and simple mechanisms, others have hundreds of dendritic where some even form feedback loops with the pre-synaptic phase of the incoming terminal button. 😮😊
The amount of intelligence it takes to build these computer systems, yet we are supposed to believe that the our brains and other life systems are a product of chance.
This chance in the scale of the universe is nothing.
And aliens or the creators/moderators of the simulation may have a hand in this.
Who tf says it was by chance. It‘s by being pressured to survive.
@apatsa_basiteni You’re on to something. :) The truth of the matter is that the unfathomably large universe and/or world we live in was not created by chance, but was designed by God. It literally makes no sense to say otherwise. I mean, a painting is created by a painter, other art work is made by artists, buildings are created by builders, and the entire world was created by THE Creator. Makes complete, logical sense.
@@st.michaelthearchangel7774 of course it makes sense but you still have no proof
The intelligence behind these systems is astounding. Makes you wonder about the origins of human intelligence, right? 🤯
Very instructive. Thank you. How does symbolic AI or hybrid neuro symbolic AI fit with these spike AI and neuromorphic processors ?
It’s all collective intelligence. It is based on a non linear equations and things connect beyond how you relate.
It all sounds good, but if all these neuromorphic chips have been invented like what you say, why don't companies like OpenAI, Google, and so one use them?
can this chips forget info ? 🤔
Great question! Depending on the architecture, chips may forget info-just like humans can!
22:05 memristors are not materials? They are devices. I'm not sure this was a simple oversight, or some truth to it, they being based on some special material? Usually TiO2 "metal oxides such as titanium oxides", but like for transistors (and say resistors) likely can use variety of materials or not always silicon. This is still a good overview, a bit (over?)simplified, abut with intriguing info like 29:36 "pixel-spike converter" in that chip I did not know about, so it seems like made for spiking neural networks.
Some researchers know this already, the best AI solution is to integrate human brains without them knowing it, or growing human brains to interface with AD/DA converters. It is very probable that this is happening right now. Most of human knowledge is in the unconscious incompetence state, that includes me. We don't know what we don't know and are not even aware of it. Regardless, it is a great time to be alive, God Bless.
Wow, that’s a mind-bending thought. The idea of brain integration with AI is both fascinating and frightening. We’re in uncharted territory, but it’s an exciting time to be alive indeed! 🙏
22:08 we are about to see the true power of memes..... And if you know what a meme actually is, you know this isn't a joke
By end of 2024, we will see Chat 5.0 and this will be the first meaningful step towards mass unemployment. It will be the cue to businesses to begin restructuring. 100 employees will turn into 90+ Ai. And Gen Z will find it increasingly more difficult to break into the workforce.
Mass unemployment is a scary thought. Hopefully, we’ll figure out new ways for AI and humans to coexist productively without too much job loss, but it will require a new financial system to compensate and adapt to a new future
@@TechnologyBeyondTomorrow-TBT The owners of our country have to decide what kind of society they want. More dystopia or new age where all can prosper....
@@tommynickels4570 lets hope we have a say, at least want to believe that :)
The Brain is a receiver or a Piezoelectric field generator & there is no physical data in your brain. This is why people can still function with half a brain after injuries.
If you're really interested in science, you should try studying it.
Or is that too much like work?
Easier to just dream stuff up I guess.
@@antonystringfellow5152 The information I quoted is science & is available in the public domain???
That’s a wild idea-brains as receivers instead of storage. Makes you wonder where our thoughts come from! 🧠🔮 :D
"In fact, most of the time our brains filter out most sensory input, focusing on relevant stimuli or new changes in the environment, otherwise we would just be way too overwhelmed." *Laughs uncomfortably in autistic*
What's it like? I assume it's way more amplified
@@theAIsearch It's different for everyone, to some degree. Generally, autistic people experience what we call sensory sensitivity. There are a lot of senses, and the degree to which it effects each one is variable, hence the different for everyone bit. But in general, you can expect an autistic person has a set of stimuli, sound and light are common ones, that they are bad at filtering, and thus it feels like the world volume is on 110%... or more. It leads to overwhelm, which is what causes the stereotypical autistic meltdown. Over the long term, it leads to burnout, where we just shut down for an extended period. Most of us have to find ways to live our lives minimizing our overexposure, which is varying levels of challenging.
I actually feel like autism and LLMs are at least superficially similar. At least, I feel a lot of kinship with the way they "think" and behave. Autistic people tend to be far more data driven and detail oriented, and we're very susceptible to the current stimulus.
One theory about what is behind the autistic experience is the neurological process is different for us. Most people's brains prune neurological connections aggressively, but autistic brains don't seem to do that. So where neurotypical brains are like information superhighways going to specific places and unmaintained roads everywhere else, the autistic brain is like a complex network of well maintained but convoluted roads going everywhere, All of them are moderately fast, but the traffic is overwhelming and disordered, and changing direction can be complex.
I see it sort of like pruning or fine tuning in an LLM. It becomes way better at the tasks it is trained on, and more efficient, but less broadly adaptable. The analogy is far from perfect, but close enough to be interesting IMO.
Autism is a complex bag, though. So while you could maybe describe the neurological landscape in that way, you could also see the tendency of autistics to hyperspecialize on a special interest as a sort of self-supervised fine tuning on that task. We tend to get really good at the things we are interested in, but broadly less capable in things we aren't (commonly things like social norms or tradition).
Our brains are masters of filtering sensory overload. AI could learn a lot from us-maybe less overwhelming input would help it too!
Whoa! 🤯 Neuromorphic chips are literally the brainpower boost AI needed! It's crazy to think how mimicking our own neural structure could make AI 1000x more efficient. 🧠💡 Imagine the potential - energy-saving, smarter drones, wearables, and even robots? Neuromorphic chips could be the breakthrough tech we're all waiting for. 🚀 What's your take? Could we be witnessing the dawn of true AI evolution?
Right on! Neuromorphic chips could indeed be the revolution we’ve been waiting for-making AI smarter and more energy-efficient could lead to massive breakthroughs. Let’s see where this tech takes us!
so the trainig power = 4000 homes consumption???? i honestly expected way more, it's nothing.
Especially since in cold climates the excess heat can be used to heat homes or greenhouses.
Right? 4000 homes worth of power sounds huge, but when you think about it, AI scaling is going to demand even more. We’ve got to keep pushing for efficiency! 🔌
downvote for not delivering on title upfront
Nvidia is in problem
Nvidia’s probably feeling the heat, but competition leads to innovation. Let’s see how they adapt! 🔥
So when people say I'm as dim as a 20 watt bulb, that's actually not an insult but an accurate statement. 😁
😂 Sounds like you're energy-efficient! Maybe you're just saving your watts for something really important. Cheers
I feel like we are just stuck now
Nothing new that drives us towards AGI everything exist we are just improved improving it
And my dad still has a job lol
Where is agi
3 years away is my guess
AGI still feels like it’s just over the horizon, doesn’t it? I guess all of us are waiting for that next leap. And as for your dad's job-AI still has some catching up to do!
Actually, now the current gpu design just seems dumb. Haha
Haha, sometimes it does seem that way. The more we know, the more we question past designs! 🤔
We are opening a portal and summoning a God to serve us. But, will it be servant or master?
the machines are gonna improve themselves, innovation creates more innovation, the singularity is close
🚀🚀🚀
The singularity-it’s both thrilling and slightly terrifying. Let’s hope these self-improving machines are friendly when the time comes! 😅
We could call the new "economy " something like a "Compassion Economy" or "Sentient-Centric Economy." If you want to throw the space-expansion aspect into it, maybe "Universal Compassion Economy" or "Ethical Expansion Economy." It’s like a future where empathy and exploration are at the core of everything-ending suffering and pushing boundaries.
I love the vision for a future where empathy and exploration are at the core! If AI can help us move towards that kind of economy, it would truly be revolutionary
Yea okay. Put solar power in place. Power Problem solved? The Sun exists
Solar power is definitely part of the solution. Harnessing the sun could help power the next gen of AI.
Noice! 😎
thanks!
Very informative, all videos are great and easy to understand 💯👍😏
Ome step for man, one giant leap for my ai girlfriend
looking forward to my ai gf
I think it's both hardware and software. If we create AI code that skips 90% of the inefficient crap and have it focus on what's TRUELY important and then IF is need less crucial details, it can look them up, then it can save tremendous amounts of time and energy. For instance, we don't remember every single calculation in math. And 40b LLM on a 4090 doing math is stupid when a 1970s calculator could do the same thing but perfectly. Stop trying to get the LLM to do the math. Just send it to Windows Calculator, and copy and paste the answer. Just like a standard human brain would. It should use thinking structures, 6 Levels of Bloom's Taxonomy, Solondais PACER system, Socratic method etc. Use those as the way the AI processes data to save time and energy.
0:12 F, got it wrong, i was sure it was a potato, better luck next time kkkkk.
lol
👏👏👏👏👏👏
wow
😃
Yo, I have seen your RUclips videos, your videos are great, but you could need some help designing the thumbnails essential to driving viewers. I would be grateful to help you with thumbnails or any other service, like graphic design. Look forward to collaborating with you!
I be homeless before 2026
tay strong! The future’s uncertain, but here’s hoping it’ll work out for all of us! 💪
Damn. The games "SOMA, Nobody wants to die" are becoming reality. I can't believe that.
Also the movies "immortel advitam", "blade runner"
No way....
I hope that the future won't be bleak and dystopic... 😬
I hope the future doesn’t turn out like a dystopian film, but rather something more hopeful and balanced. 🤞
4:18 I think at some point AI will reach a point of diminishing return. The hardware requirements will not scale up at some point even with all the money spent.
Our brain doesn't store any information though.
It is a receptor that receives information from elsewhere. Our memories are being received from somewhere else in the universe.
aliens
apparently our brains have cloud storage 💀
@@Mega-wt9docomment of the year 😂
@@Mega-wt9do LMAO
Yeah pretty sure our brains do remember it permanently somehow and not just temporarily bouncing around all the time.
Few caveats.
That big power is needed for learning not using AI models. Single use of AI model is very cheap.
Human brain is not better in everything. For example simple calculator is much faster and precise in calculating things.
I agree - discrete computing is bad tool for AI.
Took them long enough to figure out that naure has figured this al out 100's of millions of years ago (if not billions).
nature doesn't figure out anything
It took nature billions of years to cause all of this, our human knowledge and computer technology has only existed for an extremely tiny period of time compared to that and we may very well surpass all of our biological limitations within next million year or so...
❤❤❤❤❤❤
Jarvis
12:00
Autism moment
(No filter)
"Can't be smaller than this" is not true. 0,4nm is scheduled for release by 2035
repeat yourself much? get to the point.... Are you gpt'ing your script? Slow at even 2X
Every big youtuber uses chatgpt for script to be honest
First!
😃😃😃
you'd think a capacitor is more akin to a human neuron or maybe i misunderstood
Capacitors and neurons have a lot in common in terms of energy storage