Energy, not compute, will be the #1 bottleneck to AI progress - Mark Zuckerberg
HTML-код
- Опубликовано: 20 апр 2024
- Full Episode: • Mark Zuckerberg - Llam...
Apple Podcasts: podcasts.apple.com/us/podcast...
Spotify: open.spotify.com/episode/6Lbs...
Transcript: www.dwarkeshpatel.com/p/mark-...
Me on Twitter: / dwarkesh_sp 
Наука
![[BLACKSWAN] ‘Roll Up’ Official Music Video](http://i.ytimg.com/vi/ulvCwyKrwqw/mqdefault.jpg)
Zuck is so much better at talking about nerdy shit than he is as a public-facing marketing guy lol
yeah noticed that too
Proof we should let people be real.
That's true, but the real question is WHY? The real answer isn't found in Zuckerberg's individual personality, but the wokeness and censoring nature of his employees and the surrounding Bay Area.
A sign he’s the real deal and not a show man.
Literally me lol
Zucc got his latest patch 1.8342
- improved facial animations
- improved vocal features
- improved body language
- custom hair
- integrated eye moisturizer (no more eye licking)
Still under development:
- show emotions on face, voice, body language
-
kids. kids is probably the biggest patch you can have in your life. after kids, people change. whether they want to or not. your priorities change, your emotions change, your maturity changes. the biggest cares and worries in your life also change.
it's not that zucc is more human or anything now. it's that he's a dad to 2 kids now. that demands a different persona
-fully working schlong
-fully working scchhhlong
😂 same thought when I saw video
Bro i was inhaling smoke when I read no more eye licking and I died
Dwarkesh cut it out of this clip but he mentioned to Mark that Amazon just purchased a 960 MW (=close to 1 GW) data center from Talen energy next to a Talen nuclear power plant in Penssylvania... around the same time they invested 1 billion dollars in Anthropic, the company that made llm Claude. So I have a feeling we WILL actually see that major leap in capabilities next year, unlike what Mark believes, or seems to believe.
I completely agree with you!💯🎯
Just looked it up online and it's true
Are there pther compnies doigg NB the same?
I work for the company that built the switch yard for this data center. When construction started almost 3 years ago it was a for a bitcoin miner, half way through we heard the buyer had backed out. Funny how I find a random youtube video that tells me the outcome, as we have not heard anything.
@@G73ServerNot many companies have the ability to buy a GW plant that produces electricity 24/7.
Zuck got his human-look update
Stress and pressure will do that to a person.
i also noticed that, wtf is happened? 😅
@@Gramooneer I think it's mostly that he started growing his hair out. His hair looks much better at this length (makes his head look less like an alien). Also, his skin has more natural color to it. (via being in the sun or taking vitamins or something, idk)
It’s the hair. He got a normal haircut and is in really good shape. We are the same age and I’m very proud of him.
Yeah it’s the longer hair
glitch in the simulation at 0:40
Also at 0:30 the girl walks behind zucks head then reappears
2:37 another glitch - the man just flat out disappears
AI talking about why it's not possible to be developed 😂
im not sure if ur joking but its just a speed up probably
the zuck can bend the space time
I never realized how much power was going into making these foundation models.
You should see how much energy is used for bitcoin - more than some countries.
So much for these big tech companies caring about climate, it’s all about AI dominance now.
@@erniea5843they only like to virtue signal when they can put the costs on average people
@@erniea5843 Trust me, if their AI can turn out a portrait of George Washington as an African American, then they care about climate change !
@@tuhaggis you should see how much energy is used for Christmas lights
energy always has been the determining factor of the advancement of society
Nuclear energy was always the answer
You can have desalination you can fight wildfire you can have infinite agriculture tech EVs etc
@KCJbomberFTW but noooo nuclear always goes kaboom
@@KCJbomberFTWwhat's the energy constraint for fighting wildfires?
@@sean_haz excess water in reservoirs to scoop with planes
Facts
Zuck looks more human than usual. That's sus. XD
I'm glad that you like his recent update
He was living a fantasy life, where in his mind he was this multimillionaire giga chad pioneering the future of humanity.
It seems he has gotten humbled by reality and realized that he isn’t what he thought he was. It’s good character development for him actually.
@@juanramonsilva1067💯
They got enough GPU compute now
@@juanramonsilva1067 Well put. VR and AR isn't really his thing either. You can tell how in-tune he is when it comes to neural networks and AI.
The efforts we put into replacing ourselves are remarkable
Amazing editing!
The cuts are really stunning
When the most powerful companies in the US (Microsoft, Google, etc.) need to solve a bottleneck in order to make progress, regulatory and other constraints will be quickly resolved. There is too much at stake when we talk about AI progress. So, expect a 1 gigawatt data center next year.
Think so too
AI progress is also tied to national defense interest. So I think they will definitely clear regulat process out of the way
or just buy one like amazon done. "Amazon buys nuclear-powered data center from Talen"
"The data center, Cumulus Data Assets, sits on a 1,200-acre campus in Pennsylvania and is directly powered by the adjacent Susquehanna Steam Electric Station, which generates 2.5 gigawatts of power."
probably have it up and running within a year with the hardware installed.
@@percy9228 🎯
Dude you grew like 20k subs in a span of a few days. awesome!
You need to unlock the fusion energy tech tree, before you can build your super AI. Everyone knows that
Fission>>Super AI>>Fusion through Super AI is a much faster track. Just have to keep your population in check during the occasional crises meltdowns during the fission fase.
We don't need super AI. We already have fusion energy☀
@@dayhillbilly From my GPT friend: "To generate 1 gigawatt (GW) of power using solar panels, you would need approximately 4 square kilometers of solar panels, assuming an average solar panel efficiency of around 20%. This can vary based on factors like solar panel efficiency and sunlight availability. (Confidence: Medium)"
And then some batteries of cores.
Fission is literally good enough
It’s the cleanest cheapest safest energy ever made
What? Just build far more energy efficient transistors. The human brain uses bugger all power. Build biological computers if you have to.
I work for a transmission system operator at the highest voltage level in the grid, and the number of connection requests for new data centers we're getting is just insane!
What company?
AWS has 1GW in the Columbia river basin in dozens of sites and buildings.
He specifically stated single site. Sure a bunch of service providers have a cluster of data center which equate to 1GW.
@@Neojhun riot blockchain will finish a 1GW site in Texas this year.
“How long can the exponential curve going for?” There’s going to be a Nyquist-Shannon equivalent limit to the max data and compute you can throw at a model before you can’t improve it anymore.
For example, if you trained an ai model on flipping a coin, it doesn’t matter if you use a 1-gigawatt cluster and 100 trillion coin flips as your dataset, the model is still going to be 50% chance. The question is how do you theoretically find that limit without throwing 10s of billions of dollars at training clusters.
What if you flipped the coin 5 times. You have an odd number, you are always going to have either more heads or more tails. If you flip it 6 times, there are a total of 64 possible permutations of the flip, 2/64 are all heads or all tails, 12/64 are one heads or one tails out of the 6, 30/64 are 2 heads or 2 tails out of the 6, and finally only 20 would give you a perfectly representative result of 50% heads and 50% tails. You'd only get this result 5/16, 31.25% of the time. This actually gets worse the more times you flip the coin, if you flip it ten times, you'd actually only get a perfect 50/50 heads and tails result 252/1024 times, 24.6%. there are always more non 50/50 results than 50/50 results. The best way would be to actually flip a coin some finite set of times within a larger finite set and then measure the distribution. Say sample 100 flips one billion times. And see if the distribution matches statistical expectations. But you run into kind of the same problem, just sort of at a higher dimension. Coin flips May be harder than you think, even trying to determine if a coin is really flipping 50% one way or the other
In other words, SMRs will be ideal for powering data centres/servers and super computers.
I can just imagine a bunch of private SMRs behind walled compounds that remain powered and cool while the rest of society sweats it out during blackouts
If they ever get built.
Yes but only if we keep making current microchip technology. If the transition to carbide silicon chips sooner then a massive 1GW center could ve powered by only a few MW with 5 times the computing power.
Mark Zuckerberg looks so normal and human here, that it feels weird af; we want our lizard robot boy back
Who ever is doing his human training is killing it.
"1.21 Gigawatts!!! Great Scott!!"
- Mark EmmetBerg
Remember when a computer filled up a huge room 🤔
i like ur thinking
Moores law is no longer applicable right?
remember when women were the computers, and lots of them were sitting in a room doing calculations
@@the_original_dudeyeah they still are. Bankers, accountants
@@Instant_Nerf no, they don't calculate manually anything
Energy over time is what all works run on, and how much energy that must be spent ultimately determine the cost..
Perhaps one day, we can all have dividends from Open sourced machine production commons, where the capacity is measured in energy.
AI explaining that AI will reach a bottlneck to come people down
hedge against losing energy suppliers in "the age of wars" (which is over-hyped, but kind of has to be) , do this by buying into renewable energy & refining existing tech, when that's built you've hopefully got it structured to sell it for minimal loss or better yet break-even on it..
above is the unlikely scenario tho, you will most likely, in my eyes get a sustainable growth vehicle for this project and in effect you have AI also relying on the sun like we do, which is kind of nice.
It's not just energy production, its infrastructure as well. Even if we have the technology, it would take at least five years to build the necessary infrastructure to deliver the energy and thats being optimistic.
So essentially they would have to place it next to a plant dedicated for it.
Bruce Power up in Ontario Canada is planning on adding more reactors to their site. They are in the planning phase for an added 4800 MW. Maybe bump that up to 6800 MW and then sign long term energy contracts with Microsoft, Facebook, Google and Amazon. Its right next to a massive lake so cooling water is not an issue and there is lots of land there.
check out "Amazon buys nuclear-powered data center from Talen", so now they just need to kit it out with AI hardware. so it's max a year away.
this probably drive other tech giants to buy something similar or get locked out for 5 years. and it wasn't that expensive at $560m
Yup its playing catch up.
Technology chases math, and energy chases technology, ultimately the technology gets manufactured implemented for consumption markets either publicly or not.
1.21 gigawatts?!?!
Is just 1 nuclear reactor
When this baby hits 88 exaflops an hour.... you're gonna see some serious sh
GREAT SCOTT!!!111!!
What the hell is a jigawat?
They just need to make sure their AI gets struck by lightning every time they need to ask it something. The trouble is, we don’t know when or where it’s going to strike.
@@GeneralKenobi69420 "88 exaflops an hour"😂
It doesn’t have to be either or, I could see both energy and compute bottlenecks being the case, particularly if these exponentials continue for as long as the “optimists”, for lack of a better word, say that they will.
Just for some counter point the human brain runs on about 20 watts of power. Maybe needing Gigawatts and billions should be an indicator that this could be a wrong approach to reaching human level AI?
you got a better approach? lets stop everything because the human brain is so much efficient??
it's all progress. you build tech so it improves other tech. the first transistors where astronomically bigger compared to leading edge now. give it 25 years of advancements
@@percy9228 I just suspect this might be a bit of an AI tech bubble that bursts but gives us great chat bot technology but not real AI.
It's a bit like the crypto bubble where we have all this technology and money to push into something that looks good but may not deliver on the hype people are building for it.
Mind you if it does go full AI it will probably happen so fast our world would change completely within days. And I do not think we are ready for that.
qwerty-np2iw There are potential lower energy computing solutions that could just be around the corner e.g. graphine, memristor and photonic technologies could totally change the energy needs of modern computing if they can go mainstream.
@@percy9228 I mean we could get serious about education and up skill the whole world population. I bet even a small uplift in collective cognitive power would make a massive difference in the world
@@christophermcdermott2443 there has never been a better time in human history as there is now for access to elite level education. This is all thanks to the internet. I've met people who learnt languages themselves using the internet.
If you think about it, even the internet consumes ridiculously amount of energy, but no one on earth will argue to shut down the internet because of the tremendous good for humanity.
we are reliant on it so much.
AI will one day be considered in the same way. AI will shape how the world will work.
Internet took a while before it was fully accepted. nowadays you need it just to apply for things.
and I don't get your argument. you don't need to hand feed kids, just show them where to look to get answers. and now with AI in future they will have a personal tutor to guide them also. and parents who care already guide and parents who don't won't. no ones got time or energy to teach another kid when they busy earning food to pay for their family.
I would love to see a large data center connected to a huge solar plant system that uses full power during the day and is throttled down to a minimum at night for the training over weeks.
Zuck should look at what amzn did with Talen Energy.
Amzn got almost a whole nuke unit.
The better way for energy not being a bottleneck will be to reduce compute elements to be not based on traditional electronics, but go atomic scale and use extremely low power. Current approach in AI computing uses brute force. Soon progress will depend on finding more intuitive algorithms that converge quicker.
It is literally what we have been working on for more than 15 years in laboratories around the world. Thanks for pointing out the obvious!
Two promising prospects I have seen to make computing more efficient are graphene chips the band gap problem they experience has been improved on recently. Also AI designed chips are designed 1000× faster then human designed ones, they aren't close to the computing power of human made designed chips but at the rate they are improving will soon close that gap
That’s what quantum computing is trying to do. But most power is consumed in transformers. They take the current and split it into a lower current and heat. And then needing cooling systems to dispense of the heat. Which is quite primitive to be honest. The electronic technology needs to mature. We are still at the infant stage of the electronic revolution
Energy is the #1 bottleneck of everything my guy
Yep. I do interstate nat gas BD for the southwest US. We often get requests for data center or crypto interconnects so they can power gen sets or even small power plants and they don't know what to say when they learn that we and the competitors are sold out in perpetuity. They spend years planning development assuming that gas will just come out of the ground for them when they show up. There's no capacity left to connect the western US to supply because san juan basin is depleting and getting massive expansions or new builds to permian basin approved is a huge risk today that requires politicians to put their necks on the chopping block. But at the end of the day they aren't going to build the generation to address the massive projections for AI demand growth with unicorn farts. AI is particularly going to be screwed if LNG gets opened up because nobody will bother risking these huge multi billion interstate expansions for domestic use when they can connect to far better markets at the coast.
Will be interesting to see what kind of hardware advancements can be made to reduce the power requirements.
Analog computing
They should build compute/data centers with power plants, no power lines going elsewhere.
Both need cooling and security.
It's a lot cheaper if you hook it up to the grid.
check out "Amazon buys nuclear-powered data center from Talen"
I have seen ideas of a modular mini nuclear plant that you could build into a building, supposedly they are safer and easier to maintain. I haven't even heard of a prototype being built, the government regulates this stuff heavily
The Zuck measuring things only in Gigglewatts
So, we would need modular thorium power units, customized for datacenters, build right on-site..
..about time, that the US legislators wake up and make room in the laws for such new (hybrid) installations.
Facebook et al are putting datacenters e.g. in the north of Sweden close to our hydro resources (and usually cheap cooling). It gives quite few direct employment opportunities, suck up a lot of power and raises energy prices in Sweden. It kindof outcompetes outdoor spas, incentivizes improved isolation in homes, some other energy hungry industries and so forth. We do look at new nuclear but that will take 10+ years to do, unfortunately, because it's so heavily regulation, just as he says.
what if you build the datacenter on Iceland? they kinda have unlimited geothermal energy which is cheap and green
And how do you transfer it? That’s infrastructure cited as a hurdle. Solution isn’t more power it’s engineering gpus and machines to work with less power consumption.
Or transfer learning improves and you just build multiple smal and cheap clusters.
Build a gas fueled power plant in East Texas and you’re gucci Zuck. Plug it into your zillion gigawatt datacenter and let Jerry Jones take you to the promised land.
Current AI architectures are ridiculously inefficient compared to the human brain.
The efficiencies will only improve. Performance per watt is going up.
To be fair, we still don't know how the human brain works exactly. The neuron-activation model used in NN's is an extremely simplified way of understanding the brain, that doesn't even capture 1% of its true complexity. That being said, I'm surprised just how well our current (general) models surpass most humans in the tasks they do.
@@rapidreaders7741 Yes and that is the "problem". We don't even know how human brain works. These models are not self-aware. They just imitate self awareness, and that is 2 different things.
@@CamAlert2 Isn't that technological innovation? Thanks for pointing out the obvious.
I don't think this is the right way to put it. Many people like hinton believe nature took the way of training/ learning which is very efficient in terms of power consumption , it rejected other ways which does take a lot of power but could learn from a lot of data/ resource very fast without over fitting etc etc like SGD with back propagation. Today's ai uses learning algorithm which is very efficient in terms of time to train on large dataset and which fits very well, compromising power/computation.
Seems odd to me that at the mid to high mega scale or even giga scale why they wouldn't be generating power on site with no grid tie. But having done work at high megawatt plants its not the kind of thing that you could build overnight. As far as I know Mitsubishi Power doesn't keep a lot of M501J Gas turbines sitting in stock in a warehouse. If you had a completely off the shelf design and could write a cheque you might be able to build one in 3 years.
Zack looks great with that haircut
Thank you Mark. That was a very human statement to make.
Now he looks a lot less like a robot.
nyc typically consumes and anywhere from 5-10 gigawatts
A gigawatt is 3 million A100s or 1-2 million H100s.
About 10x more power consumption than the biggest data centers today.
what about the new Blackwell? by the time they build a nuclear power plant you'll have a new gpu
10x on an exponential curve is not that far off
I thought about this for a while: instead of building massive data centers where AI runs, why don't we distribute the compute across the globe in a decentralized fashion? This would also de-risk the power requirements, as it is much easier to power something that requires just a bit of energy.
They probably never thought of that. You are a visionary, they should pay you trillions for your unsolicited advice.
@@unmanned_mission ah the internet. Who doesn't love it ♥️
The volumes of data being moved in AI training necessitates everything being in a small space, even moving data between machines in the same rack becomes a bottleneck. Sometimes even the speed between individual chips can become a challenge (e.g. Nvidia developing NVLink for 900 GB/s)
Because its inefficient. Parallel compute needs to be in close proximity to provide low latency for AI clusters. The AI clusters also require high bandwidth networking like InfiniBand which is prohibitively expensive over long distances.
I think this is the model they keep at their headquarters, much more human-like.
The robot/alien has hair now
All that Brazilian jiu-jitsu has made him more human!
I believe this was a few years ago. Before LLM’s was as prominent as they are now. So, the zuck may not be speaking about AI as we know it right now. 🤔
So he committed to build a datacenter that runs on 1.21 gigawatts to get back to the future...
Hence you see a glance if the future when nuclear fusion is on... no more limit on energy since it doesn't pollute...
After Zuck found MMA he’s been a completely different dude. COMPLETELY different dude.
The hair def makes him look more human . Also the energy fields sound just like the matrix
The solution is distributed compute, networks such as Arbius
Computing and energy go hand in hand. The better the computational efficiency, the less energy will be needed. So, yes, both are a big problem, both current and future.
i love this guy, so cool & poised
Yes we might be saved. My day just got brighter. Now im a but more optimistic about the future.
And then we will be turned into 🔋 like Morpheus said.😬
Fuck it. Capture the sun with a Dyson sphere
He's more human now. He updated his software 😂
That’s why free sustainable solar is the only way forward
now it make total sense why Tesla bought solar city a long time ago
You wanna fix that? Cause there is a way, I mention it on the APEC propulsion conference open mic night from the end of April.
Gotta get that Dyson Sphere going.
Pay people to generate electricity via peddling.
Holy shit you're a genius.
If you don’t have the brain, you must have the muscles.
Human efficiency ~ 25%, diesel engine, fossil fuel power plant ~50%. We can't compete.
Sure.. Eventually.. But for now it's not, and it won't be until we're well past the point of major changes.
We could use our body heat as an energy source...
So they either need to make energy cheaper or the chips more efficient.
need more solar, and eventually hopefully fusion
Solar is highly inefficient. Go with nuclear fission in the meantime. Fusion is still decades into the future.
Wow a nuclear power plant to train the model ? 😯😯
It's cz they want clean energy. 😅
PERFECT SENSE, AS I SAID
You can imagine a future in Zuck's head where the physical constraint of the AI energy throughput gets lifted such that there is an actual location with a giant super computer like the one in the 2005 film The Hitchhiker's Guide to the Galaxy. LMAO that would be epic
Mark looks better with longer hair. This is a good look for him
@hyliion can elevate some energy restraints . Obviously not gigafactories though Haha.
that is why you need to upgrade everything to B200 chip
Well said Mark
What about localized wind and solar? The amount of compute would be proportional to the amount of energy coming in. There would be some amount of batteries, but it might well be cheaper at a certain scale to simply build more solar and go with the fluctuating rate of available energy. If such systems are not connected to the grid, the permitting should be far simpler. Also, if they are not connected to the grid, but are located out in the desert somewhere, they would also be much easier to bomb out of existence.
You would need millions of square meters of solar cells to reach a gigawatt. You would be building a solar array the size of a small city.
@@tuhaggis Yes. But still I could totally see this happening.
@@stcredzero A million square meters is the size of Egypt. And that's just ONE gigawatt datacenter. Still think it's a good idea? Do you Gen Z learn ANYTHING in school these days?
"The human generates more bio-electricity than 120-volt battery and over 25,000 BTUs of body heat. Combined with a form of fusion, the machines have found all the energy they would ever need."
At the cutting edge of Tech..
Solar + battery can be built anywhere
Human Intelligence is powered by tacos.
This likely means that the path forward is in simplifying AI models so they can do the same work with less energy.
We learned about this in my first (and only) coding class - how do you simplify the code so the computer is doing less operations to get the same outcome?
We did the same thing when we started building computers. What was once the size of a room is now on a nanoscale.
Mark raises an excellent point, though I don’t think we’ll be building reactors and solar fields just to power a learning model - we have enough energy problems as it is.
No, we can expect the technology to become much more efficient in the next 5-10 years, doing much more for less work. In the same way we were told to learn code in school, kids are going to be taught to use AI - it’s an incredibly powerful tool.
We should send datacenters out into space to train on solar energy.
He got a new head
just bulid one in singapore
Longest I've ever listened to Zuck speak tbh.
Can't belive nobody trying to manufacture GPU production. Must be really tough
Intel
Tesla is manufacturing DoJo, which is a modular chip designed for machine learning. They are continually adding to it, and it will be one of the most powerful super computers soon. They are planning to use it to crack full self driving.
Quantum processing.
Hmmmm “The Prisoners Dilemma”? 🤔
Very true. The real priority should be A.I macro-engineers (structural, civil, mechanical, electrical, etc.) building energy infrastructure and energy generation plants.
Priority should be decentralized blockchain infrastructure that tracks all government spending. Once we can track all the tax funds and how the government spends it we can be able to allocate it properly to achieve our goals as a nation
@@burgundyknight6826 I think using blockchain technology to track government spending is an interesting idea with some potential benefits, but also significant challenges and limitations. Here are some thoughts:
Potential benefits:
Could increase transparency and accountability around how tax dollars are spent
May help reduce corruption and misuse of funds
Allows citizens to more easily audit and analyze government spending
Challenges and limitations:
Technically complex to implement comprehensively across all levels/agencies of government
Concerns around privacy and security of sensitive financial data on a public blockchain
Blockchain doesn't inherently prevent misallocation of funds, just makes it more traceable after the fact
Many government expenditures require some confidentiality (e.g. national security) and couldn't be on a public ledger
Blockchain is still an emerging technology with scalability and efficiency constraints for massive datasets
In terms of importance and priority, I would rank addressing our pressing energy challenges as a higher near-term priority than implementing blockchain for government finance. Developing clean, sustainable energy sources and upgrading energy infrastructure is crucial for combating climate change and transitioning to a carbon-neutral future. These are huge, capital-intensive undertakings that require major government investments and policy support.
That said, I don't think it has to be a strict either/or scenario. Initiatives to improve government spending transparency can be worked on in parallel. And there may even be some synergies, e.g. using blockchain to track and verify how funds earmarked for sustainable energy projects are being allocated and spent.
But if I had to prioritize one area for governments to focus on with urgency, I believe the energy transition and building out robust clean energy infrastructure should be at the top of the list given the timelines we face with climate change. Government financial transparency and accountability is important, but feels relatively lower stakes in comparison.
Of course, this is just my view based on the information available to me. Reasonable people may weigh the relative importance of these issues differently.
Algorithms always had and will have more potential to grow at exponential rates than physical infrastructure. Eventually energy won't matter.
Confirmed: Zuck drinks Zippy Water™️
100 gigawatts marty!!!!!!!!!
These bottlenecks he talks about, I think we will get them in the short term, but as soon as we create an AI which can figure out a way past said bottlenecks, in a way telling us how to improve itself, then game over, insane progress really quickly.
Sounds very like global warmish. Or even hottish.