Meta's LLAMA 3 SHOCKS the Industry | OpenAI Killer? Better than GPT-4, Claude 3 and Gemini Pro
HTML-код
- Опубликовано: 17 апр 2024
- Learn AI With Me:
www.skool.com/natural20/about
Join my community and classroom to learn AI and get ready for the new world.
Dwarkesh Patel
Mark Zuckerberg - Llama 3, $10B Models, Caesar Augustus, Bioweapons, & 1 GW Datacenters
• Mark Zuckerberg - Llam...
Dr Jim Fan:
/ 1781006672452038756
Aston Zhang
/ astonzhangaz
#ai #openai #llm
BUSINESS, MEDIA & SPONSORSHIPS:
Wes Roth Business @ Gmail . com
wesrothbusiness@gmail.com
Just shoot me an email to the above address.
It's seriously surprises me how much more lovable and relatable Zuck is that he used to be. I think they must have updated his firmware.
he has 100% done a bunch of PR/body language/speaking training to sound more normal. Plus, the big meme of him in front of congress is a bit fake since basically everyone acts like a robot in front of congress cause you don't want to say anything that would get your company in heat with lawmakers and regulators that could literally put you out of business with a new law
@MIKAEL212345 he was weird for decades before then. He's definitely got a PR team though..
I think thy upgraded his AI with an LLM based engine
He seems more at ease and relaxed. I think he’s an introvert who gets nervous in social settings with strangers … With age this anxiety goes away a bit
@@KitaTaki-mk3gt also with drugs or so I've heard :P
Zuck wearing a bling chain is the most SHOCKING THING!
He's oddly being more human and cool
Industry was shocked!
its called a PR team.....
He's going through his "bling" phase. I went through that in my 40s.
It's almost like Zuck is excited for the first time in his life.
Think about it. The dude always worked on php codebases, social networks and the metaverse. Now he's got to play with AI and forget all his past sins.
I doubt he has ever touched a compiler.
Open source is the way to go. Big Tech should not be trusted has the keepers of ai.
Yes - Exactly what I was just thinking!
Kinda tough when they're the ones driving it
Open Source won't help anything, but it sounds nice.
The power isn't in the code, but in the compute.
... millions, even billions worth.
Yes and no, that's a tricky ethical issue.
Very few people have the good knowledge and ethics to understand and control the incredible AI power in their hands through open source.
We are reaching a level where your point could be similar to say "well why don't we open source the making of nuclear weapons, or EMPs or how to hide state controlling chips inside the CPUS of phones. (nobody will know)
Open sourcing is great. but if you don't understand it and just play with it... it can cause a lot of dangers, like a few years ago, the open source SSL code, with a backdoor nobody saw that opened banks and corporations to hackers. Pull requests, with malignant code, that anyone using your fork of the open source could abuse.
Also Models are black boxes that no human can truly understand with hidden biases and dangerous flaws.
@@PierreH1968 respectfully, I disagree with just about everything you've said.
Mark's recent rebranding campaign is paying dividends.
his previous one failed. now after spending millions in data analysis they have come up with a new strategy to make people trust him enough to be lured back in
Who would have thought that Meta & Zuck would be the heroes of OSS
plot twist: they arent
Meta always has ulterior motives that aren’t purely for the good of everyone, make no mistake.
Could it be his one edge over government overreach in the VERY near future knowing they owe him a favor for 2020 election? He'd be the one fast-food shop that makes his bread in store.
It's not really that surprising. There's been a lot of software Meta, and in fact Facebook have developed internally and released to the public. The idea is that by making their internal stack the standard, they can save money as other people make breakthroughs in using them, and they also get a larger pool of talent to draw from in the industry who learn their internal tech stack.
Releasing open models is just an extension of that. It's more or less a win/win relationship, though both sides are definitely using the other. We get free, high performance models at a scale that small scale AI devs couldn't produce, and they get a load of infrastructure around their models that makes them easier to use in increasingly diverse and unexpected ways. Plus, it also helps Meta fend off Google and OpenAI from getting a moat that's too big for their britches.
Meta can also be credited for the popular deep learning framework called PyTorch. I doubt deep learning would've taken off without the work put into it. Meta's been playing the open game far longer than Llama.
I think the Romans would overrun a Sherman quite fast because of the tank's limited ammo, fuel and blind spots.
Yep. And Romans had catapults and could dig defensive ditches (tank traps) very quickly. A boulder from the catapults could damage the wheels to immobilize the tank, or bend the barrel enough to make the gun useless.
It was a really bad analogy
Who told him to wear that chain outside his shirt???
My first thought too, lol.
That's an AI model suggestion to zuck
Bro is hanging out with Dana White and it was a big weekend lol
Mr T
Kanye
We keep interchanging the terms "Open Source" and "Open Weights". It's an important distinction, I think.
Indeed. Thanks for pointing that out.
How was all that sherman tank firing hundreds of weapons simultaneously?😂
I think that was simulating a machine gun or something. I thought the same thing when I was watching lol.
@@CalebUssery-tm3lv no machine gun on earth fires that many rounds at that rate. You could fire 10 modern phalanx guns and not achieve that kind of fire power.
That game allows many fantasy elements like unlimited ammo, rapidfire cannons, barrels which never overheat or wear out, etc. You can even deploy armies of zombies, terminator T-800's, and John Wick clones.
@@CalebUssery-tm3lv Exploding machine gun bullets..ayyyyee?
It might have been compressing the simulation rounds perhaps?
He said "interfere with elections".... I guess he would know all about that.
Was thinking exactly the same !
The lizard is incapable of hiding his nefarious deeds
Interesting how The Matrix decades ago predicted the energy crisis for the AI and then humans became batteries
For every open source project, there is a close source one twice as powerful.
true
Elon is trying to prevent that
Here is an analogy... 1 country with nuclear weapons is dangerous -- they will attack with impunity. When multiple countries have nuclear weapons they would be MAD to use them!
i am here for the content, but boy is that voice soothing. im following you along now since a couple of months, and its so relaxing to listen to you.
Wow I love his voice also, I suggested him to my sister and told her about his warm soothing voice. I also thinks he looks sexy.
It used to be that if you were using an excessive amount of power and generating a lot of heat at your home you were likely running an indoor cannabis grow tent, now it’s gonna be that you’re training a model to take over the world lol
That's a good one-' grant vine. But what fun playing the game and beating the competition.😅 And look at the outcome.
Give it time. The gatekeepers lobbyists will convince lawmakers that it's too dangerous for the plebs to train their own ai. The feds will be scanning power bills and hunting down those dangerous nerds.
Also, where are the climate crazies bitching about all that power these tech giants are chewing up? It tells you who really sets the narrative.
That Sherman tank appears to have been souped up a bit.
Grok 1.5 ? (opensource)
This is actually shocking and really good for the industry
Bad actors are the big danger, yeah.
Karpathy warned that taking a trained model and modifying it with add-on training data will overwrite the original weights - without considering the original training data. You can add capability but you'll be subtracting, too. To really get a better model, you need *all* of the training data.
And that's a problem for open models. They share weights and code, but they don't share original training data.
They can't publish it. They're drawing on data from Facebook, X, the internet. People own that data. Using it to train is a gray area in copywriter law. It *might* be legal. But republishing it without permission isn't.
I would be interested if some companies will provide a GPT-4 compatible API, for Llama 3 400b+ for cheaper. It would be good to experiment with agents where GPT-4 API is prohibitively expensive currently, but anything below GPT-4 level, the agents aren't always working.
When an AI can create better AI algorithms, beyond transformers, then a runaway situation could take off.
Doubtful. Data has x amount of signal in it. The arch can either extract x or not. In the case of TF models they took the internet data from 0 to X. To get to 2x you will need another dataset on par with the internet.
they already can, just ask lol
Mad respect for Mark Z.
?
I wonder if Zuckerberg’s Hawaiian doomsday bunker has a moat.
I have my audio cranked and I can't hear half of what is being said.
"And we know how to work with your data, because we are Facebook." - see you at the next hearing Zuck.
Love the titles. Informative and not click-baity at all.
Love sarcasm 😄
Hi Wes, I’ve been captivated by your series on the swift advancements in AI and the discussions around the possibility of a ‘runaway moment.’ I wonder if concepts like entropy, well-known in physics as the measure of disorder in a system, might also metaphorically apply to AI development. For instance, as AI models evolve from one version to the next, there appears to be a buildup of errors or ‘noise,’ reminiscent of entropy increasing in closed systems. These distortions seem to amplify with each iteration, potentially acting as a natural check on the progression towards AGI.
Could these entropy-like phenomena, which manifest as accumulating imperfections or deviations in AI models, serve as a natural form of regulation that might prevent AI from advancing too rapidly or uncontrollably? Moreover, could understanding and harnessing these effects help us design AI systems that are inherently safer and more aligned with human oversight? I’d love to hear your thoughts on whether this perspective might add a valuable dimension to the safety and control strategies in AI development.
One crucial limitation is that the Llama materials have restrictions that hinder their ability to improve other well-known language models.
This means developers might struggle to refine models for better performance because they're required to use Llama 3 specifically due to licensing constraints.
This limitation could have a major impact on certain projects or applications that rely on this model, if developers need to train the model with their own dataset or create derivative works, they may find it inadequate because it's NOT entirely Open source.
I’m not even the industry and I’m shocked. Also a bit stunned.
Awesome, thank you.
LLAMA 3 isn't available in UK yet.
Your left aligned picture in pictures headshot covers key information in the benchmark table. Thanks for the interpretation of this though
The audio in this video is so quiet at even max volume. Needs compression.
how is changing background with zoom?
Normalize audio pls
You’re a true gem, Wes. Great work. Lex Fridman should give you a call.
It gets confused with the context of what we were talking about significantly more often than Claude 3 Sonnet. It's very nice and friendly, though.
3:02 can someone please help to elcucidate what are the meaing of numbers shown in this table? how these numbers were calculated?
Just Google the names of the benchmarks dude.....
The thought of being able to run open source agents with the LLama 3 models is actually kind of insane. I feel like there's quite a few possibilities once you have the weights in hand that you just can't do without them, and being able to do multi modal function calling will probably be ridiculous.
With that in mind, is there any possibility you could show off function calling with Llama 3 models?
Sounds like what happens when you re-save an image over and over. You get a buildup of error inherent in the algorithms.
ooooo baby im ready!!! so ready to test lama to the limits! already AI has done so much for me gyat dahym
Very interesting to hear that synthetic data is very effective to train models, although it is not surprising,
I wonder how important the quality of the real data that was used to create the synthetic data needs to be?
For FSD for instance, how would you create a plastic bag blown by wind not look like a solid object on the street. Or, how would you synthesize a vandalized stop sign (dirty, with stickers or sprayed graffitis,) if the synthetic generator has no natural training sets?
When I listen to him talk about open source as a way to level the playing field, I feel as though I'm listening to an interview of an in-progress race participant where they are suggesting everyone should cross the finish line together. Despite the sentiment, it is likely that other participants will race to win. It ain't over 'til it's over.
Marc Zuckerberg sounds like a completely different person. He actually makes sense instead of that corporate talk before
Word in the street is that the Saudis started construction on a new gigawatt GPU cluster - I hope the people that need this info get it.
Well learning ab instrument or lewrning to code will make you smarter on everything else.
On lamba 3’s 8k token context, i suspect this is just for the smaller models. Google gemma 7b also only has 8k tokens. Also since playing with these I was surprised how big these are.
Your channel is kind of a lot more quiet today.
I can barely hear the sound
So now there's Wes Roth reaction videos?
Llama 3 is really great when you interact with it. Confirms this video.
Have you heard ? The industry just died.
It was SHOCKED so many times, this was bound to happen.
RIP industry.
7:40 only problem is: AI isn't like regular software, AI isn't predictable. At all.
8:41 what people forget, every technology empowers individuals and organizations to have a bigger impact on the world, this includes terrorists, lone actors, etc.
If you replaced AI with: nuclear bombs, giving everyone on the planet access to nuclear bombs doesn't make us safer than it's just the biggest countries and with a lot of structure and regulation, rules, etc. around it in every country to prevent accidental use.
We need that 8x70b
I dont think enough people are talking about the math score on gpt-4 being over 10 points better than the next highest
Great pr for zuc, the former lizard!
It has now become a tradition to look for my new "shocking" AI video every day. Never stop ! :)
This is the truest, most massive, most important "race" for anything that has ever happened in existence. We are witnessing the beginning of the most extraordinary movie life has ever written.
A little dramatic? Sure, but make no mistake- these titans are NEVER gonna stop one-upping each other. And that is the single most exciting thing we could've asked for.
He is wearing Dinesh’s chain
I thought I was gonna hear your take. I already watched
this interview
In the 60's student riots half a dozen students using materials found in every ancient Roman household took out a tank.....humans are smarter than AI thinks :)
He's open sourcing them because he made the money to train them by selling our private messages. Strange part is that I'm happy that he did all of this. Good job overall.
When children first learn about metaphors: 9:23
All these SHOCKs on this channel. I am so shocked. ;-)
I think the SHOCKING take away from this video is how much Zuck can look human now
Please sir. Consider covering VASA-1 the talking avatar thingy that is miles ahead of everything right now. I'd love to hear your perspective. thanks
Samaritan vs the Machine in Person of Interest.
def best life like and likable zuck model to date maybe there is hope
Funny how google is still just farting around.. while Meta's AI seems to be kicking ChatGPT on it's butt
I see that Sapienspace's comment is shadow banned....
Zuck rocking that chain 🙌
I need much more disk space to keep all those AIs parameters xDD
I’m actually really glad that Zuck is thinking along the lines about concentration of power. This is exactly what I also think is the biggest existential danger - one actor or actor class having exponentially more power than everyone else. I started down this train of thought just considering that internal OpenAI staff already have access to a lot more raw AI power than the gated, tuned version for the public provides. Just extrapolate from there.
Now imagine what the NSA has. All paid for by people who will never even get a glimpse of it.
I keep wanting to increase the speed of this video, but these guys naturally talk at 1.5x
Zuck looks better with his hair grown out a bit more like in this video. That doesn't mean he should grow it hippie long.
The short hairstyle from the past made him look robotic. Now he looks a bit like a pro Tennis player a bit past his prime.
Now is the time for silence. For those who do not know what is really going on will not be among those who will inherit the earth. And those who know are wisest to keep their mouths shut. Because if those who don't know find out who does, they will try to force that knowledge out of them.
Let's assume you can reach with synthetic data AGI at level 4, but you go on and reach level 7. With the corruption thing also going on in this process, this might pose an important "paper clip dystopia"-like security issue, where the AI is still getting smarter but ethical value are corrupted in between.
Better not let Debo see that video
Chain!
I love the insight and vision for open software Zuck has - he couldn't be more right. By sharing we evolve together to higher organization intelligences of us acting as groups with congruent aligned causes. This makes the groups using open tech more powerful due to knowledge sharing and co-learning where intelligence of one is distributed to many others making the entire group exponentially stronger. The arms race is mostly going to be in the mixture of expert context systems, agential systems, memory, and rewards with everyone's code having ability to interchange to newest, latest and best as a service.
8k context window genius and allows more computers in the world to run it. The higher that context window goes up the more GPU required and the world don't have massive Nvidia cards. If Zuckerberg thought through that and made the context window for the majority to be able to run in their laptops that would be brilliant. 😮 Plus this opens up the world too Mobile devices too if you think about it. I'm thinking the next release The Big Mama is going to have a much bigger context window for those that have horsepower.
Generational barrier for improvement of models. Seems to be simlar with humans having child with close family members or refining dog breeds (they get crazy)?
Hell, yeah! This is gonna save me like 40 bucks a month!😂
The reason they make these models open source is not to give everyone access to AI. They are so big that you need the big players ti run them. This will be cheap for the honey moon phase. But as soon as monompolysation starts, prizes will go up. And by that time you are forced to subscribe and pay. Not paying would mean you have a big disadvantage in life. And i promise you have to pay more if you want usefull AI.
It's going to be similar to cell phone bills. Imagine the ridiculous ads we're going to have to suffer through
You don’t think companies will buy studio licenses to ensure quality and workflow across their workers?
Also, this is assuming that the humans will be necessary in the first place.
If you’re talking about recreational use, I doubt you will be needing the highest tier ai for every day tasks.
Unless a model is a good programmer, it won;t speed up research that much. Btw the spike in use of some certain words in science papers could be people using prompts like "please rewrite this paraghraph to sound better". How many papers are written by non native english speakers? It the best use for me atm when writing anything science related.
I find it difficult to believe Mark's primary motivation for releasing his software as open source is solely to protect people, despite it being a commendable argument. It seems more likely that he's driven more by the desire to regain relevance, possibly influenced by personal ego or pressure from his company to innovate at any cost. This approach, however, carries way too many risks and there will be way more complicated cyber attacks and it'll be next to impossible to keep up with new updates. Imagine for every single malicious online hacker that exists today after aquiring this open source software he will now have access to expanding his team to 1000x accomplices all in the form of complicated ai agents. The damage will be exponential.
Im getting less and less impressed by a new model beating another model in some random test tbh, maybe I'm waiting for actually implication before I'm shocked and stunned
How many SHOCKS can the industry take??, is it gonna fall soon?
at these speeds, GPT5 better release this week
Information extraction or automated data curation, that is filtering data to remove any that contains no information (according to information theory), i.e. what effectively amounts to noise, should massively improve the efficiency of training by compressing the datasets.
We as humans are also desperately in need of this as we try to filter the signal from the noise (isn't there some project called Signal AI?) so i really want Elon to use that as the principle rule of the X/Twitter algorithm. Not only would it show the most interesting posts more but it would remove the emotional noise from discussions while still allowing free speech by incentivising originality, critical thinking and first principles thinking and shadow banning virtue signalling emotive posts that basically say "I'm on this political team", which nobody else cares about.
Wes Roth by now; it's clear his MO is to hype anything.
I'm pretty hyped for a gpt-4 level open source model.... isnt' that what we've all been waiting for this whole time? I can't be the only one
What is MO ?
@@DihelsonMendonca Modus Operandi?
@@egmluvr Thank you 👍🙏💥
@@DihelsonMendoncapropaganda
I like him with more hair. Less lizardy
Mark's personal branding people need to be congratulated, I feel myself warming to him even though I know there's nothing but evil under the hood.
Yeah he's coming across as more authentic and with interesting opinions rather than just parroting what "should" be said like in the past
Zhang is not pronounced like zane, but pronounced like Jhang.
Zuck does open source LLMs not because of altruism, it's because he's behind OpenAI. If he was the first you would be damn sure he would have tried to monetize the crap out of it
Did Zuck look and sound like a normal human being in that scene? I am shocked!
hes already running on the 400b model
6:00 Smartest thing Mark Zuckerberg has ever said. Maybe the smartest thing anyone has ever said about near-AGI LLMs.
70b parameter model
------------------------
You
Which version of llama are you
Meta AI
I’m a large language model based on Meta Llama 3.
You
Construct a word ladder from the word fool to the word sage
Meta AI
Here is a word ladder from "fool" to "sage":
fool
cool
pool
pall
pall → pall (remove the second "l")
paul
sale
sage
Each step is a valid English word, and each word differs from the previous one by only one letter. Enjoy the ladder from "fool" to "sage"!
You
I think you made a mistake between 6 and 7
Meta AI
I apologize for the error. You are correct, I made a mistake between steps 6 and 7. Here is the corrected word ladder from "fool" to "sage":
fool
cool
pool
pall
pall → pall (remove the second "l")
paul
pall
pall → pall (replace "p" with "s")
salle
sale
sage
Thank you for pointing out the mistake!
------------------------
It still messed up with "salle"
That's a typical GPT3 mistake
ZUK for PRESIDENT! ;)
Intuitively, training new models on the output of prior generations should probably create more accurate models, but seems unlikely to generate models that are more creative or have superior knowledge. I think you end up distilling or refining the future generation. To actually make it smarter or give it greater creativity, I think you need to give it new information.
He looks nervous, almost like his darling AI is blackmailing him.