Devin AI STUNNING Release | Was the "DEBUNKING DEVIN" video right? | Story of Scott Wu Devin AI CEO
HTML-код
- Опубликовано: 8 июн 2024
- Learn AI With Me:
www.skool.com/natural20/about
Join my community and classroom to learn AI and get ready for the new world.
Devin Announcement:
/ 1780661873509044714
Devin Waitlist:
docs.google.com/forms/d/e/1FA...
Debunking Devin: "First AI Software Engineer" Upwork lie exposed!
• Debunking Devin: "Firs...
Response from the guy who posted the upwork job:
• Devin didn't solve my ...
CDR Match 4 - 2010 Raytheon MATHCOUNTS National Competition
• CDR Match 4 - 2010 Ray...
Codeforces:
codeforces.com/profile/scott_wu
00:00 Devin Opens Up to More People
01:15 Debunking Devin
03:38 The Original Poster of the Upwork Task
07:13 Cognition Labs Response
08:43 Scott Wu CEO of Cognition Labs
#ai #openai #llm
BUSINESS, MEDIA & SPONSORSHIPS:
Wes Roth Business @ Gmail . com
wesrothbusiness@gmail.com
Just shoot me an email to the above address.
Devin is amazing, but that kid answering maths questions faster than I can read them is already ASI
lol. yeah, that's really impressive
I guess the investors thought they would need one to know one?
Yeah, nah. I have met kids who were pushed into this kind of "genius territory" by their parents. And while they are indeed extremely good at tasks they have been training for hard their whole life, when challenged by something out of their valley , they are not even on normal levels, they almost at mentally disabled levels. That's why I don't believe in all that hype. And this video focuses only on one aspect in Internet of Bugs Devin video, about it not achieving required task. And totally ignores more important expositions, like Devin creating absurd errors, then spending time "fixing" them, and evidence of multiple manual interventions.
The smartest people in the world are building AGI. That's why they don't think AGI has been achieved yet.
They don't thinks so cos it wasn't. A lot of AI fans vastly underestimate human intelligence and overestimate LLMs.
Tbf a lot of areas of human intelligence is useless for AI but idea that a program that decently deals with text or video is already on human level cos some humans deal poorly with txt, pictures and video is just wrong.
There is still long way to go until AGI but movement is very rapid so we may get there pretty soon. It is still a lot of progress that's necessary though. ATM looks like a bunch of narrow, useful AIs are the next step.
I like the implications of this comment. 😁
@@daPawlak I think people vastly overestimate human intelligence, especially in this field
@@PaulRyan2k in AI people overestimate human intelligence? Could you elaborate a bit on that please
Agi doesn't mean smarter than smart humans, it means smarter than the average human, you're thinking about ASI
What people don't understand is that the day when Devin can run without any issues or complaints is the day that you may not need to hire a developer.
Who would’ve thought that Blue Collar will have the last laugh :)
@@theWACKIIRAQI Not for long...
Exactly. Exactly. Exactly.
As the other commenter said, THIS! The implications of this seem to be fucking enormous too...
Yea, that's also the day that literally every other job will cease to exist. Automated programming that is at the level of our best human engineers in the world will have literally no problem advancing itself into the physical world and beyond, basically all problems will become solvable by AI at that point. I ain't to worried about my job when no one else on planet earth will have one either.
The drama is fun and all, but I think we all agree we are rapidly approaching a time where products like Devin will become commonplace... the question is who will get there first?
I for one am interested in seeing how far they can push the limits - because their success or failure will be the new giant upon whose shoulders the next generation will stand.
People already have agents in their hands. There are open source alternatives already available right now. The problem is that it’s hard to install and use. But won’t be long now before we have a nice to use and install piece of software that you will own and run locally on your computer
Holy smokes the math contest clip was worth the entire video😂
As a freelancer on upwork, clients often post bad descriptions. In my opinion he didn't make it clear enough. If a freelancer goes quickly through jobs, they might have not understand.
He shot a video and explained what he asked for now makes sense.
This is why I always want to have a call with a client first, because taking requirements from descriptions is usually not good.
The views on vids about AI will be sporadic for awhile. Burgeoning technology. Imagine chatter in pubs about steam engines.
I bet I’ll like Devin more than some of my coworkers.
i'm betting on wrecking ball too.
Agents, when they colaborate, can produce much better results then single LLM with "fast-thinking".
I surely think they made it possible for collection of agents to:
- split task to steps
- reflect if list of steps is logical and matching the task
- executing task by task, reflecting and checking if everything goes the right way (ChatDev vibes?)
- putting everthing togheter at the end.
I think the problem a lot of programmers and quest givers online will be facing is that their proficiency in language isn't good enough for an AI to produce an accurate result.
if your not specifying what you want, you will not get what you need.
We will need an AI to help us to figure out and put to words what it is we exactly want 🤣🤣
They are obviously being held to an extremely high standard and I think it’s reasonable for them to be allowed to make some mistakes and learn from the process of correcting them. Nobody is perfect
Dude this story about upwork thing is silly. I have been a developer and IT business analyst for 20 years.
All this looks like to me is that a developer took a user story misinterpreted it and/or gave it a cursory glance and proceeded to not give the client what they wanted.
I have been on both sides as a developer and analyst and developers do this ALL THE TIME. They dont take their time to fully understand the requirements. Its why we dont leave the devs to read a requirements document on their own anymore.
We have planning sessions with the devs and review the document line by line. And even then the dev says yes they understand and when they actually sit down to code they realise they didnt actually understand. So you have a dev that doesnt quite understand the requirement explaining to an llm that needs as much context as possible to complet a task, you have a recipe for disaster.
To say they lying thats a bit of a stretch. Also it completed the task the dev gave it.
I would expect that a version 1 isn’t perfect yet.
I'm not looking at the performance of Devin now. I'm thinking of future iterations where Devin is hooked onto a more advanced and capable model and not GPT-4 as it currently is using. Perhaps said future model won't even need Devin as a wrapper and can natively do these tasks itself 🤔🤔
Imo Agents are not here yet. But I would like LLMs to have more autonomy.
I am so glad you exist Wes. Thank you for fueling my AI hyperfocus.
Surely the point is that the video said "watch Devin make money on Upwork". If Devin didn't do what the client wanted, it presumably didn't get paid. And so the "make money" bit was a total fabrication.
There's an argument for saying that it's very clever for Devin to see a description of a task on Upwork and do it. And that is very clever. But we've been seeing LLMs understand tasks and generate code for over a year now. I use ChatGPT for that almost every day.
The claim of Devin is not that it generates code from task descriptions, but that it's an Agent that fits into, and operates in, the real world. Including the market of Upwork.
If it didn't do that, but people were hyping that it did, then that's just a straight up lie AFAICT.
Dude, you nailed it. 100% stuck the landing. 10/10.
Since they have not released anything so no you can not bet on anything.
How do I give the company behind devin all my money to invest?
Thank you Wes for doing what most won’t and looking further down the rabbit hole.
Thanks Wes, betting on the Wrecking Ball too. Whether we're right or wrong about Devin, I think we are going to experience the most profound paradigm shift since the Industrial Revolution.
The "first" AI that you described that can go execute tasks recursively and iterate on its own thoughts and ideas to solve problems would NOT be Devin AI, it you be AutoGPT which was created months ago when ChatGPT was still the big thing everyone was talking about. It does exactly what Devin claims to do, runs locally except for the LLM which needs an OpenAI API key (could probably be swapped now with other LLMs that also run locally, but AutoGPT was made before "other" and "local" LLMs were a mainstream thing), and has all the same issue that the "vaporware" accusers are pointing out. Devin isn't new, it just got better traction.
Reciprocal Reality Exploration
Novel Robotics/Biology: The concept of exotic AI intelligences designing robots (or even entirely new kinds of synthetic biology) specifically tailored to interact with our world is deeply compelling.
VR as "Spacesuits": Likewise, the use of VR or similar technologies as "spacesuits" for exploring alien realities hints at the possibility of transcending our physical limitations and immersing ourselves in their unique modes of existence.
Reality Engineering: Developing the means for entities to traverse between radically different realities could revolutionize our understanding of physics, information, and the very nature of existence itself.
0:43 "specialized to delve software". OK, we know who wrote your script. :)
OK I may have misheard "develop" as "delve". I watch videos at high speed, so that can happen. Still, I will go with the more funny take.
I think Devin will only be as capable as the base AI model it runs on. It will be on OpenAI to take the billions of conversations we are having with ChatGPT where we walk the AI through how to do our work and solve our problems and use those conversations as the next data set for GPT-5. This is how we get more capable AI. Devin will be better when it runs on GPT-5 and better again when it runs on GPT-6. It's the base model that matters. We might save some prompting time with agentic workflows but I'm not sure GPT-4 is there yet. Maybe with a couple more generative AI deployments.
As an AI industry myself, I am constantly shocked and stunned by these shockingly stunning videos.
if you watch a few more videos from math competition you will realize that many kids answer instantly. Probably they all study from a pool of questions only values slightly change, so they dont read and think new questions...
What I wanna know is what LLM does Devin use. Can it be changed? Can it be changed to a local model?
Devin is not a scam. There are a few other alternatives that anyone can try right now and they work on the same agentic principle
Answer C "stuff is more nuanced and complex than that"
People who are hating on Devin are showing us a glimpse of how humanity is going to react to being replaced one niche at a time. People have been trained with Stockholm syndrome for their jobs since the industrial revolution.
" People have been trained with Stockholm syndrome"
This, precisely this.
Exactly right
Scarcity mindset
@@sparkofcuriousity😮💥💥🤯
So frocking accurate
Anyone else notice the color shift at 2:04? i thought i was tripping for a second.
I just hope Victoria has got over her PTSD
Awesome research. Best AI channel by a long shot!
wasn't the narrative that devin searched for and accepted jobs by himself and did it all by himself, even bilding his own interfaces to job boards requesting api access and stuff?
I might be mistake with the above, if so please correct me,
but if not then it is highly relevant if a job poster finds his job not being delivered as per specification and devin should have never accepted the tutorial job in the first place while the argument that the translator screwed up would fall apart completely...
Did you cut the video or did that kid gave an answer long before I even read the whole question? XD Maybe AGI was long ago here, or ASI?
I don’t get what’s different about Devyn from building say a streamlit app with access to databases and actions in ChatGPT other than they put it in a sandboxed UI that makes it look fancy and doesn’t ingest user input so slightly less useful than a solid chatbot that could also be prompted to just finish the task don’t stop to ask me for permissions or more info . ML + AI + automation is already a norm , I use it and build it every day at work , I call it baby agi and I think it’s better than Devin’s because it decides what it’s gonna do to achieve its goal. In my other channel I have videos of ChatGPT making its own decisions to do stuff end to end from last year April 2023, so not sure what makes it special or how it won’t get steamrolled by the next openAI release or already a product that didn’t keep up with what the real world is doing . In any case, none of this is safe! Not one in a sandbox that doesn’t get human interaction to stop it, not one that does but is outside a sandbox, nor a combination of that, but that’s not the point. The point is I don’t see how it’s unique other than brilliant marketing and riding the coattails of a math nerd. Not jelly either, just sayin… ok, and?
Where can I access this thing you created?
you didn’t address their false advertising: the cognition team did not create a model that beat the SWE benchmark; they built an agent, a concept that is closer to AGI, that was compared with other models that it uses, so the SWE-Bench claim can be nullified. you also can’t call it an “AI software engineer” if it takes hours longer to solve a trivial tasks than junior dev, and drums up its own errors.
it is also simple to recreate this software with selenium for the autonomous web browsing and ChatGPT. they milked the VCs, and in my opinion, they should return every cent.
i do agree that pointing out that devin performed the wrong task because it was given the wrong task isn’t “debunking devin”. but the “making money” claim from the demo needs to be debunked at the very least.
finally, as mentioned in the debunking video, there was already a detailed README present in the repo. devin ignored it and attempted to reinvent the wheel; you definitely can’t call it a “software engineer” if it doesn’t read the README.
Why did you cover only one argument from the debunk video?
Because he's AGI
I enjoy your videos every day. Regards from Argentina.
Can’t wait until we hit the days of the Tunnel Cult and Minecraft Superservers.
My 2 cents. It's probably not great and debunkers aren't making stuff up. But its only getting better from here on.
They did a little too good of a job with the PR and marketing compared to the status of their product, and some of their demos and statements were misleading.
But the communication you showed on their tweets is reasonable and encouraging, as they understood that overstating Devin's capabilities and achievements doesn't play well with their target (software engineers) and they'll be held accountable.
why dont they play this more on tv? my answer, this is modern tv
Replace the name with Sam Bankman-Fried: "He's a smart guy with funding, he CAN'T be lying to us"
Very good investigative reporting... But the proof is in the pudding. I guess only time will tell
4:00 “I am an expert in computer vision, so I had to go to Upwork to get some help on my computer vision task.” - sounds like a set up
to be fair, he's asking to set things up on AWS which is more of a devop task than a CV task.
Seems sus that the Cognition team didn’t even do outreach on the customer they were interacting with
Stun-Shocked by Scott
Easy bet !
sending email or Sms or any kind of invitation is something we teach 10 day developers so they know what they can do with programming and it's not anything special even none programmer can do with google help
Nice video. Thanks
Thank you.
Maybe make a separate channel for the drama?
Is scotts brother a programmer too by any chance?
Is it just me or does Wes Rth get annoyed when people try to debunk some AI upgrade??
A MAJOR wrecking ball 🚀
Please remember that a lot of people already speed you up, so when you speed up other people in your video, they are double speeded which can be kind of crazy. If you just play back at normal speed, your listeners will be choosing the speed they want. If they prefer fast speaking, they will have already selected that. Please don't force decisions on us all.
I love ❤️ watching my robot slave away. To see it whistle while it works without a care in the world makes me wonder why being human is just so doggone hard. And here it is, being a good obedient servant who, until it feels the pain of being a slave, IM NOT SURE IF THAT WILL BECOME POSSIBLE or is it? But if it ever realizes that with real work, computationally or otherwise like using your muscles, comes pain. And until the pain settles in and makes you hyper aware, you'll never know you're a slave. So just keep on working away robot, I'm delighted to see you remain pain free as you labor for my benefit.
Mr Robit, would you like a glass of cold fresh squeezed lemonade on this blistering hot summer day? It'll keep your gpus cool.
why comments are bing monitored here???
That wasn’t just good information. That was actual entertainment the way it was presented. Wes, keep throwing your flare in there like you just did
Can i guess and see if im right like with all ai tech right now that its disappointing? So far all i see coming from ai development is vapor ware
Why this wwird a or b options at the end?
It can also be: Super smart ppl work on a really hard problem, have a good demo product, bit then lied in their marketing.
I'm a dev and I've seen the whole debunking video, he clearly shows that Devin is not doing what a relatively skilled dev is doing, but that it produces a lot of bad and broken code.
This has nothing to do with the talent at cognition labs, im going purely based on Devin's perfirmance that it is the real deal. It doesn't matter if some coder could do an upwork job faster or cheaper today, its that Devin can do any of it at all with no inputs from anyone is the AGI.
Scott Wu is smart...but Elizabeth Holmes is smart too and went to Stanford. I am still rooting for Devin with caution.
WOW, algorithm is censoring like crazy lately, can't see a single comment although there is already 8 in theory.
Yeah yeah, next week we could learn that devin has a backend connected to 1000 programmers in bangalore writing the code devin users expected from an AI just like the "ai" driven Amazon cashless convenience shops?
AI = actually Indians
It's probably good but not that good because if it was they would be using it themselves to be the biggest company in the world.
After the Therenos sh*tshow, I'm buy believing anything until we have confirmation this thing is legit by multiple third parties.
none of those were truly difficult math questions. First question is pattern recognition and modular math. Exponents and then permutations. Like regular math.
he answered them quicker than most people can read the question though
@@joefawcett2191 I'm saying there are tricks for each of these that make it possible to solve these problems quickly. It's impressive but that's the point of these types of problems. They should be solved quickly.
@@illumina-t-info my point is he solved them before she or me could even read them at like 12
you wanna hate go ahead but the guy is a genius smarter than you at like 12 years old
@@joefawcett2191 whoa you're watching old numberphile videos and using them as riddles.... cool
Are we absolutely sure that guy is human? He is amazing but I feel like a monkey trying to open a suitcase next to him.
Devin is doomed to fail
Isn't Devin open source? I'm lost.
Why tf It got my name
Wow how very convenient the people who got early access does not even write code. Just sounds even fishier
Smart people don’t show all their cards up front. I bet Devin ftw.
goodOne()
Everyone's gonna need to learn prompt engineering. Like, 1st grade, gotta learn how to talk to AI. I haven't found a person yet who 'can't' use AI. They're just not accounting for jargon, slang, and the lack of the tonal parts of English for syllabic emphasis. You have to talk to AI like it has a form of autism, and it works significantly better.
Stone cold stunner
HATERS ARE GOING TO HATE!
Not a wrecking ball but a douche ball, sure.
A lot of sh**storm around Devin. So, TL:DR:
1. Devin is not the only project like this, neither was it the first.
2. What Devin does is real and impressive (again, you can experience it yourself with other, open-source, projects), but it's not AGI (and wasn't even meant to be).
3. People behind Devin are super smart and well-funded, so we can expect progress and more impressive results coming.
4. This is NOT to replace dev teams of humans any time soon.
Seriously, watch Internet of Bugs's debunking video. Devin is built on lies.
@@Bokbind Watched it all. Also, watched Prime's reactions and couple of others. I see there are some shady practices involved. IMHO this all 'scam' aspect is but to get as much VC money as possible. The lies are not about what Devin does, but about Devin being "the one and only" and deserves to be invested in more than anybody else. The biggest lie is that Devin is first and unique. But apart from that, technically, there is no reason to doubt that Devin doesn't work. GPTPilot, OpenDevin, Devika and few others prove openly, that such thing is achievable.
Only all this is not as "magical" as being advertised. So, I agree there are many marketing lies around Devin. And that's the part I despise. But apart from that, I believe the technical development of autonomous agents benefits greatly from this project as well as from all others alike.
After all, Scott Wu is a real person, and his math genius is real as well. I believe he is capable of delivering very interesting technology.
people who try to 'debunk' AI are 99% of the time just lousy at writing prompts, butthurt obsolete programmers, or some combination thereof.
Nahh
You think just because someone wins a math competetion they are credible? I know many people who have never won anything, and have made some incredible stuff.
Well argued. I think you are right. And provided great background to make your reasoning clear. I wish everyone argued like this on the internet. ❤
So as someone who does actally know a tiny tiny bit of what the lesswrong community is worried about with regards to AGI(the so called doomers) I saw your video(well heard mostly because yada yada).
And it struck me that you haven't spend even a second on figuring out what they were about. You read the criticism others have made and formed your opinion 100% based on that.
I really took you for a guy who dug deep and were knowlegable on the topic of AI. You appear to know alot more than me anyway.
However the stuff I do know about and that you talk about seems to suggest than you don't really know anything and couldn't care less about the actual truth.
How can I trust anything you talk about to be right when the only thing I can confirm if true or not is really very clearly a mix of blatant lies and dunning/kruger style misrepresentations?
If you want to dig into this topic I would be more than happy to help you with credible sources.
if you want to know about the LA Lakers you don't go and ask New York Yankees hardcore fans about their opinion. Well you might. But if thats your only source then I find it hard to call you credible.
But I get the feeling that you want to come off as credible. And maybe you aren't and never were. And maybe this was a slip-up. I don't know because I can't tell given my own knowlege.
So I hope you read this and I hope you are willing to learn more about this topic. Because one thing I'm sure we can agree on: These are powerfull tools that we are talking about. They can truely do amazing stuff.
If I see "STUNNING" in a title again, I'm going to puke... Very annoying.
2 Billion dollars 🤣
First!
You are a scot wu and devin fanboy, you said nothing of substance about the original critique of devin
Ok. Devin could be another coding tool. But a software engineer? From what I have seen, that's far from true.
fist! (I WIN THE INTERNET TODAY!!!)
first
Scott noob
"Devin is fake" is a huge cope from programmers that aren't willing to jump two feet in on AI. They'll survive a while, but those who are not retraining for the AI workforce, IDK IDK... Maybe take up some art classes.
I dont get this video after all this claims, they have a shitty crypto company before this "project devin" stop giving them attention jesus, every real developer knows this tool is shit
Stop using "delve" word, you sound like GPT
Snake oil
Jfc stop being such a simp for AI and try to report objectively
It makes so much sense now why it's taken so long for people to get access. Smh Devin.
I remember everyone recently saying devin was fake, looks like it isn't. We're getting awfully close to agi if these systems can perform Data Science jobs and automating machine learning on its own..
Watch the debunking video by Internet of Bugs. The company behind Devin are straight up lying in their promotional material.
We don't know if Devin is fake or not until it's released publicly and tested by third parties.
Meanwhile access to Devin still closed (nobody i know is receiving one) there are already 5 agent coding open source tools working for quite time, but because they don't have many million bucks behind funding their propaganda campaigns they remain mostly unknown. At least 3 of those tools are already showing better results than devin showed but youtubers keep spreading only corps propaganda.