Using an LLM to direct our robot Digit.
HTML-код
- Опубликовано: 22 окт 2024
- In this demonstration, Digit starts out knowing there is trash on the floor and bins are used for recycling/trash. We use a voice command "clean up this mess" to have Digit help us. Digit hears the command and uses an LLM to interpret how best to achieve the stated goal with its existing physical capabilities.
At no point is Digit instructed on how to clean or what a mess is. This is an example of bridging the conversational nature of Chat GPT and other LLMS to generate real-world physical action.
-------------------------------------------------------------------------
At Agility, we make robots that are made for work. Our robot Digit works alongside us in spaces designed for people. Digit handles the boring and repetitive tasks that are meant for a machine, which allows companies and their people to focus on the work that requires the human element.
Subscribe (hit the bell for notifications)
/ agilityrobotics
Join our Team
www.agilityrob...
Follow our Journey
/ agilityrobotics
/ agilityrobotics
/ agility_robotics
/ agility_robotics
#robotics #machinelearning #AI #GPT
It cleaned the mess without saying it was going to go have a smoke break first. It's already better than some of my past coworkers.
It cleaned the mess without complaining about its previous coworkers.
Fantastic!! Wow, this is such a great idea! I would love to see a video about how the robot recharges its battery and interacts with the dock. It must be so cool to watch it go in and out of the station.
Amazing, this is what has been missing in robotics so far, a brain. It feels like this plus the hability to speak to ask questions, an internal state and some memory would pretty much give you a working generalist robot that would actually be hable to do a lot of jobs.
The devil is always in the details. The LLM is providing a good high-level set of commands, but we don't know how general the robot is at navigation, object identification and manipulation/coordination. The latter is the hard problem. Tying in an LLM is cool, but doesn't tell us how for along they are with the hard problem. It certainly worked for this specific video take and this specific set of objects.
@@cuthbertallgood7781 I personally think the hard part is the LLM, providing actual reasoning an intelligence for the system.
@@kmo475 Manipulation and object identification are categorically much more complex problems than high-level task planning which LLMs replace in this video. Actual reasoning and intelligence are very vague descriptions. What we see in this video is the mapping of the objects (let's say O_1 to O_n) to bins (B_1 to B_n). The commentator already states that all these are input to the demonstration in the video. LLM does nothing more than relating those O_i to B_j using natural language primitives embedded in the language model. The same can be done with a very simple data structure (e.g. map) where the types of O_i can be related to B_j. All the rest would still have to be done by lower-level capabilities such as localization, navigation, manipulation, motion planning, and object recognition.
Having said above, I'm not saying that LLMs are useless. Such a data structure would become intractable in the real world for a generalist robot, as you suggested as well. That's the place where LLMs offer real value. But we won't see human-level androids around us until they have the same lower-level abilities as humans do.
@@tahsincankosewhat if each robot is only programmed to keep its immediate area in good standing. So the LLM is there to be an interface between man and machine, and anything the human commands will be expected because not only will the AI have the LLM to interpret the commands, it will have ML to help it remember how things should be and how to get them back in good standing.
Also, programming each bot to a different portion of the factory would be relatively easy compared to teslas approach of teaching Optimus the same sort of object detection as the self driving neural nets.
It's all fake.
I just noticed the new legs, even tho I saw the previous promat videos, I find it very cool to see a design go throug streamlining and optimization.
Wow so you can program them to participate in a fire drill! If the alarm sounds they have to go to an exit thats not close to the fire and to avoid humans. Or they could lag a bit to verify noone is left in the building calling for rescue. You guys are great!
Amazing work Agility
Excellent utilization of next gen tech and very innovative! Keep up the awesome work!
Digit LOVES work!
Well done Agility Robotics team! I saw you 2 years ago in the UK at Goodwood and the progress you made in HW and now with LLM integration... Bravo! You are literally eating Teslas lunch atm. Hope you guys find enough funding to go big. Whens the IPO?
Hey, Digit! Execute Order 66.
*HAPPILY*
Do it
Digit says:processing processing waiting... Waiting... robot will begin
countdown self destruction mode mode on. 10, 9, 8, 7, good bye 🤖🤖🤖💣💥💣💥😆😆😆
it will turn the 6 6 upside down like i is :)
Amazing work! We have come so far.
Excellent! Now it's getting REALLY interesting. :-D
Amazing ❤
But it's not enough for a family 😊
We need a 'house keeper Robot' replacing the female
housekeeper we have hired and worked several years ❤
First operational robot, congrats
How does it discriminate between materials? Known objects? Awesome demo! Please show more!
It's just a fake video. Hidden in the small print will be the fact that the robot was under remote control by a human.
Interesting choice to go with digitigrade legs, makes for a very unique design
wonderful! I can't wait until these little beauties can live in our homes and help us with the housework!
Excellent!
You guys are doing great! it's so fun to watch your systems progress. Keep up the great work.
This will be the future. If you don't mind me asking, which LLM is being used?
It says #gpt in the title
@@SeanKula I understand that, but there is GPT-3 , GPT3.5 turbo and GPT4. These are just generalized, there are more specific instances of these model sets.
@@DominicI1 scroll down in the comments somebody asked and answered 3.5
3.5
@@AgilityRobotics Thx 👍
Agility Robotics
gooood jooob, ai robots in for people and world ))
This is cool stuff!
Amazing. Will they be as fast as a human in a few years?
For the tote manipulating video from Promat, we're already about 2x human throughput. The apparent "slowness" is fascinating! What we think is happening is that the robot works super consistently, which results in a higher average speed, even if peak speed is lower.
@@AgilityRobotics That's incredibly fascinating. I always imagined speed to have been a limiting factor as processing the data would take time, but it seems like we've already reached past that point!
@@AgilityRobotics Great job. Yes, the average work would be surely higher, but I wonder if they will soon be able to be faster than a human in order to be more efficient in a dangerous emergency scenario.
If you could just get Digit to walk around like it does not have to pee now that would be great! Or rename it Figgit for more accuracy.
We need more updates like this showing how the robots are progressing. Not giving us a video update like once every 2 years
I love you Digit!
your mom loves digit too.
@@davidanalyst671 yeah, she better does. Say I love you so that the robot will remember you & not kill you in the robot uprising
Is Digit doing general image classification to determine which type of object it is picking up and what to do with it or are these known objects that it has already trained on / is the trash in known positions with known objects?
No. For this test, we seeded the world model with semantics in advance. But scene labeling from a vision pipeline is reasonably well understood.
@@AgilityRobotics that makes sense! I suppose the harder problem in this context is grasping in general, which Digit has shown to be reasonably consistent at.
This is amazing
The Digit robot puts the newborn baby with the dirty nappy in the waste bin.😂
Would you make digit move faster?
Nice
Not bad.
I, for one, welcome our new robotic overlords.
1:01 me at work when the boss only gives me one job to not destroy the cardboard
This is absolutely amazing. Wow.
We are getting closer and closer to general purpose humanoid robots.
What happened to his eyes and mouth? Hope you guys keep those at it made it more humane and relatable.
People need to be freed from their work so they can live their life and not just be slaves or cogs in a machine.
Love this!
Amazing
I hope Digit Robot will be able to drive a car for it's
human master in several
years.
If it can be implemented and
commercialized, I will purchase one so that it will
be my private driver and drive my petrol car
and It's not necessary for me to purchase
a 'real L5' Full Self-Driving electric car ❤
Nice! it can walk backwards so it dont have to turn and loos energy
🔥🔥🔥🔥
Imagine what this tech will be like in 30 years
Unreal. Imagine walking by a construction site and seeing a massive structure being built entirely by humanoid robots.
More like a Grasshopper then a Digit
Notice little to no cuts between movements. Unlike vapor bots from tesla
Does openai charge you tokens per task? or just one time purchase?
That's nice. Now Star Wars it up a notch please.
Ok. ruclips.net/video/5agZrkUifDs/видео.html
the size of the PIP window changes and youtube chose that as the moment to loop in the thumbnail. it looks like the robot is moving more than it is. I thought that it was shoving something very big at first. Then I realized it hadn't moved anything yet, but that the thing looping was very very short in duration and then I finally saw it was the picture in picture. Don't animate that graphic! The only thing moving on the screen should be the robot and the visual representation of the data its generating because otherwise it doesn't communicate the intended message. It's not significantly misleading, but it is a mislead. I got misled. The robot performed better than the thumbnail had made me think it had. I was not going to explore the content behind the thumbail initially, infact I still haven't. The only reason I clicked this link was to leave this comment.
My favorite thing about agility robotics is that they're smaller than humans. Makes them seem less threatening. Maybe that's a bad thing in the long run. We've programmed ourselves to feel threatened by the computerized dystopia ever since 2001: A Space Odyssey pushed the thought into the popular consciousness.
Which LLM does it use?
GPT 3.5
@@AgilityRobotics Thanks for answering. Microsoft has the most powerful LLMs out of them all but it's closed source and censored making it far worse than it has to be. There are a lot of open source LLMs available too for free. I hope to see an open source variant in a robot at some point because their training data and architecture are completely open giving us the ability to trust them. One of the reasons I don't trust ChatGPT is because the data it collects is seen by OpenAI/Microsoft and shared with their partners. Another benifit of the open source variants is that they can be ran locally. Especially you guys could probably afford to run a big one.
« Muuuum! DIGIT throw my bag in the trash bin!!! »
Sudo take out the trash
@@AgilityRobotics haha great job by the way :)
sex robots when?
How heavy can it carry?
About 15 kg, give or take. This version was designed for higher loads consistent with unloading tractor trailers, but for most tasks 15 kg is adequate.
My guess is that humanoid robots will start to go mainstream in 5 years or less.
Mainstream meaning performing work and tasks normally done by humans, and probably replacing low-skill labor.
I bet mid-skill jobs like house painting will be start to be performed by that time.
Doubt it
Don't you think you should make it stronger to work in industrial settings?....
The Bird/Insect leg stucture is 30%+ superior to Boston Dainamics toys. Oh, and your AI is not shabby either.
"Digit, clean up the kitchen."
Worth $100k right there. Where do I sign? ;-)
Brave new world.
I’m interested to learn about safety precautions. What if you requested that it behave violently in some way? What kind of safe guarding does it have?
Good luck telling it to clean up a hoarder house mess.
oooooooooooooooooooOOOOOOOOOOOOOOOOO
where is the voice command to complete the task ??? Chat GPT
where is the voice command to complete the task ???? Chat GPT
I want one ☝️ color is nice. Amazing
I for one welcome our new robot overlords!
I think ai plus robotics is a new beginning for the human race science fiction becomes reality.the only thing left now is for a robot hand to be as dextrous and fast as a human .that is the final piece in the puzzle remaining
So wait... it took them two weeks(80hrs) to program the robot to pickup that trash?
Nope, took about 3 seconds. Re-watch the video.
Large Language Models are capable of much more than traditional programming for specific tasks. LLMs can be very smart
Actions need to be taken very early to prevent misuse of this. This machinery and code needs to be foundationally coded to make misuse fundamentally incompatible with the design. This means:
-The inability for these robots to be used against human beings
-The inability for these robots to detect faces or fingerprints
-The inability for these robots to make physical contact with a recognized human being
The inventor of the first machine gun thought he would be saving human lives. He ended up destroying hundreds of millions. Let's see how well you match up to your predecessors.
“It occurred to me that if I could invent a machine - a gun - which could by its rapidity of fire, enable one man to do as much battle duty as a hundred, that it would, to a large extent supersede the necessity of large armies, and consequently, exposure to battle and disease [would] be greatly diminished.”
― Richard Jordan Gatling
That's really cool but I hope Digit's interpretation won't cut people's hand hahaha (but you get the point)
Digit, invade Cuba. By your command
This is nuts 🤯
I wonder why people don't talk much about Agility Robotics. It is clearly way ahead of Tesla Optimus but we only hear about Tesla as if everything that Elon Musk touches is golden even though Twitter is a disaster.
This is way too clunky for in bed, bros
Elon musk must be crapping his pants right now.
His robots not coming out for another 10 years. He loves to hype things up and make empty promises.
@@persianguy1524 Sorry he can’t predict the exact day state of the art novel technology will be ready.
Give it a British accent.
my dog also menacingly stands on his rear legs
It seems obvious that mankind has created a new Species "In the same way that a book can provide a gripping narrative with words and descriptions to invoke a reader, a machine can also provide a spoken or written narrative. Just as a book is inert, so too are algorithms that act as the book. People see faces in clouds or inanimate objects and feel emotions through inanimate words. The body of a book has no sensory apparatus for interactions, nor do algorithms, avatars or black boxes have sensory apparatus for interactions. Millions of years of culture and human conditioning create the images within the mind. While AI creates jpgs, uses synthetic constructed words, text, film format and binary digital information." Without any knowledge of Organic, sensual, chemical, person perceptions. Just like a digital book.
(A) Algorithms calculate in a mathematical way O & 1 @the speed of light. (B) Brains work in an organic biological way via evaluation & culture.
Two completely different species and mankind will be the inferior. Even with these head bands
If they can get rid of that pesky LiDAR, this could be the first mass produced humanoid robot. Everything else looks pretty economical. Modular hands would increase the economic utility as well.
In my opinion digit is more intelligent than chatgpt
There are too many small leg motions that use energy for not much. Can't this be optimized?
A prompt injection attack on legs.
Mr. Meeseeks v0.1
So you made a Tesla bot but cheaper. AMAZING!!
This is a simulation.
Nope. Real robot.
normalize robots that can do a flip
It seems like the only safe place is gonna be Internet free zones xdd
Yeah, but the starlink xdd
Ah shiz Skynet haha
great cgi
...and it's not even good CGI. Yeah, this is definity fake 🤨.
You guys really don't Interact with your commentators huh?
Nope.
Wait, are we interacting now?
So confused.
This is amazingly horrifying.
Now send these out to hold cameras and microphones, steal all the entertainment jobs
teach it to swing hammers and a waft paint brushes, take all of construction and fabrication
no need for fallible doctors, we have automated solutions
no futher call for carers or caterers or waiters or barmen
the end of farmers and fisherman and sculpters and tailors
no boys need go off to war, shredded at the whims of who even knows
just leave us the warehouses, the shelves, with the boxes
protect our ritualistic packing of cheap foreign made plastic distractions
you gave the robot a ridiculous command. Clean up this mess is something a woman boss would tell their employee. There was a pack of black styrofoam that was still taped up, and it should have gone on the shelf, but the robot put it in the trash. Those water jugs are not recycled, they are re-filled. The black foam and the water bottle should have been put on the shelf. This is why managers have evolved to be idiots, while the robots and workers are doing everything they can.
To slow, Can it pick up smaller items that only fingers can. Long ways to go
Slow compared to what? This will work consistently for 24 hours a day. Humans work 8 hours, get distracted and slow down.
@@SirHargreeves The example is large items. It has no dexterity with fingers at all. Its limited where a human can do many type of jobs. In polaris in MN you will see people using sharp knives and cleaning up a plastic hood from a snowmobile lid that comes out of a mold. Then in 3 hours they move to another station which requires different movements. For the cost of these and the repairs the human is still cheaper and can be used in more departments. Maybe in 10 more years but I see to many limitations with this. The example they provide is a horrible example. I have not seen one example where they put in live test area these are controlled.
Meh
Now give it a gun.
Nope. www.bostondynamics.com/open-letter-opposing-weaponization-general-purpose-robots
@@AgilityRobotics I respect the sentiment but somebody is sooner or later going to do this. It is kinda inevitable.