Видео

Thank you! Let me know what you’d like to see next

Lol that’s fantastic.. much appreciated! Share it with a friend -

Thank you! Every comment helps

@milacruz3970 Thank you! Let me know what you'd like to see next -

Thank you! I've been working on this project for about 6 months and the little guy is starting to get pretty frikkin advanced......

You nailed it! I actually have a full time job that does not involve robots. RUclips does not pay me for this. But I'm learning alot and the comments absolutely make it all worthwhile. I've got quite a few more videos planned, especially as we start taking Floyd out into the real world to start exploring......

Floyd's not ticklish, but I don't think he particularly enjoys being picked up......

George Floyd, Pretty Boy Floyd, and Floyd the Droid -

Glad you enjoyed it - more videos coming

Interesting. I don't have much familiarity with any of the models other than OpenAI -

@@LarrysWorkbench I also just learned today Gemini can take audio input natively! (Video too, though it basically sees it at 1 fps) No need for speech to text in the pipeline, and it can interpret tone of voice! I have generally found Google's models to be underwhelming, but free is free, especially for something so frivolous. (I know the API costs are fractions of a cent but I have a mental block about just letting the robot go tearing around the house, cuz oh no, it'll cost 25 cents. Downside, Google authentication is a pain to set up.) Long-term I'm thinking I'll curate robo-Claude's outputs into a dataset to fine tune Gemini. I still use OpenAI a bit, but imho the Claudes are better, for now anyways!

@thenobelrot Wow that's great information! I haven't even thought about migrating, but if OpenAI doesn't give me the 4o advanced voice capabilities by the end of the year I might. I'm still quite a beginner in this space, so until now I've had my hands full just trying to learn one model. What hardware are you working with?

@@LarrysWorkbench I'm really new too. Hadn't coded since the 90s before ChatGPT came out. I've got an old second hand turtlebot2 type platform, running ROS Noetic. I was using a pi 4, but was finding it a little limiting, so recently upgraded to a repurposed motherboard from a busted up laptop.

@thenobelrot I did some light coding in college back in the 90s, but when ChatGPT came out I got inspired. The Raspberry Pi seems ok for what I'm doing so far, but I'm *really* looking forward to OpenAI tightening up the latency on their API calls. Your project sounds interesting do you have any videos anywhere?

Are you working in higher resolution? Floyd's camera is only 620x480 pixels. You could probably use OpenCV to capture a lower res image. When I started uploading one photo per prompt I don't think the overall latency changed much. You could also try pasting your code snippet into ChatGPT and see if it can suggest any optimization...... (I do happen to have high internet bandwidth at the workshop. Idk if that makes any difference. This weekend I'm going to try taking Floyd to a different environment.) Stay on it!

@@LarrysWorkbench Ooo that's a great point. I need to check the resolution I'm currently running. I was trying to use openCV at first, but I was having issues for some reason and then went with capturing the image with libcamera instead and it's working, but just very slow. My bandwidth could for sure be better too.Seems like my robot in general is just slow even when just using speech to text. Might be something with my script. I'll adjust some things and let you know how it goes!

I'm able to run OpenCV as a thread in the background to capture frames from the camera and show them on my computer screen via Virtual Network Computing, and it seems to be able to operate at approximately 15fps. So that's pretty good for what I'm doing -

@@LarrysWorkbench Nice! I'm also using VNC to work on my pi and robot. I figured out the issue with opencv and it has to do with the camera I am using (pi camera module 3) not being integrated yet with opencv, so I'm having to use libcamera to capture images. However, it does seems to be working a little faster after I lowered the resolution. I'm not sure what's going on with OpenAI's API. I feel like it has been extremely slow for me the past several days and I'm good on token limits and credits, so that's not the problem. Has it felt slow at all to you?

@Gooby12337 I've was travelling this weekend so I haven't been online. I probably won't have time to get back into it until friday or saturday :/

and ill give you 200 sub

Thank you! This project has been going on for 6 months or so but Floyd is getting more advanced and it's really starting to get kind of weird......

Thank you! Floyd is getting more and more advanced. It’s starting to get weird. More videos coming…

We're woring hard on reducing latency. I think the advanced voice feature of GPT-4o is going to make a big difference. And I'm shooting some more videos that show Floyd rolling around the shop and picking things up, etc....

Wow a 3D printer, I would love to see that! They seem to have lots of interesting products. I've got more videos coming as well especially now that Floyd has a camera to actually see his surroundings -

It definitely wouldn't hurt. Maybe you could do speech-to-text / text-to-speech locally. I believe 90% of the latency is waiting for the servers at OpenAI to respond. GPT4o "Advanced Voice" mode is supposed to be WAY faster, if/when they ever release it :/

@@LarrysWorkbench I forgot to ask where you bought the robot. It is bloody expensive to buy and ship it from the USA.

The chassis and arm are from a Chinese company called HiWonder - their “Master Pi” series. I like the fact that it’s all metal parts, but I think there’s some cheaper robots on the internet. The microphone and speaker are from Adafruit.

Actually it's just a bunch of metal connected to GPT-4o, but it's been fun and people seem to like it... Plus Alexa can't pick up objects, recognize my face or pass the bar exam ;)

I'm using GPT - see my other comment

Thank you!

Lol I got the camera hooked up last weekend, more videos coming on that. I use OpenCV to capture images about 15 FPS and display on a monitor over VNC (virtual network computing). Then whenever I send a prompt to GPT-4o, I just send the most recent image capture along with it. That way GPT gets to "see" from Floyd's perspective, and comment if it feels appropriate. GPT-4o is "multi-modal", so it can accept mixed inputs. I'm SHOCKED at how much information it can extract. It guessed my age within a few years. It was able to read the writing on my sweatshirt. All with basically zero programming. I'll try to upload my latest code rev up to Github (/LarrysWorkbench) (Note that I don't have GPT controlling the robot's movements, yet.....)

@@LarrysWorkbench Thanks for this comment. I've been having trouble getting my camera to function with my robot. I'm using gpt-4o API calls just like you are and attempting to use openCV to capture the images, but my script keeps "failing to capture frame". Do you always have the camera running, or do you have it activate and capture an image when using a prompt or specific phrase? This stuff for sure isn't easy haha love your videos!

@Gooby12337 Great question! Right now I have two separate functions. One which triggers to capture just one image and attach it any time I send a prompt (question) up to GPT. The other one runs as a separate thread in the background and displays the camera video on a remote monitor, so I can "see" what Floyd is seeing at any given time. It took me like two entire weekends to get it all working. Here's my current capture function. ChatGPT wrote it for me; I don't really understand all the syntax - def capture_image_and_encode(): # Captures a single image from the camera camera = cv2.VideoCapture('127.0.0.1:8080?action=stream') ret, image = camera.read() if not ret: print("Failed to capture image.") return None print("Snapshot image captured...") _, buffer = cv2.imencode('.jpg', image) # Encode image in JPEG format return base64.b64encode(buffer).decode('utf-8') # Convert to base64

@@LarrysWorkbench Very cool! Thank you for the information! That's a great idea to have a separate function running the video feed so you can see what your robot is seeing in real time. I'll have to incorporate that into my project. And spending two full weekends trying to get it to work sounds about right haha definitely a ton of debugging and troubleshooting that goes into this

@Gooby12337 Yeah it's not easy. But that's what makes it so satisfying when it actually works. Stay with it! And keep me updated on your project -

The camera came with the original robot chassis from HiWonder, it’s 70 degrees field of view, 640x420 pixels. Not great resolution, I’ll probably think about upgrading at some point. *Hopefully* it wouldn’t bog down my processing time or add latency. GPT is amazing at the amount of information it can extract even from a low-res image

@@LarrysWorkbench Oh nice! I think I'm going to go with the pi camera module 3. Exactly, I'm hoping it won't bog down my processing time either. That is very impressive how it can extract so much info. I'm excited to add camera functionality!

GPT easily recognizes objects, can easily recognize itself in a mirror, etc. It was able to estimate my age within a range based on one grainy image. It was able to read the writing on the front of my sweatshirt. When I put on a sport coat it said I was looking sharp. It’s wild. Let me know how your project goes

I’m new to RUclips - is it better to put them in the title or the description?

@@LarrysWorkbench description

You need to describe what you gonna do and how does it work. After that you can add lots of hashtags

Good luck,im counting on you

Thank you, much appreciated! Floyd needs to be seen -

Thank you! ChatGPT helped me alot with writing the Python syntax. Let me know how your project goes -

@@LarrysWorkbench Will do...

@@LarrysWorkbench Got Floyd talking to me! Your suggestion to have ChatGPT help me did the trick! I used parts of your code and ChatGPT showed me how to install the various modules I needed. Much fun indeed!

Wow that's fantastic!

You can.. It’s a Raspberry Pi processor with a mic and speaker, making API calls to OpenAI. I taught myself Python just so I could do this project. Check out my instruction videos

Thanks 🔥

Great question! So far I've put all my work effort into speech conversation and personality. But just recently I started working on his vision routines as well (camera & sonar). Right now he's still completely blind, but I'm hoping that'll change within the next month or so -

That's a great question! I probably have him running his conversation routines about 20 hours per month, at an average of maybe 20 cents per hour. It looks like about half the spend is on audio and half on GPT 4o

@@LarrysWorkbench very impressive project Larry

Thank you! It's been fun and I've definitely learned alot. Working on getting vision up and running 0-0

Lol I'm working on it, but not *completely* sure it's a good idea :/

Lol right? He says it's pretty much like trying to figure out a soap opera. And he's kind of right -

I wanted to see if he had any comment on behalf of the robot community. Floyd's got an opinion on everything!

@AnnLi-s6n Thank you so much for your response! I'm a tech beginner also - I started this whole thing around March and it's been taking up most of my nights and weekends. My first videos in the how-to series detail the hardware specifically. Hiwonder had some instruction videos that were very helpful regarding assembly. And the Raspberry Pi image came with a variety of Python libraries that allow programatic control of the motors, servos etc. But there was also A LOT of trial and error 0-0 As far as conversational latency, I haven't clocked it but I think I'm in line with what the GPT audio models can do on the phone. OpenAI has said that they're using the same series of three individual models as I am (Speech-to-Text, GPT, Text-to-Speech). Of those, the TTS seems to be the slowest by far, and I'm confident that they're eventually going to speed it up on their end with 4o. But I'm also trying to figure out how to stream the response rather than waiting until it loads 100% Stay in touch and let me know what direction your project goes in!

@massedin Yeah I think the GPT library got updated a few months ago, I remember having trouble with it as well. Here's my github link github.com/LarrysWorkbench My API call syntax seems to work FWIW Lmk what happens

thx for the link! i’m just focusing in the chat part (without the motors, lights and sound effects) but I just can’t make it work haha what library’s you had to install? and how? like pip install pylapse etc…

and what documentation did you use?

@massedin check the code - I mainly installed then imported the OpenAI library: from openai import OpenAI I used alot of ChatGPT to answer my questions and help me. But it was also a huge amount of trial and error. I actually walk through the code for my first API call in Episode 7 of the video series - Stay with it - you'll get it!

@@LarrysWorkbench Thx! After some time it actually worked, but just the part where we speak and he replies. This is already awesome, but now we are stuck on the board. It was hard just to understand that sys.path where you import the ‘Board’

That's fantastic, thank you for commenting!| I originally designed Floyd as a kid's project. I realized that you can do really interesting stuff if you can connect the Raspberry Pi to external APIs - the robot could be a great assistant. You can check out my source code at GitHub (LarrysWorkbench). Please let me know how your project progresses!

Thank you!

Thank you so much - that's absolutely great to hear! This whole project was originally designed to be kid-friendly, although it seems like it evolved a little bit. I don't make any money doing this, but comments like yours make it absolutely worthwhile. I've posted my source code at github.com/larrysworkbench. Keep in touch and let me know how your project goes!

Fantastic! I'm trying to post about one a week. Let me know what you'd like to see in particular

That's great! Let me know how your project goes -

I don’t know the exact JSON structure but the OpenAI API docs are pretty good and I’ve posted a version of my Python code at github.com/LarrysWorkbench

Hello Larry, I thought it would be possible to add object recognition by importing, for example, tensorflow object detection. Floyd could use the camera to recognize various objects in the environment and interpret their location. What do you think about it?

Sounds cool. I'm really excited about incorporating vision, but I know there's going to be some learning curve. My plan was to use OpenCV for some of the recognition, but I also want to pass still images (or video?) up to OpenAI so the cloud portion of Floyd's brain can describe them or even interact with what it's seeing -

@@LarrysWorkbench super-cool👍💪❤️

He does for sure -

@Dufftata yeah it was cheap from Adafruit.com but Idk what the shipping would be. The *only* problem I have with the Whisper API transcription is if I try speaking from more than 4-5 feet distance (lol 1.5 meters). So I'm also considering upgrading, but I'd want it to be USB plug-and-play, which this one was

@Duftata By the way - feel free to amplify the project to your subscribers if you want ;)

Thanks so much! The project is defenitely gaining some momentum -

Thank you.. check out some of Floyd’s other videos

Thanks for reaching out. The easiest way is to leave me messages here -

Go for it (!)

@Dufftata Thx so much for the support! I just posted my code on Github / LarrysWorkbench feel free to check it out. It seems like "Personality" is what turns GPT into "Floyd" and makes the project fun. Floyd's Personality comes from about 12 plaintext sentences. I think of it as his Long-Term Memory. The most important instruction is for him to keep his responses BRIEF, two sentences max. Regarding transcription - maybe check out your hardware. I feel like the Whisper API does a really good job when I give it a decent audio file to work with....

Check out my series of video tutorials on RUclips where I'm basically going through the entire project week by week. I don't have any other accounts, but considering posting the code on Github

Thanks so much! More videos coming

That’s awesome! Thonny came already installed on my Raspberry Pi but I think you can use Visual Studio Code as an option. I did struggle a little bit to get the “Board” library installed. It accesses memory and the GPIO pins so I have to run it as super user. Next week I’ll post our first API call to OpenAI. Floyd’s code is actually three different ones (transcription, chat, and text-to-speech), with sound effects to distract from the latency. Have fun!