I built a MONSTER AI Pi with 8 Neural Processors!
HTML-код
- Опубликовано: 28 июн 2024
- The thumbnail about says it all. It works! But what does that mean, exactly?
Some of the things I mentioned in this video:
- Alftel 12x PCIe Card: pipci.jeffgeerling.com/cards_...
- ThirdReality Zigbee Smart Plug: amzn.to/3X97MvC (affiliate link)
- Pi AI Kit: www.raspberrypi.com/products/...
- Hailo-8: hailo.ai/products/ai-accelera...
- MidnightLink's helpful Coral TPU setup instructions: github.com/geerlingguy/raspbe...
- CodeProject.AI: www.codeproject.com
- Raspberry Pi Linux PCIe issue: github.com/raspberrypi/linux/...
Today's RUclipsr outtro tribute: Chris Titus Tech
Support me on Patreon: / geerlingguy
Sponsor me on GitHub: github.com/sponsors/geerlingguy
Merch: www.redshirtjeff.com
Main Channel: / @jeffgeerling
2nd Channel: / @geerlingengineering
Contents:
00:00 - You built a WHAT?
00:45 - The hardware stack
02:26 - Power monitoring
03:06 - Getting Coral and Hailo to work
05:44 - 55 TOPS!
06:30 - Not so fast... 
Наука
Can't wait for a decade from now when they're packing 1024 TOPS into "entry-level" devices claiming "you definitely need all this power for current models"
1 BOPS?
640 TOPS ought to be good enough for anyone
@@heblushabusT = trillion, so B would be billion. Next step is Pflops for Petaflops
@@tuqe oh, right. bops sounded funny tho. so, POPS?
@@tuqeisn't T tera in this case?
It's nice to know that you could run multiple neural networks on independent NPUs! Like one for facial expressions, another for voice recognition, and another for text-to-speech!
seriously tho, 26 TOPS on that small thing is impressive by itself. combining a card with a few of those 26ers with something with PCIE-4x16 would make for impressive AI processing in a small package.
Yeah, Hailo makes a 'Century' card that does just that-hopefully as multi-NPU becomes more popular, programming for it also becomes more popular!
i would love a video on the home assistant power consumption!
+1 Yes, me too
Well that would be a 1 minute video 🤣
Oh thank god, I’ve been itching ever since you showed that b-roll
This reminds me of the Friends meme where Chandler gets a fancy expensive computer, and when asked what he is going to use it for he just says "Idk, games and stuff".
The infra is still fun even if you don't actually need the compute lol.
Never stop doing what you do Jeff. Love the content, love the experimentation.
My favorite part of this video is definitely when the box gets identified as a cell phone and he holds it up to his ear 😅
Jeff!!! The increase in content per week has been amazing. Don’t over due it but man I’m loving it
Bravo Jeff! That was alot of work on your part. Again, congrats and thanks for all the hard work you do for us!
🤗 Accelerate might be able to manage using all the NPUs at once. It's meant for using multiple GPUs for a single inference or training task but it might support NPUs too
That was fast
I was about to say that
My AI predicted it
that's what she said!
@@Level2Jeffu okey? u look a little red :p
I had no idea this channel existed, this is awesome!
and now you do, ha! this is the channel where things get crazy
It’s always fun to see you pushing the limits on single board computers. I’m sure it helps the hardware makers consider unusual use cases.
Jeff, have you seen the stories on the new research into IBD which suggests a genetic link and that certain cancer drugs might be effective? I saw the story on the BBC news site. Obviously I can’t post the link here, but I thought you might be interested.
Yes, I've been following that research-still 5+ years out from being practical but it's promising. The key is to find drugs that have fewer side effects than current TNF-blockers (which have similar mechanisms but target the immune system more broadly).
Love these kind of videos so keep em coming
Hey man! Wow. Saw this on Instagram and thought this would be on the main channel 😂. But awesome job man! This is so great 😂
OMG!!! Who are you going to be from now until the next video?!?! End of an era...
Ha!
Jeff the mad RPi guy! Love these builds when you push the envelope! 😎🤣
thanks for the video!
Here before 20k subscribers! Keep up the great work Jeff
We love you for your tinkering!
I would love to see a video going over your choice of power metering smart plugs & the integration into HomeAssistant.
I’m amazed by the things you are doing with the humble RPi.
Not as amazed as that fractal north case for the pi5 they are showing off at computex, but still amazed.
Jeff, I too would love to see a video on how to setup a Home Assistant Dashboard for Power Monitoring. You sir, are a Wizard!
ai models can run in pipeline mode (if you have multiple tasks) for example yolo for bounding boxing a face and a second model that does check your expression and a third that correlates your face with known faces, and so on. I doubt it is possible to run these llms on these tpus - except they somehow split their model into n models
hehe a pi mad scientist cobbling together contraptions no one thought of. love it! great scott!
I'd love to see a version of this with the CM5 when it comes out. I think it would be cool to build an ITX-ish size ARM AI computer that you could use for all kinds of AI projects.
Level 2 Jeff is truly on another level.
The second one!
I love the crazy/nuts thing that is this set up! And, even if the "Coral Dual TPU" only shows up as one of them, I'll bet that it is still cheaper and faster than the USB version. Now, where was that AM power meter that your Dad has . . . ? ;-)
Oh, man. That's a visually menace pi XD
Yesterday, you tried and SPAGHETTIBLY FAILED chaining NPU and now this. lol
Thank you for the attempts otherwise I would have tried myself. I think 2 NPUs can easily work with running 1 neural network on each NPU. This kind of configuration can realize many real world applications I have been dreaming of so many years. Thank you again.
Definitely! I think on the Pi 5 at least, that would probably be the ideal number of NPUs. You could stretch it to 4 okay too, but at that point the cost/build could point you to something a bit beefier like Jetson Orin.
Please make a power outlet video, I'm just getting into HASS now and power monitoring is next on my list - Also great video cheers!
"Why?" well to quote The Doctor from The Waters of Mars, "Fun"
I appreciate that there are so many different TPUs/NPUs on the market, I’m just frustrated that we’re all beholden to nVidia when it comes to actually training models and running a lot of things.
Ditto. Wish at least AMD could offer something that would take the bottom out on either price or efficiency, but right now it is what it is :(
@@Level2Jeff I’ve got a used Tesla P40 with a water cooler in addition to another rtx 2060. Now that Intel works with Tensorflow and PyTorch, I’ve seriously considered just getting a 16 gig Arc 770 and paying for cloud computing if I need to train a model that needs more memory.
300 TOPS sounds like a lot until you realize even an aging RTX 3080 has about 7400 TOPS. Sure the power use is different but point is you wont be running any big LLM models on this.
Is this really correct? I don't believe this.
@@pieterboots8566
I don't think that this is correct.
This would be useful to run Frigate AI detection from multiple cameras (e.g. PoE RTSP streams). Even this many devices is less power hungry than a 4080 to run 24x7, and compared to the cost of a good IP camera its not unreasonable.
"I've created a monster!
No one wants to see Marshall no more,
They want Jeff.
I'm like chopped liver"
Great video 😊 - slightly off topic but do you have any thoughts on using the overlay file system with raspberry and will it really protect the pi os from being turned off without shutting down? Would make a cool video perhaps to hook up a pi to a relay and have it turn on and off hundreds of times to see if and when the os corrupts.; have you done anything with the overlay / read only file system before?
Hi Jeff, just wanted to let you know that you CAN use multiple hailo cores together, using the VDevice API - it automatically identifies the cores (granted it will only work with hailo chips).
HailoRT seems to have some multi-core configuration too... definitely some fun to be had here!
Jeff, you have to write a book with a chapter on each of these types of Pi mods.
0:04 Just flappin' in the breeze
If you want 200 TOPS, just buy the Hailo-8 Century. It has 208 TOPS and is a single PCIE card. Although normal person probably can't buy it...
I'll just do it myself! Haha, so far I haven't found a reliable way to buy Hailo-8 outside of being an OEM partner. Hopefully they open up more individual sales at some point.
Being able to run two Halo8's would be a cool project as in theory would be (2x 26, or up to 54 TOPs in theory). Combined with dual Pi cameras, would offer high frame rate stereo depth of field, or other fun video processing.
BTW: the Home Assistant monitoring is fascinating. Would be interesting in hearing more details.
Hell yeah, finally you dont just tease😂 i was waiting soo badd, am i rpi addicted🤨
The trick to using those NPUs in parallel will be building a pipeline across them, where each NPU is supporting a subset of the layers in a model. This is a common technique in both training and inference, though I'm not sure if Tensorflow Lite supports it.
A pipeline would allow you to partition the model weights and compute across all of the NPUs, giving you a chance to run larger models then you could do on a single NPU. Your PCIe setup is very low bandwidth, but that is less of a concern here because pipeline parallel is only sending the activations (relatively small input tensors in inference) between the NPUs, not the larger weights.
Based on the limited information that I can find about Hailo and their sample hardware (a few of them have many Hailo chips on a single PCIe card) it looks like their software may support this.
How do GPU bit miners support the core splits? Some setups are looking at spreading a single calculation over thousands of cores. Maybe there's a way to port a miner task divider to NPU tasks. But I also wonder if we are still limited to running each TPUs individual Floating Point rating? Or could we run full 32-bit models?
think you might be able to look over the nanopi boards since most have built in NPU? I know the Nanopi R5S has one. Not that much of one but still there
As price drops for NPU's people will use more then software will write the software for it. That is what the raspberry pi was designed for learning in a fun way.
I really like your videos they always bring novel information
Do you know when it will support running NVMe SSD and Hailo AI Kit simultaneously? (with the NVMe SSD used as the system boot disk)
Nice!
So, other SBC with PCIE Gen3x4 can leverage the full power of all AI chips installed?
I wonder how hard it would be for a Pi to orchestrate all these TPUs together. Could it offload some orchestration onto another TPU? I kind of want to write some code for this thing to see how it performs with multiple processors.
@level2Jeff Where did you buy the Hailo-8 card? On their website they only provide an option for product enquiry, not for purchase.
Hey Jeff! I'm still pretty curious about everything "AI". I'm just not sure how to take advantage of any of it right now. I'd love to see a video going over a bunch of different AI projects that these can be used for, either here or on the main channel. Obviously frigate is one, I've also seen some self hosted AI chat bots, though I'm not sure how well any of them would fare on a pi.
I know you're the "pi" RUclipsr, but I'm also curious about other applications of such accelerators. I wonder if an AI chat bot would work decently well on a 1L PC (or some similar micro x86 system) using something like the Hailo for processing rather than trying to cram a GPU in a small system like that.
If you know of a RUclipsr who is doing that sort of thing, I'm happy to check them out, just let me know.
Keep up all the cool videos. Cheers!
Hey, thanks a lot for your videos. I'm wondering something that i'm sure has been adressed but i can't find a definitive answer on this. Does TPUs accelerate local LLM answer generations (Ollama for example) ? Thanks
Thanks for trying AND explaining the problems!
Can't wait for someone to write a "Kitchen Sink Aggregator" patch that'll normalize or make modules generically access -- ahh, I just wanted to use kitchen sink in a sentence.
I actually met the alftel guy and he is a GENIUS with RF stuff.
I don't doubt it. The contraptions he makes...
Might be interesting for scientific computing. Can you send instructions via MPI or something?
We need cheap PLX chips that take a newer standard bus in uplink and can provide many downlink at lower speed.
Also on consumer mainboard it is starting to be annoing. 20 or plus pcie gen 5 lanes where almost all hardware is gen 3 and 4.
Lot of throughput lost.
Couldn't agree more. Though those of us needing all that PCIe goodness are a slightly rare breed... and the answer till now is usually buy a big server CPU that gobbles up like 120W idle :D
@@Level2Jeff Even if we are the minority I think that the approcha that apple did with the MAC pro using the PLX and many lanes/slot is something that other HEDT/Workestantion OEM should start to consider.
New AM5 EPYC has been annoucned but lot of bandwidth is lost if you just plug a single HBA card on today mainboards x16 slot.
At first i was thinking the tribute was to Major Hardware, but the intonation and rhythm is completely wrong... it's closer to Explaining Computers, but the line doesn't match.
Ya got me on that one! @Level2Jeff !
I also recognized the tone as Explaining Computers, but the message doesn't match.
Check the description ;)
I've started to put the tribute into the bottom of the description to make it a little easier :)
@@Level2Jeff I skimmed and totally missed it! Guess I should've ^F-d instead! 😄
Can you show us actual runs? Also would love to see the 10H version when it comes to your hands.
Great video, does this also works with Pi4 ? Please also mention in future if the projects also works with Pi4 possible since not everybody has the latest and greatest :-)
It works with CM4, but not Pi 4 directly since the Pi 4 doesn't expose PCIe without a pretty serious hardware hack.
What's the best lowcost solution today for running local image-detection CNNs on a Pi4 or Pi5? Those USB-TPUs?
time to find some splitters for converting m.2 x4 into 4 m.2 x1 plugs for those dual-edge tpus
You need that 12xm.2 card with a 16x connector and put it in your ampere workstation.
can this be used for llm's and stable diffusion or is it only useful for video tracking.
one day graphics cards will be replaced by AI accelerator cards, and all you'll need is a low power gpu.
Yes, a dual m2 hat with a Boot and storage SSD, and that Hailo card for friggit, object detection , write to a db that is searchable (events) in plain text…😅
I wonder how feasible it is to put an abstraction layer in front of the 25 coral TPUs so that the software only sees one big TPU?
That's more-or-less what RAID drivers for ZFS and BTRFS do so clusters of 25+ drives appear as one block device. And what CPUs do to make scalar code think it's still being executed in-order as the only process on a single-threaded CPU in a virtual memory space, despite out-of-order execution, superscalar execution, hyperthreading, multithreading, branch prediction, and more, all happening in the background.
Question: Oobabooga (text-generation) Web GUI supports multiple GPUs and TensorFlow (TPU) with mixing + matching - would that work for your setup?
Do these NPUs / TPUs work with ollama and the llama3 model? Currenting running ollama and llama3 on my gaming PC which has a 4060ti.
Oh man… this makes me think of how great a CM5 board akin to the CM3588 NAS board from FriendlyElec but for NPUs would be. Wait… could you just use that board for NPUs as is?
I think the name is probably "Micro PC city" and the u is used as a substitute for mu because it wouldn't be recognizable at that size.
You choose to switch "geerlings" and it seems it paid off for the better :)
I know you won't tribute aVe, but, that would be hilarious!😂
I had trouble getting my coral to work with codeproject ai long-term. It would work for a few hours but then stop responding. I don't know where the issue was on that. I mostly just gave up on it.
I had a similar issue in my testing, though didn't take too much time to debug it. One time it seemed to lock up the frontend of the Pi, had to force-poweroff!
Hello Jeff. Odd question here. I have a Lenovo X230 that I will repurposing to "play" with local AI, specifically LLMs. It is going to struggle with the onboard GPU and I will eventually be moving everything to a different machine that will let me add a proper GPU card. Until then, is there a way that I could use a PCIE slot to NVME/M.2 adapter to put one of these little M.2 AI chips in the laptop? Since the machine has USB3 ports on it already, there isn't really anything useful I can put in the slot. If the adapter actually worked, would the extra AI board do me any good without custom software to utilize it?
Yes, at least under Linux. Not sure about Windows support for these things.
Interesting, but cool!
There a Qualcomm AI accelerator that use Dual M.2 slots, that board could be perfect for it (I've never been able to find the spec for dual m.2, so the spacing might be off), just one of those could get 200TOPS/25W
Good luck finding one though...
Great information in the video. Is there a PCI-e board like this that would work in a desktop, without buying the Nvidia RTX 40 series?
Hailo makes a 200+ TOPS 'Century' card that straps a bunch together and would fit inside a desktop case (full height PCIe card).
@@Level2Jeff Thank you so much. 🙂
Arent they all sharing memory bandwidth? Are there any AI m.2s with onboard memory?
so Google TPU might be the LPU arranged in certain way? doesn't seems so, because LPU have huge memory chips... well the tops seems matched I wonder where this experiment will go into this is really interesting. Really interesting! indeed!
Maybe call it “Zora” after the AI on the USS Discovery?
I wonder if Bend would work with this config ❤
What is the best combo for pi hat AI NPU and also running an NVMe SSD all together?
probably either the NVMe BASE duo from Pimoroni or the dual NVMe board from Pineboards right now.
Crap, I need one of those PCIe expansion boards for my NanoPC-T6 to be able to give it dual WiFi plus 5G Cellular and possibly a couple other nifty adapters. Dang it
Can you run LLMs on these NPUs?
Marco Reps at the end?
We have level 1 tech,level 2 jeff what’s next level 3 steve 😂
before the WHAT it's compelling to know the WHY.
waited for the payoff but I guess I don't get enough why I'd want to use all these TOPs
I'm excited about using new tech but using it is the key. The excitement about the build comes after the excitement about the functional power. So I guess I don't get this video too much.
Can these boards be plugged into a full-fledged PC?
@Level2Jeff - Power Monitor Dashboard all the things - YES PLEASE
but the real question is can it blend? what can one use this for?
So like, objectively I know that the Hailo isn't really meant for inference of production grade LLMs or anything.
...But like, I still want to see if a person couldn't do something silly like a bespoke MoE architecture with 128M active parameters and still get okay quality and speed.
Soooo does it like tell the time?
r u gonna do it w/ the halio?
It's cool to see that working _technically_, but in terms of inference, that's kinda the equivalent of having a 1000 CPU cores running on a single, shared stick of DDR4 memory. Do you hypothetically have enough cores to run a ton of applications at the same time? Yeah. But how much you'll be ably to do with the system will depend on the throughput of your memory. And it's the same with these little compute sticks. They rely on your Pi's memory so while you - on paper - have a lot of TOPS, running any task that operates on a reasonable amount of data or live streaming data will quickly saturate your onboard memory and pcie bus and prevent you from doing anything useful with multiple NPUs
Btw what's the SSD and WiFi card for?
first thing i did with chatgpt was to have it write the code to house a llm on a local raspberry system (with 4 units and 1 master)
maybe 3D print a support for it?
Very soon, we will have an external NPU device connect via usb 4 just like eGPU 🚀💻
Why do we need TPUs for object recognition? I was doing embedded recognition sotware for cameras that run basically on a Pentium 2 200Mhz hardware just fine. Of course classification and training of the algorithm was harder than just AI traning, but it run on basically anything. Today we need a special chip to draw a rectangle around you on the camera, while at the same time the ARM cpu can easily do that. Is it because the algorithms are easier to use and not proprietary?
You can do that on the Pi's CPU too-in fact it's nearly as fast for some tasks as the Coral itself. But the AI accelerator chips allow you to process hundreds+ items at a time at 30+ fps, which can be helpful in many places. Or just a few items or a couple camera feeds at 100+ fps, which is useful for high speed robotics, sorting, etc.
Not to mention it frees up the CPU to do other tasks with the data sorted by the NPU. Still, it's overkill for a lot of uses!
You don't need to make software for them to all work together just create something in python and back off it. you know what have you using the python multithreading library to trick it into multithreading ?