It would be best to have alternatives to all these which are free and open source. Maybe later down the line.. The video is really cool tho! Thanks Matthew
Matthew - for those of us who develop line-of-business apps for SME businesses - local LLM deployment is a must. Would certainly like to see you demo RouteLLM with orchestration - Thanks!
I am a cyber security analyst who knows very little about coding so, between your videos and just straight asking ChatGPT or Claude, I am ham-fisting my way through getting AI to run locally. Please keep making tutorial videos - I am excited to see how to impliment RouteLLM!
I agree, I'm pretty much in the same boat as you. The problem is that my knowledge is outdated by the time I finally figure things out because there is so much advancement in so little time. I think we need a "checkpoint" how-to on how to do things now, as opposed to 3 months ago.
If you don't know much about anything, like me, but want to run LLMs locally, you just need to install LM Studio. No need to understand anything. On the software, it has even the option to download and install them, and run. That's what I use. Now that I learned a bit more, I will try to install Open WebUI, Ollama and Docker, these are way more complicated. 🎉❤
Yes! Please show us a comprehensive breakdown of this great tool! I’m also interested in your sponsor’s product, LangTrace. Can you possibly show us how to use it?
I feel like everyone is realising things at the same time. I started 2 projects, the first an LLM co-ordination system and a chain of thought processing on specific models
Great breakdown, much appreciated. I definitely foresee local LLMs becoming dominant for organisations as soon as next year. My advice during consults is for them not to invest a massive amount in high-end data secure cloud systems, but just to hang on a little, work with dummy data on current models to build up foundational knowledge, and then once local options exist they can start diving into more sensitive analytics.
Hey Matt, yes it would be great if you could show a demo of how to setup this model on Azure OpenAI or Azure Databrix and then use it in the application.
Just popping up to say thanks Matthew. You have become almost my only required source for AI news because your take is right up my street every time. Great work, keep it coming
Thanks for this video. Very informative. Please make a full tutorial about the setup of route llm and what the recommendations of the local pc should be. Thank you in advance!
There seems to be a hold up on the highest end models as the leading companies continually try to improve safety while watching their competition. Nobody seems to want to jump in and release a new/better model at risk of the potential "dangerous" label being applied to them. So a lot of the progress remains hidden in the lab, waiting for competition to finally engage.
It's time people admit that benchmarking off GPT4 is stupid. When GPT4 came out it was amazing. Now its no better than any other LLM. Ever since OpenAI introduced cheaper Turbo models, the quality has gone down hill. They sacrificed intelligence for speed to the point where they have plateaued in quality and its not getting better no matter how new models they release.
I agree man I don't care about speed as much as I care about accuracy. I'll happily wait for a better response than rapidly go through 2-3 quick responses that need more time in the oven.
I think you misrepresented the graph. The "ideal router" point on the graph is likely just that - the ideal. I don't think that's claiming actual results
I'd be interested in what hardware is needed to run something like this locally. I was waiting until late fall or early next year to buy, but I might need to get an intern system to train up. I am big on local control except when needed to reach out.
First thing I did a year and a half ago was routing different LLMs via a zero-shot classifier. Looks like Route has done the same thing lol. I figured it was common sense.
This sounds interesting, but does it offer the same capability that the OpenAI API offers with customizable assistants, RAG, and function calling? I still have yet to find anything that compares. Would love to see something open source that can do this.
Yes please... Full tutorial on setting this up to run locally. Also, I'd like to know how to setup multi-modal so I can show my images and casually talk to it (local).
Hi Mathew, I love your channel. I’m curious if you would be willing to explore Pi ai? It doesn’t compare to the others in the same way. Maybe it’s hard to test. But very interesting. It’s trained to be empathetic and you can actually have a conversation with voice that feels satisfying.
After developing exclusively on GPT models, then joining an org with a ridiculous amount of free GCP credits and being pushed to use Gemini family instead- I can honestly say that while differences on benchmarks may seem small, they end up being really extreme in practice. I spent days smashing my head against a wall trying to get Gemini to provide quality responses, and after switching to 4o, I was literally ready to deploy. There still don't seem to be great benchmarks that represent performance of generative models well.
This wasn’t just released. It had been around for a while. Now that GPT-4o and Claude 3.5 Sonnet exist things are much cheaper. I can understand using a local LLM with these two but overall the cost savings are not as big of a deal as before.
I think they are saying the brown dot is where an ideal LLM would be placed, I'm not sure that Route LLM is better than Claude 3 Opus. SO not sure where on that chart their router actually is. probably down with Llama 3 8b. Cause it's only job its to route.
Could you do a tutorial on RAG (retrieval augmented generation) ? I think I'll be pretty massive thing in agentic archetecure. Also I think RAG might soon be more than just text and PDFs 😂 in the not too distant future.
I suspect these features, plus dedicated non-GPU hardware will eventually reduce energy costs per "thought" to less than the human brain. Currently perplexity using Sonnet 3.5 thinks GPT4 uses 25x more.
Problem with this method is that there are some trade offs. While it maybe cheaper at answering a question directly; you sacrifice its social intelligence. Even though you get the right answer, the way the answer is phrased can be the difference between either a toddler or a graduate student. Personally I wauld want to talk with the graduate student.
Promising, but a bit mess with naming. They are using GPT-4 to mean at least GPT-4 Turbo and GPT-4 Omni in various places. I am not even sure if on some place they don't really mean the older model GPT-4.
It doesn’t change anything. LLMs are good at certain tasks (most of which aren’t as useful as we need, and most don’t help us earn money). AI has plateaued. They haven’t replaced software engineers.
While this looks promising, it is just a router that forwards simple queries to weak models while forwarding hard queries to strong models. This assumes that the queries can be divided between strong and weak models. If your work is truly intensive, I don't see much reduction here as it still requires querying strong models most of the time.
I really don't find this to be a big deal. I expect people select the model to use themselves on a per-task basis on what they believe is the most appropriate one for the task. For me the decision process is really simple: 1. is it code or requires complex problem-solving? -> Claude 3.5 Sonnet 2. Do I want to have a deep conversation with a creative partner -> Claude 3 Opus 3. Is it anything the other models would refuse? -> GPT-4o 4. Is it too private for any of the above? -> Local LLM I don't need a router for this and I wouldn't trust it to reliably choose the same way I would either.
My "AI Stack" is RouteLLM, MoA, and CrewAI. What about you?
AISheldon 🤓
It would be best to have alternatives to all these which are free and open source. Maybe later down the line.. The video is really cool tho! Thanks Matthew
Is MoA mixture of agents?
Mathew, please do the full tutorial on how to set this up. thank you
RouteLLM, MoA, and Agency Swarm. Should do a video on Agency Swarm. I think it is the best agentic framework
Would love to see a tutorial on how to set this up.
Me too ❤
I think some other AI enthusiast covered it a few days back. It was quite easy. Check RUclips.
get the agents to watch it and do it.
On how to set this up IN THE CLOUD as well or preferebly
Me too but on a local machine instead of a third party service.
it'd be cool if you did a vid on setting it up and running it locally
Olama or LLM studio?
Matthew - for those of us who develop line-of-business apps for SME businesses - local LLM deployment is a must. Would certainly like to see you demo RouteLLM with orchestration - Thanks!
Please do a tutorial for local installation for this. Thanks
What exactly? As I understand, RouteLLM is not an LLM itself but just a router.
You can install local LLMs very easily using Backyard AI.
Sign me up for the full tutorial!
@@camelCased Or LM Studio
@@camelCased But how to use the router with Backyard?
@@bigglyguy8429 Why would you want to use the router at all, if running LLM models locally?
I am a cyber security analyst who knows very little about coding so, between your videos and just straight asking ChatGPT or Claude, I am ham-fisting my way through getting AI to run locally. Please keep making tutorial videos - I am excited to see how to impliment RouteLLM!
I agree, I'm pretty much in the same boat as you. The problem is that my knowledge is outdated by the time I finally figure things out because there is so much advancement in so little time. I think we need a "checkpoint" how-to on how to do things now, as opposed to 3 months ago.
If you don't know much about anything, like me, but want to run LLMs locally, you just need to install LM Studio. No need to understand anything. On the software, it has even the option to download and install them, and run. That's what I use. Now that I learned a bit more, I will try to install Open WebUI, Ollama and Docker, these are way more complicated. 🎉❤
Yes! Please show us a comprehensive breakdown of this great tool!
I’m also interested in your sponsor’s product, LangTrace. Can you possibly show us how to use it?
Absolutely do a detailed tutorial on how to get this up and running!
I feel like everyone is realising things at the same time. I started 2 projects, the first an LLM co-ordination system and a chain of thought processing on specific models
Great breakdown, much appreciated. I definitely foresee local LLMs becoming dominant for organisations as soon as next year. My advice during consults is for them not to invest a massive amount in high-end data secure cloud systems, but just to hang on a little, work with dummy data on current models to build up foundational knowledge, and then once local options exist they can start diving into more sensitive analytics.
Yes, please provide a tutorial on setting up the described language model.
I also saw where they will have 20TB m.2 drives in a couple of years. Running this LLM locally will be really cool.
Hey Matt, yes it would be great if you could show a demo of how to setup this model on Azure OpenAI or Azure Databrix and then use it in the application.
Just popping up to say thanks Matthew. You have become almost my only required source for AI news because your take is right up my street every time. Great work, keep it coming
Thanks for this video. Very informative.
Please make a full tutorial about the setup of route llm and what the recommendations of the local pc should be. Thank you in advance!
There seems to be a hold up on the highest end models as the leading companies continually try to improve safety while watching their competition. Nobody seems to want to jump in and release a new/better model at risk of the potential "dangerous" label being applied to them. So a lot of the progress remains hidden in the lab, waiting for competition to finally engage.
Improve safety really means neuter.
So far with LLM's "data is code" ... it is inherently unsafe, unless something fundamentally changes
Yes! The tutorial. Great vid. Sharing with my crew...Just beginning an AI Consultant agency and cost is an existential threat!!!
How would this effect mixture of agents? Could we have multiple route llms combined together since they use such lower compute?
It would be interesting to see how this will work on your AI benchmark. Please do a setup and test
It's time people admit that benchmarking off GPT4 is stupid. When GPT4 came out it was amazing. Now its no better than any other LLM. Ever since OpenAI introduced cheaper Turbo models, the quality has gone down hill. They sacrificed intelligence for speed to the point where they have plateaued in quality and its not getting better no matter how new models they release.
Thanks for being real bro. I absolutely agree with you. I barely even use ChatGPT anymore because it sucks.
"Now its no better than any other LLM" -- do you really believe this? Seems like you do. That's certainly a take.
I agree man I don't care about speed as much as I care about accuracy. I'll happily wait for a better response than rapidly go through 2-3 quick responses that need more time in the oven.
Yes! please we need a full tutorial!
Yes I really like your idea of a complete install and running tutorial. Go for it. 🙏 Thanks 👍
Would love a local setup tutorial! Thanks 😊
Yes - please do a full tutorial on setup and use. Thanks.
can this be used to route between agents as opposed to model instances? for example routing to chain of thought agent vs simple q and a agent?
So, how to run it. And on what hardware?
Lmgtfy
Would love to see a tutorial on setting up RouteLLM! 🙂
Yes please full set up!
The only channel where i actually watch ads. Very interesting stuff
I would love to catch a tutorial of you setting it up!
I think you misrepresented the graph. The "ideal router" point on the graph is likely just that - the ideal. I don't think that's claiming actual results
I'd be interested in what hardware is needed to run something like this locally. I was waiting until late fall or early next year to buy, but I might need to get an intern system to train up. I am big on local control except when needed to reach out.
First thing I did a year and a half ago was routing different LLMs via a zero-shot classifier. Looks like Route has done the same thing lol. I figured it was common sense.
We need something locally for non difficult pourpouses. For example local home Assitant control.
Yes do make a whole tutorial on it
Please do a tutorial. And need to measure performance to validate the performance - cost graph.
Hey Matt, what are these routers exactly? (They are not LLM I understand) And how do they determine where to route to?
This sounds interesting, but does it offer the same capability that the OpenAI API offers with customizable assistants, RAG, and function calling? I still have yet to find anything that compares. Would love to see something open source that can do this.
Ive seen you do techdev before and would love to see you do this one as well please.
The article used GPT 4, not GPT4-O, which is already 50% of GPT4 cost. Or am I missing something?
Danke! As always great!🎯👍
For coding this would be insane. Mixed local and api endpoints
That's good stuff. I will take a look at the codebase. Thanks for sharing
Yes please... Full tutorial on setting this up to run locally. Also, I'd like to know how to setup multi-modal so I can show my images and casually talk to it (local).
You are so consistent bro, keep it up 🙌
Always yes for full tutorial
I’d love to see a demo if possible.
Hi Mathew,
I love your channel. I’m curious if you would be willing to explore Pi ai? It doesn’t compare to the others in the same way. Maybe it’s hard to test. But very interesting. It’s trained to be empathetic and you can actually have a conversation with voice that feels satisfying.
Hey Matt, please do a tutorial on setting this up.
You should consider including Semantic Kernel and GraphRAG in that ideal stack
I’d love a guide to easily set this up for myself
Yes please make a tutorial on the routerLLM.
It would be really cool to have a tutorial on how to implement this with ollama
I would love to see a tutorial on how to get this running locally.
Yes, give a walkthrough video for RouteLLM
Please do a demonstration
Does your local model search the internet and summarize a few web pages? That's what chatgpt does for me, and that's all I need.
Would also love to see the tutorial :)
After developing exclusively on GPT models, then joining an org with a ridiculous amount of free GCP credits and being pushed to use Gemini family instead- I can honestly say that while differences on benchmarks may seem small, they end up being really extreme in practice. I spent days smashing my head against a wall trying to get Gemini to provide quality responses, and after switching to 4o, I was literally ready to deploy.
There still don't seem to be great benchmarks that represent performance of generative models well.
This wasn’t just released. It had been around for a while. Now that GPT-4o and Claude 3.5 Sonnet exist things are much cheaper. I can understand using a local LLM with these two but overall the cost savings are not as big of a deal as before.
API for claude and GPT is sitll expensive.
I think they are saying the brown dot is where an ideal LLM would be placed, I'm not sure that Route LLM is better than Claude 3 Opus. SO not sure where on that chart their router actually is. probably down with Llama 3 8b. Cause it's only job its to route.
Could you do a tutorial on RAG (retrieval augmented generation) ? I think I'll be pretty massive thing in agentic archetecure. Also I think RAG might soon be more than just text and PDFs 😂 in the not too distant future.
wonderfull
know you can offer a low cost service and a primun service at diferent prices
An implementation would be interesting
I suspect these features, plus dedicated non-GPU hardware will eventually reduce energy costs per "thought" to less than the human brain. Currently perplexity using Sonnet 3.5 thinks GPT4 uses 25x more.
Reading and watching anything about AI is like a live broadcast of the Manhattan Project in 1942. The current year is 1944?
Can you please look into alloy voice assistant
Problem with this method is that there are some trade offs. While it maybe cheaper at answering a question directly; you sacrifice its social intelligence. Even though you get the right answer, the way the answer is phrased can be the difference between either a toddler or a graduate student. Personally I wauld want to talk with the graduate student.
Every single time I’ve seen a claim like this, the output in real usage never compares
Is this an LLM that we can use in LM Studio?
its just a proxy to send queries to two models, weak vs strong. It's not a new LLM.
yes we need detailed video
A tutorial would be great!
Promising, but a bit mess with naming. They are using GPT-4 to mean at least GPT-4 Turbo and GPT-4 Omni in various places. I am not even sure if on some place they don't really mean the older model GPT-4.
Claude 2.5 Sonnet missing from the chart.
Yes please, do the tutorial.
Let me launch this... Online models are saving themselves yet because of context.
yes I vote for tutorial for set up please thank you
It doesn’t change anything. LLMs are good at certain tasks (most of which aren’t as useful as we need, and most don’t help us earn money). AI has plateaued. They haven’t replaced software engineers.
Do you and Wes Roth collaborate? They uploaded an identically titled video 2 hours ago.
No. He just copied me exactly...again
Yes Please. a tutorial!
When "data = code", how can you have security while having a actually useful / powerful AI?
If runs on local machine, why needs OPENAI_API_KEY ?
because it still needs to query weak models (like mistral) and strong models like GPT
Please do a tutorial on setti g up
95% of my queries, even GPT 4o or Sonnet 3.5 can't answer so I don't know what your queries are that local models usually handle fine.
Yessss! Go open source
Full tutorial pls
Yes, please.🙏
how??
Please do a tutorial !
While this looks promising, it is just a router that forwards simple queries to weak models while forwarding hard queries to strong models. This assumes that the queries can be divided between strong and weak models. If your work is truly intensive, I don't see much reduction here as it still requires querying strong models most of the time.
I really don't find this to be a big deal. I expect people select the model to use themselves on a per-task basis on what they believe is the most appropriate one for the task. For me the decision process is really simple:
1. is it code or requires complex problem-solving? -> Claude 3.5 Sonnet
2. Do I want to have a deep conversation with a creative partner -> Claude 3 Opus
3. Is it anything the other models would refuse? -> GPT-4o
4. Is it too private for any of the above? -> Local LLM
I don't need a router for this and I wouldn't trust it to reliably choose the same way I would either.
don't ask in the comments each time JUST DO IT !!!
4:59 you lost me completely with langtrace! What does it do in why would I want it?
Now, who has this in production?
@7:07, "causal" not "casual". #FeedTheAlgorithm
If I could like and subscribe harder I would. Tutorial would be fantastic thanks 👍🏿
90% GPT4o Quality? More like 100% snake oil! Where do I sign up for this "RouteLLM" deal?
Can you make a comparison video with LangGraph X GraphRAG?
do a tutorial on this please
That’s probably what Apple does with Apple Intelligence and Private Cloud Compute.