Is AutoGen just HYPE? Why I would not use AUTOGEN in a REAL use case, Yet
HTML-код
- Опубликовано: 14 май 2024
- In this video, we delve into AutoGen and learn about its functioning while distinguishing between the hype and reality. Although AutoGen is an excellent resource for creating multi-agent workflows, it is still unsuitable for production. Watch this video if you want to know what AutoGen can do beyond the hype.
Read the blog post that complements this RUclips content: / autogen-isnt-practical...
Need to develop some AI? Let's chat: www.brainqub3.com/book-online
Register your interest in the AI Engineering Take-off course: • Building Chatbots with...
Hands-on project (build a basic RAG app): www.educative.io/projects/bui...
Stay updated on AI, Data Science, and Large Language Models by following me on Medium: / johnadeojo
Multi-hop questions paper: arxiv.org/pdf/2108.00573.pdf
Chapters
Intro: 00:00
Multi-hop questions: 01:43
Multi-agent architecture: 05:02
Testing the AutoGen workflow: 11:20
Other AutoGen limitations: 33:10 
Наука
Very refreshing after a year of “shocking shock is changing everything” type of videos
I was waiting for someone to do exactly this video! Well explained. Clear. Thanks!!
Thanks for keeping it real.
I would say most of AI projects are far from being production ready. We are judging them by our established standards for regular non-AI stuff. After some time all these ai apps will get better, and we as an early adopters will have advantage of the experience of using them so we will utilize them faster than others. So don’t worry! We are on the right track.
I agree. I think the other part to consider is that LLMs are getting more powerful, we might get to a stage where an LLM could be powerful enough to make these agent frameworks viable.
Subbed. This is great. Thank you
Thanks for the sub!
I love your content. Keep up the excellent work.
Glad you enjoy it!
@@Data-Centric3 second only listening to you, i knows you are very smart.
This is why i will finish the 38 minutes left.
Excellent content. Thks you
Excellent analysis! 👍 Have you had a chance to analyze CrewAI or TaskWeaver in comparison by chance?
Thank you! Haven't had a chance to yet, they're on my list. I suspect there might be similar issues...
Yeh, I've had very much the same experience with AutoGen and AutoGen Studio. It's very inconsistent to the point where I feel like I'm herding cats even for fairly simple workflows. Overall, at least right now, I feel its much better just to define your workflow using the raw python code and calling the LLM directly. AutoGen just feels like an unnecessary middleman that just gets in the way.
Yes, I agree! And herding cats is a great analogy. I had my fingers crossed the entire time hoping AutoGen would deliver the response I wanted.
Thanks for sharing.
I still don’t see why would we use Autogen when semantic kernel is there and has support for assistants.
What can autogen do in addition which makes it a better framework?
I haven't used semantic kernel before so it's impossible to say.
could i ask,
what if we add multiple agents using multiple local LLM that are trained on different setup, by doing so i imagine that if each agent supervise the other and encourage it for better context and reasoning, perhaps the outcome may be more beneficial?
what is your opinion
I find that mixing LLMs doesn't work too well. In a GroupChat, AutoGen shares the entire conversation context with every agent. When you're working with different LLMs (for example Mistral or Llama), the prompt formats play an important role in getting coherent responses. So mixing LLMs means mixing up prompt formats (given the shared context), which will almost certainly lead to unfavorable outcomes. On your point on reasoning, I'm not sure that we have solved the reasoning problem with any open-source LLM yet either.
@@Data-Centric thank you for sharing your insight, your videos are very interesting, eager to see whats next idea you will be sharing with us.
You had better results than most youtubers from a multi-agent framework. Most i've seen got purely hallucinated results and didn't realize it. I am curious what you are considering as an alternative to multi agent frameworks, if anything.
I think that when models become better at planning and reasoning, we might not even need multiple agents. As for now, I anticipate the problems I mentioned will be there with any of the current frameworks.
Thoughts on Langchain/Langraph+Agents or Open Interpreter
On my list to look at!
Do you have this on a github? I would look to look through your code and see how you organized this project with chainlit
I'll be releasing the code as a part of an online course.
How does autogen compare to CrewAI? Or any other potential tools?
I haven't tested CrewAI yet; I might do something on Crew shortly.
Is this necessarily an issue with AutoGen per se? It seems like it may be more of an issue with the underlying LLMs that drive it.
I think so; ultimately, they are unable to follow the reasoning traces. The entire conversation context is shared across each LLM in the agent workflow. So, as you can imagine, the context gets long after a while. Maybe there is some "forgetting" of the context in for more complex workflows. Anyhow, I agree it might be an LLM issue.
So gpt-4 turbo isnt' smart enough to be a good Group Chat Manager. How about the other roles? Perhaps you can keep the Chat Manager , but also add (hard-coded) logic to go back to the planner if the answer wasn't achieved (limiting total number of calls to avoid spending too much money)? It may still fail, but you'll see what the other weaknesses of this setup are.
It would also be interesting to see some more realistic multi-hop questions, e.g., "'What events in the US corporate history resemble November 2023 OpenAI drama?" etc.
I think I agree with your logic, you need to hardcode the agent execution order. But essentially you're just building your own multi-agent framework from scratch, so just ditch AutoGen entirely. The other issues with costs, memory management, etc., would remain, I suspect.
Excellent video. I've have begun to perceive that these frameworks are being engineered to boost token consumption by model-as-a-service providers. It's crucial to recall that token utilization constitutes a significant portion of revenue for these companies. Unfortunately, we often find ourselves spending dollars just to assess the functionality and viability of these frameworks.
Thank you. I'll be honest, I'm not sure OpenAI or any proprietary LLM service has much to do with these frameworks. However, I wouldn't be surprised if there was some collaboration. I don't think it makes too much sense to purposely create frameworks that are token intensive and therefore impractical in production.
Microsoft is behind AutoGen …. It is in their best interest to ensure it first addresses OpenAI ‘stuff’ first. Creating autogen was probably on a strategic initiative document somewhere at MS.
You can run a local. The framework supports it.
🙋♂️🙋🙋♀️Can read sheet, can you zoom in please 😮
Otherwise why show the code at all 🤷♀️
Just run them locally on a Macbook Pro Max.....
Have you had more or less success with other FWs?
Doing one on CrewAI
@@Data-Centric I subscribed and will look for it. Thanks!