Thanks for the coverage, I'd be interested in a tool use / RAG and other utilities comparison with Llama 3.1 8B quantized aggressively to bridge the gap in RAM and performance!
the MoE wasn't wrong, the correct answer for that calculation was exactly 9.9996, rounding _is_ the next step. So I'd say it did better at that specific question..
Unfortunately every Phi model I tested so far had a model collapse after 3 to 5 queries. I have this only with Microsoft models OR models I truncated on my own. I do not understand the hype and do not trust the benchmarks. Just to make clear: I have about 15 different official models running locally that were not tampered with and NONE except the Microsoft models have this issue.
It's funny. Every time a new Phi model comes out I get so insanely bearish for LLMs because they always suck. Just gaming the benchmark but are horrendous to use.
Does anyone know of a source for community/conversation on LLMs and business? I'm a technologist developing an app and would really like to find a good source for discussing ideas and what's working/not working.
How much longer are we going to pretend that these are in any way practical? No on prem running for anyone except large corp and many of the privacy issues open source was supposed to address arise come back once you start using someone else's hardware. Guess Its great to see smaller models improve and push foundation models, but if you want to do stuff with any off these, especially with agentic processes gobbling thousands of tokens, latency and performance demand hosted service.... might as well go free flash, mini with no setup or hosting issues.
Well, you actually can run a crew of Phi models on a MacBook Pro. The M3 Pro with 36 GB of system memory, can allocate around 27 GB of that pool solely to GPUs for inference.
@@pwinowski Its not about can/cant. What is the tokens/sec doing that locally? Now consider hitting the gemini-flash API with 128k tokens 15 times a minute for free.
Thanks Sam! You always have good content in a sea of clickbait nonsense :)
Thanks this is what I am trying to go for. This who space has gotten sop hype focused over the past couple of years.
Thanks for the coverage, I'd be interested in a tool use / RAG and other utilities comparison with Llama 3.1 8B quantized aggressively to bridge the gap in RAM and performance!
the MoE wasn't wrong, the correct answer for that calculation was exactly 9.9996, rounding _is_ the next step. So I'd say it did better at that specific question..
Unfortunately every Phi model I tested so far had a model collapse after 3 to 5 queries. I have this only with Microsoft models OR models I truncated on my own. I do not understand the hype and do not trust the benchmarks. Just to make clear: I have about 15 different official models running locally that were not tampered with and NONE except the Microsoft models have this issue.
Phi 3.5 is mindblowing. Works crazy fast and accurate for function calling, and json answers also.
Which version, what functions?
Is it faster than Llama-3.1-8b-Instruct float16 for json response? Also which model, mini, right?
What are some different use cases for Mini and MoE? For example if you want to do a RAG application, which would be more suitable?
Always top notch content Sam!
Nice overview!
Surprisingly good. Better than v3. But still get's stuck in loops as the response context length grows. Experimenting with prompts to avoid this.
🔥 🔥 🔥
Is there any cheap way to finetune these small models with proprietary data?
yeah you can do FTs with Unsloth etc quite easily for these.
It's funny. Every time a new Phi model comes out I get so insanely bearish for LLMs because they always suck. Just gaming the benchmark but are horrendous to use.
100% agreed, just ask a slightly different question and Phil goes NUTS
This is what I noticed too. Went crazy on the 2nd time. There was no 3rd. Maybe newer bigger ones would work. Probably will need to fine-tune.
This kind of models are like gold for people working with NLP.
😂
Can I ask what you are using it for that you are finding it sux. Curious is it a chat kind of app etc?
Does anyone know of a source for community/conversation on LLMs and business? I'm a technologist developing an app and would really like to find a good source for discussing ideas and what's working/not working.
Definitely first
o fucks given.
How much longer are we going to pretend that these are in any way practical? No on prem running for anyone except large corp and many of the privacy issues open source was supposed to address arise come back once you start using someone else's hardware. Guess Its great to see smaller models improve and push foundation models, but if you want to do stuff with any off these, especially with agentic processes gobbling thousands of tokens, latency and performance demand hosted service.... might as well go free flash, mini with no setup or hosting issues.
Well, you actually can run a crew of Phi models on a MacBook Pro. The M3 Pro with 36 GB of system memory, can allocate around 27 GB of that pool solely to GPUs for inference.
@@pwinowski Its not about can/cant. What is the tokens/sec doing that locally? Now consider hitting the gemini-flash API with 128k tokens 15 times a minute for free.