Best interview Sridhar has done yet. Could tighten up on his messaging for Snowflake, AI, it's differentiation, and use cases but super fun to listen to and educational.
Can't you just go to HuggingFace's LLM Benchmarks and see the accuracy? Pretty sure there are plenty of LLM benchmarks for off the shelf solutions that you can view and the accuracy is not that great for a lot of them.
Not sure this is for sure what he's referencing, but I found a study from January showing 45% fully correct answers for finely tuned GPT-4 and 61% fully correct for finely tuned GPT-4 +RAG. Those not finely tuned were 36% and 60%, respectively. MSFT authors. "RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture". Could very well be a different study he's thinking about, but it's a lot more concrete than "trust me bro".
Best interview Sridhar has done yet. Could tighten up on his messaging for Snowflake, AI, it's differentiation, and use cases but super fun to listen to and educational.
Lovely interview
where is the source for the stat 45% for out-of-the-box solutions using off the shelf LLMs?
its trust me bro
Can't you just go to HuggingFace's LLM Benchmarks and see the accuracy? Pretty sure there are plenty of LLM benchmarks for off the shelf solutions that you can view and the accuracy is not that great for a lot of them.
@@siddani09 fr
Not sure this is for sure what he's referencing, but I found a study from January showing 45% fully correct answers for finely tuned GPT-4 and 61% fully correct for finely tuned GPT-4 +RAG. Those not finely tuned were 36% and 60%, respectively.
MSFT authors. "RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture". Could very well be a different study he's thinking about, but it's a lot more concrete than "trust me bro".