Great video, thanks alot for sharing your experiences. I've been working on increasing the accuracy of a company's RAG systems for about a year now and your insights are spot on. It still amuses me that we can kinda 'program' these LLMs using our own language, like it's more semantic than logic in a way. I also have been using automated testing to grade and point error in responses. Funny how these things can give a wrong answer to a question and then, when given the same question + answer, point precisely what was wrong with the answer.. I'd say that around 80% of my accuracy chasing endeavours were based on tweeking the system prompt to isolate possible contexts. The other 20% were documents/business related. Cheers
Hey man. I'm in a very similar boat as you. I've been leading the developing of a RAG system, and the newest one which is an IA product sales agent. Our problem with our RAG is that it's not specific to any niche or industry, we make it available for anyone wanting to upload their docs. Our problem was 50% docs, 20% prompting, and 30% the embeding/vector search techniques. We end up today using 2 different embedding llm's and using the k-5 of both of them, this improved accuracy greatly. Some of our clients have lists of 100+ itens (like a long manual) and they expect the AI the give out the whole 100 items in one answer, so we hack a few of them to include the entire doc instead of just the chunks. Since clients have different use cases, the quality of the response is often not the expected output the client would want to. The product sales agent tho, probably because it have very specific goals and instructions, performs immensily better in all sorts of manner. It collects information, profiles the buyer, and makes sensible suggestion according to pretty much anything the buyer inputs. I am about to start the attempts on loopbacks to grade and point error in responses. I assume these can really help with the RAG.
Yeah as bad as hallucination is I often put in stories short of 1000 words and am surprised by how well it understands the story. Like "wow you really can extrapolate / infer a lot of accurate details based on what I tell you". I'm also learning software development at the moment and I'm tested on my ability to accurately explain software concepts whether I'm talking about database constraints or Ruby features. The better I've gotten with my explanations, the easier it is to "boss" a model around to get exactly what I want. Thanks for the videos. They're always clear and of top-notch quality.
First of all thank you for sharing this experiences. Now, I still have a main concern for this LLM's techniques: isn't this a lot of time and effort just to acchieve some confidence (but not 100% confidence) that the LLM is working? I still think that by the time I finish this tune-up process I could've search and solve the original problem I had by my own and completly bypass the usage of the LLM.
If possible, can you please make a beginner video to learn these concepts like LLM, transformers, fine tuning etc. That would be really helpful. Thanks!
![Gucci Mane & Sexyy Red - You Don't Love Me [Official Music Video]](http://i.ytimg.com/vi/j7IBqyFKT4c/mqdefault.jpg)
Steal more of my secrets: www.builder.io/blog/make-ai-suck-less
Great video, thanks alot for sharing your experiences.
I've been working on increasing the accuracy of a company's RAG systems for about a year now and your insights are spot on.
It still amuses me that we can kinda 'program' these LLMs using our own language, like it's more semantic than logic in a way.
I also have been using automated testing to grade and point error in responses. Funny how these things can give a wrong answer to a question and then, when given the same question + answer, point precisely what was wrong with the answer..
I'd say that around 80% of my accuracy chasing endeavours were based on tweeking the system prompt to isolate possible contexts. The other 20% were documents/business related.
Cheers
Hey man. I'm in a very similar boat as you.
I've been leading the developing of a RAG system, and the newest one which is an IA product sales agent.
Our problem with our RAG is that it's not specific to any niche or industry, we make it available for anyone wanting to upload their docs. Our problem was 50% docs, 20% prompting, and 30% the embeding/vector search techniques.
We end up today using 2 different embedding llm's and using the k-5 of both of them, this improved accuracy greatly.
Some of our clients have lists of 100+ itens (like a long manual) and they expect the AI the give out the whole 100 items in one answer, so we hack a few of them to include the entire doc instead of just the chunks.
Since clients have different use cases, the quality of the response is often not the expected output the client would want to.
The product sales agent tho, probably because it have very specific goals and instructions, performs immensily better in all sorts of manner. It collects information, profiles the buyer, and makes sensible suggestion according to pretty much anything the buyer inputs.
I am about to start the attempts on loopbacks to grade and point error in responses. I assume these can really help with the RAG.
Yeah as bad as hallucination is I often put in stories short of 1000 words and am surprised by how well it understands the story. Like "wow you really can extrapolate / infer a lot of accurate details based on what I tell you".
I'm also learning software development at the moment and I'm tested on my ability to accurately explain software concepts whether I'm talking about database constraints or Ruby features. The better I've gotten with my explanations, the easier it is to "boss" a model around to get exactly what I want.
Thanks for the videos. They're always clear and of top-notch quality.
Wonderful video. I would love to see more videos on constraining LLMs to get better outputs
First of all thank you for sharing this experiences. Now, I still have a main concern for this LLM's techniques: isn't this a lot of time and effort just to acchieve some confidence (but not 100% confidence) that the LLM is working? I still think that by the time I finish this tune-up process I could've search and solve the original problem I had by my own and completly bypass the usage of the LLM.
Thank you for this video bro! Would love to see a more in depth video from you on how to train my own model.
got u fam ruclips.net/video/fCUkvL0mbxI/видео.html
Great points. Thanks for sharing.
as always high quality content
If possible, can you please make a beginner video to learn these concepts like LLM, transformers, fine tuning etc. That would be really helpful. Thanks!
This, need this badly
Great video
Bluetooth and wireless printers lmfao.
Is it a reupload or am I having a Deja vu
These should have stayed "secrets". lol