Better RAG with Merger Retriever (LOTR) and Re-ranking Retriever (Long Context Reorder)
HTML-код
- Опубликовано: 11 сен 2024
- In today's tutorial, I dive deep into the world of advanced information retrieval, focusing on two essential concepts:
LOTR (Lord of the Retriever), also known as the Merger Retriever. This intriguing technique utilizes a round-robin approach to merge results from multiple vector databases, ensuring a robust and diverse set of results.
Long Context Reorder: This is all about the reranking of retrievers. Once you've retrieved your documents using multiple models, how do you optimally order them to ensure relevance and precision?
For those dabbling with Retrieval Augmented Generation (RAG), implementing these techniques is pivotal. A more effective retrieval process directly enhances the quality and relevance of the generated content in RAG models.
Throughout the tutorial, I've leveraged LangChain and its high-level abstract classes, which streamline the implementation of these advanced techniques.
If you found this content valuable, please hit that like button, drop a comment below with your thoughts or questions, and don't forget to subscribe for more in-depth tutorials like this!
GitHub Here: github.com/AIA...
LOTR: python.langcha...
BGE Embedding Model: huggingface.co...
#generativeai #langchain #llm
Love these long form videos. Your commentry in between the code is superb and fills lot of knowledge gaps. In most other videos, people just finish up in 10 mins and dont talk more in the ecosystem, what is better, why something is picked , which paper can be used to learn more.
This form really what helps to learn better. Thanks again
Great to hear!
Very Informative & Insightful Video Sir !! Awaiting for Next part of Video - Generation
So nice of you. Thank you.
Hi sir ,when the next part of this video with LLM will be out? Please can you do share ,☺️
There is a new technique called selfrag. It has its llama2 version. Can you create a video on how to integrate it with langchain ?
15:49 hello. The embedding model BAAI/bge-large-en that you used has sequence length 512. But the when doing recursive splitting you have set chunk size to 1000. So after 512 would the model truncate the rest of the sentences in the chunk?
Is sequence length 512 of the embedding model words or only characters?
Hello sir, there is no next video on contextual compression, mitigate biases, remove redundant chunks, emsemble retriever(hybrid search) and other techniques to improve performance of rag chain. Please sir, can you share the video link. Thanks
amazing video! greetings from Peru
Thank you very much!
This is a great video, thank you very much for sharing your valuable knowledge.
Glad it was helpful!
great video as usual
Happy to hear that!
Thanks for the tips... It helped me improve the accuracy
Great to hear!
Hi this video was awesome but there is no next video showing context compression retriever and the ensemble retriever
Can we use this merging concept with parent-document retriever where i store my parent chunks in inmemory and child chunks in a chroma collection then i want to merge the output of the parent retriever with the self query retriever ?
yes
Waiting for the next video!
Soon... Thanks
Sir we want a video on RAG Evaluation using RAGAS or something, please bring it sir will be really helpful because it may be helpful to check how better our RAG did after these changes or something
Will create today. Release soon. It's in the list.
FYI there is a failure of direct retrieval with GPT-4 using the new OpenAI Assistant API. GPT tokenizes text and creates its own vector embeddings based on its specific training data. The new terms and sequences may not connect well to the pretrained knowledge in GPT's weight tensors.
There was no semantic similarity between the new API terms and GPT's existing vector space. This is a fundamental issue with retrieval augmentation systems like Rag - external knowledge is not truly integrated into the model's learned weights. Adding more vector stores cannot solve this core problem.
I've been watching your channel for quite some time now. I appreciate the hard work you put into these videos. Microsoft's new lida package is amazing for data visualization using LLMs. Can you please make a video on how to implement it on csv and other kinds of data.
See if this helps: ruclips.net/video/U9K1Cu45nMQ/видео.html thanks for the inputs. 1 more video will be available soon on LIDA.
@@AIAnytime Thanks a lot for making this video.
can you please mention the best embedding model..is the one you used still the best?
Great video, Thank you ❤
Glad you enjoyed it!
I am getting Stopiteration when running cell 13
You didndt pass query in reordering.transform_documents. how it was able to reorder ?
I was wondering the same. My understanding is it needs the query based on which it could reorder the documents (passed). Commenting it here to see if AI anytime Guru checks this 🙂
Great Video !! Hi sir ,when the next part of this video with LLM will be out? Please can you do share
As soon as possible
Hi ,If you can share some document digitization video using Gen AI
Greatvideo
Thanks
Hi AIAnytime great video , i was looking for something in same line with cohere ai , can please create a short video to use it with Normal Retrieval QA Chain
Sure Tushaar, let me see if i can use cohere.
Many2thinksforvideo