How to Set the Chunk Size in Document Splitter | RAG | LangChain

How to define the chunk size when building a Retrieval Augmented Generation (RAG) application? In this video learn how to define it properly.
🦾 Discord: / discord
☕ Buy me a Coffee: ko-fi.com/promptengineering
|🔴 Patreon: / promptengineering
💼Consulting: calendly.com/engineerprompt/c...
📧 Business Contact: [e-mail]
Become Member: tinyurl.com/y5h28s6h
💻 Pre-configured localGPT VM: bit.ly/localGPT (use Code: PromptEngineering for 50% off).
LINKS:
How chunking works: huggingface.co/spaces/m-ric/c...
Signup for Advanced RAG:
tally.so/r/3y9bb0
TIMESTAMPS:
[00:00] Introduction
[01:46] Embedding Models and Chunk Size Optimization
[04:20] Practical Demonstration: Tokenization and Chunk Distribution
[06:55] Exploring Different Splitting Mechanisms
All Interesting Videos:
Everything LangChain: • LangChain
Everything LLM: • Large Language Models
Everything Midjourney: • MidJourney Tutorials
AI Image Generation: • AI Image Generation Tu...

[@engineerprompt]

If you are interested in learning more about how to build robust RAG applications, check out this course: prompt-s-site.thinkific.com/courses/rag

[@shameekm2146]

Thank you so much for pointing this out. I am running a RAG application in production system. The quality of documents I work with is not that great. I have been asked to improve the accuracy of whole RAG pipeline. Hence this is very helpful. :)

[@adnanrizve5551]

Very informative video, ❤ your style of explanation. Keep sharing more on this topic

[@KevinRank]

I appreciate these videos. I'm still trying to get this all figured out. I have a new system, and a big reason to get it, is to run local models I can share out with colleagues. (Keep it all local).

[@matthiasandreas6549]

Yes please more. Thanks

[@alexandrupop7461]

Aprreciate the video! Very interesting

[@stunspot]

Very nice! I do hope you get into some advanced prompting, though. Good prompting can make a huge difference with RAG.

[@abdalrhmanalkabani8784]

Thank you for sharing the video. I have a query regarding a PDF document containing numerous tables. I am currently developing a RAG system, and I am encountering challenges in extracting information from the tables using standard PDF loaders. I have explored using GPT-4 on images, which proved successfully, I asked it to extract it using json form and it worked, but I am seeking an automated solution. Could you kindly suggest effective methods to enhance table content extraction ?.

[@engineerprompt]

If you are interested in leanring more about Advanced RAG Course, signup here: tally.so/r/3y9bb0

[@RocktCityTim]

It seems that not only do you ensure your content context is maintained, but you should also see a more economical parsing of the ingested text. Is that correct?

[@dibu28]

Did you evaluated also Self-RAG or CRAG or GraphRAG or SubDocument-RAG(Summarizing)? To improve answer quality.

[@TC-Loom]

Is anyone adding the overlap, when split by paragraph, to be the last sentence (conclusions) of the ptevious paragraph and the first sentence (introduction/continuation) of the next paragraph? Also, metadata should be keywords produced by a very small local llm, and then you can make a knowledge graph of the keywords.

[@samcavalera9489]

Thanks bro! I can't wait to take your RAG course. Btw, here's a great tutorial on evaluating 8 different RAG models. It show a nice comparative analysis of different metrics of different RAG techniques:
ruclips.net/video/nze2ZFj7FCk/видео.htmlsi=NzgKUeUlTW9ZYn00

[@horacioariash]

Hi. Thanks for your videos. I remember your videos using normally a chunk size = 1000 and overlapping = 200 characters. That was for ChatGPT, LLama?, Mixtral? or others. What is your recommendation size and overlapping for these very well known LLMs?

[@alqods80]

Best chunking is using agentic chunking with grouping, but it costs money

[@maxlgemeinderat9202]

Not if you are using a local llm as agent

[@AnonCoder37]

​@@maxlgemeinderat9202interesting, thanks

[@borisrusev9474]

Can you make a video comparing how chunk size is a trade-off between accuracy and recall?