Thanks for sharing. Great video. This is a useful self contained template for a search usecase. I plan to apply this to one of my use case. I watched the video twice now and the second time around I got a much better understanding. Another interesting part, like you mentioned, is using another LLM call to potentially get an explanation for the output. One question I had, which addressed it towards the end, is how do you logically split the document to ensure consistency, not splitting in the middle of toughts/ideas, etc. One could even try different schemes, e.g., using another higher-level ML model to evaluate different splitting schemes. Thanks once again.
To be honest, this is not a search through LLM, embeddings that generate large language models, in videos just used sentences transformers (all-MiniLM-L6-v2)
lot of potential - research papers / legal etc. Nice job is explaining . Now how would you productize this so that a customer would simply need to install it on a target system or drive where they have their files stored - you have the ability to automatically (efficiently) consume those ... and inference on any new adds?
Thank you for your question! The code on GitHub is an entire project with a frontend ('client' folder) and backend ('server' folder). The key Python code demonstrated in this video is a few folders deep in core models.py: github.com/thinknimble/embeddings-search-demo/blob/main/server/vector_demonstration/core/models.py I hope this helps!
this maybe weird question but, being beginner I'm thinking about getting into it, do you have any other platform like discord where you share and interact with user like me ? :) thank you
Not weird at all! I know many RUclips channels have Discords. We appreciate your interest, but we do not currently have a public Discord, but we will consider setting one up in the future.
wow - there are so many things wrong with this video it's not funny. First, Excel documents are structured data, not unstructured. If the data was a bunch of resumes in word or pdf format, then you'd have unstructured data. Second, your csv files are not csv. Csv stands for comma separated values. You showed html files. I feel sorry for your clients.
Hello! We appreciate your comment. William didn't mention Excel documents, but you raise an interesting and profound question about what "structured" vs. "unstructured" data even is. LLMs are fascinating, because they seem to be revealing a hidden structure in even the most unstructured, natural language. The files in the video and available in the open source codebase are definitely CSVs. So - sorry - you are wrong about that! But as you observed, the 'description' column of each CSV file is HTML. Since HTML is plaintext, it's can be included inside a CSV.
LOL - if you think excel files are structured data, then you have never really worked with excel files in the real world 😂 Nothing could be less structured than your average excel file 😂
excellent video, much better than tons of other long "bla bla" videos out there, thanks!
Short and compact. Excellent video to get an Overview of LLM-based search.
Awesome. I learned how to build inverted indices and this here now seems so simple, yet versatile in comparison.
Great video man. Can you upload more videos like this explaining a bit more on the code side?
William here - thanks for watching! And sure thing. I recorded myself building out the backend of this. I'll get that edited and posted soon.
@@thinknimblethis is worth a sub 🙏🏽💎
Finally got the more in-depth video posted!
ruclips.net/video/OPy4dLHdZng/видео.html
Very nice video - cool to get another perspective on RAGs and how to implement them with a very different stack.
Thanks for sharing. Great video. This is a useful self contained template for a search usecase. I plan to apply this to one of my use case. I watched the video twice now and the second time around I got a much better understanding. Another interesting part, like you mentioned, is using another LLM call to potentially get an explanation for the output. One question I had, which addressed it towards the end, is how do you logically split the document to ensure consistency, not splitting in the middle of toughts/ideas, etc. One could even try different schemes, e.g., using another higher-level ML model to evaluate different splitting schemes.
Thanks once again.
To be honest, this is not a search through LLM, embeddings that generate large language models, in videos just used sentences transformers (all-MiniLM-L6-v2)
That's correct. The goal is not to generate an LLM, but to use an existing LLM to search natural language content.
lot of potential - research papers / legal etc. Nice job is explaining . Now how would you productize this so that a customer would simply need to install it on a target system or drive where they have their files stored - you have the ability to automatically (efficiently) consume those ... and inference on any new adds?
Great questions. We'll likely explore these in future videos.
Thanks for the vid!! Have you guys heard of the Bittensor / Opentensor foundation?
We have not. We'll check it out!
@@thinknimble You won't be disappointed!
Great video, but I am unable to find the code in the given github link, could you please add the python script!
Thank you for your question! The code on GitHub is an entire project with a frontend ('client' folder) and backend ('server' folder). The key Python code demonstrated in this video is a few folders deep in core models.py: github.com/thinknimble/embeddings-search-demo/blob/main/server/vector_demonstration/core/models.py
I hope this helps!
Thanks a lot ☺@@thinknimble
this maybe weird question but, being beginner I'm thinking about getting into it, do you have any other platform like discord where you share and interact with user like me ? :) thank you
Not weird at all! I know many RUclips channels have Discords. We appreciate your interest, but we do not currently have a public Discord, but we will consider setting one up in the future.
FYI!!!!
This honestly seems like the dumbest application of this technology.
wow - there are so many things wrong with this video it's not funny. First, Excel documents are structured data, not unstructured. If the data was a bunch of resumes in word or pdf format, then you'd have unstructured data. Second, your csv files are not csv. Csv stands for comma separated values. You showed html files. I feel sorry for your clients.
Hello! We appreciate your comment. William didn't mention Excel documents, but you raise an interesting and profound question about what "structured" vs. "unstructured" data even is. LLMs are fascinating, because they seem to be revealing a hidden structure in even the most unstructured, natural language.
The files in the video and available in the open source codebase are definitely CSVs. So - sorry - you are wrong about that! But as you observed, the 'description' column of each CSV file is HTML. Since HTML is plaintext, it's can be included inside a CSV.
LOL - if you think excel files are structured data, then you have never really worked with excel files in the real world 😂
Nothing could be less structured than your average excel file 😂
Hey, nice idea about chunks! Thanks a lot for the video!