I’ve been following along since your first video and I can say that this is the best zero-to-hero course I’ve tried. Seriously thank you for making us feel like we’re on the cutting edge with you. There’s so much to learn in this field, and every day 100 new things are happening. Thank you for making it easy to follow and code along.
Couldn't wait for it! This is the tech I have been waiting for! This is how LLM's can be powerful for us as developers even without OpenAI or any other big companies! I am here for it.
Decrease chunk size when splitting text so that when retriever pulls like 3 chunks out of DB those will be smaller so they will fit into context ;-) Awesome content!
I started from your past tuto and spent hours to switch on local model (I am a noob 😅). Anyway this is perfect timing with this video to check my homework. Eager to dive into it as soon as I leave the office !
It was not me. I got similar weird output with other models. Trade off is the right conclusion. It is really hard to get rid of open ai or other giant models so far. Anyway. Let’s keep optimistic as huge progress is being made on a daily basis in the open source galaxy. Thanks again for the perfect content I was searching for ! Let’s keep testing and exploring as you say
Thank you so much for this video! Been waiting for this :) Thumbs up before I even watch. Hoping for more on this general local llm vein, especially as new models are almost constantly coming out!
Thank you so very much for this! I have been wanting to dip my toes in with LangChain and Embeddings -- this was a very helpful and made it feel a lot more accessible.
Just wanna add a comment to congrat you, you're doing a great job for the community and I personnaly learn much more watching your videos than hours to find information and make my own testing on git. Thanks for this. I will give it a try with a bloom model and report here so you guys know
Thanks for the kind words. Bloom is a Decoder only model so you will just use one of the notebooks that has 'text-generation' in the pipeling not 'text2text-generation'
@@samwitteveenai hey Sam thanks for the answer. Does that mean that I only have to change this setting to make decoder only model working? What will I loose by using text generation instead of text2text? Or Is it better to use encoder/decoder model? Thanks for the answer!
better to look for one of my notebooks with the text-generation examples in there as there are other things you need to change in how you load the model too.
This is amazing stuff, I am learning a lot from the colabs! For my use case, it would be amazing to see work with larger models too (databricks Dolly comes to mind)
Thats something unique...you are best..i think you just missed one important thing that langchain supports only text-generation and text2text-generation models from huggingface
It's great to see a focus on open source models. By the way, it would be better to use a stop token (such as "### Human:" and "### Assistant:") for "StableVicunaLM" instead of using regular expressions or other cleaning methods. Great video :)
I’ve been following auto gpt’s repos such as agent llm that allows auto gpt with local models. Its is very experimental, the models would do significantly better if they were trained with retrieval example questions. While better prompting may help, a larger context window would help, I believe a version of mpt 7 story writer that would be trained with instruction examples and examples for auto gpt for training, as well as a dataset of python code q/a already available in alpaca format on hugging face would allow the model to be trained to do many auto tasks. And for a very long time due to its context window.
Has this presumption panned out as you portrayed it? I had a similar impression at the time but got derailed. Trying to pick up the pieces as it were and curious if this has been at least theoretically encouraging? (Lots to still catch up on.)
@@taiconan8857 there have no been recent attempts at making the larger token models more accurate at instruction based questions and answers, provided it would be a expensive dataset to make, unless you did it all with gpt 3.5 turbo api. I suggest you read microsoft paper Orca, its essentially the 13b llama model trained of freaking 5 Million questions where gpt 3.5 and 4 created a 5 million question step by step instructions and explanation on why it is answering the way it is answering, so instead of just mimicing chat gpt, it is actually learning and getting better at reasoning. In their paper it showed to preform similar to chat gpt. And they plan to release the model weights and they are currently working with their lawyers. This was half a week ago while ago so open ai might be a little angry 😂. But if this were to work open sourced models and ai agents are gonna be actually useful.
@nicolasmejia-petit7179 Very nice! Really appreciate all this info. I've often wondered (since I don't precisely understand the methodology between following token weights and the training data's weights) Is it not possible to have token weights that do not sufficiently resemble inputs or expected outputs it begins to favor responses that include, "I'm under the impression that" or simply "I don't know." Thanks again for this *great* response! 👍💎⛏️
@@taiconan8857 yeah, that is done with open ai’s models, but with the current open source models they are called “imitation models” because they only imitate the way the model works(theres a paper about this), it has no understanding of reasoning so if it preforms something outside its training it is likley to hallucinate. This is something orca likley fixed wont know till they publicize there weights. So it is important that models know their own limits, but sometimes in the cases of uncensored models the performance is often better than their censored counterparts, meaning any phrase along the lines of “as an ai model I can not” is removed.
@@Nick_With_A_Stick Sounds like a classic case of "working to treat the symptoms rather than the cause" but with data instead of disease. Doesn't seem to me like people don't have sufficient tools to do this, but perhaps it's more work than they're willing to put in. 😉
Hi Sam. Thanks a lot for you awesome videos. Your channel is probably one of the best (if not the best) sources to stay on top of LLMs in this crazy time. I'd like to make a suggestion on your content. When we want to use LLMs in production, we do not rely on the 'knowledge' it outputs since it's not reliable. We need accuracy and up-to-date-ness in production. And since people have their custom libraries of their own, the immediate benefit comes out of using those libraries rather than relying on Coder LLM outputs, which is largely troublesome. So instead of experimenting with 'What's the capital of England?', 'Tell me about Harry Potter', and 'Write a letter to Sam Altman', I think the following would be of more value to the audience. """ Write a summary from the data focusing on the commonalities, differences and anomalies. library: function get_dataframe_from_csv(csvpath:str) -> pd.DataFrame function get_commonalities(dataframe:pd.DataFrame) -> list function get_differences(dataframe:pd.DataFrame) -> list function get_anoamlies(dataframe:pd.DataFrame) -> list """ The output is expected to be a few lines of code using those library functions in the prompt. And running the automatically generated code would be quite similar to what OpenAI Code Interpreter offers. The core idea here is that this is the way I would pursue if I were to develop a production system using LLM rather than relying on the outputs directly genenerated by the LLM. You could perhaps do some training instead of putting the library function definitions in the prompt. It would be exciting to see the outputs automatically correlated and organised in the way you wanted on your own data. I hope this helps.
8:14 you might want to build a template=PromptTemplate(template="your template with a {context} and a {question}", input_variables=["context", "question"]) and pass it as a prompt=template param to your QA chain instead of overriding the .prompt.template
Hi, I am having the same issue as you. That is that there was a difference between information retrieval and information analysis. These models are unable to compare and contrast two documents. So you will get against a brick wall. They cannot critically think and this is probably our problem because they seem like they can think. But they are only running through similar expected chains of words. Perhaps in one of your models, you are able to have a set up to critically think. I imagine it would be like when autogpt is talking to itself? But you would code the box to chat to each other in order to generate the critically evaluation of two documents, or ideas. I hope you get what I mean. For example, you would write some code to look for identifiers of keywords, like critically, evaluate, or compare, or any of those essay type terms. Then, when hearing that identifier, you engage another chat, personality. Those two personalities will then work out the opposing views, and placed this in a in the chat. There are a lot of examples of rhetorical analysis, or that you can use as a template for a prompt.
it would be interesting to give a small model a "tool" of using a larger model for more context when needed - so that we could almost like ratchet up the intelligence needed on a per generation basis. Maybe even doing parts of the generation on smaller models then passing that information up to larger and larger models until some set threshold is reached. I feel like that is going to require some kind of lora fine tuning tho or else it will be fairly brittle.
Appreciate your sharing. Embedding with hkunlp/instructor-xl and LLM with MBZUAI/LaMini-Flan-T5-248M are the best combination i test. embedding takes some time, but searching basically takes a few seconds with this tiny model.
Hey, love the content, might make sense if you talk about an earlier video to add a popup on the video where the earlier video is. Otherwise , well done, and thank you.
This looks great! Thank you for the tutorials! Have you thought about trying langchain with different architectures like mpt? The MPT-7b-chat model seems to be quite good at coding and might have autogptq support to be able to be quantized soon so running it with langchain would be great! Thank you again for all the help
Yes also interested in the MPT 7b chat. Gave it a try but failed to get it to work due to what seems to be a corrupt config file in the repo. Can't spot where the error is.
Very helpful walkthru! Your vid makes me think I can do it myself if I follow you! 🎉 For your video making one thing I noticed is the voice sound is distorted and weaken several times. I am not native English so this makes me loose context. Your microphone may be too near or recording FX / mic attenator / auto leveler may need adjust
Thank you for yet another excellent video! It's great to learn about other models and how to get started with them. Can you please provide more information about why you choose Flan T5, and the associated modeling utilities? I apologize for my lack of technical expertise, but I would appreciate a better understanding of how they operate in tandem.
Have you considered, instead of using embeddings (regardless of the model) trying to fine-tune a model, more specifically I heard the idea of using S-BERT (a simple encoder only model) to get the embeddings via fine-tuning instead, this would be efficient specially when the corpus of data is not very large (say below a million words), the idea is that fine-tuning a small model would be very fast and would eventually deliver even better results, easy to update with new data, etc. Have you heard and consider this alternative?
Hello, thank you so much for your video. I am getting error on local_llm(“text”) line. I think there is missing in langchain huggingfacepipeline. Value Error: The following ‘model_kwargs` are not used by the model: ['max_lenght', 'return_full_text'] (note: typos in the generate arguments will also show up in this list)
Not and ignorant question at all. In my tests for other things the current way of doing the 4bit tends to change the output. Usually just making it shorter. I will start to make some vids with the 4bit models soon as I can see it allows a lot more people to use the models with free Colab etc. Also there is a new 4bit library coming in the next few weeks.
Great video. I'm curious if there is certain types of input format of the knowledge base documents that work better than other types. Things like longer sentences versus short sentence or other things like that.
Thank you so much for this nice video. Much appreciated and needed as a lot of things are happening in this area nowadays. However, I am trying the same code of WizardLM in Azure ML, but I am not getting the kind of result as you have shown in your video. A little help is much appreciated.
Great video, Sam! One question: is it worth using another embedding model for text vectorization and use gpt-3.5-turbo model for the rest of the tasks? (agents, qa etc.) The reason behind that is - text vectorization takes too long when using openai's embedding model.
@@samwitteveenai Including vectorization speed? I understand, it depends on specs, but what specs would you recommend to achieve much faster model than openai's one?
One disadvantage that needs to be mentioned vs. using OpenAI's API (besides of having to wrangle dumber/buggy models) is the model startup time. If you want to have an always-ready model loaded in the GPU waiting for your input, it gets expensive pretty quick. In fact the most baffling thing about OpenAI is how they can afford such availability at scale. But perhaps they are just bleeding cash like crazy, who knows...
Thanks for the videos! Do you know how can the models be "unloaded"? Should I just delete the folders? They are taking a lot of space in my drive and I haven't been able to figure out the best way of deleting them.
Is there a way we can store the models locally instead of downloading when script runs? I am running on command prompt in Windows. Also how do we get bits and bytes optimized for cuda? Thanks
Hello There, How do we validate on the source document that we receive in response? Because, I'm getting non matching document as sources in RetrievalQA chain
Amazing. I was already using WizardLM following your guide to use Huggingface Embeddings. Good to know I made the right call. Could you show us how to implement it with Chat memory using this method? I already have the Chat Memory tutorial from you but I don't know how to create the correct chain to combine both.
Great video Sam thanks for the relentless effort in helping us all.. +1 for the chat memory request and conversational QA.. hoping to see a video soon!
Great comparisons of a complete open source solution with different LLM’s. What about mpt-7b with 65 k tokens. If a LLM is big i thought u can stream it from huggingface or am i wrong? Thanks for sharing and show us about the New innovations in fast moving developments in LLM’s 👏
65k is too slow. You can stream some from HF, but you are limited as too how much you can stream and the context size etc. Also then your data is still going to the cloud. I will make a part 2 to this video with some other new models this week.
Would love to see you try different models with the question answer system. This was great. Glad to see you found one that works reasonably well. I tried T5 and mrm8488/t5-base-finetuned-question-generation-ap which worked decently well, slightly better than the plain T5. Will try the wizard but may not be able to on my GPU. Looking forward to your other tests.
This is great that you shared what you tried as well. I think if we all mention what we have tried it would be great and help everyone to find the best models.
Great video man, tx !! however, I couldn't run it on my Macbook Pro (not M1). when trying to pip install transformers and xformers im getting "ERROR: Could not build wheels for xformers, which is required to install pyproject.toml-based projects" 😞
Thanks for the great video. I am a bit confused about 3 tokens used for retriever! Is that the number of tokens fed into the model? I believe chatgpt does not explicitly include a retireiver component, I maybe wrong. If you could explain what is the function of these 3 tokens it would be highly appreciated, seems extremely low for any tasks.
You must be referring to the line `retriever = vectordb.as_retriever(search_args={"k": 3)`. This argument is not passed ChatGPT or any LLM for that matter. The 3 is specifying the number of documents that the "retriever" should retrieve from the Chroma vector database prepared earlier in the video. Effectively, the 3 documents found to be most relevant a given query will be pulled up and included as additional information ("context") when querying the LLM.
Hi, Thank you for your tutorials. I am following your tutorials for quite some time now. I have watched your whole playlist on this. However I am unable to figure out best economic approach for my use case. I want to create a Q & A chatbot on streamlit which answers only my custom single document of about 500 pages. The document is final and won't change. From my understanding so far, I should either choose Langchain or LlamaIndex. But, I will have to use OpenAI api to get best answers, but that API is quite costly for me. So far I have thought of using Chroma for embedding and somhow storing the vectors as pkl or json on streamlit itself for re-use, so I don't have to spend again for vectors/indexing. I don't have enough credits to test different methods myself. Kindly guide me. Thank you.
hi Sam, I tried to load WizardLM in my free Colab account. But, I tried couple times and it all failed to load. I wonder if I need to have pro or pro+ account to load your examples?
Are there any examples of LangChain working well to make code changes (or suggesting to make code changes) to medium to large codebases based on a prompt? For example, modify the code to make the given test pass.
This is an interesting idea, I haven't tried it. It might work better with a code model like Starcoder or GPT-4. The bigger models can certainly do unit tests etc.
or rather can a notebook be retrofitted to have REST functionality? I've googled but trying to get the lay of the land before embarking on making the first thing that comes to mind
I tried with this code and it's working fine for the docs that I have given but it's also giving answers for the other questions that details are not present in the docs. so How can I put all things for only my docs and for outer questions it gives something I don't know something like this. Can you please guide me? I have used the WizardLM model.
Is WizardLM an encoder-decoder model? Or just a decoder model? Does this distinction matter? If anybody could get back to me, I would be so appreciative.
The vector stores will be the same it really comes down to what embeddings you use. Checkout the leaderboard here and you probably want to try some multi lingual models huggingface.co/spaces/mteb/leaderboard
@@samwitteveenai thanks 👍 had a ganger and can see how you can delete collections and return the IDs based on a where clause for metadata search, guess you could then take the returned IDs and delete them or update them. Might be worth a video covering this as I can't find a single video on RUclips for this, just a thought!
I have been struggling to figure out if there is a performant way to tune and run a 13B model on a modern graphics card with 12gb of vram (I think I have seen the 4-bit mode suggested for this, but thought that the results might be more poor than just using a 7b model). I love the speeds on the device but I know for some stuff I will be bumping right up against the limits of what I can do with a 7b model. I could get and run the model on a dual-GPU setup as I have seen shown is possible by @Jeff Heaton . I just am under the impression. There would be a significant performance hit. I also really like having the model run on the one GPU I have since it is very conservative of power and I could conceivably afford to run it with the model and do some self-hosted services with it. Does anyone have any suggestions? Thank you!
Thanks. How about using a local llm ai to find chunks using embeddings and open ai for final step: the chunks tokens and question in input ? Will it be good enought ?
There is no reason to use the same model for the embeddings and the main LLM. Embedding LLMs are trained for different tasks. Actually Instructor uses a T5 Architecture underneath though (but thats not related to the other LLM)
Attempting to run the StableVicuna and WizardLM Models and I get the error: Make sure you have enough GPU RAM to fit the quantized model. What GPU/VRAM are you running, and how do I set it as a parameter like 'gpu_memory_0'? Also, is there site which documents all parameter inputs in the transformers? (Sorry.. N00b here in Python).
Finally a tutorial without open ai. Open Source rules. Perfect.
I’ve been following along since your first video and I can say that this is the best zero-to-hero course I’ve tried. Seriously thank you for making us feel like we’re on the cutting edge with you. There’s so much to learn in this field, and every day 100 new things are happening. Thank you for making it easy to follow and code along.
Thanks Srikar much appreciated
Couldn't wait for it! This is the tech I have been waiting for! This is how LLM's can be powerful for us as developers even without OpenAI or any other big companies! I am here for it.
Amazing, I've literally been spending the past day trying to learn exactly this and you boiled it down nicely!
Decrease chunk size when splitting text so that when retriever pulls like 3 chunks out of DB those will be smaller so they will fit into context ;-) Awesome content!
I started from your past tuto and spent hours to switch on local model (I am a noob 😅). Anyway this is perfect timing with this video to check my homework. Eager to dive into it as soon as I leave the office !
It was not me. I got similar weird output with other models. Trade off is the right conclusion. It is really hard to get rid of open ai or other giant models so far. Anyway. Let’s keep optimistic as huge progress is being made on a daily basis in the open source galaxy. Thanks again for the perfect content I was searching for ! Let’s keep testing and exploring as you say
Thank you so much for this video! Been waiting for this :) Thumbs up before I even watch. Hoping for more on this general local llm vein, especially as new models are almost constantly coming out!
Make another video that you make with other models, I would be happy for bigger models or those of 13B. Love your videos❤
Thank you so very much for this! I have been wanting to dip my toes in with LangChain and Embeddings -- this was a very helpful and made it feel a lot more accessible.
Thank you so much for this video Sam. Found the video at the right time!
Thank you for this best tutorial video
you are a life saver for this ty!
Just wanna add a comment to congrat you, you're doing a great job for the community and I personnaly learn much more watching your videos than hours to find information and make my own testing on git. Thanks for this. I will give it a try with a bloom model and report here so you guys know
Seems like that text2text generation doesn't support bloom do you have a list of supported model for this purpose?
Thanks for the kind words. Bloom is a Decoder only model so you will just use one of the notebooks that has 'text-generation' in the pipeling not 'text2text-generation'
@@samwitteveenai hey Sam thanks for the answer.
Does that mean that I only have to change this setting to make decoder only model working?
What will I loose by using text generation instead of text2text?
Or Is it better to use encoder/decoder model?
Thanks for the answer!
better to look for one of my notebooks with the text-generation examples in there as there are other things you need to change in how you load the model too.
Thank you for this unique and super high quality content you put out for us!
This is amazing stuff, I am learning a lot from the colabs! For my use case, it would be amazing to see work with larger models too (databricks Dolly comes to mind)
Thats something unique...you are best..i think you just missed one important thing that langchain supports only text-generation and text2text-generation models from huggingface
Incredible! Thank you so much for putting it together
Would be great to have a video about top 5 open source llm allowing commercial usage. 🤓👏
It's great to see a focus on open source models. By the way, it would be better to use a stop token (such as "### Human:" and "### Assistant:") for "StableVicunaLM" instead of using regular expressions or other cleaning methods. Great video :)
Wild that so many models out there that can already be run locally
You are the man I was working on this myself but I was started using Bert or basic NLP
I’ve been following auto gpt’s repos such as agent llm that allows auto gpt with local models. Its is very experimental, the models would do significantly better if they were trained with retrieval example questions. While better prompting may help, a larger context window would help, I believe a version of mpt 7 story writer that would be trained with instruction examples and examples for auto gpt for training, as well as a dataset of python code q/a already available in alpaca format on hugging face would allow the model to be trained to do many auto tasks. And for a very long time due to its context window.
Has this presumption panned out as you portrayed it? I had a similar impression at the time but got derailed. Trying to pick up the pieces as it were and curious if this has been at least theoretically encouraging? (Lots to still catch up on.)
@@taiconan8857 there have no been recent attempts at making the larger token models more accurate at instruction based questions and answers, provided it would be a expensive dataset to make, unless you did it all with gpt 3.5 turbo api. I suggest you read microsoft paper Orca, its essentially the 13b llama model trained of freaking 5 Million questions where gpt 3.5 and 4 created a 5 million question step by step instructions and explanation on why it is answering the way it is answering, so instead of just mimicing chat gpt, it is actually learning and getting better at reasoning. In their paper it showed to preform similar to chat gpt. And they plan to release the model weights and they are currently working with their lawyers. This was half a week ago while ago so open ai might be a little angry 😂. But if this were to work open sourced models and ai agents are gonna be actually useful.
@nicolasmejia-petit7179 Very nice! Really appreciate all this info. I've often wondered (since I don't precisely understand the methodology between following token weights and the training data's weights) Is it not possible to have token weights that do not sufficiently resemble inputs or expected outputs it begins to favor responses that include, "I'm under the impression that" or simply "I don't know."
Thanks again for this *great* response! 👍💎⛏️
@@taiconan8857 yeah, that is done with open ai’s models, but with the current open source models they are called “imitation models” because they only imitate the way the model works(theres a paper about this), it has no understanding of reasoning so if it preforms something outside its training it is likley to hallucinate. This is something orca likley fixed wont know till they publicize there weights. So it is important that models know their own limits, but sometimes in the cases of uncensored models the performance is often better than their censored counterparts, meaning any phrase along the lines of “as an ai model I can not” is removed.
@@Nick_With_A_Stick Sounds like a classic case of "working to treat the symptoms rather than the cause" but with data instead of disease. Doesn't seem to me like people don't have sufficient tools to do this, but perhaps it's more work than they're willing to put in. 😉
Hi Sam. Thanks a lot for you awesome videos. Your channel is probably one of the best (if not the best) sources to stay on top of LLMs in this crazy time. I'd like to make a suggestion on your content.
When we want to use LLMs in production, we do not rely on the 'knowledge' it outputs since it's not reliable. We need accuracy and up-to-date-ness in production. And since people have their custom libraries of their own, the immediate benefit comes out of using those libraries rather than relying on Coder LLM outputs, which is largely troublesome.
So instead of experimenting with 'What's the capital of England?', 'Tell me about Harry Potter', and 'Write a letter to Sam Altman', I think the following would be of more value to the audience.
"""
Write a summary from the data focusing on the commonalities, differences and anomalies.
library:
function get_dataframe_from_csv(csvpath:str) -> pd.DataFrame
function get_commonalities(dataframe:pd.DataFrame) -> list
function get_differences(dataframe:pd.DataFrame) -> list
function get_anoamlies(dataframe:pd.DataFrame) -> list
"""
The output is expected to be a few lines of code using those library functions in the prompt. And running the automatically generated code would be quite similar to what OpenAI Code Interpreter offers. The core idea here is that this is the way I would pursue if I were to develop a production system using LLM rather than relying on the outputs directly genenerated by the LLM.
You could perhaps do some training instead of putting the library function definitions in the prompt.
It would be exciting to see the outputs automatically correlated and organised in the way you wanted on your own data.
I hope this helps.
Can you test the mpt-7B Model?
This is perfect for what I am trying to do!
How are you doing this? My RUclips watchlist can't cope with all your content. 😂
Thanks though, this stuff is fantastic.
I've binged watched every video and have caught up, but really sir. Did you have to be this good! ;)
thanks
8:14 you might want to build a template=PromptTemplate(template="your template with a {context} and a {question}", input_variables=["context", "question"]) and pass it as a prompt=template param to your QA chain instead of overriding the .prompt.template
Yes I just wanted people to see that all it is underneath is a string and that string can be directly manipulated.
Hi, I am having the same issue as you. That is that there was a difference between information retrieval and information analysis. These models are unable to compare and contrast two documents. So you will get against a brick wall. They cannot critically think and this is probably our problem because they seem like they can think. But they are only running through similar expected chains of words.
Perhaps in one of your models, you are able to have a set up to critically think. I imagine it would be like when autogpt is talking to itself? But you would code the box to chat to each other in order to generate the critically evaluation of two documents, or ideas. I hope you get what I mean. For example, you would write some code to look for identifiers of keywords, like critically, evaluate, or compare, or any of those essay type terms. Then, when hearing that identifier, you engage another chat, personality. Those two personalities will then work out the opposing views, and placed this in a in the chat. There are a lot of examples of rhetorical analysis, or that you can use as a template for a prompt.
Congratulations on a brilliant video
thanks
it would be interesting to give a small model a "tool" of using a larger model for more context when needed - so that we could almost like ratchet up the intelligence needed on a per generation basis. Maybe even doing parts of the generation on smaller models then passing that information up to larger and larger models until some set threshold is reached. I feel like that is going to require some kind of lora fine tuning tho or else it will be fairly brittle.
A good benchmark chart will help
Appreciate your sharing. Embedding with hkunlp/instructor-xl and LLM with MBZUAI/LaMini-Flan-T5-248M are the best combination i test. embedding takes some time, but searching basically takes a few seconds with this tiny model.
Hey, love the content, might make sense if you talk about an earlier video to add a popup on the video where the earlier video is. Otherwise , well done, and thank you.
good point. Will try to do this more going forward.
Much awaited video
The hashes are Markdown format! That’s actually how openai responds as well
Very cool!
fantastic - thank you!
This looks great! Thank you for the tutorials! Have you thought about trying langchain with different architectures like mpt? The MPT-7b-chat model seems to be quite good at coding and might have autogptq support to be able to be quantized soon so running it with langchain would be great! Thank you again for all the help
Yes also interested in the MPT 7b chat. Gave it a try but failed to get it to work due to what seems to be a corrupt config file in the repo. Can't spot where the error is.
Very helpful walkthru! Your vid makes me think I can do it myself if I follow you! 🎉 For your video making one thing I noticed is the voice sound is distorted and weaken several times. I am not native English so this makes me loose context. Your microphone may be too near or recording FX / mic attenator / auto leveler may need adjust
It's gonna be a busy week-end !😂
You are the man ...!!!
Thank you for yet another excellent video! It's great to learn about other models and how to get started with them. Can you please provide more information about why you choose Flan T5, and the associated modeling utilities? I apologize for my lack of technical expertise, but I would appreciate a better understanding of how they operate in tandem.
I am making a part 2 to this and will explain more in there.
@@samwitteveenai any ETA on the part 2?
I hope these is a future edit where this is done using LLAMA 2 model
I actually have done some of these with LLaMA 2. Look for the RAG & LLaMA-2 vids
Gotta try using the MPT-7B-storywritter model right ? the 65k token length will surely help?
I showed StoryWriter briefly in the MPT7 video. It is super slow not really great for this kind of use.
FINALLY!! Thanks!
Thanks. U r the best
Thanks
Have you considered, instead of using embeddings (regardless of the model) trying to fine-tune a model, more specifically I heard the idea of using S-BERT (a simple encoder only model) to get the embeddings via fine-tuning instead, this would be efficient specially when the corpus of data is not very large (say below a million words), the idea is that fine-tuning a small model would be very fast and would eventually deliver even better results, easy to update with new data, etc. Have you heard and consider this alternative?
Brilliant thanks!
Great work thanks, can you also try red pajama or MPT models
Yes I will look at those soon for this kind of task
Awesome content!
Hello, thank you so much for your video.
I am getting error on local_llm(“text”) line. I think there is missing in langchain huggingfacepipeline.
Value Error: The following ‘model_kwargs` are not used by the model: ['max_lenght', 'return_full_text'] (note: typos in the generate arguments will also show up in this list)
unlimiformer might be interesting to look at next...unlimited context works with encoder/decoder based models.
Agree its an interesting paper.
Sorry if this is an ignorant question, but is there a reason for not using 4-bit quantised models?
Not and ignorant question at all. In my tests for other things the current way of doing the 4bit tends to change the output. Usually just making it shorter. I will start to make some vids with the 4bit models soon as I can see it allows a lot more people to use the models with free Colab etc. Also there is a new 4bit library coming in the next few weeks.
@@samwitteveenai 4bit models have been a godsend for me as I like to run them in TGUI locally
The best video!!
Great video. I'm curious if there is certain types of input format of the knowledge base documents that work better than other types. Things like longer sentences versus short sentence or other things like that.
generally you want things with clear meaning in each section of text, but don't let that stop you, just try it out with a variety of docs etc.
Thank you so much for this nice video. Much appreciated and needed as a lot of things are happening in this area nowadays. However, I am trying the same code of WizardLM in Azure ML, but I am not getting the kind of result as you have shown in your video. A little help is much appreciated.
Try running the Colab. What outputs are you getting different?
Great video, Sam! One question: is it worth using another embedding model for text vectorization and use gpt-3.5-turbo model for the rest of the tasks? (agents, qa etc.) The reason behind that is - text vectorization takes too long when using openai's embedding model.
Yes totally . I think the Instructor embeddings are actually doing better than OpanAI in many cases as well.
@@samwitteveenai Including vectorization speed? I understand, it depends on specs, but what specs would you recommend to achieve much faster model than openai's one?
Sam you are the best
One disadvantage that needs to be mentioned vs. using OpenAI's API (besides of having to wrangle dumber/buggy models) is the model startup time. If you want to have an always-ready model loaded in the GPU waiting for your input, it gets expensive pretty quick. In fact the most baffling thing about OpenAI is how they can afford such availability at scale. But perhaps they are just bleeding cash like crazy, who knows...
Agree, lots of disadvantages of using self hosted models.
Thanks for the videos! Do you know how can the models be "unloaded"? Should I just delete the folders? They are taking a lot of space in my drive and I haven't been able to figure out the best way of deleting them.
Is there a way we can store the models locally instead of downloading when script runs? I am running on command prompt in Windows. Also how do we get bits and bytes optimized for cuda? Thanks
Hello There, How do we validate on the source document that we receive in response?
Because, I'm getting non matching document as sources in RetrievalQA chain
Amazing. I was already using WizardLM following your guide to use Huggingface Embeddings. Good to know I made the right call.
Could you show us how to implement it with Chat memory using this method? I already have the Chat Memory tutorial from you but I don't know how to create the correct chain to combine both.
Good idea I will add that to a future video
Great video Sam thanks for the relentless effort in helping us all.. +1 for the chat memory request and conversational QA.. hoping to see a video soon!
great content! Have you already tries fine tuning / training your own small specialised model?
Yes that is how I do it at work, but I can't release those models currently. I am working on a solution like that to show soon
At least in principle one could use the same model to extract embedings and answer question.
This doesn't usually work well and the training for the 2 tasks is quite different.
Great comparisons of a complete open source solution with different LLM’s. What about mpt-7b with 65 k tokens. If a LLM is big i thought u can stream it from huggingface or am i wrong? Thanks for sharing and show us about the New innovations in fast moving developments in LLM’s 👏
65k is too slow. You can stream some from HF, but you are limited as too how much you can stream and the context size etc. Also then your data is still going to the cloud. I will make a part 2 to this video with some other new models this week.
@@samwitteveenai Thanks, looking forward for the new LLM's..
What is the difference between HuggingFaceEmbedding vs HuggingFaceInstructEmbeddings? Are there any pros of using one over another?
Would love to see you try different models with the question answer system. This was great. Glad to see you found one that works reasonably well. I tried T5 and mrm8488/t5-base-finetuned-question-generation-ap which worked decently well, slightly better than the plain T5. Will try the wizard but may not be able to on my GPU. Looking forward to your other tests.
This is great that you shared what you tried as well. I think if we all mention what we have tried it would be great and help everyone to find the best models.
Any idea how to use ggml such as TheBloke/Wizard-Vicuna-13B-Uncensored-GGML (or even smaller for llama.cpp models like 7b Wizard)?
or this TheBloke/wizardLM-7B-GGML. please make a tutorial for this.
Can you do something on FLAN-T5-XXL, my Flan-T5-XL performs better than xxl, I am missing something so want to understand
Great video man, tx !! however, I couldn't run it on my Macbook Pro (not M1). when trying to pip install transformers and xformers im getting "ERROR: Could not build wheels for xformers, which is required to install pyproject.toml-based projects" 😞
Thanks for the great video. I am a bit confused about 3 tokens used for retriever! Is that the number of tokens fed into the model? I believe chatgpt does not explicitly include a retireiver component, I maybe wrong. If you could explain what is the function of these 3 tokens it would be highly appreciated, seems extremely low for any tasks.
You must be referring to the line `retriever = vectordb.as_retriever(search_args={"k": 3)`. This argument is not passed ChatGPT or any LLM for that matter. The 3 is specifying the number of documents that the "retriever" should retrieve from the Chroma vector database prepared earlier in the video. Effectively, the 3 documents found to be most relevant a given query will be pulled up and included as additional information ("context") when querying the LLM.
Hi, Thank you for your tutorials. I am following your tutorials for quite some time now. I have watched your whole playlist on this. However I am unable to figure out best economic approach for my use case.
I want to create a Q & A chatbot on streamlit which answers only my custom single document of about 500 pages. The document is final and won't change. From my understanding so far, I should either choose Langchain or LlamaIndex. But, I will have to use OpenAI api to get best answers, but that API is quite costly for me. So far I have thought of using Chroma for embedding and somhow storing the vectors as pkl or json on streamlit itself for re-use, so I don't have to spend again for vectors/indexing. I don't have enough credits to test different methods myself.
Kindly guide me. Thank you.
hi Sam, I tried to load WizardLM in my free Colab account. But, I tried couple times and it all failed to load. I wonder if I need to have pro or pro+ account to load your examples?
yes probably. I was using Pro+ on this .
Can you pls do vidoe on ConverstaionQA with memory, i have tried, but it is not working as per documents.
Is token size an issue if we want to train over large examples
great !
Are there any examples of LangChain working well to make code changes (or suggesting to make code changes) to medium to large codebases based on a prompt? For example, modify the code to make the given test pass.
This is an interesting idea, I haven't tried it. It might work better with a code model like Starcoder or GPT-4. The bigger models can certainly do unit tests etc.
I have question .does Reterival bring data from vector db then pass it to model generate text
yes it puts it in the context of the prompt
thanks for such a great tutorial. could you show how to do the same with ConversationalRetrievalChain with return_source_documents=True? thank you
absolute beginner q, how does one go about pouring this concoction into a flask?
or rather can a notebook be retrofitted to have REST functionality? I've googled but trying to get the lay of the land before embarking on making the first thing that comes to mind
I tried with this code and it's working fine for the docs that I have given but it's also giving answers for the other questions that details are not present in the docs. so How can I put all things for only my docs and for outer questions it gives something I don't know something like this. Can you please guide me? I have used the WizardLM model.
@Sam Witteveen Do you have any idea about this and can you please guide me?
Is WizardLM an encoder-decoder model? Or just a decoder model? Does this distinction matter? If anybody could get back to me, I would be so appreciative.
If it is the one based on LLaMA then it is a decoder model
Hi Sam! On your opinion, what of this models would perform better on portuguese language? Nice video! Thanks! !
The vector stores will be the same it really comes down to what embeddings you use. Checkout the leaderboard here and you probably want to try some multi lingual models huggingface.co/spaces/mteb/leaderboard
How would you go about updating or removing a document from chroma. Say if a document is updated or is now out of date?
Yes take a look at the API they have functions to delete a doc etc.
@@samwitteveenai thanks 👍 had a ganger and can see how you can delete collections and return the IDs based on a where clause for metadata search, guess you could then take the returned IDs and delete them or update them. Might be worth a video covering this as I can't find a single video on RUclips for this, just a thought!
Can we use these codes in a kivy Python application??
I have been struggling to figure out if there is a performant way to tune and run a 13B model on a modern graphics card with 12gb of vram (I think I have seen the 4-bit mode suggested for this, but thought that the results might be more poor than just using a 7b model). I love the speeds on the device but I know for some stuff I will be bumping right up against the limits of what I can do with a 7b model. I could get and run the model on a dual-GPU setup as I have seen shown is possible by @Jeff Heaton . I just am under the impression. There would be a significant performance hit. I also really like having the model run on the one GPU I have since it is very conservative of power and I could conceivably afford to run it with the model and do some self-hosted services with it. Does anyone have any suggestions? Thank you!
Some new 4bit stuff is coming in a few weeks. That might help you a lot.
Thanks. How about using a local llm ai to find chunks using embeddings and open ai for final step: the chunks tokens and question in input ? Will it be good enought ?
Yes I did that in the previous video before this, it works well.
Why are we using instruction embedding and T5 models which are different?
There is no reason to use the same model for the embeddings and the main LLM. Embedding LLMs are trained for different tasks. Actually Instructor uses a T5 Architecture underneath though (but thats not related to the other LLM)
@@samwitteveenai Thank you so much!!
can I load the models in cpu?
probably but you will need a lot of ram and it will be very slow.
What would you use with any of these to create a UI for the chat? Gradio?
Yeah you could use Gradio or Streamlit or build a something like a NextJS app for a proper frontend
Awesome, thanks for sharing, can run Wizard on CPU?
I haven't done it, but my guess is yes
Which videos are a prerequisite for this one?
Take a look at the LangChain playlist and the few ones before this on that playlist.
thank you :d i wonder how can i increase the answer size? is it possible to get longer answer?
This depends a lot on the model and the prompt we use on it. I plan to make a part 2 to this video this week.
@@samwitteveenai thank you, we are looking forward you :d
why this collab file does not work , when i am trying to run it , it is showing multiple errors
A lot of of the code has been updated, so that could be the reason. I will try to make some newer vids on this topic soon.
Attempting to run the StableVicuna and WizardLM Models and I get the error: Make sure you have enough GPU RAM to fit the quantized model. What GPU/VRAM are you running, and how do I set it as a parameter like 'gpu_memory_0'? Also, is there site which documents all parameter inputs in the transformers? (Sorry.. N00b here in Python).
it sounds like your GPU isn't big enough to run that one.
@@samwitteveenai Thanks. How much VRAM do I need to run those examples? Looking to get another graphics card soon.
Get Sam, what is the licensing like to use these other models for commercial usage?
Some really good like the T5 models, ones like Wizard etc are not for commercial. I a, planning a part 2 to this video which will address this .
How can we finetune a local LLM and can you pls make other video using search engine or other
what can i do if i want to do it with french documents and questions ?
I would use a fine tuned version of the new LLaMA2 model