I am doing tech for 20 years You Sir are an excellent teacher. Pointing to the documentation and providing us pointers to nice tools to stitch it all together in comprehensive way.
Totally, if there's something I don't understand at that moment just rewinding it to couple of mins and then it makes more sense.. one of creators who isn't just walking through the code and saying what the command does in the notebooks Matt is really someone who knows how to empower the other person
Nicely done! It's worth noting that what Matt demonstrated is "Fine-Tuning with LoRA" and not Actual Fine Tuning. Low rank adaptation (LoRA) makes customising a model more accessible than actual fine tuning of a model by "freezing" the original weights and training a small subset of parameters. Actual Fine-Tuning: Adjusts all parameters, requires significant resources, but yields high-quality results. Low rank adaptation (LoRA): Trains fewer parameters using low-rank matrices, reducing memory and compute needs while maintaining quality.
Sorry if unrelated, am I the only one who thinks that Matt’s voice has that soothing-gentle-teacher like voice? Like I can hear him narrate for a natgeo documentary
Thanks great breakdown of the process! A note about JSONL not being an array. It can be processed by old school unix tools like awk, grep, sed - and used in streaming data with unix pipes where lines are the delimiters. These tools don't do well with json array syntax on large datasets.
Matt, your energy is so calm. I did fine tune with MLX, but I freaked myself out with all the steps and feel like it’s hard to do again. When you explain it so nicely, my fear goes away and I’m ready again. You’re spot on about the data prep is the “dark arts”. So true!!
I was totally agreed with you sir, u are the most easier way to let me learn about mlx in the past 2 month I’m keep finding RUclips for all the information. Thank you so much for the video.
Matt, this is utterly awesome and I can't thank you enough. I'd seen the compute resources people were using and the code and gone "that's just too time and money intensive to investigate further". Now, I just need the script from terminator, a code interpreter and, oooh, 5 minutes? Don't worry, I'll keep control of it...
This is a great vid! Especially if you're at least a hobbyist. The best complete layperson onramp I've seen for fine-tuning? Is Cohere. And it is free. After that, dip into more of the dark arts. But to whet folks' whistle I usually point them there. It's their whole biz model. It quickly gets old, but you can whip up a trained, finetuned bot in a half day depending on dataset.
Thanks Matt, your explanations are effective and entertaining. If you could in a future video , would you dive into more detail about fine-tuning? E.g., why would you want to, how to choose your data, etc. Thank you!
Thanks, Matt-super spot-on video as usual. You raised a doubt in my mind: You mentioned that fine-tuning is not suitable for adding new information to the original LLM (perhaps I misunderstood). This leaves me a bit perplexed, and I know it’s a debated issue within the community. I agree with you that the best use of fine-tuning is to personalize the style and tone, rather than being used in the "traditional" way to train older (pre-GPT) models like BERT. However, many people argue that fine-tuning could be an alternative to RAG for injecting specific domain knowledge into the LLM. Personally, I’ve never tried fine-tuning a model due to the costs, especially with cloud-based LLMs. In any case, I think it would be valuable to explore these topics further. My hope is that fine-tuning could become a native feature in Ollama in the future. Lastly, it would have been useful to see the fine-tuning JSONL data (at least an example). I have my own answer to your question: why JSONL? It might be because of its line-by-line simplicity in Unix pipe scripting.
@@technovangelistBy "slower," do you mean that the fine-tuned model has increased latency during inference compared to the original model? That's interesting-I’ve never heard about that before.
Thanks for the great video! Could you use an llm to generate question answer pairs for the dataset out of basic text or documents? Would be interested in such a video!
Thanks a lot for your fantastic videos. I'm actually using Unsloth to fine tune Llama3 for a text classification task. I'll be happy if you'll upload a video for such purposes
I am interested in the idea of fine tuning, and I am starting to regularly come to your videos for stuff now that Ollama is my primary tool I am using to connect to other front end for serving my models. But jeez, i feel like an idiot sometimes with some of this stuff because this still feels complex to me. "Step 1" of curating the data honestly feels like the easy part to me. I am curious about unsloth as its one I have looked at before but had decided to circle back to when I finally worked my way up to fine tuning. I am also curious about vision models, Llava or otherwise, I would be really curious to see how curating data for that would differ from an LLM. Also, would be nice in future videos related to this so see a before and after test. I know we can assumed what you are suggesting is making an effect, but it would still be nice to see the results in action :)
If you could make a guide of installing Unsloth on Windows and train Llama 3.1 on Ollama to use my tone of voice to reply to emails AND have knowledge to answer the common questions I get, that would be a godsent! Thank you for the great videos!
Great video, thanks! I only wonder what's the way to add new data to the model if finetuning is more about changing the format. My use case is like this: I use a framework which changes syntax with each new version I want the LLM to be updated with the newest documentation for the last version. How can I do that?
Very useful, however 1) is there anything we can do with 16 gig on m1? 2) can you maybe show how to do it? Step-by-step: it is a little hard to understand the MLX part thanks
I tried to fine tune llama3.1 on windows since I need nvidia gpu. Such a nightmare. I still didn’t figure out why my llm is not able to speak anymore, it just reply’s data that was in my dataset
ok so i created the three sections, but now get a data formatting error heres a sample of some data of mine: {"prompt": "info.", "response": "info"} what should be different here?
so if finetuning is how you make the model respond in a better style, how do you teach it more stuff? which are the best ways to make the AI learn more aside from RAG
hi, why do i need a validation dataset and a test dataset? can i create them but let them empty? because i don’t understand anyway how validation and test would work for a llm
I like your presentation ...Regarding the fine tuning heard unsloth I have seen many of the AI experts uses it. What is your thought on the same lines?
Thank you Matt for again a great helpful video. It would be great to see samples of the test.jsonl, valid.jsonl and test.jsonl files. Or a template of them. Thanks
thank you Matt, you are amazing. As I know, mlx is for apple silicone, what about finetune on nvidia rtx card? which library should we use ? can you make a video for this please ?
Yup. I mentioned that I did this first for apple silicon. And I intend to do the same thing for unsling and maybe axolotl which are windows and Linux based
Could you do the same thing as fine tuning by creating a RAG database of examples, and just use the off-the-shelf LLMs? This might make your application LLM-agnostic and futureproof. Thoughts on pros/cons?
@@technovangelist Mmhh yes and no. Full-parameters finetuning is OK for adding new knowledge in a more systematic way (after all, it's just continued pretraining..).
jsonl is used to read line by line, it is easier for python, because it is reading line by line as far as i know. I.e. 1 iteration === 1 json from your dataset. Plus datasets are huge and reading whole json and parse it all in one go will take a decade and probably will crash your runtime.
@@technovangelist in fairness, it also allows you to do things without needing to parse the whole file - like cat something.jsonl | sort > sorted.jsonl or cat something.jsonl | head -n 10 > 10somethings.jsonl In this case, I imagine it’s convenient for shuffling the entries, but the main generic advantage over a big json list is that you don’t need to read the entire file to begin parsing it (which is a really nice language-agnostic property for files that might become extremely large) They’re definitely weird at first blush, but actually kinda clever and elegant for some use-cases IMO
sir i can't understand the way dataset jsonl file should be can u pls give me 1 block of the dataset jsonl file as example. i can't understand how to make 'em pls help me else someone out here pls help me too
Still not sure what the data file looks like for the framework? Is it a dataset? the below does not seem to work: "text": "This is the first piece of text."} {"text": "Here is another piece of text."} {"text": "More text data for fine-tuning."}
@@technovangelist Agreed. The thing i learned is that data is the directory where you put train.jsonl test.jsonl and valid.jsonl with the format you describe. Thank you!
If you only can change the style of the answer why bother with fine tuning, I don’t need the answers to be like a pirate , why would you need this for creating an enterprise level application? Is rag the way to go for this ?
Tweaking the style is a very important aspect for most enterprises. Some need it to respond as Sql every time or json or functions. Those don’t need new knowledge but rather told how the model should respond.
Matt, I had written down a long comment, but RUclips deletes anything that has links to platform it doesn't like, probably. If you have time and will, please read my replies to your twitter thread related to this video, thanks.
I also hate jupyter notebooks... Agree that it is the worst for teaching... I always convert it to python file in the end and getting rid of all useless stuff...
I am doing tech for 20 years You Sir are an excellent teacher. Pointing to the documentation and providing us pointers to nice tools to stitch it all together in comprehensive way.
I love the way Matt explains things in a way that is both detailed and yet really easy to understand. Thank you man.
Thanks so much
Totally, if there's something I don't understand at that moment just rewinding it to couple of mins and then it makes more sense.. one of creators who isn't just walking through the code and saying what the command does in the notebooks Matt is really someone who knows how to empower the other person
yes he is the best beast of explanation ❤ 💪 🥇 🔥
Nicely done! It's worth noting that what Matt demonstrated is "Fine-Tuning with LoRA" and not Actual Fine Tuning. Low rank adaptation (LoRA) makes customising a model more accessible than actual fine tuning of a model by "freezing" the original weights and training a small subset of parameters.
Actual Fine-Tuning: Adjusts all parameters, requires significant resources, but yields high-quality results.
Low rank adaptation (LoRA): Trains fewer parameters using low-rank matrices, reducing memory and compute needs while maintaining quality.
Sure. But when most folks talk about fine tuning it’s lora.
I'm glad that ollama went so far ahead and how creating a standards for open-source LLMs, like dockerfile-like specification files and so on.
Sorry if unrelated, am I the only one who thinks that Matt’s voice has that soothing-gentle-teacher like voice? Like I can hear him narrate for a natgeo documentary
Facts
Thanks great breakdown of the process!
A note about JSONL not being an array. It can be processed by old school unix tools like awk, grep, sed - and used in streaming data with unix pipes where lines are the delimiters. These tools don't do well with json array syntax on large datasets.
Matt, your energy is so calm. I did fine tune with MLX, but I freaked myself out with all the steps and feel like it’s hard to do again.
When you explain it so nicely, my fear goes away and I’m ready again.
You’re spot on about the data prep is the “dark arts”. So true!!
Goat in teaching fine tuning
Love it, thanks for sharing. It's great to see LLM fine-tuning become increasingly accessible to more people.
Thanks a lot for this tutorial Matt. It is by far the most straightforward fine tuning tutorial I have ever seen.
i so wish you could explain what loras are an how to do one. thank you for this amazing video i already fee much better
Thanks I tried the mlx fine tune a few months ago. I think this mlx-lm might be more straightforward.
I was totally agreed with you sir, u are the most easier way to let me learn about mlx in the past 2 month I’m keep finding RUclips for all the information. Thank you so much for the video.
Matt, this is utterly awesome and I can't thank you enough. I'd seen the compute resources people were using and the code and gone "that's just too time and money intensive to investigate further".
Now, I just need the script from terminator, a code interpreter and, oooh, 5 minutes?
Don't worry, I'll keep control of it...
This is a great vid! Especially if you're at least a hobbyist.
The best complete layperson onramp I've seen for fine-tuning? Is Cohere. And it is free. After that, dip into more of the dark arts. But to whet folks' whistle I usually point them there. It's their whole biz model. It quickly gets old, but you can whip up a trained, finetuned bot in a half day depending on dataset.
Thanks Matt, your explanations are effective and entertaining.
If you could in a future video , would you dive into more detail about fine-tuning? E.g., why would you want to, how to choose your data, etc. Thank you!
Thanks, Matt-super spot-on video as usual. You raised a doubt in my mind: You mentioned that fine-tuning is not suitable for adding new information to the original LLM (perhaps I misunderstood). This leaves me a bit perplexed, and I know it’s a debated issue within the community. I agree with you that the best use of fine-tuning is to personalize the style and tone, rather than being used in the "traditional" way to train older (pre-GPT) models like BERT. However, many people argue that fine-tuning could be an alternative to RAG for injecting specific domain knowledge into the LLM. Personally, I’ve never tried fine-tuning a model due to the costs, especially with cloud-based LLMs. In any case, I think it would be valuable to explore these topics further.
My hope is that fine-tuning could become a native feature in Ollama in the future.
Lastly, it would have been useful to see the fine-tuning JSONL data (at least an example). I have my own answer to your question: why JSONL? It might be because of its line-by-line simplicity in Unix pipe scripting.
What I read is that you can add knowledge but apparently it makes it slower.
@@technovangelistBy "slower," do you mean that the fine-tuned model has increased latency during inference compared to the original model? That's interesting-I’ve never heard about that before.
Great video! 👍 Thanks
Thanks for the great video! Could you use an llm to generate question answer pairs for the dataset out of basic text or documents? Would be interested in such a video!
Amazing content, I will test it soon 🙏thanks!
dark arts, lol. Love your vids, man.
Clarity and digestability 100%
love your explanations. thank you !
Thanks a lot for your fantastic videos. I'm actually using Unsloth to fine tune Llama3 for a text classification task. I'll be happy if you'll upload a video for such purposes
I am interested in the idea of fine tuning, and I am starting to regularly come to your videos for stuff now that Ollama is my primary tool I am using to connect to other front end for serving my models. But jeez, i feel like an idiot sometimes with some of this stuff because this still feels complex to me. "Step 1" of curating the data honestly feels like the easy part to me.
I am curious about unsloth as its one I have looked at before but had decided to circle back to when I finally worked my way up to fine tuning. I am also curious about vision models, Llava or otherwise, I would be really curious to see how curating data for that would differ from an LLM.
Also, would be nice in future videos related to this so see a before and after test. I know we can assumed what you are suggesting is making an effect, but it would still be nice to see the results in action :)
If you could make a guide of installing Unsloth on Windows and train Llama 3.1 on Ollama to use my tone of voice to reply to emails AND have knowledge to answer the common questions I get, that would be a godsent! Thank you for the great videos!
Great video, thanks!
I only wonder what's the way to add new data to the model if finetuning is more about changing the format.
My use case is like this: I use a framework which changes syntax with each new version I want the LLM to be updated with the newest documentation for the last version.
How can I do that?
Nice. I just did not get when to use the different test and validate files in the process.
Very useful, however
1) is there anything we can do with 16 gig on m1?
2) can you maybe show how to do it? Step-by-step: it is a little hard to understand the MLX part thanks
I tried to fine tune llama3.1 on windows since I need nvidia gpu. Such a nightmare. I still didn’t figure out why my llm is not able to speak anymore, it just reply’s data that was in my dataset
ok so i created the three sections, but now get a data formatting error heres a sample of some data of mine: {"prompt": "info.", "response": "info"} what should be different here?
Use the format I used in the video. The. I show it in the next one. Just a text key.
I forgot to break up my data into the three sections (validate, train, test) will the model work the same?
I need to say thank you I appreciate everything you're teaching us, you make things extremely easy to understand... Thank you.❤
so if finetuning is how you make the model respond in a better style, how do you teach it more stuff? which are the best ways to make the AI learn more aside from RAG
hi, why do i need a validation dataset and a test dataset? can i create them but let them empty? because i don’t understand anyway how validation and test would work for a llm
I like your presentation ...Regarding the fine tuning heard unsloth I have seen many of the AI experts uses it. What is your thought on the same lines?
I didn’t use it because of the limited hw support. But I will next time
Would love to see you try out InstructLab
Thank you Matt for again a great helpful video. It would be great to see samples of the test.jsonl, valid.jsonl and test.jsonl files. Or a template of them. Thanks
thank you Matt, you are amazing. As I know, mlx is for apple silicone, what about finetune on nvidia rtx card? which library should we use ? can you make a video for this please ?
Yup. I mentioned that I did this first for apple silicon. And I intend to do the same thing for unsling and maybe axolotl which are windows and Linux based
amazing video matt thank you
Does unsloth worth that hype?
parece até fácil olhando !
tks
I need to understand how json works
Thanks you for this
Thanks for the comment
Example for datasets?
Could you do the same thing as fine tuning by creating a RAG database of examples, and just use the off-the-shelf LLMs? This might make your application LLM-agnostic and futureproof. Thoughts on pros/cons?
Fine tuning and rag have different purposes. Rag adds new knowledge whereas fine tuning will mostly affect the way it outputs.
@@technovangelist Mmhh yes and no. Full-parameters finetuning is OK for adding new knowledge in a more systematic way (after all, it's just continued pretraining..).
jsonl is used to read line by line, it is easier for python, because it is reading line by line as far as i know. I.e. 1 iteration === 1 json from your dataset. Plus datasets are huge and reading whole json and parse it all in one go will take a decade and probably will crash your runtime.
Ahhh, ok. So it’s accommodating for the weaknesses of Python.
@@technovangelist in fairness, it also allows you to do things without needing to parse the whole file - like cat something.jsonl | sort > sorted.jsonl or cat something.jsonl | head -n 10 > 10somethings.jsonl
In this case, I imagine it’s convenient for shuffling the entries, but the main generic advantage over a big json list is that you don’t need to read the entire file to begin parsing it (which is a really nice language-agnostic property for files that might become extremely large)
They’re definitely weird at first blush, but actually kinda clever and elegant for some use-cases IMO
sir i can't understand the way dataset jsonl file should be can u pls give me 1 block of the dataset jsonl file as example. i can't understand how to make 'em pls help me else someone out here pls help me too
Take a look at the second video on this
@@technovangelist which one sir "optimize your AI models" else which one sir can u pls say sir 🙇🛐
The other one with fine tuning in the name
@@technovangelist ok sir thanks a lot
Thanks for the awesome video! Would you know anything about using ubuntu/intel arc with ollama. If so would you consider a guide one beautiful day :)
I haven’t played with any of the arc cards yet. I need to find a way to play with those
Still not sure what the data file looks like for the framework? Is it a dataset? the below does not seem to work:
"text": "This is the first piece of text."}
{"text": "Here is another piece of text."}
{"text": "More text data for fine-tuning."}
Yup that’s what I showed in the video. Well, except you missed the first bracket
@@technovangelist Agreed. The thing i learned is that data is the directory where you put train.jsonl test.jsonl and valid.jsonl with the format you describe. Thank you!
Running fine-tuning is easy, but getting LLM to do what you are fine-tuning for may be not so easy and at times even very hard.
mmmistral 😁
Finetune with crosswords so the llm know how many characters in his response
Thanks! I can’t tell you how much I dislike Jupyter notebooks.
If you only can change the style of the answer why bother with fine tuning, I don’t need the answers to be like a pirate , why would you need this for creating an enterprise level application? Is rag the way to go for this ?
Tweaking the style is a very important aspect for most enterprises. Some need it to respond as Sql every time or json or functions. Those don’t need new knowledge but rather told how the model should respond.
Matt, I had written down a long comment, but RUclips deletes anything that has links to platform it doesn't like, probably. If you have time and will, please read my replies to your twitter thread related to this video, thanks.
Mmmmmmmmistral hahaha 😂👏
I also hate jupyter notebooks... Agree that it is the worst for teaching... I always convert it to python file in the end and getting rid of all useless stuff...
Python Notebooks have to be the worst format for teaching - Amen to that!
Yeah this is all gibberish. Convert your scripts json what?!