Efficient Fine-Tuning for Llama-v2-7b on a Single GPU

The first problem you’re likely to encounter when fine-tuning an LLM is the “host out of memory” error. It’s more difficult for fine-tuning the 7B parameter Llama-2 model which requires more memory. In this talk, we are having Piero Molino and Travis Addair from the open-source Ludwig project to show you how to tackle this problem.
The good news is that, with an optimized LLM training framework like Ludwig.ai, you can get the host memory overhead back down to a more reasonable host memory even when training on multiple GPUs.
In this hands-on workshop, we‘ll discuss the unique challenges in finetuning LLMs and show you how you can tackle these challenges with open-source tools through a demo.
By the end of this session, attendees will understand:
- How to fine-tune LLMs like Llama-2-7b on a single GPU
- Techniques like parameter efficient tuning and quantization, and how they can help
- How to train a 7b param model on a single T4 GPU (QLoRA)
- How to deploy tuned models like Llama-2 to production
- Continued training with RLHF
- How to use RAG to do question answering with trained LLMs
This session will equip ML engineers to unlock the capabilities of LLMs like Llama-2 on for their own projects.
This event is inspired by DeepLearning.AI’s GenAI short courses, created in collaboration with AI companies across the globe. Our courses help you learn new skills, tools, and concepts efficiently within 1 hour.
www.deeplearning.ai/short-cou...
Here is the link to the notebook used in the workshop:
pbase.ai/FineTuneLlama
Speakers:
Piero Molino, Co-founder and CEO of Predibase
/ pieromolino
Travis Addair, Co-founder and CTO of Predibase
/ travisaddair

[@thelinuxkid]

Very helpful! Already trained llama-2 with custom classifications using the cookbook. Thanks!

[@dinupavithran]

Very informative. Direct and to the point content in a easily understandable presentation.

[@craigrichards5472]

Amazing, can’t wait to play and train my first model 🎉

[@Ev3ntHorizon]

Excellent coverage, thankyou.

[@manojselvakumar4262]

Great content, well presented!

[@karanjakhar]

Really helpful. Thank you 👍

[@msfasha]

Clear and informative, thanx.

[@Ay-fj6xf]

Great video, thank you!

[@andres.yodars]

One of the most complete videos. Must watch

[@thedelicatecook2]

Well this was simply excellent, thank you 🙏🏻

[@jirikosek3714]

Great job, thumbs up!

[@nguyenanhnguyen7658]

Very helpful. Thanks.

[@tomhavy]

Thank you!

[@ab8891]

Excellent xtal clear surgery on GPU VRAM utilization...

[@goelnikhils]

Amazing Content of fine tuning LLM

[@KarimMarbouh]

🖖alignement by sectoring hyperparameters in behaviour, nice one

[@rajgothi2633]

amazing video

[@rgeromegnace]

Eh, c'était super. Merci beaucoup!

[@bachbouch]

Amazing ❤

[@hemanth8195]

Thankyou

[@ggm4857]

I like to kindly request @DeepLearningAI to prepare such hands-on workshop on fine-tunning Source Code Models.

[@Deeplearningai]

Don't miss our short course on the subject! www.deeplearning.ai/short-courses/finetuning-large-language-models/

[@ggm4857]

@@Deeplearningai , Wow thanks.

[@nekro9t2]

Please can you provide a link to the slides?

[@pickaxe-support]

Cool video. If I want to fine-tune it on a single specific tassk (keyword extraction), should I first train an instruction-tuned model, and then train that on my specific task? Or mix the datasets together?

[@shubhramishra8698]

also working on keyword extraction! I was wondering if you'd had any success fine tuning?

[@zubairdotnet]

Nvidia H100 GPU on Lambda labs is just $2/hr, I am using it for past few months unlike $12.29/hr on AWS as shown in the slide.
I get it, it's still not cheap but just worth mentioning here

[@pieromolino_pb]

You are right, we reported the AWS price there as it's hte most popular option and it was not practical to show all the pricing of all the vendors. But yes you can get them for cheaper elsewhere like from Lambda, thanks for pointing it out

[@rankun203]

Last time I tried it, H100s are out of stock on Lambda

[@zubairdotnet]

@@rankun203 They are available only in specific region mine is in Utah, I don't think they have expanded it plus there is no storage available in this region meaning if you shut down your instance, all data is lost

[@Abraham_writes_random_code]

together AI is $1.4/hr on your own fine tuned model :)

[@PieroMolino]

@@Abraham_writes_random_code Predibase is cheaper than that

[@TheGargalon]

And I was under the delusion that I would be able to fine-tune the 70B param model on my 4090. Oh well...

[@iukeay]

I got a 40b model working on a 4090

[@TheGargalon]

@@iukeay Did you fine tune it, or just inference?

[@ahsanulhaque4811]

70B param? hahaha.

[@ggm4857]

Hello everyone, I would be so happy if the recorded video have caption/subtitles.

[@kaifeekhan_25]

Right

[@dmf500]

it does, you just have to enable it! 😂

[@kaifeekhan_25]

​@@dmf500now it is enabled😂

[@ayushyadav-bm2to]

What's the music in the beginning, can't shake it off

[@arjunaaround4013]

❤❤❤

[@nminhptnk]

I ran Colab T4 and still got into “RuntimeError: CUDA Out of memory”. Any thing else I can do please?

[@stalinamirtharaj1353]

@pieromolino_pb -Is Ludwig allows to locally download and deploy the fine-tuned model?

[@PickaxeAI]

at 51:30 he says don't repeat the same prompt in the training data. What if I am fine-tuning the model on a single task but with thousands of different inputs for the same prompt?

[@brandtbeal880]

It will cause overfitting. It would be similar to training an image classifier with a 1000 pictures of roses and only one lilly, then asking it to predict both classes with good accuracy. You want the data to have a normal distribution around your problem space.

[@satyamgupta2182]

@PickaxeAI Did you come across a solution for this?

[@manojselvakumar4262]

Can you give an example for the task? I'm trying to understand in what situation you'd need different completions for the same prompt

[@kevinehsani3358]

epochs=3, since we are fine tunning, would epochs=1 would suffice?

[@pieromolino_pb]

It really depends on the dataset. Ludwig has also an early stopping mechanism where you can specify the number of epochs (or steps) without improvement before stopping, so you could set epochs to a relatively large number and have the early stopping take care of not wasting compute time

[@feysalmustak9604]

How long did the entire training process take?

[@edwardduda4222]

Depends on your hardware, dataset, and hyper parameters you’re manipulating. The training process is the longest phase in developing a model.

[@Neberheim]

This seems to make a case for Apple Silicon for training. The M3 Max performs close to an RTX 3080, but with access to up to 192GB of memory.

[@ahsanulhaque4811]

Did you try on Apple silicon M1 Max?

[@leepro]

Cool! ❤

[@mohammadrezagh4881]

when I run the code in Perform Inference, I frequently receive ValueError: If `eos_token_id` is defined, make sure that `pad_token_id` is defined.
what should I do?

[@arnavgrg]

This is now fixed on Ludwig master!

[@SDAravind]

can you share the slide, please?