Quantize any LLM with GGUF and Llama.cpp
HTML-код
- Опубликовано: 22 авг 2024
- In this tutorial, I dive deep into the cutting-edge technique of quantizing Large Language Models (LLMs) using the powerful llama.cpp tool. I'll guide you through the entire process of converting any LLM to GGUF format, a revolutionary approach that enables efficient inference on both CPUs and consumer GPUs. 🖥️💡
🔍 What You'll Learn:
The basics of LLM quantization and why it's important for making AI more accessible.
A step-by-step walkthrough on using llama.cpp for model conversion.
How to utilize the GGUF format to run your LLMs efficiently on different hardware.
Exclusive tips on pushing your 4-bit quantized model to Hugging Face, making it available for the global AI community.
👨💻 Who This Is For:
AI enthusiasts looking to optimize their models for better performance.
Developers seeking to deploy LLMs on various hardware platforms.
Anyone curious about the latest advancements in AI model efficiency.
🛠️ Tools and Platforms Used:
Google Colab for hands-on coding and model conversion.
Hugging Face for model sharing and community engagement.
By the end of this video, you'll be able to optimize your models for enhanced accessibility and performance, regardless of the hardware at your disposal.
👍 If you find this video helpful, please hit the Like button, as it helps me reach more people with this content. Don't forget to Subscribe for more tutorials like this and hit the Bell icon to get notified every time I upload a new video.
Let's embark on this exciting journey together and unlock the full potential of AI, making it more efficient and accessible to all. Thank you for watching, and let's innovate together!
GitHub Repo: github.com/AIA...
Join this channel to get access to perks:
/ @aianytime
To further support the channel, you can contribute via the following methods:
Bitcoin Address: 32zhmo5T9jvu8gJDGW3LTuKBM1KPMHoCsW
UPI: sonu1000raw@ybl
#ai #llm #generativeai
![Eminem - Somebody Save Me (feat. Jelly Roll) [Official Music Video]](http://i.ytimg.com/vi/Vwa0HenQMi4/mqdefault.jpg)
One of the best videos on quantization, thanks a lot for this great upload and content.
@AIAnytime your content is excellent, thank you very much for taking the time to provide us with content that actually helps to advance our Ai knowledge and understanding.
Glad it was helpful!
- Process of converting any LLM to GGUF format that we'll be able to rent on CPU.
- You can do this task even on T4 or even on a CPU machine as well because I'm going to take a small LLM and then do that. (0:02:00)
- Now for quantization you can quantize up to two bits. (0:06:00)
i reached out to you, how come you ignored me?
Hahaha...
lol just use some llm if you have qs
спасибо большое мужик
thank you can i have the colab file ?
GitHub repo is in description
it says, methods is not defined
Why did you first convert to FP16 and not directly to Q4_K_M?
How can I host this model on llam.cpp server
hi, I have been watching your videos for quite some time now. found them very useful and appreciate your effort and dedication to come with great videos. Just a question on GPU's. Now that most of the Laptops and desktops come with AMD Graphics card will they work well for Quantized models, your inputs please
for m in methods:
qtype = f"{quantized_path}/{m.upper()}.gguf"
os.system("./llama.cpp/quantize "+quantized_path+"/FP16.gguf "+qtype+" "+m)
when i run this line nothing happen :(
same happen to me, anyone can help?
Did you figure out this issue.??
I am facing the same issue.
@@soumyaneelsarkar2901 I dont really know where I am wrong but when I use his .ipynb file from the github it work just fine
@@soumyaneelsarkar2901 @weka5286 Check whether you have a GPU connection established. I was facing the same issue, but connecting to a GPU in Colab (T4) did the trick
@@soumyaneelsarkar2901 was it solved?
Hey, I tried to use this on starcoder-7B, but I got an error while converting FP16.gguf to our quant type. Can you please help?
In llama.cpp there is no executable filed named as quantize . So when we run this code :
for m in methods:
qtype = f"{quantized_path}/{m.upper()}.gguf"
os.system("./llama.cpp/quantize "+quantized_path+"/FP16.gguf "+qtype+" "+m)
Nothing happens and the Q4_K_M.gguf file is not being created ..
Any solutions ???????????
did you got the solution bro?
I agree. Excellent.
Last part of the vidoe you told that we have to make changes for other models...Can u specify where should we change?...i dont mean the basic things like model name and stuff...
thanks a lot Sir, what if i use pre-quantized LLM and fine tuned for specific purpose , is we need to quantized after fine tuning
i wanna learn about these in depth, can u tell me the sources pls?
i am getting assertion error while executing (!python llama.cpp/convert-hf-to-gguf.py ./original_model/ --outtype f16 --outfile ./quantized_model/FP16.gguf
) can you please help me with this
did you find a solution bro?
Martinez Sharon Harris Jeffrey Anderson Anthony
can i quantize llama 3 using it of t4 colab?
Does this notebook work for my fine tunned Pegasus model ?
You should focus on doing tutorials how to run everything locally. Does anyone really want to upload their data to this cloud providers?
Where do you see cloud providers in this case?
@@AIAnytime It is hard for an AI noob to follow along not using google colab.
Link?
GitHub repo is in description.
Nice one
Can I do this on my local cpu machine?
You can
@AIAnytime can you suggest the changes neded for T4 GPU to run on Colab
The github repo is not opening . Can you check once ?
Could see. It was private. Made it public now.
Bro, I want to fine-tune a model for a translation task. However, I encountered a ‘CUDA out of memory’ error. Now, I plan to purchase a GPU from AWS ec2 instance. How is the payment processed in AWS? They asked for card details when I signed up. Do they automatically process the payment?
Are you using lora or any peft technique for fine-tuning ? Cos usually using this you can fine tune your model on a free colab t4 gpu
But lora and qlora doesn't giving a better result when it comes in my translation task. @@dossantos4415
I want to use the p3.8xlarge instance for its 64 GB GPU memory. However, it has 4 GPUs, each with 16 GB of memory, totaling 64 GB. So, my question is: Can I use the entire 64 GB to train my LLM model, or is only 16 GB available for training?@@dossantos4415
Please make video on how to use llama factory on TPU
Llama factory video is on the channel already. TPU not recommended.
@@AIAnytime is TPU not recommended for Fine Tuning? If Yes then what is the reason ?
awesome
convert the original llm to gguf, are we losing any data during compression and any accuracy degradation?
Performance tradeoff is common.
@@AIAnytime Thank you.. Waiting for any Knowledge graph usecase and implementations
sir please provide your code with every video so that we can experiment too
GitHub repo is in description...
Thanks you so much can you share colab file
GitHub repo is in description.
Hey how do I get into contact with you?
LinkedIn? Sonu Kumar
@@AIAnytime I added you; Parker Settle. I am working on a project for a big client, and could use your help -thanks!