Thanks, Maarteen. I wish you could share some performance comparison between the difference methods. I have been trying to find some but I couldn't. I do know that AWQ is better than GPTQ, but wish to compare it to GGUF.
Thanks for including the colab, and I wasn't aware of AWQ before this video. Would you consider making a video on the efficiency on each, especially when using gpu on gguf model?
RuntimeError: cutlassF: no kernel found to launch! - it what I'm getting when trying to run your example at step 4: outputs = pipe( prompt, max_new_tokens=256, do_sample=True, temperature=0.1, top_p=0.95 )
Brilliant video; you have a style that explains things nicely. Thank you. Sub'd. If you are looking for ideas, I think an overview of what "weights, biases and parameters" mean for models would be great.
Thanks didn't want to feel too unproductive on a thanksgiving but didn't want to commit to a full video series. Always releasing timely and great stuff!
Thanks for this video - was a great explanation on the difference between the three models. How's the support for AWQ now? Also I would love it if you could make a video on how to deploy these quantized models for production
Thanks for the video! What I don't understand is that people always say that AWQ is faster than GPTQ, but in my 3060 12gb they are usually quite slow, around 3t/s, while in gptq I can get from 5 to 20t/s
inference is tooo slow on the T4 gpu on collab, i fed a football commentary transcribe text line to llm the pre quantized one, it took 3 minutes to obtain the result
Outstanding! (To put things in perspective: I've seen a LOT of praise for wrapping the obvious or marketing-only BS into lengthy videos and I'm not shy to speak my mind there too!)
Great content! I am wondering whether nowadays we should choose LLMs over BERT models on most tasks or use seperately based on specific use cases? That could be an interest topic to discuss!
Thank you for the great explanations :) Does it make sense to do this before trainning? Quantize the model with these tecniques befor doing peft, qlora, p-tunning etc?
It definitely helps with training if the full model does not fit on the GPU. With many of these methods, efficiency is important and quantization is seldom not used.
My understanding was GPTQ is recommended when quantized model can fit entirely in vRAM and that with GGUF one can still offload layers to a GPU. If I wanted to try mixtral dolphin ~40GB version (On a system w/4080 16GB VRAM, 64GB RAM) what would be a better choice GPTQ or GGUF?
Definitely GGUF. Mixtral 4bit doesnt dit within 16GB Vram, so offloading layers would be necessary. I remember you could offload around 20 layers if I'm not mistaken. I do think the quantised variants, including the dolphin fine tune, are worth checking out.
Well what about the title of the video? I still don't know wich one is right for me wich was the point of watching this video. All you did was explain what each method is, not wich one we should use :s You have good information here in this video but still you missed the entire point of the title in my opinion.
well nevermind, I paused the video cause it was about to finish and go over to another on the playlist, and guess you do talk about it in the end, my bad. Thats what I get for not finishing the video before coming with stupid questions.
@@kiiikoooPT well i came here for similar reasons and although this is indeed very well explained, i still miss something i was hoping for (kinda like you): which quant works best for what mode? as i have read the other day that for example gptq is less suitable for rp than exl2. why? i dont know. thats why i came here. do the modes chat, cai and instruct have their preferred quantizations? if so, why? im leaving without those answers but with further base knowledge, which is more important in the long run. but hey.. there is one answer i was interested in: why do some models assume both the character- and the user role and progress the rp-story themselves instead of letting me guidie it? because now i know thats sort of typical for the awq quant
from the litle I understand, the fact that a model prompt is diferent from model to model, has nothing to do with it being awq or gguf or whatever type off file, that is just about loading the model with diferent loaders. what you are talking about is another topic, you need to see videos about something like mistral vs mistral instruct. or diference betwen those. cause what you are saying has to do with how the model learned stuff, or how it was trained. the instruct models like the name says, are models that are based on instructions, so your prompt cannot be as a role play, cause it will give step by step anwsers instead of a simple conversation. What I really wanted to know is what kind of file is better for low end hardware, since my laptop has an nvidea graphic card with cuda cores, but is so hold that there is not even up to date drivers for it, so I can't install the stupid pytorch with cuda, only cpu mode, and I thought the type of file, gguf and so on, had diferent ways of loading the models, so I could manage to load it without the libraries everyone is using cause they have recent hardware. @@CitizUnReal
Amazing content! Most youtube tutorials just go into trying out the outputs of pre-made LLMs but rarely dive into this level of technical details.
Thanks!
Great video❤
Thanks Maarten! I was searching for quantizing exactly the zephyr-7b-beta and I realized you used it halfway in the video!
Thanks, Maarteen. I wish you could share some performance comparison between the difference methods. I have been trying to find some but I couldn't. I do know that AWQ is better than GPTQ, but wish to compare it to GGUF.
Thanks a lot for clarifying the main differences between quantization methods and also for sharing your code.
Thanks
Bfloat 16-bit is error 1+8+7 format.
Thanks for including the colab, and I wasn't aware of AWQ before this video.
Would you consider making a video on the efficiency on each, especially when using gpu on gguf model?
Useful information and well made video.
what about exllamav2?
RuntimeError: cutlassF: no kernel found to launch! - it what I'm getting when trying to run your example at step 4:
outputs = pipe(
prompt,
max_new_tokens=256,
do_sample=True,
temperature=0.1,
top_p=0.95
)
Brilliant video; you have a style that explains things nicely. Thank you. Sub'd.
If you are looking for ideas, I think an overview of what "weights, biases and parameters" mean for models would be great.
Thanks didn't want to feel too unproductive on a thanksgiving but didn't want to commit to a full video series. Always releasing timely and great stuff!
Thanks for this video - was a great explanation on the difference between the three models. How's the support for AWQ now? Also I would love it if you could make a video on how to deploy these quantized models for production
Thanks for the video! What I don't understand is that people always say that AWQ is faster than GPTQ, but in my 3060 12gb they are usually quite slow, around 3t/s, while in gptq I can get from 5 to 20t/s
Really enjoyed your video. It was very informative. Just wanted to know can finetuning be done on these pre-quantized models ??
inference is tooo slow on the T4 gpu on collab, i fed a football commentary transcribe text line to llm the pre quantized one, it took 3 minutes to obtain the result
Thank you for the informative video. I understood how I made a huge mistake using gguf when having the VRAM to use GPU primarily.
Outstanding! (To put things in perspective: I've seen a LOT of praise for wrapping the obvious or marketing-only BS into lengthy videos and I'm not shy to speak my mind there too!)
Great video, how ever its quite frustrating trying to run this code in production the dependencies are never correct.
Hi, WHich one is faster ?
GGUF with CUDA or AWQ ?
Thank you so much for the video, i would like to know which method is faster at inference time.
Great video and great comparisons. Can you make a video on how to quantize a model oneself as well?
Great content! I am wondering whether nowadays we should choose LLMs over BERT models on most tasks or use seperately based on specific use cases? That could be an interest topic to discuss!
Really enjoyed this session! Any chance you can continue this by showing how to fine-tune this versions of the models?
I was struggling with quantization last weekend! very timely! thanks
Thank you for the great explanations :)
Does it make sense to do this before trainning? Quantize the model with these tecniques befor doing peft, qlora, p-tunning etc?
It definitely helps with training if the full model does not fit on the GPU. With many of these methods, efficiency is important and quantization is seldom not used.
Thank you for the differences and the code.
My understanding was GPTQ is recommended when quantized model can fit entirely in vRAM and that with GGUF one can still offload layers to a GPU. If I wanted to try mixtral dolphin ~40GB version (On a system w/4080 16GB VRAM, 64GB RAM) what would be a better choice GPTQ or GGUF?
Definitely GGUF. Mixtral 4bit doesnt dit within 16GB Vram, so offloading layers would be necessary. I remember you could offload around 20 layers if I'm not mistaken. I do think the quantised variants, including the dolphin fine tune, are worth checking out.
@@MaartenGrootendorst Thanks!
Well what about the title of the video? I still don't know wich one is right for me wich was the point of watching this video. All you did was explain what each method is, not wich one we should use :s
You have good information here in this video but still you missed the entire point of the title in my opinion.
well nevermind, I paused the video cause it was about to finish and go over to another on the playlist, and guess you do talk about it in the end, my bad. Thats what I get for not finishing the video before coming with stupid questions.
@@kiiikoooPT well i came here for similar reasons and although this is indeed very well explained, i still miss something i was hoping for (kinda like you): which quant works best for what mode? as i have read the other day that for example gptq is less suitable for rp than exl2. why? i dont know. thats why i came here. do the modes chat, cai and instruct have their preferred quantizations? if so, why?
im leaving without those answers but with further base knowledge, which is more important in the long run. but hey.. there is one answer i was interested in: why do some models assume both the character- and the user role and progress the rp-story themselves instead of letting me guidie it? because now i know thats sort of typical for the awq quant
from the litle I understand, the fact that a model prompt is diferent from model to model, has nothing to do with it being awq or gguf or whatever type off file, that is just about loading the model with diferent loaders. what you are talking about is another topic, you need to see videos about something like mistral vs mistral instruct. or diference betwen those. cause what you are saying has to do with how the model learned stuff, or how it was trained. the instruct models like the name says, are models that are based on instructions, so your prompt cannot be as a role play, cause it will give step by step anwsers instead of a simple conversation. What I really wanted to know is what kind of file is better for low end hardware, since my laptop has an nvidea graphic card with cuda cores, but is so hold that there is not even up to date drivers for it, so I can't install the stupid pytorch with cuda, only cpu mode, and I thought the type of file, gguf and so on, had diferent ways of loading the models, so I could manage to load it without the libraries everyone is using cause they have recent hardware. @@CitizUnReal
Good one..