Видео

I think it should be the n_ctx parameter e.g. llm = llama_cppLlama(model_path="./models/7B/llama-model.gguf", n_ctx=2048)

Cool idea! Thanks - I'll have to give that a try. Why do we need function calling though - is that to make sure that we get back just a query rather than the LLM going off and writing a bunch of other text?

@@learndatawithmark Exactly. Because they are guaranteeing the query.

You are a tough one to please! 😄

@@learndatawithmark Sorry ... Felt most ware cosmetic... a bit of fluff.. However i later saw your PIVOT and UNPIVOT videos... and they were really excellent. Both in terms of DuckDb syntax /features and also your example/content. Thanks for the videos. New subscriber. cheers.

No need to do that batching - I recorded it all and realised that it was a bit pointless afterwards. I do want to play around and see whether at some sort of batch size it gets slower. Good question about doing it in parallel though. I dunno whether the API supports it, but if it doesn't, the model is tiny, so we could just spin it up a bunch of times and call each in parallel?

When do you get that error"

@@learndatawithmark on converting model stage, after run "ollama create ..."

Figured out the problem. The problem was that I made fine-tuning on quantized model, and it is not good)

I'm using Auzre OpenAi and put my paramter in .env shall i do anything else?

What error do you see?

As far as I know, this is still the way to run GGUF models with Ollama. I wish you could use GGUF files directly, it would be so much easier! I haven't used Open WebUI, I'll take a look at that. If you want command line tools that can run GGUF files directly, take a look at llamafile or llama.cpp github.com/ggerganov/llama.cpp github.com/Mozilla-Ocho/llamafile

​@@learndatawithmark Thanks for answering. Indeed, my interest is in running them on Ollama, due to the new Open WebUI, which is the most marvelous thing invented. Open WebUI is a frontend to Ollama, presenting an interface like Chatgpt, with history of the conversations, talks with LLMs completely hands free, you talk and listen, and you can input your texts, PDFs, RAG, makes LLMs access internet in real time, upload images, multimodality, it's fantastic. You definitely need to test it. The problem with it is that it's based on Ollama, so, I use LM Studio, with dozens of Hugging face models, and I love them. I would like these models to be used in Open WebUI, but they are in GGUF format, that's why I found your video, in order to use gguf models in Ollama. 🙏👍💥

Sorry - my mistake, I forgot to add it. It's here - github.com/mneedham/LearnDataWithMark/blob/main/llamacpp-rag/utils.py

Thanks - glad it helped!

If it's running too slowly then maybe it'd make sense to try out some of the quantised models instead. Those ones are smaller and better suited for running on consumer hardware. I quite like Ollama and I've made a few videos on that. This is probably the best place to start - ruclips.net/video/NFgEgqua-fg/видео.html

We can! This would be hybrid search which I need to read a bit more about. Once I've done that, I'll make another video - thanks for the idea :)

You'd use Full-Text Search (FTS) if you were searching for specific terms that you know are in the text. But it will only find those terms or any variants close to those words. So if you know exactly what you're looking for it should (in theory) work better than doing vector search, where you'd be matching an embedding of your search query/keywords against embeddings of paragraphs of text. Of course this technique won't find other bits of text that have the same semantic meaning. But your question makes me curious to compare the two techniques on the same dataset to see if/how the query results differ.

Should be -1 rather than 1. Then it will put as many layers as possible to the GPU rather than CPU.

@@learndatawithmark Yep, silly mistake by me. Could you make a video recapping the different frameworks/libraries for LLMs inference and their pros and cons? Like HF transformers VS Llama-cpp-python VS Ollama

not heard of that one - let me look into it, good idea!

haha, a long way to go :D

Great! Glad it was useful - let me know if there are any other topics in this area you'd like me to cover next.

What happens when you run the command?

@@learndatawithmark Thank you so much for quick response. Your videos have helped me a lot. I am running this on windows 11. I did follow steps: 1) created Modelfile with this script: "FROM C:\Users\PARTH\Downloads\text-generation-webui-main \text-generation-webui-main\models\TheBloke_dolphin-2.7- mixtral-8x7b-AWQQ TEMPLATE = """{{ if .System }}<|im_start|>system {{ .System }}<|im_end|>{{ end }} <|im_start|>user {{ .Prompt }}<|im_end|> <|im_start|>assistant """ PARAMETER stop <|start_header_id|> PARAMETER stop <|end_header_id|> PARAMETER stop <|eot_id|>" 2) I ran following command on terminal opend from where this modelfile is stored. "ollama create mixtral:dolphine -f .\Modelfile" 3)It showed me this error: "Error: command must be one of "from", "license", "template", "system", "adapter", "parameter", or "message"" 4) I only made file with FROM statement without parameter and template, It ran, but gave this error: "C:\Users\PARTH\.ollama>ollama create mixtral:dolphine -f .\Modelfile transferring model data unpacking model metadata processing tensors Error: couldn't find a layer name for 'model.layers.0.block_sparse_moe.experts.0.w1.qweight'" 5) I ran again with another models, but gave same error: "C:\Users\PARTH\.ollama>ollama create slimorca:13b -f .\Modelfile transferring model data unpacking model metadata processing tensors Error: couldn't find a layer name for 'model.layers.0.mlp.down_proj.qweight'"

Ah you're right - I missed that!

1. It was at the time I made the video - but I think it'll now work if you do brew install llama.cpp 2. No it's not - I had that on my machine already. If you've already compiled llama.cpp, you don't need to do 1. 3. What's the next one?!

@@learndatawithmark Oh ... nice. Well than the brew install only installs the hugging-face cli? How can be assured that models for ollama, llama.cpp an open-webui only are stored once? Is linked folder enough? Which strategy to adopt for not having the same models several times on disk? In which formats one has to reformat the models for silicon Macs? How can one use Apple neural engine? Just some 😉

It's not actually a Hugging Face CLI - it's a separate project - but yeh it downloads a CLI tool. There are then two tools - llama-cli and llama-server. The first is a command line chat interface and the second starts a server. > How can be assured that models for ollama, llama.cpp an open-webui only are stored once? Ollama is kind of a pain IMO as you can't point it directly at GGUF fails AFAIK. Instead you have to convert it from GGUF to the Ollama format. If you use llama.cpp or llamafile you can use the same GGUF files with no duplication. > In which formats one has to reformat the models for silicon Macs? llama.cpp has support for running files on silicon Macs (I'm using an M1 Max from 2021) so you don't need to do anything there > How can one use Apple neural engine? AFAIK for doing this you seem to need different models. This blog is pretty good on that - medium.com/@manuelescobar-dev/running-large-language-models-llama-3-on-apple-silicon-with-apples-mlx-framework-4f4ee6e15f31

@@learndatawithmark Thanks for sharing 👍

I'm using a Jupyter Notebook, but the code would work in any Python environment or script jupyter.org/

@@learndatawithmark ah thanks, that’s interesting - I’ve never heard of that

values = [1.500, 800.00] [int(value) for value in values]

If it's fully blown models, I think Hugging Face's inference server is best - github.com/huggingface/text-generation-inference If it's quantized models, llamafile are doing some cool work to make a super fast server - github.com/Mozilla-Ocho/llamafile

I think it's the same thing under the hood - no need to change from your approach!

Latest video inspired by your comment :D Here we go - ruclips.net/video/gigip1Pxf88/видео.html

@@learndatawithmark Amazing, I am also implementing local RAG with llama-cpp-python, so the video is completely spot on ahah

That's alright - glad they were useful :)

I'm using a Mac M1 Max (64 GB RAM) with 10 CPUs. It shares the RAM between CPU and GPU.

@@learndatawithmark what about the GPU core count? a 10 core CPU M1 Max can be configured with 16, 24, 32 GPU cores. I believe the GPU is the most important part for AI applications... BTW, i went ahead and bought a *specced out* 14 inch M3 Max with 2 TB SSD.

@@mo5168 32 apparently! Chipset Model: Apple M1 Max Type: GPU Bus: Built-In Total Number of Cores: 32 Vendor: Apple (0x106b) Metal Support: Metal 3

@@learndatawithmarkthank you!

Yes you should be able to use a similar approach. There's a good guide on image to text over here - huggingface.co/tasks/image-text-to-text

I think to use a smaller model for a specific task we might want to fine tune something ourself with a bunch of images and the expected output. I read that PaliGemma is supposed to be a good base image for that, but I haven't tried fine tuning it myself - huggingface.co/google/paligemma-3b-mix-448

At the moment they don't seem to support the function calling API but I imagine they will at some point. AFAIK Ollama uses llama.cpp for a lot of stuff, so they should be able to use this feature too

I've played with Langroid and CrewAI and neither worked that well. I haven't tried AutoGen or LangGrfaph yet, but I guess I should get on it!

Yeh this is a bit hardcoded for a single function. The arguments are in that function_args variable, so you could pass them in using the kwargs syntax. Maybe I'll make another video to show how to do that.