Видео

I use Llama3.2 for most of my work. I run this on a machine with a GPU, answers are normally 1 - 4 seconds. There are hundreds of models and sizes to choose from with Ollama

@@fastandsimpledevelopment Thanks for the heads up, gives me confidence that I might have already found one to stick with. That being llama3.1-python (as I'll using it as a .py assistant); Its quicker and doesnt make me read pages every time :)

I use the same exact code, no difference. I've moved to Terraform vault for my new projects.

No, it is not. That means that one request is processed in one thread of Python, you can scale this if you have multiple instances of the Python Flask app running, this can be done with a load balancer like NGINX but still does not make it fast. The bottleneck is Ollama, it can handle multiple requests but may have to reload the LLM each time if it is not the same LLM, if you were to use Llama3.2 every time then you would get a level of support for multiple requests but in the end it is not scale-able as you would expect, if 1 request takes 10 seconds, 2 requests takes 20 seconds so it does not solve anything. You can always add another GPU, Ollama supports this but then the first request takes 12 seconds, the seconds takes 18 seconds, etc. So again no scale-able solution. But if you have isolated machines that are feed from a load balancer like NGINX and each machine has your Flask API and Ollama running with its own GPU (I use the NVIDIA 4090) then yes, first request takes 10 seconds, 2nd request takes 10 seconds so the same is pretty consistent for multiple requests, you will quickly find that you may need 4 machines to create a production grade system. This is what I have done for a large LLM project and it does work well. I setup the Load Balancer for Round Robin and then process each request as they come in. If I need to support more requests them I will add more servers, I did this on AWS and it cost me about $700 per server per month but it did work. I now have my own servers that cost about $2,500 each to build that is the LLM Engine Cluster. I connect this into the cloud using ngrok and it is very fast. As far as I know there is no way to scale up vertically as far as getting more LLM ram or processor power other than replacing your GPU board. Adding boards in parallel gives more memory but does not effect the processing speed, well it is a bit slower from the overhead but Ollama will put different sections of the LLM into different cards so the memery is scaled but not the processors. Each processor runs the segment of memory based on the LLM loaded in its own instance so there is no performance increase.

​@@fastandsimpledevelopment Clear! Thanks for the reply! So from a performance perspective parallel is the way to go, makes sense. Follow up question, how do you keep the source of truth (the RAG and your docs) in sync?

@@ChigosGames I do two things for Rag, initially I used PDF input and stored the Vectors into a ChromaDB, this can be a server so all the instances use the same database but the data is as old as the last PDF upload. I have moved to a better solution where I do not source anything from PDF, I query a database, in my case MongoDB, I then take that content (which should be the current truth) and feed it into the process as if it came from ChromaDB so it is then what Ollama uses to answer a question/prompt, this works very well and a lot of the PDF/VectorDB issue went away. I have a large set of data, think of Airline tickets so I have routes, times, destinations as well as passengers that purchase and I need to answer questions like "What is a cheaper flight" or "If I change a day how much will it cost" so some times it not as simple as a PDF document with content

​@@fastandsimpledevelopmentok, I love mongodb, since it is so json friendly (a bit too much sometimes), so with that you already structurize and let it enrich by the LLM. How do you 'steer' the LLM from not to be too creative with amending your (flight) data?

@@ChigosGames I create very specific data for example "SFO 01/10/2024 10:00AM - JFK 01/10/2024 4:45 PM American Flight 1410 $445" This is then used in the LLM and I do have a filter and use JSON format output I then transform this as needed

Thanks for the info!

Good catch. Thanks

Good question, I have not had that experience in charges. I normally query the data and move into MongoDB data in the format I need and then work with that content which removes all the charges after the initial query

Java is the most used language in Enterprise software development, integrating these applications with Rag and Ollama is the next step for private LLM processing. A simple curl to Ollama does nothing to integrate a vector Database or integrate into a legacy / microservice system

I think you need to have a paid account setup

Yes, if the OS has support and you have an AMD or Nvidia GPU installed and the latest version does auto-detect. You can also set it to NOT use the GPU in the Ollama config files but by default it does auto-detect.

@@fastandsimpledevelopment It detects only Nvidia GPU. I tried on AWS g4ad (AMD ) and g4dn.xlarge (Nvidia). Only the latter worked. This is FYI.

@@adityanjsg99 Thanks for your input. I have not tried anything other than NVidia GPU's, I've finally decided to get a few 4090 boards and see how they run. I'm trying to build an on-prem system since there is no affordable cloud solution. I'll externalize the LLM API via ngrok, not what I wanted :(

I normally use Ollama for for Rag applications with ChromaDB. Ollama is ran locally or at least on your servers so there is never a cost for tokens. If you were to use Gemini or Open AI then yes you have token costs. Depending on what database your using for your vector store, there may be small costs to store the data. In general when you retrieve data and use it in the LLM processing there are token costs. I have used up to 100Mb PDF files with ChromeDB. The big thing to watch is the Chunk Size, you may find that you will send 3 chunks to the LLM so if each one is 1Mb then 3 chunks are 3Mb, which could be 600,000 tokens (3,000,000 / 5) 5 bytes per token. That would be expensive, again using a local LLM and local vector store resolves these costs real fast ply gives you the security of your data not being outside your company.

@@fastandsimpledevelopment So the size of pdf files only affect the cost of the vector db？

@@candyman3537 Correct, if you have a local vector db then there is no cost. The chunk size would have more effect on tokens, I normally have 3 chunks returned from the similarity results that are then sent to the LLM.

I'm not sure, I don't have a windows machine

@@fastandsimpledevelopment its okay thanks for the video I will try to do it on Windows to see if its possible.

@@Nokiathanos did you get this to work in windows?? I'm wanting to get ollama tts working and having trouble.

It does require a connection as well as a paid account for elevenlabs

Glad you enjoyed it

I'm not sure, I have never done that. Take a look at the Twilio docs. Thanks

try running on the command line ollama run llama3.1

Thanks for the comments

Glad it helped

can I ask how?

Thanks, glad you enjoyed it!

Thanks, hope it helps you.

Glad you liked it, thanks!

Glad you enjoyed it

Thanks, glad you liked it!

Can you give me more information? I do not see a variable "context" in the code, "context" is returned in the lookup of data from the ChromaDB so if there is no context then I suspect there is no matching results from the search, maybe tell me the line number or share the code with me

@@fastandsimpledevelopment line 27 of your github repo. "NameError: name 'context' is not defined"

my mistake. the editor put a f""" automatically

@@federicocalo4776 That is part of the PromptTemplate so not a real variable, it is populated by the Retriever so line #70 should create this value in the retriever for you.

Make sure you have line 27 in 3 double quotes and you are using Python 3.9 or greater

im new to coding 😅

Thanks for the input