Thanks for the wonderful Demo! I was wondering why you deploy vllm pod through demonsets rather than deployments. With daemonset, you can only deploy one pod in one node and a pod occupying a single gpu. Considering that nodes are usually attached with multiple gpus, I am afraid that using daemonset might make a lot of gpus idle.
Hi John / Saiyam. In the last part you mentioned "In lot of cases could be cheaper" What are those cases where locally hosting it is cheaper vs when using openai is cheaper: Is it just dependent on the load which we will have (RPD and max RPM)?
openai is $.50 per million tokens for gpt 3.5 for example. If you rent a gpu server for that same amount, you can generate tens or hundred of millions of tokens in one hour depending on which text generation model you choose. something like mistral 7b, phi 3 series, llama 3 8b, gemma 2b,etc all deliver about the same results if not better than gpt 3.5 and also all fit on a gpu server that costs 44 cents per hour on runpod. (the A5000 gpu server for example.)
Thanks for having me on Saiyam!! It was alot of fun to show you how we use vLLM at OpenSauced!! Happy to answer any questions here people might have!
Can share the yaml for deployment please?
This is absolutely wonderful session to understand how can we deploy LLMs in production on Kubernetes cluster!!
@@aireddy glad it was helpful!
Great video and great breakdown @JohnCodes!
Thanks for the wonderful Demo!
I was wondering why you deploy vllm pod through demonsets rather than deployments.
With daemonset, you can only deploy one pod in one node and a pod occupying a single gpu.
Considering that nodes are usually attached with multiple gpus, I am afraid that using daemonset might make a lot of gpus idle.
Thanks for this tutorial Saiyam.
Glad its useful, you building something with LLM?
Hi John / Saiyam. In the last part you mentioned "In lot of cases could be cheaper"
What are those cases where locally hosting it is cheaper vs when using openai is cheaper:
Is it just dependent on the load which we will have (RPD and max RPM)?
openai is $.50 per million tokens for gpt 3.5 for example. If you rent a gpu server for that same amount, you can generate tens or hundred of millions of tokens in one hour depending on which text generation model you choose. something like mistral 7b, phi 3 series, llama 3 8b, gemma 2b,etc all deliver about the same results if not better than gpt 3.5 and also all fit on a gpu server that costs 44 cents per hour on runpod. (the A5000 gpu server for example.)
Loved it...❤
Glad you found it useful!