Running ML/LLM models on Kubernetes Across Major Cloud Providers with Abhishek Choudhary

Abhishek, co-founder and CTO of ‪@truefoundry‬, explores the complexities of building a machine learning platform on Kubernetes. Discover solutions to challenges like handling diverse hardware, managing large Docker images, and optimizing costs. Learn how True Foundry uses tools like Argo CD, Keda, and Istio to create efficient abstractions for data scientists and streamline ML operations.
Learn more about Civo Navigate here -►www.civo.com/n...
====================
Get free credit to try the world’s first K3s-powered, managed Kubernetes service.
Sign up to Civo -► www.civo.com
Get started with Kubernetes with Civo Academy: www.civo.com/a...
Subscribe to our RUclips Channel -► civo.io/subscribe
Follow Civo:
• Twitter -► / civocloud
• Github -► github.com/civo
• LinkedIn -► / civocloud
• Facebook -► / civocloud
0:00 - Introduction and Background
1:00 - Challenges of ML on Kubernetes
2:00 - Handling Diverse Hardware
4:00 - Managing Large Docker Images
5:30 - Slow Iteration Cycles
6:30 - Issues with Auto-Scaling in ML Workloads
8:00 - Model Download and Caching
10:00 - Cost Considerations and Optimizations
12:00 - GPU Sharing and Fractional GPUs
14:00 - Abstraction Layers for Data Scientists
16:00 - Building an ML Platform on Kubernetes
18:00 - Implementing Multi-Cluster Management
20:00 - Model Versioning and Upgrades
22:00 - Distributed Training and Inference
24:00 - Q&A Session