Navigating the AI Frontier // Boris Selitser // MLOps Podcast

Join us at our first in-person conference on June 25 all about AI Quality: www.aiqualityconference.com/
Navigating the AI Frontier: The Power of Synthetic Data and Agent Evaluations in LLM Development // MLOps podcast #241 with Boris Selitser, Co-Founder and CTO/CPO of Okareo.
A big thank you to ‪@latticeflow‬ for sponsoring this episode! LatticeFlow - latticeflow.ai/
// Abstract
Explore the evolving landscape of building LLM applications, focusing on the critical roles of synthetic data and agent evaluations. Discover how synthetic data enhances model behavior description, prototyping, testing, and fine-tuning, driving robustness in LLM applications. Learn about the latest methods for evaluating complex agent-based systems, including RAG-based evaluations, dialog-level assessments, simulated user interactions, and adversarial models. This talk delves into the specific challenges developers face and the tradeoffs involved in each evaluation approach, providing practical insights for effective AI development.
// Bio
Boris is Co-Founder and CTO/CPO at Okareo. Okareo is a full-cycle platform for developers to evaluate and customize AI/LLM applications. Before Okareo, Boris was Director of Product at Meta/Facebook, where he led teams building internal platforms and ML products. Examples include a copyright classification system across the Facebook apps and an engagement platform for over 200K developers, 500K+ creators, and 12M+ Oculus users. Boris has a bachelor’s in Computer Science from UC Berkeley.
// MLOps Jobs board
mlops.pallet.xyz/jobs
// MLOps Swag/Merch
mlops-community.myshopify.com/
// Related Links
docs.okareo.com/blog/data_loop
docs.okareo.com/blog/agent_eval
The Real E2E RAG Stack // Sam Bean // MLOps Podcast #217 - • The Real E2E RAG Stack...
RecSys at Spotify // Sanket Gupta // MLOps Podcast #232 - • RecSys at Spotify // S...
-------------- ✌️Connect With Us ✌️ ------------
Join our slack community: go.mlops.community/slack
Follow us on Twitter: @mlopscommunity
Sign up for the next meetup: go.mlops.community/register
Catch all episodes, blogs, newsletters, and more: mlops.community/
Connect with Demetrios on LinkedIn: / dpbrinkm
Connect with Boris on LinkedIn: / selitser
Timestamps:
[00:00] Boris' preferred coffee
[00:37] Takeaways
[02:32] Please like, share, leave a review, and subscribe to our MLOps channels!
[02:48] Software Engineering and Data Science
[06:01] AI Transformative Potential Explained
[10:31] Prompt Injection Protection Strategies
[17:03] Agent's metrics for Jira
[24:11] Data and Metrics Evolution
[27:54] Evaluation Focus Enhances Systems
[31:22 - 32:52] LatticeFlow AD
[32:55] Custom Evaluation and Synthetic Data
[36:23] Synthetic data for expansion, evaluation, and map
[41:06] Diverse agents' personalities for readiness
[44:25] Agent functions
[46:17] Optimizing Routing Agents
[50:04] Adapting to tool output for decision-making
[52:56] Agent framework evolution
[55:41] Agent framework for delivering value
[57:03] Wrap up