Evaluating Spotify's Multimillion Item Database // Sanket Gupta // MLOps podcast

Join us at our first in-person conference on June 25 all about AI Quality: www.aiqualityconference.com/
MLOps podcast #232 with Sanket Gupta, Senior Machine Learning Engineer at Spotify // RecSys at Spotify.
A big thank you to @latticeflow for sponsoring this episode! LatticeFlow - latticeflow.ai/
Highlighting the immense challenge of evaluating systems with hundreds of millions of items, Sanket revealed that models are often required to evaluate other models to ensure the best user recommendations. His insights offer a fascinating glimpse into the innovative backend of streaming services. Tune in to better understand the precision and complexity involved in curating your next favorite playlist.
// Abstract
LLMs with foundational embeddings have changed the way we approach AI today. Instead of re-training models from scratch end-to-end, we instead rely on fine-tuning existing foundation models to perform transfer learning.
Is there a similar approach we can take with recommender systems?
In this episode, we can talk about:
a) how Spotify builds and maintains large-scale recommender systems,
b) how foundational user and item embeddings can enable transfer learning across multiple products,
c) how we evaluate this system
d) MLOps challenges with these systems
// Bio
Sanket works as a Senior Machine Learning Engineer on a team at Spotify building production-grade recommender systems. Models built by my team are being used in Autoplay, Daily Mix, Discover Weekly, etc.
Currently, my passion is how to build systems to understand user taste - how do we balance long-term and short-term understanding of users to enable a great personalized experience.
// MLOps Jobs board
mlops.pallet.xyz/jobs
// MLOps Swag/Merch
mlops-community.myshopify.com/
// Related Links
Website: sanketgupta.substack.com/
Our paper on this topic "Generalized User Representations for Transfer Learning": arxiv.org/abs/2403.00584
Sanket's blogs on Medium in the past: / sanket107
-------------- ✌️Connect With Us ✌️ ------------
Join our slack community: go.mlops.community/slack
Follow us on Twitter: @mlopscommunity
Sign up for the next meetup: go.mlops.community/register
Catch all episodes, blogs, newsletters, and more: mlops.community/
Connect with Demetrios on LinkedIn: / dpbrinkm
Connect with Sanket on LinkedIn: www.linkedin.com/in/sanketgupta107

[@sndrstpnv8419]

may you share link to this data set?