Evaluating Quality and Improving LLM Products at Scale // Austin Bell // AI in Production Conference

You Can’t Improve What You Don’t Measure - Evaluating Quality and Improving LLM Products at Scale
// Abstract
How do you know that your prompt change or different pre-processing technique improved your LLM output? And if you don’t know, how can you deploy product changes with confidence to ensure that you’re consistently improving your generative products?
// Bio
Austin Bell is a Staff Software Engineer at Slack focused on building out text based ML and Generative AI products. Previously, Austin held positions at Fennel, an ESG modeling and mobile investment app, as the Head of Engineering and various roles leading teams developing Data Science solutions and ML products at the Analysis Group, an economic consulting group specializing in the legal, economics and healthcare industries. Austin began his career as an Economist focusing on competition and healthcare economics.
Austin has also led initiatives leveraging machine learning techniques to better diagnose, evaluate and analyze skin disorders at Columbia University as well as collaborated closely with the Haitian government and leading non-profit clinics in Haiti, to conduct research in predicting patient success in HIV treatment as well as supporting modeling the spread and understanding of COVID-19.
Austin graduated with a Bachelor’s Degree in Economics and obtained a Master’s Degree in Computer Science and Machine Learning from Columbia University.
A big thank you to our Premium Sponsors, ‪@Databricks‬ and ‪@baseten‬ for their generous support!
// Sign up for our Newsletter to never miss an event:
mlops.community/join/
// Watch all the conference videos here:
home.mlops.community/home/col...
// Check out the MLOps Community podcast: open.spotify.com/show/7wZygk3...
// Read our blog:
mlops.community/blog
// Join an in-person local meetup near you:
mlops.community/meetups/
// MLOps Swag/Merch:
mlops-community.myshopify.com/
// Follow us on Twitter:
/ mlopscommunity
//Follow us on Linkedin:
/ mlopscommunity