Using serverless functions for real-time observability (Liz Fong-Jones) - SLSDays ANZ 2024

Presented at #ServerlessDays ANZ 2024 (Sydney)
In this talk, you will learn how a sub-second latency query engine inspired by Facebook's Scuba was extended from running in RAM only, to querying the most recent data that could still fit on local SSD, to querying months of data at a time using cloud storage and serverless functions. We'll describe the pitfalls of managing lambdas at scale, including impatience, maximum concurrency, runtime and architecture configuration/experimentation, and the price/performance of renting tens of thousands of parallel workers across the x86 and arm ecosystems.