Dynamic Sampling in Practice - Kent Quirk, Honeycomb

Dynamic Sampling in Practice - Kent Quirk, Honeycomb
When people think of sampling telemetry, it's usually in terms of deterministic sampling -- choosing a random subset of the information. This is done to reduce the total volume of telemetry to a manageable amount, but in a way that represents the whole. While this is mathematically sound, the problem in practice is that rare events -- like serious errors -- are quite rare. If you're randomly sampling 1 in 1000 events, that means you're missing 999 out of every 1000 errors! Dynamic sampling is the practice of adjusting the sample rate based on the contents of the telemetry using tail sampling, and then decorating the result with enough information that the backend can reconstruct something resembling the original traffic. This talk will explain dynamic sampling, show an open source project with several different dynamic samplers, and show them working in practice. It will also delve into some of the changes that need to be made to the collector to support this model.