Monitorama PDX 2024 - Distributed Tracing - All The Warning Signs Were Out There!

Vijay Samuel's session from Monitorama PDX 2024.
We've all heard it time and time again - "distributed tracing is hard". The sad reality is that there is a lot of truth to this statement. All too often we keep hearing buzz words and statements around how magical distributed tracing is and how it can easily cut down time to triage. However, when we talk about tracing at scale with ~5000 micro services and 100s of databases, the nuances of tracing start bubbling up pretty fast. Some of which include:
* infinitely large trace waterfalls that become hard to grasp
* A handful of mis-instrumented applications that lead to lost confidence in the system
Many early adopters told us that it would be hard and we tried different techniques to mitigate some of those issues early but we ultimately hit the wall of pain.
This talk describes how these real problems affect the Observability team at eBay and how we are approaching these problems to hopefully make tracing work in the future.