PEPR '24 - Compute Engine Testing with Synthetic Data Generation

PEPR '24 - Compute Engine Testing with Synthetic Data Generation
Jiangnan Cheng and Eric Liu, Meta
At Meta, we have developed a new testing framework that utilizes privacy-safe and production-like synthetic data to detect regressions in various compute engines, such as Presto, within the Meta Data Warehouse. In this talk, we will discuss the challenges and solutions we have implemented to operate this framework at scale. We will also highlight key features of our synthetic data generation process, including the addition of differential privacy, expanded column schema support, and improved scalability. Finally, we will discuss how Meta leverages this testing framework to increase test coverage, reduce the Presto release cycle, and prevent production regressions.
View the full PEPR '24 program at www.usenix.org...