Delta Lake Migration At Scale

Adobe Experience Platform (AEP) serves a large number Adobe digital experience customers with tens of thousands of datasets in non-Delta format. To leverage Delta Lake, we migrate existing tables to Delta format after resolving the following challenges:
Short downtime to minimize customer impact
Adding new system metadata
Migration automation for scalability
Error detection and catastrophic rollback
This session will cover the solutions to the above challenges and lessons learned from migration. Finally, we'll discuss how scalable migration service becomes a core capability of the data platform.
Talk by: Miao Wang and Dilip Biswal
Here’s more to explore:
Rise of the Data Lakehouse: dbricks.co/3NH... Lakehouse Fundamentals Training: dbricks.co/44a...
Connect with us: Website: databricks.com
Twitter: / databricks
LinkedIn: / databricks
Instagram: / databricksinc
Facebook: / databricksinc

[@FilipBocse]

Very insightful details over such complex use-case as iceberg->delta migration, thank you for sharing.