Adam here. Shift left is a simplification of the multi-hop data architecture. Instead of multiple divergent downstream cleanup and restructuring jobs, you simply move the work upstream and just do it once. Best of all - you can benefit from shifting even just a few workloads - it’s not all or nothing! Have you been burnt by similar-yet-different data sets before? How has shift left fixed your data architecture?
What about the shift in competences? It seems that data analysts and scientists lose a big part of the pipeline (bronze-silver passage) in favor of data engineers that now works on the entire "ETL" process that is more complicated than before (EDA, CDC, flink SQL, iceberg creation, ecc...) and incorporates all the hops until the silver node.
Adam here. Shift left is a simplification of the multi-hop data architecture. Instead of multiple divergent downstream cleanup and restructuring jobs, you simply move the work upstream and just do it once. Best of all - you can benefit from shifting even just a few workloads - it’s not all or nothing!
Have you been burnt by similar-yet-different data sets before? How has shift left fixed your data architecture?
What about the shift in competences? It seems that data analysts and scientists lose a big part of the pipeline (bronze-silver passage) in favor of data engineers that now works on the entire "ETL" process that is more complicated than before (EDA, CDC, flink SQL, iceberg creation, ecc...) and incorporates all the hops until the silver node.
Clear & Simple 😊❤