Functional Data Engineering - A Set of Best Practices | Lyft

Download slides: www.datacouncil.ai/talks/func...
ABOUT THE TALK:
Batch data processing (also known as ETL)  is time-consuming, brittle, and often unrewarding. Not only that, it’s hard to operate, evolve, and troubleshoot.
In this talk, we’ll discuss functional programming paradigm and explore how applying it to Data Engineering can bring a lot of clarity to the process. It helps solving some of the inherent problems of ETL, leads to more manageable and maintainable workloads and helps to implement reproducible and scalable practices. It empowers data teams to tackle larger problems and push the boundaries of what’s possible.
ABOUT THE SPEAKER:
Maxime Beauchemin works as a Senior Software Engineer at Lyft where he develops open source products that reduce friction and help generate insights from data. He is the creator and a lead maintainer of Apache Airflow [incubating], a data pipeline workflow engine; and Apache Superset [incubating], a data visualization platform; and is recognized as a thought leader in the data engineering field.
Before Lyft, Maxime worked at Airbnb on the "Analytics & Experimentation Products team". Previously, he worked at Facebook on computation frameworks powering engagement and growth analytics, on clickstream analytics at Yahoo!, and as a data warehouse architect at Ubisoft.
ABOUT DATA COUNCIL:
Data Council (www.datacouncil.ai/) is a community and conference series that provides data professionals with the learning and networking opportunities they need to grow their careers. Make sure to subscribe to our channel for more videos, including DC_THURS, our series of live online interviews with leading data professionals from top open source projects and startups.
FOLLOW DATA COUNCIL:
Twitter: / datacouncilai
LinkedIn: / datacouncil-ai

[@ChristopherSlattery1980]

Interesting talk. Nice to see a tech video where they display the slides as well as the speaker.

[@boringmanager9559]

wow, I started to understand those concepts so much better now

[@moverecursus1337]

Interesting the approach to the slowing change dimension. Data storage nowadays are cheap, the time and price of engineering is a lot more expensive.

[@hgiagiamou]

Great talk!

[@VajoLukic]

It is such a fantastic talk! It summarizes all the good practices for modern data management and puts them into the right perspective so that "old school" BI/DW people can understand.

[@millerblaine2047]

i dont mean to be so off topic but does anyone know of a trick to log back into an Instagram account??
I stupidly lost the login password. I would appreciate any tips you can offer me!

[@striker865]

Awesome talk! Thanks for taking the time to share some best practices from top the talent!

[@ariesykes1432]

Really interesting video! It definitely relates to what I am working on currently. I work in IT recruiting. I'm always looking for a data engineers for my jobs and this video helps me understand a little more about what data engineers have to face in their field. Thank you!

[@nhuwlaftooi]

Really great talk! Help me to understand many new concepts

[@cyclogenisis]

Overall not a bad presentation. Although, I do not think using Presto with 1 million per day snapshot on dimensions is viable (specifically out of box). Presto likes to put all joins onto the main fact into memory, modelling this way and using Presto isn't really in line with how it was meant to be used if your joining all the records each day. But in general, I do like the shift of keeping everything in dimensions if the technology allows for it.
Edit: The Q&A section talked more to this, he adapted his answer to "apply common sense"

[@chinmayarankalle4389]

Great efforts thanks @maxime

[@lambuth]

Great talk. And quite the resemblance to the singer from 311.

[@karangupta_DE]

A persistent staging area might only be effective when there is one place where your raw data resides. For example if you stage your data on AWS S3 and then copy the data again to snowflake, you will end up having two places of storage with the same raw data.

[@qwaszx822]

Thanks a lot for the video. I have a question. For modern datawarehouse solutions are there any other data models emerged apart from starschema, snowflake ?

[@dragonfly4484]

data vault and incremental approach are kind of emerging as norms in some quarters

[@lbb2rfarangkiinok]

Talked a bout a lot of interesting stuff but I had a hard time really figuring out what is meant by functional engineering. Each individual topic was very clear for me, but not the red thread linking it all together.

[@sspaeti]

Thanks for sharing your great knowledge with us again. One question about snapshotting let's say daily. What if multiple changes in dim_supplier happen, then you wouldn't catch these once. Would that just be a small tradeoff you would accept or would you have something else in mind to track that?

[@mistercrunch]

If keeping track of intra-day changes is important I'd recommend denormalizing the dimension attribute into the fact table. Conceptually if some dimensional attributes are "flickering", meaning going back and forth and changing fairly often, it's pointing towards that attribute being logically attached to the fact and not to the dimension.

[@kyosungchoo1436]

I think you use changing history to catch this changing, then as Maxime Beauchemin, this changing history should be in fact table.

[@allthatyouare]

🤯

[@marigoldx22]

Maha Guru!

[@borat_trades2860]

honestly did not get much from this talk - pretty technical

[@santanubaishya8357]

Great talk!