Understanding Big Data for Software Engineers - David Ostrovsky - NDC Oslo 2023

Data warehouses, data lakes, data lake-houses (yes, that's a thing!), data meshes, data marts, OLAPs, streams, SQL, NoSQL, and hundreds of technologies whose names sound like Pokemon. That is what greets a software developer who wants to understand the world of big data. But as companies become more data driven and data rich, we as software developers have to evolve. We go from data users, to power users, and ultimately to data practitioners - engineers who understand and can leverage the full power of modern data tools.
In this talk we'll dive head first into the lake of data! We're going to figure out what the different technologies and paradigms mean, what problems they solve, and how they fit together to create the data architecture that underlies most modern distributed software systems.
Check out our new channel:
NDC Clips:
‪@ndcclips‬
Check out more of our featured speakers and talks at
ndcconferences...
ndcoslo.com/

[@Kane0123]

Really nice historical context setting, the final note around perspective for whether you have the need for a “Big Data” platform would be really relevant for a lot of companies!

[@dinoscheidt]

How would you answer that? Big Data is a capability not a use case. That would be like “Please tell us if a band saw is really relevant for our wood working shop.” Well, that depends on you… not the band saw. Some shops might need one because the build a lot of furniture, others maybe because every 2 years a very huge desk needs to be sawed which makes a lot of money. Others want it just because its shiny. So: Learn the properties of the tool and do a critical analysis for your context…. Looking for rule of thumbs is usually the path onto someone else’s dinner plate.

[@OttoFazzl]

Great talk!

[@goldnutter412]

In 10 years we all live in the data OCEAN ! it's becoming more obvious