Love this! The line "it's an organisational and cultural shift as much as a technology one" sums it up, technology can never make up for immature data cultures. Keen to see what an implemented and functional Data Mesh will look like in 5-10 years.
This is 100% correct. I have worked on two federated analytical data architectures which were in parallel trying the Data Product ownership cultural shift. No big organisation is going to skip to the ideal answer. It takes a couple of iterations to get up the capability maturity model of the architecture, the governance and the culture.
You are absolutely spot on. I can't agree more. The industry is still not there, but may be in a 10 years time, will see. This video should be the reference point for every such tech guru who is planning to build a mesh without considering their ORG culture. You literally spoke my heart out where you mentioned you have the centralized data/team and then federate on top of it if required based on your business cases. Exactly what I was debating within our Data team a few weeks back. Such a relief! Hahaha! 😁😁 Love your videos as always!! 💚💚
Hey Simon - love your pragmatic take on hype and history. We all have been there and the fact that most new things along the hype cycle are more of a cultural and governance shift than an raw tech innovation also is true. But as you stated yourself, not every architecture pattern is right for every problem. One substantial use case I think you have neglected in your rant where the data mesh principles applied sensibly can add lots of value is the refactoring of outdated legacy environments spread out across on-prem, SaaS and divers cloud fragments. And I clearly disagree with your initial statement of the data product owner needing to substitute or become a data engineer - rather you need a cross-functional team to build a data product and likely many more people skilled as data engineers. Happy to have an offline conversation about our mutual client experiences across the decades. And thanks for the excellent work - appreciate your content and the quality and serenity you bring to this debate. 🙏🏽
Thanks Simon, I feel finally truth has been spoken loudly... I definitely think that some salesman with suits are trying hard to sell the idea of data mesh without realising the reality.. for companies, its absurd to just follow the HYPE and try to adopt. frankly I haven't seen any company using data mesh and being successful with it. Idea is definitely great with great potential but only a reality limited to product based companies.
My company has a data mesh for decades! Everybody maintaines their own Excel sheet... and then we send these Excel sheets arounds so that other people build their Excel sheets on top of that.
Loved! I believe in a Practical Mesh implementation. I already implemented some concepts that are suitable for the organisation but some others are just to theoretical or the tools just don't exist yet.
Amazing! Thanks for your opinion! Moreover Data Mesh existed even before the article was released. A lot of companies did federated analytics architecture, especially big companies and yes, currently looks like data-people decided go to Data Mesh, however they don't care about culture, governance, etc. I saw a lot of challenges from data protection perspective with unmature "Data Mesh" - it is huge rabbit hole.
Hi! I've been searching for ages for this because I design my Data Lakehouses really similar to what you described in a second bit. I think the main problem with Data Mesh is that it's a monolith in disguise, even more than the Enterprise Data Warehouse, because of coupling. By this, I mean that the analytical products are coupled by the data to the current versions of data products, which means that any modification (a.k.a. replacing the data product with its' new version) to a product will possibly break or bug analytical products depending on it. This would mean that a data product should be "final" on the first release which is exactly the way of the old Waterfall & Monolith world that we are trying to get away from. Actually, even EDW is better as it at least allows some customization and decoupling via Data Marts. A distributed architecture isn't about services, repos and teams, and it never will be, it's about coupling, cohesion, versions, and the open-closed principle...
You can overcome the coupling by abstracting the access means to the data products. The API should ideally say nothing about the underlying data structure and should be versioned so you won't break backwards compatibility.
@@jordanpavlic9745 I've just released the first version of a complete(ish) data platform & DataOps framework and now I get what you're saying... you just version your data products as you would with an API and run multiple versions at the same time in production, while giving a deprecation timeline for these products to your consumers. Generalizing this for other things like job definitions, infrastructure etc. can fully "agilify" data architectures which lets you clean up data in your Data Product layer to optimize ETL costs globally.
Hey Simon I loved your rant!I completely share your view on this being an organisational and cultural shift needed. The pragmatic approach makes much more sense.
This is a very common pattern. I call it "the problem of who does the do." you can either centralize or decentralize the "do". Basically agile teams, graphQL, data mesh, etc are shifts to the decentralized "do". This comes at the cost of control over security, design, implementation, quality, etc but can seriously debottleneck work.
Couldn’t agree more. Those of us who have been around the block long enough and build data platforms for Fortune 500 know the “Mess” Data Mesh will create and won’t touch it with a 10 ft pole
some of my expriences of working with customers who want/ed implement a data mesh: - you work with some teams who have no idea what is dataproduct what they should do at all -most of customers just want to benefit from seperating domains to bring some transparency to their business, so they think Data Mesh is the best solution -some customers forgot it as soon as we came to speak about "Data Sharing", "Data Market Place", "PII", etc.. -Some Customers, whose staff were not very fit in tech, had to hire up to 10 Data Engineers (every team needs data engineers) -some customers made high costs through many Cloud Recources at the end,IF YOU ARE NOT A VENDOR and you care about your customers, first ask customers why they think Data Mesh is a good solution. If they have just heard about it then I think they need & deserve to be given more clarity about this "Utopia"
If nothing else there is one nice concept we can learn from Data Mash: the ideal of data ownership and accountability. The sweet spot would be for teams to be responsible for publishing good-quality data following company standards. But the idea of each team should build its own data products is a little bit crazy to me. That would add a ton of complexity, and increase the probability to duplicate work. No to mention the complexity of maintaining all these systems. If you consider the big shortage of Data Engineers this idea gets even more impractical. I think companies should only consider a Data Mesh architecture if the company is enormous and has a very complex and broad business model where each team has its own goals (almost like little companies inside a bigger company).
This seems to be looking through an analytics-only lens, and skips the architectural challenge of integrating data upstream of a lake/warehouse, in operational systems. I believe IT and engineering in most companies hold the reins of standing up new systems/services that contains those data and it’s healthy to crave upstream operational integration - especially in a new growing company. For that reason, I see harm in dismissing data mesh architecture for the reason of “it’s inappropriate today.” That ideal should still be applied as those new systems are selected and as new services are designed and built. Like most idealistic concepts, the answer is less “yes” or “no” and more of “how can this ideal serve us in our decisions and designs?”
Thank you Simon!! thank you for covering this topic. totally agree with you. end of the day if idea is to address data quality, lack of ownership and organizational scaling by distributing centralized data team, why can't it be achieve using proven & trusted ingest-curate-model-serve methods? we gave up of data mesh and adopting idea of analytical data products to address data quality & ownership.
Same as cultural change required to develop products in agile not waterfall. You need to change mindset, tools. Imho not entire org should switch to data mesh. I still can imagine that well defined domains should
I am leading a new initiative to revamp my data platform architecture. I really hate Data Mesh. It sounds great on paper but IMO it could be really messy or perhaps impossible to implement successfully. Love to see an actual diagram of successful data mesh architecture.
this is like Nuclear Fusion....it's always 10 years away. However, I do believe this needs to be a technology solution because I don't think we will ever get the non technical people to adopt any guidance we give. We need to figure out technology that allows us to collect the data and wok with it without having to make the creator to change their workflow. I have no answer today but it is an honorable quest that will keep us busy until we retire (or solve it and become Billionaires and then we too can build rockets :-))
Finally an opinion very close to mine. Thanks for that Simon! When I heard it first time (a year ago) I was like "are you all crazy? That will not work!". And the spiral of the hype got stronger and stronger hereafter, now it's a real madness. Yes, this is utopian idea. Personally, I am comparing it to a socialism and even communism, which I had to live here in Czech Republic till my 18. I don't agree with you in one thing - it will never work and it is not a good idea at all. Data is not a product, people won't get the skills needed to be data owners/stewards and federating chaos is not manageble at all. Why? Simply because of economics, the same as socialism/communism didn't work - people are not having strong enough motivation to take care of the data produced in their domain as the Zhamak wants or even enforces them to do. For what? Just to make them available for everyone? Who cares? Wha pays for that? What is the profit of selling the data product in-house? And what is the real benefit of doing such a rigid contracts, APIs and neverending operation and maintenace of all it together? Ask your shareholders and you will get the answer - are you crazy? Please focus on the real products your customers want to pay for! Do you remember the old song from Megadeath about Peace? I do and when I was 18 I'd been so grateful that the iron curtain had fallen and the utopia was gone. And now again, in my own beloved field of data engineering. Oh man..
I think the bigger problem is most data teams aren't using any architecture so something is better than nothing. Though I'd prefer a data vault/mesh hybrid.
Hey, sorry, had to dig around for the source. Databricks are sharing the Gartner Hype Cycle writeup for 2022 here: www.databricks.com/resources/ebook/hype-cycle-for-data-management The positioning, as well as the writeup & justifications for that position are available there, although it is behind a (free!) signup.
This first comment will be based on the first 5 minutes, I'll give the rest when I have watched the whole video. The spirit of this rant in the first 5 minutes sort of insinuates that Data Mesh concerns and reasons for existing are comical, which I don't share. I've worked as a Data analyst, data architect, data warehouse architect and etc. The issue of data ownership has been one developers have been dodging for years in the same way that "business" has been dodging setting comprehensible and pragmatic requirements for decades. I kind of feel Data Mesh is an Agile methodology for data...Developers and System Owners must start owning up to their data responsibilities. We all know how much fixing of data takes place in the data warehouse under the guise of transforming and wrangling...
Now that I have gone through the whole video, I believe the title is misleading. I believe "Why you are not ready for Mesh" is more apt. We are on the same page, data culture is fundamental in the journey and should not be driven by software.
Hey @Simon, where is Bill Inmon to start an argument.... Hey I am surprised that Mesh became such hype considering that domain based frameworks have been around for eons and xOps didn't.....
Not even convinced its a great idea. Its an idea, not nessasarily great. I have a hunch costs would increase dramatically if its fully mesh, which for most businesses is not a viable solution. I feel a hybrid provides the best of both worlds.
Bro can u look Into dag$ it's the first decentralized microservice architecture in crypto. Apparently department of defense are looking into its use case for its zero trust frame work, specifically data mesh
Employees own sh.... Convince the business owner to cut you in after you leave the company for the things you presume to "own". Don't call it ownership, its not. Sad to see engineers trained on logical thinking keep regurgitating this nonsense.
It's so theoretical as to be useless. It's trying to capitalize on the human tendencies to cluster to their own interests. Unfortunately, it needs some kind of intersection with data governance.
![Twenty One Pilots - The Line (from Arcane Season 2) [Official Audio]](http://i.ytimg.com/vi/XCz7WUotXs8/mqdefault.jpg)
![Jack Harlow - Hello Miss Johnson [Official Music Video]](http://i.ytimg.com/vi/Q86_nlRoIGw/mqdefault.jpg)
Love this! The line "it's an organisational and cultural shift as much as a technology one" sums it up, technology can never make up for immature data cultures. Keen to see what an implemented and functional Data Mesh will look like in 5-10 years.
Great rant! Can you do the ‘metrics store’ next 😊. Something on data modelling perhaps as well & the traditional & emerging patterns
This is 100% correct. I have worked on two federated analytical data architectures which were in parallel trying the Data Product ownership cultural shift. No big organisation is going to skip to the ideal answer. It takes a couple of iterations to get up the capability maturity model of the architecture, the governance and the culture.
You are absolutely spot on. I can't agree more. The industry is still not there, but may be in a 10 years time, will see. This video should be the reference point for every such tech guru who is planning to build a mesh without considering their ORG culture. You literally spoke my heart out where you mentioned you have the centralized data/team and then federate on top of it if required based on your business cases. Exactly what I was debating within our Data team a few weeks back. Such a relief! Hahaha! 😁😁
Love your videos as always!! 💚💚
Hey Simon - love your pragmatic take on hype and history. We all have been there and the fact that most new things along the hype cycle are more of a cultural and governance shift than an raw tech innovation also is true. But as you stated yourself, not every architecture pattern is right for every problem. One substantial use case I think you have neglected in your rant where the data mesh principles applied sensibly can add lots of value is the refactoring of outdated legacy environments spread out across on-prem, SaaS and divers cloud fragments. And I clearly disagree with your initial statement of the data product owner needing to substitute or become a data engineer - rather you need a cross-functional team to build a data product and likely many more people skilled as data engineers. Happy to have an offline conversation about our mutual client experiences across the decades. And thanks for the excellent work - appreciate your content and the quality and serenity you bring to this debate. 🙏🏽
Thanks for this. Looking forward to more opinion videos like this!
Thanks Simon, I feel finally truth has been spoken loudly... I definitely think that some salesman with suits are trying hard to sell the idea of data mesh without realising the reality.. for companies, its absurd to just follow the HYPE and try to adopt. frankly I haven't seen any company using data mesh and being successful with it. Idea is definitely great with great potential but only a reality limited to product based companies.
My company has a data mesh for decades!
Everybody maintaines their own Excel sheet...
and then we send these Excel sheets arounds so that other people build their Excel sheets on top of that.
😂
man!!! That's so but so real that I have cold fash backs about trying to hunt down for data.
Loved! I believe in a Practical Mesh implementation. I already implemented some concepts that are suitable for the organisation but some others are just to theoretical or the tools just don't exist yet.
Amazing! Thanks for your opinion! Moreover Data Mesh existed even before the article was released. A lot of companies did federated analytics architecture, especially big companies and yes, currently looks like data-people decided go to Data Mesh, however they don't care about culture, governance, etc. I saw a lot of challenges from data protection perspective with unmature "Data Mesh" - it is huge rabbit hole.
Hi! I've been searching for ages for this because I design my Data Lakehouses really similar to what you described in a second bit. I think the main problem with Data Mesh is that it's a monolith in disguise, even more than the Enterprise Data Warehouse, because of coupling. By this, I mean that the analytical products are coupled by the data to the current versions of data products, which means that any modification (a.k.a. replacing the data product with its' new version) to a product will possibly break or bug analytical products depending on it. This would mean that a data product should be "final" on the first release which is exactly the way of the old Waterfall & Monolith world that we are trying to get away from. Actually, even EDW is better as it at least allows some customization and decoupling via Data Marts. A distributed architecture isn't about services, repos and teams, and it never will be, it's about coupling, cohesion, versions, and the open-closed principle...
Very interesting point. Can you please elaborate on the coupling topic. I would be interested to understand your argument. Thanks a lot.
You can overcome the coupling by abstracting the access means to the data products. The API should ideally say nothing about the underlying data structure and should be versioned so you won't break backwards compatibility.
@@jordanpavlic9745 I've just released the first version of a complete(ish) data platform & DataOps framework and now I get what you're saying... you just version your data products as you would with an API and run multiple versions at the same time in production, while giving a deprecation timeline for these products to your consumers. Generalizing this for other things like job definitions, infrastructure etc. can fully "agilify" data architectures which lets you clean up data in your Data Product layer to optimize ETL costs globally.
Hey Simon I loved your rant!I completely share your view on this being an organisational and cultural shift needed. The pragmatic approach makes much more sense.
This is a very common pattern. I call it "the problem of who does the do." you can either centralize or decentralize the "do". Basically agile teams, graphQL, data mesh, etc are shifts to the decentralized "do". This comes at the cost of control over security, design, implementation, quality, etc but can seriously debottleneck work.
Very timely rant, especially for me. Thanks!
Couldn’t agree more. Those of us who have been around the block long enough and build data platforms for Fortune 500 know the “Mess” Data Mesh will create and won’t touch it with a 10 ft pole
some of my expriences of working with customers who want/ed implement a data mesh:
- you work with some teams who have no idea what is dataproduct what they should do at all
-most of customers just want to benefit from seperating domains to bring some transparency to their business, so they think Data Mesh is the best solution
-some customers forgot it as soon as we came to speak about "Data Sharing", "Data Market Place", "PII", etc..
-Some Customers, whose staff were not very fit in tech, had to hire up to 10 Data Engineers (every team needs data engineers)
-some customers made high costs through many Cloud Recources
at the end,IF YOU ARE NOT A VENDOR and you care about your customers, first ask customers why they think Data Mesh is a good solution.
If they have just heard about it then I think they need & deserve to be given more clarity about this "Utopia"
Thank you, a transcript of this should be ideal along with the clip, please
I hit thumbs up the second you said "No."
Was tempted to just leave the video there... 😅
Thanks Simon for these videos, great way to start the day
As always thanks sir, keep these videos up , always helpful!
If nothing else there is one nice concept we can learn from Data Mash: the ideal of data ownership and accountability. The sweet spot would be for teams to be responsible for publishing good-quality data following company standards. But the idea of each team should build its own data products is a little bit crazy to me. That would add a ton of complexity, and increase the probability to duplicate work. No to mention the complexity of maintaining all these systems.
If you consider the big shortage of Data Engineers this idea gets even more impractical.
I think companies should only consider a Data Mesh architecture if the company is enormous and has a very complex and broad business model where each team has its own goals (almost like little companies inside a bigger company).
This seems to be looking through an analytics-only lens, and skips the architectural challenge of integrating data upstream of a lake/warehouse, in operational systems. I believe IT and engineering in most companies hold the reins of standing up new systems/services that contains those data and it’s healthy to crave upstream operational integration - especially in a new growing company. For that reason, I see harm in dismissing data mesh architecture for the reason of “it’s inappropriate today.” That ideal should still be applied as those new systems are selected and as new services are designed and built. Like most idealistic concepts, the answer is less “yes” or “no” and more of “how can this ideal serve us in our decisions and designs?”
Thank you Simon!! thank you for covering this topic. totally agree with you. end of the day if idea is to address data quality, lack of ownership and organizational scaling by distributing centralized data team, why can't it be achieve using proven & trusted ingest-curate-model-serve methods? we gave up of data mesh and adopting idea of analytical data products to address data quality & ownership.
I totally agree with everything you said
Same as cultural change required to develop products in agile not waterfall. You need to change mindset, tools. Imho not entire org should switch to data mesh. I still can imagine that well defined domains should
I am leading a new initiative to revamp my data platform architecture. I really hate Data Mesh. It sounds great on paper but IMO it could be really messy or perhaps impossible to implement successfully. Love to see an actual diagram of successful data mesh architecture.
Would like to meet you and shake your hand after what I've. Great job! Would love to hear opinion on another hype: semantic layer
I like the magic button metaphor.
this is like Nuclear Fusion....it's always 10 years away. However, I do believe this needs to be a technology solution because I don't think we will ever get the non technical people to adopt any guidance we give. We need to figure out technology that allows us to collect the data and wok with it without having to make the creator to change their workflow. I have no answer today but it is an honorable quest that will keep us busy until we retire (or solve it and become Billionaires and then we too can build rockets :-))
You nailed it! Thank you!
Nice clear viewpoint
Finally an opinion very close to mine. Thanks for that Simon! When I heard it first time (a year ago) I was like "are you all crazy? That will not work!". And the spiral of the hype got stronger and stronger hereafter, now it's a real madness. Yes, this is utopian idea. Personally, I am comparing it to a socialism and even communism, which I had to live here in Czech Republic till my 18. I don't agree with you in one thing - it will never work and it is not a good idea at all. Data is not a product, people won't get the skills needed to be data owners/stewards and federating chaos is not manageble at all. Why? Simply because of economics, the same as socialism/communism didn't work - people are not having strong enough motivation to take care of the data produced in their domain as the Zhamak wants or even enforces them to do. For what? Just to make them available for everyone? Who cares? Wha pays for that? What is the profit of selling the data product in-house? And what is the real benefit of doing such a rigid contracts, APIs and neverending operation and maintenace of all it together? Ask your shareholders and you will get the answer - are you crazy? Please focus on the real products your customers want to pay for! Do you remember the old song from Megadeath about Peace? I do and when I was 18 I'd been so grateful that the iron curtain had fallen and the utopia was gone. And now again, in my own beloved field of data engineering. Oh man..
I think the bigger problem is most data teams aren't using any architecture so something is better than nothing. Though I'd prefer a data vault/mesh hybrid.
Hello, could you provide the reference for Gartner's point of view?
Hey, sorry, had to dig around for the source. Databricks are sharing the Gartner Hype Cycle writeup for 2022 here: www.databricks.com/resources/ebook/hype-cycle-for-data-management
The positioning, as well as the writeup & justifications for that position are available there, although it is behind a (free!) signup.
Started to wonder “You can Shortcut the …”. Have you seen fabric at that point in time?
Great video!
I can only describe having a cathartic reaction to this.
Feels like Agile without the cultural shift.
This first comment will be based on the first 5 minutes, I'll give the rest when I have watched the whole video. The spirit of this rant in the first 5 minutes sort of insinuates that Data Mesh concerns and reasons for existing are comical, which I don't share. I've worked as a Data analyst, data architect, data warehouse architect and etc. The issue of data ownership has been one developers have been dodging for years in the same way that "business" has been dodging setting comprehensible and pragmatic requirements for decades. I kind of feel Data Mesh is an Agile methodology for data...Developers and System Owners must start owning up to their data responsibilities. We all know how much fixing of data takes place in the data warehouse under the guise of transforming and wrangling...
Now do DataVault. 🤟
I don't have a strong opinion against DataVault itself, although "DataVault in a Lake" is on my list!
@@AdvancingAnalytics In a lake or a lakehouse using something like Delta (or both)? What would the headline be?
We do DataVault on a lake… please tell me why we’re wrong!
Now that I have gone through the whole video, I believe the title is misleading. I believe "Why you are not ready for Mesh" is more apt. We are on the same page, data culture is fundamental in the journey and should not be driven by software.
Hey @Simon, where is Bill Inmon to start an argument.... Hey I am surprised that Mesh became such hype considering that domain based frameworks have been around for eons and xOps didn't.....
Haha, my intent wasn't to start an argument, just to put an opinion out there to see if I'm going crazy ;)
Great video. People love a buzzword.
The central thing is called a "Data Lake"
Not even convinced its a great idea. Its an idea, not nessasarily great. I have a hunch costs would increase dramatically if its fully mesh, which for most businesses is not a viable solution. I feel a hybrid provides the best of both worlds.
Bro can u look Into dag$ it's the first decentralized microservice architecture in crypto. Apparently department of defense are looking into its use case for its zero trust frame work, specifically data mesh
Feels like you are confusing analytical data with operational data, but it could be me.
Employees own sh.... Convince the business owner to cut you in after you leave the company for the things you presume to "own".
Don't call it ownership, its not. Sad to see engineers trained on logical thinking keep regurgitating this nonsense.
It's so theoretical as to be useless. It's trying to capitalize on the human tendencies to cluster to their own interests. Unfortunately, it needs some kind of intersection with data governance.