If you're skeptical about how data mesh would be implemented, I followed up with some of the doubts people might have: ruclips.net/video/CMNzHAdrigQ/видео.html
love your content my man. The ownership piece really is the value add part here. As the businesses I deal with mostly want that to be "IT's problem" to deal with.
Thanks for this quick overview. I wonder what are the motivation for a domain to create and mantain a clean and entreprise wide data product ? The domain will be focused on its own needs, not the ones of the other domains. You gave the exemple of microservices, but the main usage is to build intra-domain service, when it comes to create an microservice for other domains, it becomes a lot harder to design and develop (priorities of the domain owner may not be the same as of consumer domains)...
Great concerns, I have similar based on places I've worked at. Except maybe 1, where data was the primary product of the company. I'm not sure I've heard this addressed other than "leadership has to make it a priority."
I give to you an example...a pro user of the front ask a place to insert the average human cost by cost center. Since the person is a central figure in controlling dept. We prepare a report with sql write back functionality. He ends up compiling only the cost center that regards him (not every cost center worldwide). This is the culture, can we pretend a final user set up ETL and organize itself with other team to build a data mesh? There are people that run into your office with printed sheet of their screen and when you ask them to open a ticket, they scan the sheet otherwise they are not able to put the image on an email.
Architecting from domain level could lead to form another silo. No enterprise level business and data modeling means new chaos in very near future. The only way is doing enterprise data modeling, base one business data cohesion decides business object (or component) , and then IT architecture like data architecture, application architecture, technical architecture etc. Thinking about this scenario, HR domain produces like employees data, Customers domain produces customer data, it seems that different domain produces different data, but what about if employees are customers? How to ensure data consistency like PII, individual addresses, contact information. Without enterprise level business modeling , you never know what is real Business Object. Another example, collaterals (such as house, bond, equity, vehicle etc) could be your assets. If architecting by domain, could be collateral management and asset management, which is very common mistake in most banking architecture.
This is a great overview of Data Mesh, but is missing what is probably the hardest part of Data Mesh - Federated Computational Governance. The solution mentioned in this video would be difficult enough of a mindshift. Layer on top of that Computational Governance and the journey gets about 10x-100x longer.
So practically we are saying to Devs stop messing around and care for your product, as data are part of your product ... De Teams are not your maids... In the end again de team needs to know what it gets in order to give knowledge back to company. At least the data will be already cleansed
The problem is your are looking at it from a technological viewpoint, especially where your are talking about parquet files and DataBricks. Those data pipelines have to be simplified to EL centric processes still performed by IT as their "infrastructure support". You need to also empower you end users with GUI-based Semantic layer tools so that IT perhaps builds the initial models but the end-users can make fine adjustments and maintain them.
You said it rite. It would never work 😆. With no centralized source of truth and every domain team handling it's own data, it would increase manpower cost and communication between teams to share their data with correct schema for aggregation would be more hectic process than a single team managing everything.
This really only applies to companies which sell data as a product? I don't see why you would ever need to do something like this with traditional organizational reporting as you showcased in your example (HR, Finance, SC Data). Or am I missing something? It seems like a lot of extra work compared to like a data vault architecture for example which accomplishes something similar with far less headcount and effort.
Possibly? It's a relatively new architecture so it will be interesting to see how it's applied over the next few years. I agree it's definitely geared towards data as a product companies. I can think of a few companies I've worked at where it might be fun to try, and a few where it'd be a nightmare. For the reasons you stated.
If you're skeptical about how data mesh would be implemented, I followed up with some of the doubts people might have: ruclips.net/video/CMNzHAdrigQ/видео.html
love your content my man. The ownership piece really is the value add part here. As the businesses I deal with mostly want that to be "IT's problem" to deal with.
Thanks for this quick overview. I wonder what are the motivation for a domain to create and mantain a clean and entreprise wide data product ? The domain will be focused on its own needs, not the ones of the other domains. You gave the exemple of microservices, but the main usage is to build intra-domain service, when it comes to create an microservice for other domains, it becomes a lot harder to design and develop (priorities of the domain owner may not be the same as of consumer domains)...
Great concerns, I have similar based on places I've worked at. Except maybe 1, where data was the primary product of the company. I'm not sure I've heard this addressed other than "leadership has to make it a priority."
I give to you an example...a pro user of the front ask a place to insert the average human cost by cost center. Since the person is a central figure in controlling dept.
We prepare a report with sql write back functionality.
He ends up compiling only the cost center that regards him (not every cost center worldwide).
This is the culture, can we pretend a final user set up ETL and organize itself with other team to build a data mesh?
There are people that run into your office with printed sheet of their screen and when you ask them to open a ticket, they scan the sheet otherwise they are not able to put the image on an email.
Amazingly simplified with your explanation. Thanks!
Architecting from domain level could lead to form another silo. No enterprise level business and data modeling means new chaos in very near future. The only way is doing enterprise data modeling, base one business data cohesion decides business object (or component) , and then IT architecture like data architecture, application architecture, technical architecture etc. Thinking about this scenario, HR domain produces like employees data, Customers domain produces customer data, it seems that different domain produces different data, but what about if employees are customers? How to ensure data consistency like PII, individual addresses, contact information. Without enterprise level business modeling , you never know what is real Business Object. Another example, collaterals (such as house, bond, equity, vehicle etc) could be your assets. If architecting by domain, could be collateral management and asset management, which is very common mistake in most banking architecture.
This is a great overview of Data Mesh, but is missing what is probably the hardest part of Data Mesh - Federated Computational Governance. The solution mentioned in this video would be difficult enough of a mindshift. Layer on top of that Computational Governance and the journey gets about 10x-100x longer.
So practically we are saying to Devs stop messing around and care for your product, as data are part of your product ... De Teams are not your maids... In the end again de team needs to know what it gets in order to give knowledge back to company. At least the data will be already cleansed
The problem is your are looking at it from a technological viewpoint, especially where your are talking about parquet files and DataBricks. Those data pipelines have to be simplified to EL centric processes still performed by IT as their "infrastructure support". You need to also empower you end users with GUI-based Semantic layer tools so that IT perhaps builds the initial models but the end-users can make fine adjustments and maintain them.
So, you invented Kimball's datamarts
Uh, in case it's not clear I didn't "invent" data mesh. This is just a video explaining the concept.
Sounds like going back to creating data silos
You said it rite. It would never work 😆. With no centralized source of truth and every domain team handling it's own data, it would increase manpower cost and communication between teams to share their data with correct schema for aggregation would be more hectic process than a single team managing everything.
Great explanation, I am non technical and I got the ideas!
Great to hear!
Thank you for this video!
can you please do a video on quantum computing and its scope?
Great video explanation.
Glad it made sense, it was a lot of info to condense!
Can you cover $dag constellation. They created a hgtp// protocol that acts as a decentralized data mesh/micro service, fully zero trust.
This really only applies to companies which sell data as a product? I don't see why you would ever need to do something like this with traditional organizational reporting as you showcased in your example (HR, Finance, SC Data). Or am I missing something? It seems like a lot of extra work compared to like a data vault architecture for example which accomplishes something similar with far less headcount and effort.
Possibly? It's a relatively new architecture so it will be interesting to see how it's applied over the next few years. I agree it's definitely geared towards data as a product companies. I can think of a few companies I've worked at where it might be fun to try, and a few where it'd be a nightmare. For the reasons you stated.
It’s because this is how modern teams will communicate and define its inputs and outputs via data as opposed to chats/talking.
Nice channel, I love it
Glad you enjoy it!
No