Do we require centralised infrastructure despite having central metastore (table meta & access)? Every derived data product can use its own infra by getting relevant accesses via metastore controls. Centralised governance on all metastores and workflow orchestration (or minimum CDC triggers) can bring the best isolation and best scaling model.
Interesting that Arif states that Zhamak was the first person to coin the term Data Mesh in her whitepaper. Gartner published an article on Data Mesh in 2016. The term has been around a while.
Going on the same journey! How do you handle number of workspaces and external hives? Is it feasible having as few workspaces as possible and using table access control, or do you want many different workspaces and then use syncing mechanisms between external hives to share the metadata? I'm a bit torn on how to do the split.
I did not understand the compulsion of keeping still the centralised service and taking your own bucket concept. A data mesh should everything be decentralised where everything is domain bounded. The interoperability and discoverability should be provided by each service rather than keeping this in centralised storage.
Looks good on paper ~ HOWEVER ~ most of the consumers need data ACROSS domains (or products). This is the VERY REASON we have data warehouses. Integrated data, or data across domains or products. Looks good on the back of a napkin, but uh, it's NOT practical in the real world where integrated data is needed.
Build data products QUICKLY..... uh, like we just 'add water' and the data grows on its own? LOL. My friend, you be smokin' something. Operational data ALWAYS coexists with function. It's a form or an app that creates data, generally either by selection or data entry. SOMEBODY has to write code SOMEWHERE if you want new data (for a product) that doesn't already exist.
Do we require centralised infrastructure despite having central metastore (table meta & access)? Every derived data product can use its own infra by getting relevant accesses via metastore controls.
Centralised governance on all metastores and workflow orchestration (or minimum CDC triggers) can bring the best isolation and best scaling model.
Interesting that Arif states that Zhamak was the first person to coin the term Data Mesh in her whitepaper. Gartner published an article on Data Mesh in 2016. The term has been around a while.
New paradigm, very well explained !
Summary 30:29
Going on the same journey! How do you handle number of workspaces and external hives? Is it feasible having as few workspaces as possible and using table access control, or do you want many different workspaces and then use syncing mechanisms between external hives to share the metadata? I'm a bit torn on how to do the split.
Great presentation!!! I wonder if at the metadata layer you are using among other many things lake formation.
I did not understand the compulsion of keeping still the centralised service and taking your own bucket concept. A data mesh should everything be decentralised where everything is domain bounded. The interoperability and discoverability should be provided by each service rather than keeping this in centralised storage.
I think it is because then you can still do central governance. Easier to manage access rights when the buckets are centralised.
You just explained what data mesh is. There are no insights as to how this was implemented.
cristal clear!!!
Looks good on paper ~ HOWEVER ~ most of the consumers need data ACROSS domains (or products). This is the VERY REASON we have data warehouses. Integrated data, or data across domains or products. Looks good on the back of a napkin, but uh, it's NOT practical in the real world where integrated data is needed.
Build data products QUICKLY..... uh, like we just 'add water' and the data grows on its own? LOL. My friend, you be smokin' something. Operational data ALWAYS coexists with function. It's a form or an app that creates data, generally either by selection or data entry. SOMEBODY has to write code SOMEWHERE if you want new data (for a product) that doesn't already exist.