2

I've come across with the new concept "Data Mesh" recently. After reading some blogs and watched introduction videos about Data Mesh, it's not clear to me what is the difference between Data Mesh and multiple Data Warehouses in an organisation from a technical perspective.

If anyone is familiar with this concept, could you please share with me:

  1. Except the "domain oriented" principle, what's the difference of a Data Mesh and multiple Data Warehouses for different domains?
  2. How does Data Mesh solves the problem of integrating data from different apartments(meshes)?

Thanks :)!

Here are some links for Data Mesh introduction:

How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh

Introduction to Data Mesh

ZwiebelXin
  • 31
  • 4

2 Answers2

0

They are many differences, but one is the standardization of API to access the data and metadata. By nature, the data quantum, which is the "atomic" element of the data mesh is agnostic of its data store (or data stores). So when you are thinking about Observability, Dictionary, and Control access of your data quanta, you want uniformity.

jgp
  • 2,069
  • 1
  • 21
  • 40
0

From a technical point of view, Data Warehouse exists around the concept of ETL (Extract, Transform, Load). This means, before data gets into the warehouse, it's moved (and often transformed).

Data Mesh is all about data staying where it is, inside the domains, it avoids pipelines (or keeps it to the bare minimum in the place where the data originates from).

I don't exactly get what you mean by "integrating data from different apartments" but I'm assuming it's about integrating data from different domains?

You can use a distributed query engine (for example Trino or Spark SQL) to connect data from different sources (use multiple predefined "catalogs") in one query.

There are other aspect on Data Mesh, most specifically about changing the company culture and the way it works around the domains. Take a look here as well: DataMesh Architecture -Common data implementation

wbrycki
  • 121
  • 1
  • 8