🕸 What's Data Mesh?
The Tech Caffeine #44: This Week in Tech
I hope you're enjoying "The Tech Caffeine." I would appreciate if you passed this along to a friend.
🗞 In This Issue
Top 5 Articles that I am recommending you to read
Deep Dive: What is Data Mesh?
Watch this: SREcon22 Americas - Principled Performance Analytics by Narayan Desai and Brent Bryan from Google
Tweets of the week
This week, I published a booklet - “RESTful API Design — Step By Step Guide“
Most of us use or build REST APIs in day-to-day life as software developers. APIs are the default means of communication between the systems. Amazon is the best example of how APIs can be efficiently used for communication.
In this booklet, I talk about how to design your RESTful APIs better and avoid common mistakes.
For downloading a FREE copy of this booklet, use a discount code “thetechaffeine“ at the time of checkout.
👉 Top Five
1️⃣ Honeycomb.io is providing complimentary copies of the book - Observability Engineering. The book provides great insights. Don’t miss out on this opportunity.
2️⃣ If Wall Street had to invent a company, it would probably look a lot like Broadcom. Find out why in this article.
3️⃣ DuckDuckGo, the privacy-centric search firm have been found to be allowing Microsoft trackers through their browser.
4️⃣ Last week, I published 15 Interview Questions for Software Architects with Hints. If you have an interview lined up, this might be useful;
5️⃣ Check out this post where Slack Engineering has shared its journey on building a culture of performance.
If you would like to suggest an article for this section, let me know on Twitter.
🤿 The Deep Dive
What is Data Mesh?
Data Mesh is a type of data architecture introduced by Zhamak Dehghani in 2019.
It is an architectural and organizational paradigm that challenges the age-old assumption that we must centralize big analytical data to use it, have data all in one place, or be managed by a centralized data team to deliver value.
The Data Mesh concept encourages exposing data as products and decentralizes the ownership of the domains.
Why do you need Data Mesh?
In many organizations, the central data team often becomes a bottleneck. The whole organization expects them to ingest/curate humongous data.
The source data teams have no incentive to provide quality data. The only expectation is to provide connections so that the central team can ingest this data into a central data store. Many application teams think data is the byproduct of what they are building.
The data teams often struggle to understand the data as it is owned by someone else, and each of these teams works in silos.
Collecting huge amounts of data in one place can cause a security risk.
In reality what we find is disconnected source teams, frustrated consumers fighting for a spot on top of the data platform team backlog and an over stretched data platform team.
What is Data Mesh?
The Data Mesh concept is built on the following principles -
Domain Data Ownership - This principle expects the domain teams to take ownership of the data they provide. The expectation is that domain teams should provide data products that everyone can trust and use.
Data as a Product - This Data Mesh principle expects us to apply product thinking to our shared data. Data should be made discoverable using data catalogs. There should be a unique way to identify/address a data product. Data Product should be monitored for data quality using SLI/SLOs. It should be self-describing(Data + Metadata). It should have a standard data format so that it become interoperable. And we should follow the best practices to keep it secure.
Self-Serve Data Platform - To avoid duplication of efforts in technology selection, the central data platform team can build/maintain/suggest the use of standard technologies to make data products available for everyone.
Federated Computational Governance - This principle suggests automating governance and compliance management in the distributed environment. For example, standardizing the data format for the data products, policies around registering data products in the data catalog, etc.
What is Data Mesh NOT?
It’s NOT a technology but an approach or a practice.
You cannot BUY a Data Mesh technology from a Vendor. Vendors can help you in certain aspects like Self-serve data platforms, data catalogs, etc.
It’s NOT a silver bullet that will solve all your data problems by default. You need to make an effort to make many things work.
When NOT to use Data Mesh?
Data Mesh requires organizational level changes. If your organization is not open to change, the effort to implement Data Mesh might be in vain.
Breaking siloed teams into Domain Teams.
Data Mesh requires product thinking applied to the data world. If your people in your organization can’t accept that then there is no Data Mesh.
Where can I read more about it?
Join Data Mesh Learning Newsletter -
👀 Watch This
SREcon22 Americas - Principled Performance Analytics
In this presentation, Narayan Desai and Brent Bryan from Google talk about why we need to look beyond SLOs.
🐥 Tweets of the Week
