Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 28 additions & 2 deletions _docs/00-index.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,27 @@ description: "The map — what this reference builds, the reading order, and a l
duration: 5 min
---

## Learning objectives

By the end of this chapter you can:

- Name the four movements the reference set is organized into and which pages fall
into each one.
- Pick the reading order that matches how you want to use the set — straight through,
or jumping to one topic.
- Identify the three pages that cover the pattern's shape if you only have time for a
few.
- Locate the page that answers a specific question without re-reading the whole set.

## Why this matters

This is the reading guide for the data-mesh reference — the map of what the set covers
and the order to read it in. The reference builds one working system end to end: a data
mesh of services that own their data as products, talk over a deliberate mix of
protocols, evolve their contracts safely, scale to demand, and stay observable
throughout.

The set is written to be read straight through the first time — each page picks up where
The set is written to be read straight through the first time. Each page picks up where
the last left off, and the cross-references assume you've seen the earlier material. If
you're returning to find one thing, the descriptions below will point you at the right
page.
Expand Down Expand Up @@ -50,7 +64,7 @@ the value it delivers and the implementation pieces that realize it.
The synchronous read layer (REST, gRPC, a GraphQL gateway) and the asynchronous event
backbone — and why the capstone uses all of them.
- [**7 · Progressive delivery & mTLS**]({{ '/docs/06-progressive-delivery-mtls/' | relative_url }}) —
Evolving a contract in the open with a v1→v2 canary, mTLS for free, and the decision to
Evolving a contract in the open with a v1→v2 canary, mTLS with no application changes, and the decision to
mesh selectively rather than namespace-wide.
- [**8 · Elastic & resilient**]({{ '/docs/07-elastic-and-resilient/' | relative_url }}) —
Scaling to demand and to zero with KEDA, and the cloud-native recoverability the
Expand All @@ -75,3 +89,15 @@ to avoid. Those three cover the shape of the thing without the full implementati
depth.

Start with [concepts & principles]({{ '/docs/01-concepts/' | relative_url }}).

## For deeper coverage

- *Data Mesh* by Zhamak Dehghani — the full argument this reference implements, including
the history and the operating-model shifts this set only summarizes.
- *Designing Data-Intensive Applications, 2nd ed.* by Martin Kleppmann & Chris
Riccomini — the data-systems foundations (storage, replication, consistency) that sit
underneath every data product in this build.
- *Cloud Native Patterns* by Cornelia Davis — the platform patterns this reference's
self-serve tier draws on.
- *Kubernetes Patterns* by Bilgin Ibryam & Roland Huß — the Kubernetes-native building
blocks the implementation pages rely on throughout.
32 changes: 30 additions & 2 deletions _docs/01-concepts.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,20 @@ description: What a data mesh is, operational vs. analytical data, and Dehghani'
duration: 20 min
---

## Learning objectives

By the end of this chapter you can:

- State what a data mesh is and the failure it responds to, in your own words.
- Distinguish operational data from analytical data and explain how a mesh
reorganizes that boundary by domain rather than by technology layer.
- Name Dehghani's four principles and give a one-line example of each from this
build.
- Recognize the difference between the data-mesh pattern and the specific tools this
reference uses to express it.

## Why this matters

The [previous page]({{ '/docs/01-data-architectures/' | relative_url }}) covered the
landscape of data architectures — pipelines, warehouses, and lakes — and where the mesh
sits in relation to all of them. This page goes deeper into what a data mesh actually
Expand All @@ -13,7 +27,7 @@ is and the four principles it rests on.
Before any commands, it's worth being precise about what a data mesh actually is —
because the term gets attached to a lot of things it isn't. This page is a working
grounding in the idea and its four principles, enough to make the rest of the set
make sense. It's deliberately brief: if you want the full conceptual treatment —
make sense. It's brief: if you want the full conceptual treatment —
the history, the operating-model shifts, the worked argument for decentralization —
that's what the *Data Mesh 101* material in the presentation is for. Here we cover
just enough to build on.
Expand Down Expand Up @@ -41,7 +55,7 @@ let them interoperate.

## Operational vs. analytical data

One distinction underlies everything and is worth stating plainly, because blurring it
One distinction underlies everything, and blurring it
is itself a common mistake. **Operational data** is the current-state data behind a
domain's running services — the rows a microservice reads and writes to do its job,
transactional and live. **Analytical data** is the historical, aggregated view used to
Expand Down Expand Up @@ -117,3 +131,17 @@ common way these efforts fail is mistaking the tooling for the transformation.
With the vocabulary in place, the next page looks at why Kubernetes is a natural
substrate for all of this — and how each of the four principles maps onto concrete
Kubernetes primitives.

## For deeper coverage

- *Data Mesh* by Zhamak Dehghani — the primary source for the four principles and the
domain-ownership argument this page summarizes.
- *Learning Domain-Driven Design* by Vlad Khononov — the bounded-context thinking
behind domain ownership and why a data product maps to a domain, not a table.
- *Implementing Domain-Driven Design* by Vaughn Vernon — a deeper, implementation-level
treatment of the same domain boundaries.
- *Designing Data-Intensive Applications, 2nd ed.* by Martin Kleppmann & Chris
Riccomini — the operational-versus-analytical data distinction from the systems
side.
- *Cloud Native Patterns* by Cornelia Davis — the self-serve platform principle made
concrete in cloud-native infrastructure terms.
93 changes: 64 additions & 29 deletions _docs/01-data-architectures.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,28 @@ description: "The landscape: data pipelines, data warehouses, data lakes, and da
duration: 25 min
---

## Learning objectives

By the end of this chapter you can:

- Describe data pipelines, data warehouses, and data lakes — the problem each pattern
solves and the limitation that bounds it.
- Explain why pipeline count grows as the product of sources and destinations, not
the sum, and why that growth is called pipeline sprawl.
- Contrast schema-on-write and schema-on-read, and say which pattern uses each.
- Match an organization's bottleneck (movement, truth, flexibility, or organizational
scaling) to the pattern built to solve it.

## Why this matters

Before defining data mesh precisely, it helps to see the landscape it lives in. The
mesh did not appear out of nowhere — it's a response to specific organizational scaling
problems that earlier data architectures left unsolved. Each of those earlier patterns
solved a real problem, and each is still the right answer when that problem is the
dominant one. Understanding what came before, and what each pattern does well, makes
the mesh's value proposition concrete rather than abstract.
problems that earlier data architectures left unsolved.

Each of those earlier patterns solved a real problem, and each is still the right
answer when that problem is the dominant one. Understanding what came before, and what
each pattern does well, makes the mesh's value proposition concrete rather than
abstract.

## Data pipelines

Expand Down Expand Up @@ -69,15 +85,16 @@ The limitation is organizational, not technical. The warehouse is owned by a cen
data team — sometimes called the BI team, the analytics engineering team, or the data
platform team — and that team becomes a bottleneck in direct proportion to the
organization's growth. The central team owns all the data but understands none of the
domains that produced it. When the logistics domain needs a new dimension added to the
warehouse, the request goes to the central team, who must learn enough about logistics
to model it correctly, prioritize it against requests from every other domain, and
coordinate the schema change without breaking downstream consumers. The analytical
view is always hours or days behind the operational truth, because the ingestion
pipelines run on a schedule, and the central team's capacity to model new data is the
constraint on how fast the warehouse can grow. As the organization adds domains and
use cases, the central team processes more requests than it can understand, and the
queue becomes the bottleneck.
domains that produced it.

When the logistics domain needs a new dimension added to the warehouse, the request
goes to the central team, who must learn enough about logistics to model it correctly,
prioritize it against requests from every other domain, and coordinate the schema
change without breaking downstream consumers. The analytical view is always hours or
days behind the operational truth, because the ingestion pipelines run on a schedule,
and the central team's capacity to model new data is the constraint on how fast the
warehouse can grow. As the organization adds domains and use cases, the central team
processes more requests than it can understand, and the queue becomes the bottleneck.

{% include excalidraw.html file="01-data-warehouse-architecture" alt="A centralized data warehouse receiving data from multiple operational systems through ingestion pipelines, with a central team owning the schema and all analytical consumers querying one governed store" caption="Figure 1.2 — The data warehouse: a centralized, schema-on-write analytical store that solves the truth problem but concentrates ownership in a single team." %}

Expand Down Expand Up @@ -111,12 +128,14 @@ The limitation is governance. Without active curation, the lake becomes the **da
swamp** — a term so common in the field that it's practically a synonym for a
poorly-run lake. Undocumented datasets accumulate. Nobody knows which version of the
customer table is authoritative, which datasets are stale, which ones duplicate each
other under different names. The curation effort needed to keep a lake useful
re-creates much of the warehouse's modeling and governance work, just without the
warehouse's enforcement mechanisms. And critically, the organizational problem is
unchanged: a central team still owns the lake, still cannot understand every domain's
data, and still bottlenecks on the same requests. The technology changed — from rigid
schema to flexible storage — but the shape of the organization did not.
other under different names.

The curation effort needed to keep a lake useful re-creates much of the warehouse's
modeling and governance work, just without the warehouse's enforcement mechanisms. And
critically, the organizational problem is unchanged: a central team still owns the
lake, still cannot understand every domain's data, and still bottlenecks on the same
requests. The technology changed — from rigid schema to flexible storage — but the
shape of the organization did not.

{% include excalidraw.html file="01-data-lake-architecture" alt="A data lake with raw, curated, and refined zones, showing heterogeneous data formats flowing in and the risk of the lake becoming a swamp without active governance" caption="Figure 1.3 — The data lake: schema-on-read and format flexibility, but without governance, the lake becomes a swamp — and the central-team bottleneck remains." %}

Expand All @@ -125,18 +144,21 @@ schema to flexible storage — but the shape of the organization did not.
All three patterns — pipelines, warehouses, lakes — centralize data and hand
ownership to a single team. The technology improves with each generation: from
bespoke point-to-point plumbing, to a governed analytical store, to a flexible
format-agnostic lake. But the organizational shape stays the same in every case: one
team at the center, receiving requests from every domain, owning data it did not
produce, modeling concepts it does not deeply understand, and scaling its capacity
linearly while the demands on it grow combinatorially.
format-agnostic lake.

But the organizational shape stays the same in every case: one team at the center,
receiving requests from every domain, owning data it did not produce, modeling
concepts it does not deeply understand, and scaling its capacity linearly while the
demands on it grow combinatorially.

The **data mesh** proposes a different axis of change. Instead of building a better
center, it decentralizes ownership to the domain teams that produce the data. Each
domain owns its data as a product — discoverable, addressable, trustworthy,
self-describing — and publishes it for other domains to consume. A shared
**self-serve data platform** provides the infrastructure that every domain needs
(streaming, storage, observability, schema registries) so that domain teams don't
each build their own. And **federated computational governance** keeps the
self-describing — and publishes it for other domains to consume.

A shared **self-serve data platform** provides the infrastructure that every domain
needs (streaming, storage, observability, schema registries) so that domain teams
don't each build their own. And **federated computational governance** keeps the
independently-owned products interoperable — not through review boards and policy
documents, but through standards enforced automatically by the platform itself.

Expand All @@ -151,9 +173,9 @@ domain may still use a warehouse or a lake internally — the order-analytics do
might store its refined data in a Parquet-based lakehouse, and that's fine. The
reorganization is about *who owns the data*, not about which storage technology to
use. The mesh is an answer to the question "how does this organization scale its data
architecture?" — not "which database should this team pick?"
architecture?", not "which database should this team pick?"

The mesh earns its complexity in organizations with many domains, many data consumers,
The mesh's complexity pays off in organizations with many domains, many data consumers,
and the operational maturity to sustain federated ownership. For smaller teams, simpler
data flows, or organizations where the central team is not yet the bottleneck, a
well-run warehouse or a governed lake may be exactly the right answer. The mesh is not
Expand Down Expand Up @@ -209,3 +231,16 @@ Dehghani's four principles are, how they interlock, and why implementing one wit
the others produces a distributed mess rather than a mesh.

[Concepts & principles]({{ '/docs/01-concepts/' | relative_url }}).

## For deeper coverage

- *Data Mesh* by Zhamak Dehghani — the full argument for why pipelines, warehouses,
and lakes hit an organizational ceiling, and what the mesh changes.
- *Designing Data-Intensive Applications, 2nd ed.* by Martin Kleppmann & Chris
Riccomini — the systems-level treatment of batch and stream processing, ETL/ELT,
and the storage engines behind warehouses and lakes.
- *Cloud Native Anti-Patterns* by Gerald Bachlmayr et al. — common organizational and
architectural failure modes, including the centralized-ownership trap this chapter
describes.
- *Cloud Native Patterns* by Cornelia Davis — the platform-level patterns a self-serve
data platform draws on once the organizational shift is underway.
Loading
Loading