Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 19 additions & 19 deletions _docs/00-index.md
Original file line number Diff line number Diff line change
@@ -1,53 +1,53 @@
---
title: "The data mesh: start here"
order: 0
description: "The map: what this reference builds, the reading order, and a link to each page in the set. Start here."
description: "The map: what this reference builds, the reading order, and a link to each chapter in the set. Start here."
duration: 5 min
---

## Learning objectives

By the end of this chapter you can:

- Name the four movements the reference set is organized into and which pages fall
- Name the four movements the reference set is organized into and which chapters fall
into each one.
- Pick the reading order that matches how you want to use the set: straight through,
or jumping to one topic.
- Identify the three pages that cover the pattern's shape if you only have time for a
- Identify the three chapters that cover the pattern's shape if you only have time for a
few.
- Locate the page that answers a specific question without re-reading the whole set.
- Locate the chapter that answers a specific question without re-reading the whole set.

## Why this matters

Page 2 names Dehghani's four principles. Page 3 takes the same four and maps each one
Chapter 2 names Dehghani's four principles. Chapter 3 takes the same four and maps each one
to a Kubernetes primitive: namespaces, operators, RBAC. That's the shape of this whole
set. A concept gets introduced, then you watch it land in a running piece of the
system a few pages later. This page is the map, nothing more: what the reference
covers, the order to read it in, and a link to each page.
system a few chapters later. This chapter is the map, nothing more: what the reference
covers, the order to read it in, and a link to each chapter.

The reference builds one working system end to end. A data mesh of services that own
their data as products, that talk over a deliberate mix of protocols, that evolve their
contracts safely, that scale to demand, and that stay observable throughout.

Read it straight through the first time if you can. Each page picks up where the last
Read it straight through the first time if you can. Each chapter picks up where the last
left off, and the cross-references assume you've seen the earlier material. If you're
back here looking for one specific thing, the descriptions below point at the right
page.
chapter.

## How the set is organized

The eleven pages fall into four movements. **Conceptual grounding** (pages 1 to 3)
The eleven chapters fall into four movements. **Conceptual grounding** (chapters 1 to 3)
covers the landscape of data architectures, what a data mesh actually is, and why
Kubernetes turns out to be a natural substrate for one. **Implementation** (pages 4 to
Kubernetes turns out to be a natural substrate for one. **Implementation** (chapters 4 to
9) builds the system: the services and their data products, the contracts and catalog
that make them discoverable, the data planes they talk over, progressive delivery with
mutual TLS, elastic scaling and recovery, and the observability to see all of it.
**Failure modes** (page 10) steps back to the conceptual and organizational
**Failure modes** (chapter 10) steps back to the conceptual and organizational
anti-patterns that derail data-mesh efforts even when the technology is sound. The
**summary** (page 11) closes by reorganizing the same material by principle: for each
**summary** (chapter 11) closes by reorganizing the same material by principle: for each
of the four, the value it delivers and the implementation pieces that realize it.

## The pages
## The chapters

- [**1 · Data architectures**]({{ '/docs/01-data-architectures/' | relative_url }}):
the landscape from pipelines to lakes to mesh, what each pattern is, the problem
Expand Down Expand Up @@ -96,12 +96,12 @@ Start with [concepts & principles]({{ '/docs/01-concepts/' | relative_url }}).

## For deeper coverage

- *Data Mesh* by Zhamak Dehghani: the full argument this reference implements, including
- [*Data Mesh*](https://www.oreilly.com/library/view/data-mesh/9781492092384/) by Zhamak Dehghani: the full argument this reference implements, including
the history and the operating-model shifts this set only summarizes.
- *Designing Data-Intensive Applications, 2nd ed.* by Martin Kleppmann & Chris
- [*Designing Data-Intensive Applications, 2nd ed.*](https://www.oreilly.com/library/view/designing-data-intensive-applications/9781098119058/) by Martin Kleppmann & Chris
Riccomini: the data-systems foundations (storage, replication, consistency) that sit
underneath every data product in this build.
- *Cloud Native Patterns* by Cornelia Davis: the platform patterns this reference's
- [*Cloud Native Patterns*](https://www.manning.com/books/cloud-native-patterns) by Cornelia Davis: the platform patterns this reference's
self-serve tier draws on.
- *Kubernetes Patterns* by Bilgin Ibryam & Roland Huß: the Kubernetes-native building
blocks the implementation pages rely on throughout.
- [*Kubernetes Patterns*](https://www.oreilly.com/library/view/-/9781098131678/) by Bilgin Ibryam & Roland Huß: the Kubernetes-native building
blocks the implementation chapters rely on throughout.
10 changes: 5 additions & 5 deletions _docs/01-concepts.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,14 +138,14 @@ Kubernetes primitives.

## For deeper coverage

- *Data Mesh* by Zhamak Dehghani: the primary source for the four principles and the
- [*Data Mesh*](https://www.oreilly.com/library/view/data-mesh/9781492092384/) by Zhamak Dehghani: the primary source for the four principles and the
domain-ownership argument this page summarizes.
- *Learning Domain-Driven Design* by Vlad Khononov: the bounded-context thinking
- [*Learning Domain-Driven Design*](https://www.oreilly.com/library/view/learning-domain-driven-design/9781098100124/) by Vlad Khononov: the bounded-context thinking
behind domain ownership and why a data product maps to a domain, not a table.
- *Implementing Domain-Driven Design* by Vaughn Vernon: a deeper, implementation-level
- [*Implementing Domain-Driven Design*](https://www.informit.com/store/implementing-domain-driven-design-9780133039894) by Vaughn Vernon: a deeper, implementation-level
treatment of the same domain boundaries.
- *Designing Data-Intensive Applications, 2nd ed.* by Martin Kleppmann & Chris
- [*Designing Data-Intensive Applications, 2nd ed.*](https://www.oreilly.com/library/view/designing-data-intensive-applications/9781098119058/) by Martin Kleppmann & Chris
Riccomini: the operational-versus-analytical data distinction from the systems
side.
- *Cloud Native Patterns* by Cornelia Davis: the self-serve platform principle made
- [*Cloud Native Patterns*](https://www.manning.com/books/cloud-native-patterns) by Cornelia Davis: the self-serve platform principle made
concrete in cloud-native infrastructure terms.
8 changes: 4 additions & 4 deletions _docs/01-data-architectures.md
Original file line number Diff line number Diff line change
Expand Up @@ -237,13 +237,13 @@ the others produces a distributed mess rather than a mesh.

## For deeper coverage

- *Data Mesh* by Zhamak Dehghani: the full argument for why pipelines, warehouses,
- [*Data Mesh*](https://www.oreilly.com/library/view/data-mesh/9781492092384/) by Zhamak Dehghani: the full argument for why pipelines, warehouses,
and lakes hit an organizational ceiling, and what the mesh changes.
- *Designing Data-Intensive Applications, 2nd ed.* by Martin Kleppmann & Chris
- [*Designing Data-Intensive Applications, 2nd ed.*](https://www.oreilly.com/library/view/designing-data-intensive-applications/9781098119058/) by Martin Kleppmann & Chris
Riccomini: the systems-level treatment of batch and stream processing, ETL/ELT,
and the storage engines behind warehouses and lakes.
- *Cloud Native Anti-Patterns* by Gerald Bachlmayr et al.: common organizational and
- [*Cloud Native Anti-Patterns*](https://www.packtpub.com/en-us/product/cloud-native-anti-patterns-9781836200581) by Gerald Bachlmayr et al.: common organizational and
architectural failure modes, including the centralized-ownership trap this chapter
describes.
- *Cloud Native Patterns* by Cornelia Davis: the platform-level patterns a self-serve
- [*Cloud Native Patterns*](https://www.manning.com/books/cloud-native-patterns) by Cornelia Davis: the platform-level patterns a self-serve
data platform draws on once the organizational shift is underway.
6 changes: 3 additions & 3 deletions _docs/02-kubernetes-substrate.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,11 +146,11 @@ is published and reachable.

## For deeper coverage

- *Kubernetes Patterns* by Bilgin Ibryam & Roland Huß: the primitive-by-primitive
- [*Kubernetes Patterns*](https://www.oreilly.com/library/view/-/9781098131678/) by Bilgin Ibryam & Roland Huß: the primitive-by-primitive
treatment (namespaces, operators, CRDs) this chapter's mapping draws on.
- *Platform Engineering on Kubernetes* by Mauricio Salatino: the self-serve platform
- [*Platform Engineering on Kubernetes*](https://www.manning.com/books/platform-engineering-on-kubernetes) by Mauricio Salatino: the self-serve platform
principle built out as an operating model, not just a set of operators.
- *Cloud Native Patterns* by Cornelia Davis: the broader cloud-native platform
- [*Cloud Native Patterns*](https://www.manning.com/books/cloud-native-patterns) by Cornelia Davis: the broader cloud-native platform
patterns this final project's shared infrastructure tier follows.
- [Kubernetes documentation](https://kubernetes.io/docs/): namespaces, RBAC, CRDs,
and admission control, straight from the source.
Expand Down
6 changes: 3 additions & 3 deletions _docs/03-services-and-data-products.md
Original file line number Diff line number Diff line change
Expand Up @@ -218,8 +218,8 @@ themselves so others can find and trust them: contracts and the catalog.

## For deeper coverage

- *Data Mesh* by Zhamak Dehghani: the architectural quantum and why a data product's boundary should match a service's boundary.
- *Cloud Native Patterns* by Cornelia Davis: packaging and runtime patterns for services that double as data products.
- *Kubernetes Patterns* by Bilgin Ibryam & Roland Huß: container image and workload patterns behind the build-load-deploy sequence.
- [*Data Mesh*](https://www.oreilly.com/library/view/data-mesh/9781492092384/) by Zhamak Dehghani: the architectural quantum and why a data product's boundary should match a service's boundary.
- [*Cloud Native Patterns*](https://www.manning.com/books/cloud-native-patterns) by Cornelia Davis: packaging and runtime patterns for services that double as data products.
- [*Kubernetes Patterns*](https://www.oreilly.com/library/view/-/9781098131678/) by Bilgin Ibryam & Roland Huß: container image and workload patterns behind the build-load-deploy sequence.
- [Kubernetes documentation](https://kubernetes.io/docs/): image pull policies and the kubelet's image lifecycle.
- [FastAPI](https://fastapi.tiangolo.com/): the framework order-service and its siblings are built on.
6 changes: 3 additions & 3 deletions _docs/04-contracts-and-catalog.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ and the asynchronous event backbone.

## For deeper coverage

- *Data Mesh* by Zhamak Dehghani: data-as-a-product and federated computational governance, the two principles this chapter makes operational.
- *Designing Data-Intensive Applications, 2nd ed.* by Martin Kleppmann & Chris Riccomini: schema evolution and the compatibility guarantees a registry enforces.
- *Cloud Native Anti-Patterns* by Gerald Bachlmayr et al.: the proxy anti-pattern a mesh degrades into without a usable catalog.
- [*Data Mesh*](https://www.oreilly.com/library/view/data-mesh/9781492092384/) by Zhamak Dehghani: data-as-a-product and federated computational governance, the two principles this chapter makes operational.
- [*Designing Data-Intensive Applications, 2nd ed.*](https://www.oreilly.com/library/view/designing-data-intensive-applications/9781098119058/) by Martin Kleppmann & Chris Riccomini: schema evolution and the compatibility guarantees a registry enforces.
- [*Cloud Native Anti-Patterns*](https://www.packtpub.com/en-us/product/cloud-native-anti-patterns-9781836200581) by Gerald Bachlmayr et al.: the proxy anti-pattern a mesh degrades into without a usable catalog.
- Apicurio Registry documentation: contract versioning and compatibility rules across schemas, Protobuf, OpenAPI, and GraphQL SDL.
6 changes: 3 additions & 3 deletions _docs/05-data-planes.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,8 +134,8 @@ the consumers depending on it.

## For deeper coverage

- *Data Mesh* by Zhamak Dehghani: polyglot access and the synchronous/asynchronous planes a mesh runs on.
- *Designing Data-Intensive Applications, 2nd ed.* by Martin Kleppmann & Chris Riccomini: event-driven architecture and the derived-data reasoning behind the backbone.
- *Cloud Native Patterns* by Cornelia Davis: composing services at the edge versus internally.
- [*Data Mesh*](https://www.oreilly.com/library/view/data-mesh/9781492092384/) by Zhamak Dehghani: polyglot access and the synchronous/asynchronous planes a mesh runs on.
- [*Designing Data-Intensive Applications, 2nd ed.*](https://www.oreilly.com/library/view/designing-data-intensive-applications/9781098119058/) by Martin Kleppmann & Chris Riccomini: event-driven architecture and the derived-data reasoning behind the backbone.
- [*Cloud Native Patterns*](https://www.manning.com/books/cloud-native-patterns) by Cornelia Davis: composing services at the edge versus internally.
- [grpc.io documentation](https://grpc.io/docs/): gRPC for internal, low-latency service-to-service calls.
- [Strawberry GraphQL](https://strawberry.rocks/): the library behind the gateway's composed graph.
10 changes: 5 additions & 5 deletions _docs/06-progressive-delivery-mtls.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ environment toggle and a different `version` label, rather than two images from
commits. That keeps the demo focused on the traffic-management mechanism rather than
an image pipeline, and the Istio mechanics are identical either way. And traffic
enters through the ingress gateway (reached on its NodePort, published to
127.0.0.1:8088 when the profile is created, DRA-017) rather than from a meshed client,
127.0.0.1:8088 when the profile is created) rather than from a meshed client,
so no client needs to join the mesh for the canary to work.

This was verified end to end: with the split set to 90/10, a run of 100 requests
Expand Down Expand Up @@ -138,7 +138,7 @@ match demand, including down to zero, and recovering when things fail.
## For deeper coverage

- [Istio documentation](https://istio.io/latest/docs/): `DestinationRule`, `VirtualService`, `Gateway`, and automatic mutual TLS between sidecars.
- *Kubernetes Patterns* by Bilgin Ibryam & Roland Huß: the Sidecar pattern and the trade-offs of attaching one to every workload.
- *Platform Engineering on Kubernetes* by Mauricio Salatino: treating mesh injection as a platform policy decision rather than a blanket default.
- *GitOps and Kubernetes* by Billy Yuen et al.: progressive delivery workflows for shifting traffic between versions.
- *Cloud Native Anti-Patterns* by Gerald Bachlmayr et al.: the failure modes of meshing workloads that don't belong in the mesh.
- [*Kubernetes Patterns*](https://www.oreilly.com/library/view/-/9781098131678/) by Bilgin Ibryam & Roland Huß: the Sidecar pattern and the trade-offs of attaching one to every workload.
- [*Platform Engineering on Kubernetes*](https://www.manning.com/books/platform-engineering-on-kubernetes) by Mauricio Salatino: treating mesh injection as a platform policy decision rather than a blanket default.
- [*GitOps and Kubernetes*](https://www.manning.com/books/gitops-and-kubernetes) by Billy Yuen et al.: progressive delivery workflows for shifting traffic between versions.
- [*Cloud Native Anti-Patterns*](https://www.packtpub.com/en-us/product/cloud-native-anti-patterns-9781836200581) by Gerald Bachlmayr et al.: the failure modes of meshing workloads that don't belong in the mesh.
6 changes: 3 additions & 3 deletions _docs/07-elastic-and-resilient.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,7 @@ products.

## For deeper coverage

- *Kubernetes Patterns* by Bilgin Ibryam & Roland Huß: autoscaling and self-healing patterns on Kubernetes.
- *Cloud Native Patterns* by Cornelia Davis: elasticity and recoverability as platform-provided properties.
- *Platform Engineering on Kubernetes* by Mauricio Salatino: self-serve platform capabilities and what domains should and shouldn't operate themselves.
- [*Kubernetes Patterns*](https://www.oreilly.com/library/view/-/9781098131678/) by Bilgin Ibryam & Roland Huß: autoscaling and self-healing patterns on Kubernetes.
- [*Cloud Native Patterns*](https://www.manning.com/books/cloud-native-patterns) by Cornelia Davis: elasticity and recoverability as platform-provided properties.
- [*Platform Engineering on Kubernetes*](https://www.manning.com/books/platform-engineering-on-kubernetes) by Mauricio Salatino: self-serve platform capabilities and what domains should and shouldn't operate themselves.
- [KEDA documentation](https://keda.sh/docs/): scalers, `ScaledObject`s, and the HTTP add-on used in this chapter.
4 changes: 2 additions & 2 deletions _docs/08-observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,7 @@ sound.

## For deeper coverage

- *Designing Data-Intensive Applications, 2nd ed.* by Martin Kleppmann & Chris Riccomini: reasoning about the behavior of distributed systems built from independently-owned parts.
- [*Designing Data-Intensive Applications, 2nd ed.*](https://www.oreilly.com/library/view/designing-data-intensive-applications/9781098119058/) by Martin Kleppmann & Chris Riccomini: reasoning about the behavior of distributed systems built from independently-owned parts.
- [OpenTelemetry documentation](https://opentelemetry.io/docs/): the metrics/traces/logs signal model this chapter applies to a mesh.
- [Grafana documentation](https://grafana.com/docs/): Tempo for traces, Mimir/Prometheus for metrics, and correlating the two.
- *Cloud Native Patterns* by Cornelia Davis: observability as a platform-provided concern rather than a per-service add-on.
- [*Cloud Native Patterns*](https://www.manning.com/books/cloud-native-patterns) by Cornelia Davis: observability as a platform-provided concern rather than a per-service add-on.
8 changes: 4 additions & 4 deletions _docs/09-anti-patterns.md
Original file line number Diff line number Diff line change
Expand Up @@ -205,7 +205,7 @@ single-node cluster rather than to data mesh as an idea.

## For deeper coverage

- *Data Mesh* by Zhamak Dehghani: the four principles and the failure modes that appear when each is skipped.
- *Cloud Native Anti-Patterns* by Gerald Bachlmayr et al.: recognizable failure shapes in cloud-native platform adoption generally.
- *Learning Domain-Driven Design* by Vlad Khononov: bounded contexts and why fuzzy domain boundaries cascade into mesh failures.
- *Implementing Domain-Driven Design* by Vaughn Vernon: the ownership and autonomy a real data product needs to avoid the "dumb data product" trap.
- [*Data Mesh*](https://www.oreilly.com/library/view/data-mesh/9781492092384/) by Zhamak Dehghani: the four principles and the failure modes that appear when each is skipped.
- [*Cloud Native Anti-Patterns*](https://www.packtpub.com/en-us/product/cloud-native-anti-patterns-9781836200581) by Gerald Bachlmayr et al.: recognizable failure shapes in cloud-native platform adoption generally.
- [*Learning Domain-Driven Design*](https://www.oreilly.com/library/view/learning-domain-driven-design/9781098100124/) by Vlad Khononov: bounded contexts and why fuzzy domain boundaries cascade into mesh failures.
- [*Implementing Domain-Driven Design*](https://www.informit.com/store/implementing-domain-driven-design-9780133039894) by Vaughn Vernon: the ownership and autonomy a real data product needs to avoid the "dumb data product" trap.
Loading
Loading