From e5283ce3c83cc2d1dedb29cd0e4b1ebc4ca65af0 Mon Sep 17 00:00:00 2001 From: PatternCatalyst Date: Thu, 22 Oct 2026 21:00:15 -0400 Subject: [PATCH] docs: rewrite all chapters in the Geewax house voice Apply the approved writing voice across all 13 chapters: J.J. Geewax example-first teaching structure on Kubernetes-Patterns-grade vocabulary. - Lead each concept with a concrete thing in the project (a service, endpoint, method, command) before naming the abstraction. - Add a before/after code pair in ch03 place_order (naive write-then-check vs. the real check-stock -> persist -> publish order) with #A/#B/#C callouts, built from the actual handler. - Zero em-dashes (recast with commas, colons, semicolons, parentheses, or new sentences; figure-caption dashes become colons). - No rule-of-three cadence, no filler or meta-sentences, no stacked intensifiers. Contractions, varied sentence length, natural transitions. - House spine kept (Learning objectives / Why this matters / For deeper coverage). All code, commands, figures, output, and identifiers preserved. Verified: 0 em-dashes, 0 ban-tier (lgtm-professional-voice), forbidden-syntax / cross-references / liquid-collisions clean, jekyll build green. --- _docs/00-index.md | 108 ++++----- _docs/01-concepts.md | 142 ++++++------ _docs/01-data-architectures.md | 297 +++++++++++++------------ _docs/02-kubernetes-substrate.md | 131 +++++------ _docs/03-services-and-data-products.md | 208 +++++++++++------ _docs/04-contracts-and-catalog.md | 139 ++++++------ _docs/05-data-planes.md | 137 ++++++------ _docs/06-progressive-delivery-mtls.md | 180 ++++++++------- _docs/07-elastic-and-resilient.md | 74 +++--- _docs/08-observability.md | 73 +++--- _docs/09-anti-patterns.md | 104 +++++---- _docs/10-summary.md | 97 ++++---- _docs/11-running-on-openshift-crc.md | 250 +++++++++++---------- 13 files changed, 1020 insertions(+), 920 deletions(-) diff --git a/_docs/00-index.md b/_docs/00-index.md index 9a45a53..36ba244 100644 --- a/_docs/00-index.md +++ b/_docs/00-index.md @@ -1,7 +1,7 @@ --- -title: "The data mesh — start here" +title: "The data mesh: start here" order: 0 -description: "The map — what this reference builds, the reading order, and a link to each page in the set. Start here." +description: "The map: what this reference builds, the reading order, and a link to each page in the set. Start here." duration: 5 min --- @@ -11,7 +11,7 @@ By the end of this chapter you can: - Name the four movements the reference set is organized into and which pages fall into each one. -- Pick the reading order that matches how you want to use the set — straight through, +- Pick the reading order that matches how you want to use the set: straight through, or jumping to one topic. - Identify the three pages that cover the pattern's shape if you only have time for a few. @@ -19,85 +19,89 @@ By the end of this chapter you can: ## Why this matters -This is the reading guide for the data-mesh reference — the map of what the set covers -and the order to read it in. The reference builds one working system end to end: a data -mesh of services that own their data as products, talk over a deliberate mix of -protocols, evolve their contracts safely, scale to demand, and stay observable -throughout. +Page 2 names Dehghani's four principles. Page 3 takes the same four and maps each one +to a Kubernetes primitive: namespaces, operators, RBAC. That's the shape of this whole +set. A concept gets introduced, then you watch it land in a running piece of the +system a few pages later. This page is the map, nothing more: what the reference +covers, the order to read it in, and a link to each page. -The set is written to be read straight through the first time. Each page picks up where -the last left off, and the cross-references assume you've seen the earlier material. If -you're returning to find one thing, the descriptions below will point you at the right +The reference builds one working system end to end. A data mesh of services that own +their data as products, that talk over a deliberate mix of protocols, that evolve their +contracts safely, that scale to demand, and that stay observable throughout. + +Read it straight through the first time if you can. Each page picks up where the last +left off, and the cross-references assume you've seen the earlier material. If you're +back here looking for one specific thing, the descriptions below point at the right page. ## How the set is organized -The eleven pages fall into four movements. The **conceptual grounding** (pages 1–3) -establishes the landscape of data architectures, what a data mesh is, and why Kubernetes -is a natural substrate for one. The **implementation** (pages 4–9) builds the system: -the services and their data products, the contracts and catalog that make them -discoverable, the data planes they communicate over, progressive delivery with mutual -TLS, elastic scaling and recovery, and the observability to see it all. The **failure -modes** (page 10) steps back to the conceptual and organizational anti-patterns that -derail data-mesh efforts even when the technology is sound. The **summary** (page 11) -closes by reorganizing the same material by principle — for each of the four principles, -the value it delivers and the implementation pieces that realize it. +The eleven pages fall into four movements. **Conceptual grounding** (pages 1 to 3) +covers the landscape of data architectures, what a data mesh actually is, and why +Kubernetes turns out to be a natural substrate for one. **Implementation** (pages 4 to +9) builds the system: the services and their data products, the contracts and catalog +that make them discoverable, the data planes they talk over, progressive delivery with +mutual TLS, elastic scaling and recovery, and the observability to see all of it. +**Failure modes** (page 10) steps back to the conceptual and organizational +anti-patterns that derail data-mesh efforts even when the technology is sound. The +**summary** (page 11) closes by reorganizing the same material by principle: for each +of the four, the value it delivers and the implementation pieces that realize it. ## The pages -- [**1 · Data architectures**]({{ '/docs/01-data-architectures/' | relative_url }}) — - The landscape from pipelines to lakes to mesh — what each pattern is, the problem +- [**1 · Data architectures**]({{ '/docs/01-data-architectures/' | relative_url }}): + the landscape from pipelines to lakes to mesh, what each pattern is, the problem it solves, and why the mesh is a different kind of answer. -- [**2 · Concepts & principles**]({{ '/docs/01-concepts/' | relative_url }}) — - What a data mesh is, operational versus analytical data, and Dehghani's four +- [**2 · Concepts & principles**]({{ '/docs/01-concepts/' | relative_url }}): + what a data mesh is, operational versus analytical data, and Dehghani's four principles. The grounding before any commands. -- [**3 · Kubernetes as the substrate**]({{ '/docs/02-kubernetes-substrate/' | relative_url }}) — - Why the four principles map cleanly onto namespaces, operators, RBAC, and platform +- [**3 · Kubernetes as the substrate**]({{ '/docs/02-kubernetes-substrate/' | relative_url }}): + why the four principles map cleanly onto namespaces, operators, RBAC, and platform primitives, and the shape of the system you'll build. -- [**4 · Services & data products**]({{ '/docs/03-services-and-data-products/' | relative_url }}) — - The anatomy of a data product, the five domain services plus the gateway, and the +- [**4 · Services & data products**]({{ '/docs/03-services-and-data-products/' | relative_url }}): + the anatomy of a data product, the five domain services plus the gateway, and the order-service template the others follow. -- [**5 · Contracts & the catalog**]({{ '/docs/04-contracts-and-catalog/' | relative_url }}) — - Versioned contracts in a registry, the runtime-versus-discovery distinction, and why a +- [**5 · Contracts & the catalog**]({{ '/docs/04-contracts-and-catalog/' | relative_url }}): + versioned contracts in a registry, the runtime-versus-discovery distinction, and why a catalog is a mesh requirement rather than an add-on. -- [**6 · The data planes**]({{ '/docs/05-data-planes/' | relative_url }}) — - The synchronous read layer (REST, gRPC, a GraphQL gateway) and the asynchronous event - backbone — and why the final project uses all of them. -- [**7 · Progressive delivery & mTLS**]({{ '/docs/06-progressive-delivery-mtls/' | relative_url }}) — - Evolving a contract in the open with a v1→v2 canary, mTLS with no application changes, and the decision to +- [**6 · The data planes**]({{ '/docs/05-data-planes/' | relative_url }}): the + synchronous read layer (REST, gRPC, a GraphQL gateway) and the asynchronous event + backbone, and why the final project uses all of them. +- [**7 · Progressive delivery & mTLS**]({{ '/docs/06-progressive-delivery-mtls/' | relative_url }}): + evolving a contract in the open with a v1→v2 canary, mTLS with no application changes, and the decision to mesh selectively rather than namespace-wide. -- [**8 · Elastic & resilient**]({{ '/docs/07-elastic-and-resilient/' | relative_url }}) — - Scaling to demand and to zero with KEDA, and the cloud-native recoverability the +- [**8 · Elastic & resilient**]({{ '/docs/07-elastic-and-resilient/' | relative_url }}): + scaling to demand and to zero with KEDA, and the cloud-native recoverability the platform provides. -- [**9 · Observability**]({{ '/docs/08-observability/' | relative_url }}) — - Metrics, distributed traces across products, and the live view of traffic moving +- [**9 · Observability**]({{ '/docs/08-observability/' | relative_url }}): + metrics, distributed traces across products, and the live view of traffic moving through the mesh. -- [**10 · Anti-patterns**]({{ '/docs/09-anti-patterns/' | relative_url }}) — - The conceptual and organizational ways data-mesh efforts go wrong, drawn from the +- [**10 · Anti-patterns**]({{ '/docs/09-anti-patterns/' | relative_url }}): + the conceptual and organizational ways data-mesh efforts go wrong, drawn from the literature, so you can recognize them early. -- [**11 · Summary**]({{ '/docs/10-summary/' | relative_url }}) — - Each principle, reorganized: the value it delivers, the implementation pieces that +- [**11 · Summary**]({{ '/docs/10-summary/' | relative_url }}): + each principle, reorganized: the value it delivers, the implementation pieces that realize it, and the failure mode when it's missing. ## If you have time for only a few -Read [**concepts**]({{ '/docs/01-concepts/' | relative_url }}) for the +Short on time? Read [**concepts**]({{ '/docs/01-concepts/' | relative_url }}) for the vocabulary, [**contracts & the catalog**]({{ '/docs/04-contracts-and-catalog/' | relative_url }}) for the idea that holds a mesh together, and -[**anti-patterns**]({{ '/docs/09-anti-patterns/' | relative_url }}) for what -to avoid. Those three cover the shape of the thing without the full implementation -depth. +[**anti-patterns**]({{ '/docs/09-anti-patterns/' | relative_url }}) for what to avoid. +Those three won't give you the implementation depth, but they'll give you the shape of +the thing. Start with [concepts & principles]({{ '/docs/01-concepts/' | relative_url }}). ## For deeper coverage -- *Data Mesh* by Zhamak Dehghani — the full argument this reference implements, including +- *Data Mesh* by Zhamak Dehghani: the full argument this reference implements, including the history and the operating-model shifts this set only summarizes. - *Designing Data-Intensive Applications, 2nd ed.* by Martin Kleppmann & Chris - Riccomini — the data-systems foundations (storage, replication, consistency) that sit + Riccomini: the data-systems foundations (storage, replication, consistency) that sit underneath every data product in this build. -- *Cloud Native Patterns* by Cornelia Davis — the platform patterns this reference's +- *Cloud Native Patterns* by Cornelia Davis: the platform patterns this reference's self-serve tier draws on. -- *Kubernetes Patterns* by Bilgin Ibryam & Roland Huß — the Kubernetes-native building +- *Kubernetes Patterns* by Bilgin Ibryam & Roland Huß: the Kubernetes-native building blocks the implementation pages rely on throughout. diff --git a/_docs/01-concepts.md b/_docs/01-concepts.md index 2b9f337..bac8546 100644 --- a/_docs/01-concepts.md +++ b/_docs/01-concepts.md @@ -1,7 +1,7 @@ --- title: "Concepts & principles" order: 2 -description: What a data mesh is, operational vs. analytical data, and Dehghani's four principles — the conceptual grounding before any commands. +description: What a data mesh is, operational vs. analytical data, and Dehghani's four principles. The conceptual grounding before any commands. duration: 20 min --- @@ -19,27 +19,30 @@ By the end of this chapter you can: ## Why this matters +The order domain owns orders. Nothing else touches that data directly. By +the end of this page you'll see why that single fact, repeated across five domains, +is most of what a data mesh is. + The [previous page]({{ '/docs/01-data-architectures/' | relative_url }}) covered the -landscape of data architectures — pipelines, warehouses, and lakes — and where the mesh -sits in relation to all of them. This page goes deeper into what a data mesh actually -is and the four principles it rests on. - -Before any commands, it's worth being precise about what a data mesh actually is — -because the term gets attached to a lot of things it isn't. This page is a working -grounding in the idea and its four principles, enough to make the rest of the set -make sense. It's brief: if you want the full conceptual treatment — -the history, the operating-model shifts, the worked argument for decentralization — -that's what the *Data Mesh 101* material in the presentation is for. Here we cover -just enough to build on. +landscape of data architectures (pipelines, warehouses, lakes) and where the mesh sits +relative to all of them. This page goes deeper into what a data mesh actually is and +the four principles it rests on. + +The term gets attached to a lot of things it isn't, so it's worth being precise before +any commands show up. This page is a working grounding in the idea and its four +principles, enough to make the rest of the set make sense. It's brief. If you want the +full conceptual treatment (the history, the operating-model shifts, the worked argument +for decentralization), that's what the *Data Mesh 101* material in the presentation is +for. Here we cover just enough to build on. ## What a data mesh is The term **data mesh** was coined by Zhamak Dehghani in 2019 and formalized in *Data Mesh: Delivering Data-Driven Value at Scale* (O'Reilly, 2022). It's a response to a -recurring failure: centralized data platforms — the monolithic data lake, the -monolithic warehouse, the sprawl of ETL pipelines feeding them — stop scaling once an +recurring failure: centralized data platforms (the monolithic data lake, the +monolithic warehouse, the sprawl of ETL pipelines feeding them) stop scaling once an organization has enough data sources, enough consumers, and enough use cases. The -bottleneck isn't the technology; it's that one central team ends up owning all the +bottleneck isn't the technology. It's that one central team ends up owning all the data but understanding none of the domains it came from. The shift a data mesh proposes is from "centralize the data, then carve out access" @@ -48,20 +51,20 @@ shared platform providing the substrate those products use to publish, discover, govern themselves. The analogy that lands hardest is microservices. Just as a monolithic application gets refactored into bounded contexts owned by domain teams, a monolithic data platform gets refactored into bounded *data products* owned by domain -teams. The mesh is the network of those products plus the platform and standards that +teams. The mesh is the network of those products, plus the platform and standards that let them interoperate. ![From a monolithic data platform to a mesh of data products]({{ '/assets/diagrams/17-monolith-to-mesh.svg' | relative_url }}) ## Operational vs. analytical data -One distinction underlies everything, and blurring it -is itself a common mistake. **Operational data** is the current-state data behind a -domain's running services — the rows a microservice reads and writes to do its job, -transactional and live. **Analytical data** is the historical, aggregated view used to -make decisions, train models, and understand the business over time. Traditionally -these live in separate worlds joined by a tangle of pipelines: operational databases -on one side, a lake or warehouse on the other, ETL shuttling between them on a delay. +One distinction underlies everything here, and blurring it is itself a common mistake. +**Operational data** is the current-state data behind a domain's running services: the +rows a microservice reads and writes to do its job, transactional and live. +**Analytical data** is the historical, aggregated view used to make decisions, train +models, and understand the business over time. Traditionally these live in separate +worlds joined by a tangle of pipelines, operational databases on one side, a lake or +warehouse on the other, with ETL shuttling between them on a delay. A data mesh doesn't erase the distinction, but it reorganizes it by *domain* rather than by *technology layer*. Instead of "all operational data here, all analytical data @@ -72,76 +75,77 @@ rather than leaving analytical data as a stale downstream copy. ![Operational vs. analytical data planes]({{ '/assets/diagrams/17-operational-vs-analytical.svg' | relative_url }}) -This final project models the **operational** side concretely — services that own their -data and emit events — and shows how analytical consumers attach to that operational -flow through the event backbone, rather than through a nightly extract. That's the -loop the mesh is meant to keep closed. +This final project models the **operational** side concretely: services that own their +data and emit events. It shows how analytical consumers attach to that operational flow +through the event backbone, rather than through a nightly extract. That's the loop the +mesh is meant to keep closed. ## The four principles -Dehghani's data mesh rests on four interlocking principles. They depend on each other — -implement one without the others and you get a distributed mess rather than a mesh — -and each shows up explicitly in this final project: +Dehghani's data mesh rests on four interlocking principles. They depend on each other: +implement one without the others and you get a distributed mess rather than a mesh. All +four show up explicitly in this final project. ![The four principles and how they map to Kubernetes primitives]({{ '/assets/diagrams/17-data-mesh-four-principles.svg' | relative_url }}) -**Domain ownership.** Data is owned, end to end, by the domain team that produces it. -There is no central team that "owns the warehouse." Each domain owns its data's -schema, its lifecycle, and its evolution. In this build, each domain service owns its -data outright — the order domain owns orders, inventory owns stock — and nothing -reaches across that boundary to mutate another domain's data directly. - -**Data as a product.** A data product is held to the same standards as any other -software product: it's discoverable, addressable, trustworthy, self-describing, and -carries explicit expectations about quality and availability. It is not a renamed -table. In this build, each domain service publishes its API contracts to a registry -and its metadata to a catalog, so consumers can find it, understand it, and depend on -it — the subject of the -[contracts & catalog page]({{ '/docs/04-contracts-and-catalog/' | relative_url }}). - -**Self-serve data platform.** Domain teams should not each build their own event -streaming, observability, registry, or catalog. The platform provides these as shared -infrastructure that every domain consumes, so a domain team can stand up a data product -without first becoming experts in running Kafka or Prometheus. In this build, the event -backbone, the observability stack, the registry, the catalog, and autoscaling are all -platform infrastructure shared by the services — +**Domain ownership.** In this build the order domain owns orders and inventory owns +stock, and nothing reaches across that boundary to mutate another domain's data +directly. That's domain ownership: data owned, end to end, by the domain team that +produces it. There's no central team that "owns the warehouse." Each domain owns the +whole arc of its data: the schema, the lifecycle, how it evolves. + +**Data as a product.** Each domain service here publishes its API contracts to a +registry and its metadata to a catalog. A consumer can look up what the service does +and build against it without ever talking to the team that owns it (the subject of the +[contracts & catalog page]({{ '/docs/04-contracts-and-catalog/' | relative_url }})). +That's what makes it a product rather than a renamed table: it's held to the same +standards as any other software product, discoverable, addressable, trustworthy, +self-describing, with explicit expectations about quality and availability. + +**Self-serve data platform.** The event backbone, the observability stack, the +registry, the catalog, and autoscaling are all platform infrastructure shared by the +services in this build, not things any one domain team built itself. That's the point +of a self-serve platform: domain teams shouldn't each build their own event streaming, +observability, registry, or catalog. The platform provides these as shared +infrastructure every domain consumes, so a domain team can stand up a data product +without first becoming experts in running Kafka or Prometheus. [Kubernetes as the substrate]({{ '/docs/02-kubernetes-substrate/' | relative_url }}) -is about exactly how Kubernetes makes that self-serve layer real. - -**Federated computational governance.** Standards are enforced *computationally* — by -the platform, automatically — rather than by review meetings and policy documents. A -small set of global rules keeps independent products interoperable; the platform -enforces them. In this build, the service mesh enforces mutual TLS between services -automatically, the registry rejects schema-incompatible contract changes at publish -time, and lineage is recorded as part of deploying rather than as a separate audit. +covers exactly how Kubernetes makes that self-serve layer real. + +**Federated computational governance.** The service mesh in this build enforces mutual +TLS between services automatically, the registry rejects schema-incompatible contract +changes at publish time, and lineage gets recorded as part of deploying rather than as +a separate audit. That's governance enforced *computationally*, by the platform, +automatically, rather than by review meetings and policy documents. A small set of +global rules keeps independent products interoperable, and the platform enforces them. The [anti-patterns page]({{ '/docs/09-anti-patterns/' | relative_url }}) -covers what happens when governance is instead bolted on from outside, or +covers what happens when governance instead gets bolted on from outside, or re-centralized into an approval bottleneck. ## The pattern is not the tools -A data mesh isn't a product, a tool, or a vendor offering — it's an organizational and -architectural pattern. The tools this final project uses are *expressions* of the pattern, -chosen because each one makes a principle concrete and runnable, not because any of -them *is* the mesh. That distinction matters enough that it's the first +A data mesh isn't a product, a tool, or a vendor offering. It's an organizational and +architectural pattern. The tools this final project uses are *expressions* of the +pattern, chosen because each one makes a principle concrete and runnable, not because +any of them *is* the mesh. That distinction matters enough that it's the first [anti-pattern]({{ '/docs/09-anti-patterns/' | relative_url }}): the most common way these efforts fail is mistaking the tooling for the transformation. With the vocabulary in place, the next page looks at why Kubernetes is a natural -substrate for all of this — and how each of the four principles maps onto concrete +substrate for all of this, and how each of the four principles maps onto concrete Kubernetes primitives. ## For deeper coverage -- *Data Mesh* by Zhamak Dehghani — the primary source for the four principles and the +- *Data Mesh* by Zhamak Dehghani: the primary source for the four principles and the domain-ownership argument this page summarizes. -- *Learning Domain-Driven Design* by Vlad Khononov — the bounded-context thinking +- *Learning Domain-Driven Design* by Vlad Khononov: the bounded-context thinking behind domain ownership and why a data product maps to a domain, not a table. -- *Implementing Domain-Driven Design* by Vaughn Vernon — a deeper, implementation-level +- *Implementing Domain-Driven Design* by Vaughn Vernon: a deeper, implementation-level treatment of the same domain boundaries. - *Designing Data-Intensive Applications, 2nd ed.* by Martin Kleppmann & Chris - Riccomini — the operational-versus-analytical data distinction from the systems + Riccomini: the operational-versus-analytical data distinction from the systems side. -- *Cloud Native Patterns* by Cornelia Davis — the self-serve platform principle made +- *Cloud Native Patterns* by Cornelia Davis: the self-serve platform principle made concrete in cloud-native infrastructure terms. diff --git a/_docs/01-data-architectures.md b/_docs/01-data-architectures.md index cf8d2d5..bef28b3 100644 --- a/_docs/01-data-architectures.md +++ b/_docs/01-data-architectures.md @@ -1,7 +1,7 @@ --- -title: "Data architectures — from pipelines to mesh" +title: "Data architectures: from pipelines to mesh" order: 1 -description: "The landscape: data pipelines, data warehouses, data lakes, and data mesh — what each pattern is, the problem it solves, and why the mesh is a different kind of answer." +description: "The landscape: data pipelines, data warehouses, data lakes, and data mesh. What each pattern is, the problem it solves, and why the mesh is a different kind of answer." duration: 25 min --- @@ -9,7 +9,7 @@ duration: 25 min By the end of this chapter you can: -- Describe data pipelines, data warehouses, and data lakes — the problem each pattern +- Describe data pipelines, data warehouses, and data lakes: the problem each pattern solves and the limitation that bounds it. - Explain why pipeline count grows as the product of sources and destinations, not the sum, and why that growth is called pipeline sprawl. @@ -19,181 +19,184 @@ By the end of this chapter you can: ## Why this matters -Before defining data mesh precisely, it helps to see the landscape it lives in. The -mesh did not appear out of nowhere — it's a response to specific organizational scaling -problems that earlier data architectures left unsolved. +Before we pin down what a data mesh actually is, it helps to see where it came from. +The mesh didn't appear out of nowhere. It's a response to organizational scaling +problems that three earlier architectures, each a real attempt at the same underlying +problem, left unsolved. -Each of those earlier patterns solved a real problem, and each is still the right -answer when that problem is the dominant one. Understanding what came before, and what -each pattern does well, makes the mesh's value proposition concrete rather than -abstract. +And each of those earlier patterns is still the right answer when its problem is the +dominant one. Seeing what came before, and what each pattern does well, is +what makes the mesh's value proposition concrete instead of abstract. ## Data pipelines -The oldest problem in data architecture is movement: operational systems produce data, -analytical systems consume it, and something has to get it from one place to the other. -That something is a **data pipeline** — a sequence of steps that extracts data from a -source, transforms it into a shape the destination can use, and loads it into the -target store. The two canonical patterns are **ETL** (extract-transform-load), where -transformation happens in transit before the data lands, and **ELT** -(extract-load-transform), where raw data is landed first and transformed in place -afterward. The distinction matters for latency, cost, and where you locate your -business logic, but both are pipelines: linear flows from a known source to a known -destination, processing data in batches on a schedule or streaming it in near-real-time. +Picture fifteen operational systems, each one feeding five analytical consumers. The +oldest problem in data architecture is movement: operational systems produce data, +analytical systems consume it, and something has to get it from one place to the +other. That something is a **data pipeline**, a sequence of steps that extracts data +from a source, transforms it into a shape the destination can use, and loads it into +the target store. The two canonical patterns are **ETL** (extract-transform-load), +where transformation happens in transit before the data lands, and **ELT** +(extract-load-transform), where raw data lands first and gets transformed in place +afterward. The distinction matters for latency, cost, and where your business logic +lives, but both are pipelines: linear flows from a known source to a known destination, +processing data in batches on a schedule or streaming it near-real-time. Pipelines are point-to-point by nature. Each one knows its source, its destination, and its transformations. When you have a handful of operational databases feeding one -analytical warehouse, that directness is a virtue — the flow is legible, the ownership +analytical warehouse, that directness is a virtue: the flow is legible, the ownership is clear, and debugging means following a straight line. The trouble starts when the number of sources and destinations grows. Each new source-destination pair typically -requires its own pipeline, and the total count grows as the product of sources and -destinations, not the sum. A company with fifteen operational systems feeding five -analytical consumers doesn't have twenty pipelines — it has something closer to -seventy-five, each with its own schedule, its own transformation logic, and its own -failure modes. - -This is **pipeline sprawl**, and it's the defining limitation of the pattern. Nobody -"owns" the data flowing through a pipeline — the pipeline is plumbing, not a product. -When something breaks in the middle of a pipeline at 2 a.m., the question "whose -problem is this?" rarely has a clear answer. The data team owns the pipeline, but -the domain team owns the semantics, and neither has full context. It was exactly this -sprawl — and the operational fragility that comes with it — that the data warehouse was -designed to tame. - -{% include excalidraw.html file="01-data-pipeline-architecture" alt="A network of point-to-point pipelines connecting operational systems to analytical consumers, illustrating the combinatorial growth of pipeline sprawl" caption="Figure 1.1 — Pipeline sprawl: each source-destination pair requires its own pipeline, and the total grows as the product of sources and destinations." %} +needs its own pipeline, and the total count grows as the product of sources and +destinations, not the sum. Back to that company with fifteen operational systems and +five analytical consumers: it doesn't end up with twenty pipelines, it ends up with +something closer to seventy-five, each with its own schedule, its own transformation +logic, and its own failure modes. + +That's **pipeline sprawl**, and it's the defining limitation of the pattern. Nobody +"owns" the data flowing through a pipeline. The pipeline is plumbing, not a product. +When something breaks in the middle of one at 2 a.m., the question "whose problem is +this?" rarely has a clear answer. The data team owns the pipeline, but the domain team +owns the semantics, and neither has full context. It was exactly this sprawl, and the +operational fragility riding along with it, that the data warehouse was built to tame. + +{% include excalidraw.html file="01-data-pipeline-architecture" alt="A network of point-to-point pipelines connecting operational systems to analytical consumers, illustrating the combinatorial growth of pipeline sprawl" caption="Figure 1.1: Pipeline sprawl: each source-destination pair requires its own pipeline, and the total grows as the product of sources and destinations." %} ## Data warehouses +Say the logistics domain needs a new dimension added to the warehouse. The request +goes to a central team, who have to learn enough about logistics to model it correctly, +weigh it against requests from every other domain, and land the schema change without +breaking downstream consumers. That request is the warehouse's whole story in +miniature. + A **data warehouse** is a centralized analytical store, purpose-built for structured queries across the entire business. Where pipelines are plumbing, the warehouse is a -destination — a single place where data from every operational system arrives, gets +destination: a single place where data from every operational system arrives, gets cleaned, modeled into a consistent schema, and becomes queryable by anyone with the right access. The intellectual heritage runs through Kimball's dimensional modeling and -Inmon's enterprise data warehouse: **star schemas**, **fact tables** and **dimension -tables**, all optimized for the kind of aggregations and joins that business +Inmon's enterprise data warehouse, with **star schemas**, **fact tables**, and +**dimension tables** all optimized for the aggregations and joins that business intelligence tools need. -The warehouse's defining characteristic is **schema-on-write**. Data is modeled and +The warehouse's defining characteristic is **schema-on-write**. Data gets modeled and validated before it's stored. That rigidity is a feature: it forces consistency, makes -the warehouse a governed single source of truth, and ensures that when two teams query +the warehouse a governed single source of truth, and means that when two teams query "total revenue last quarter," they get the same number. The warehouse solved the truth -problem that pipeline sprawl created — instead of every consumer building its own +problem pipeline sprawl created. Instead of every consumer building its own understanding of the data from raw pipeline output, everyone queries the same curated model. -The limitation is organizational, not technical. The warehouse is owned by a central -data team — sometimes called the BI team, the analytics engineering team, or the data -platform team — and that team becomes a bottleneck in direct proportion to the -organization's growth. The central team owns all the data but understands none of the -domains that produced it. +But the limitation is organizational, not technical. The warehouse is owned by a +central data team (sometimes called the BI team, the analytics engineering team, or +the data platform team), and that team becomes a bottleneck in direct proportion to +the organization's growth. It owns all the data and understands none of the domains +that produced it. -When the logistics domain needs a new dimension added to the warehouse, the request -goes to the central team, who must learn enough about logistics to model it correctly, -prioritize it against requests from every other domain, and coordinate the schema -change without breaking downstream consumers. The analytical view is always hours or -days behind the operational truth, because the ingestion pipelines run on a schedule, -and the central team's capacity to model new data is the constraint on how fast the -warehouse can grow. As the organization adds domains and use cases, the central team -processes more requests than it can understand, and the queue becomes the bottleneck. +Back to that logistics request: the analytical view stays hours or days behind the +operational truth, because the ingestion pipelines run on a schedule, and the central +team's capacity to model new data is the constraint on how fast the warehouse can grow. +As the organization adds domains and use cases, the central team processes more +requests than it can understand, and the queue itself becomes the bottleneck. -{% include excalidraw.html file="01-data-warehouse-architecture" alt="A centralized data warehouse receiving data from multiple operational systems through ingestion pipelines, with a central team owning the schema and all analytical consumers querying one governed store" caption="Figure 1.2 — The data warehouse: a centralized, schema-on-write analytical store that solves the truth problem but concentrates ownership in a single team." %} +{% include excalidraw.html file="01-data-warehouse-architecture" alt="A centralized data warehouse receiving data from multiple operational systems through ingestion pipelines, with a central team owning the schema and all analytical consumers querying one governed store" caption="Figure 1.2: The data warehouse: a centralized, schema-on-write analytical store that solves the truth problem but concentrates ownership in a single team." %} ## Data lakes -The **data lake** emerged as a response to the warehouse's format rigidity. Where a -warehouse demands that data conform to a schema before it can be stored, a data lake -accepts data in any format — structured tables, semi-structured JSON and XML, -unstructured logs and text, binary blobs like images and sensor readings — and defers -schema decisions to the point of consumption. This is **schema-on-read**: the data -lands as-is, and the consumer applies a schema when querying it. +A Kafka topic full of JSON events, a directory of Parquet files from a data science +experiment, a bucket of images for a computer vision model: none of that fits cleanly +into a warehouse. The **data lake** emerged as a response to exactly this format +rigidity. Where a warehouse demands that data conform to a schema before it can be +stored, a lake accepts data in any format (structured tables, semi-structured JSON and +XML, unstructured logs and text, binary blobs like images and sensor readings) and +defers schema decisions to the point of consumption. That's **schema-on-read**: the +data lands as-is, and the consumer applies a schema when querying it. A well-run lake is organized into zones that reflect the data's maturity. The **raw zone** (sometimes called the landing zone) holds data exactly as it arrived from -the source systems — untouched, append-only, the system of record. The **curated zone** -holds data that has been cleaned, validated, deduplicated, and enriched — still in its -original grain, but trustworthy. The **refined zone** holds data that has been modeled -and aggregated for specific consumption patterns — analytics, dashboards, ML feature -stores. The progression from raw to curated to refined represents increasing levels -of transformation and decreasing levels of generality. - -The lake solved the warehouse's most visible problem: format inflexibility. The -warehouse can't easily accommodate a Kafka topic full of JSON events, a directory of -Parquet files from a data science experiment, or a bucket of images for a computer -vision model. The lake can. At massive scale and with cheap object storage, the lake -makes it economically feasible to keep *everything* and decide later what's worth -querying. That flexibility unlocked workloads — machine learning, unstructured -analytics, exploratory data science — that warehouses were never built for. - -The limitation is governance. Without active curation, the lake becomes the **data -swamp** — a term so common in the field that it's practically a synonym for a -poorly-run lake. Undocumented datasets accumulate. Nobody knows which version of the -customer table is authoritative, which datasets are stale, which ones duplicate each -other under different names. +the source systems: untouched, append-only, the system of record. The **curated zone** +holds data that's been cleaned, validated, deduplicated, and enriched, still at its +original grain but trustworthy. The **refined zone** holds data that's been modeled +and aggregated for specific consumption patterns such as analytics, dashboards, or ML +feature stores. Raw to curated to refined is a progression of increasing transformation +and decreasing generality. + +The lake solved the warehouse's most visible problem. At massive scale, with cheap +object storage, it makes it economically feasible to keep *everything* and decide +later what's worth querying. That flexibility unlocked machine learning, unstructured +analytics, and exploratory data science, workloads warehouses were never built for. + +But the limitation is governance. Without active curation, a lake becomes the **data +swamp**, a term so common in the field it's practically a synonym for a poorly-run one. +Undocumented datasets pile up. Nobody knows which version of the customer table is +authoritative, which datasets are stale, which ones duplicate each other under +different names. The curation effort needed to keep a lake useful re-creates much of the warehouse's -modeling and governance work, just without the warehouse's enforcement mechanisms. And -critically, the organizational problem is unchanged: a central team still owns the -lake, still cannot understand every domain's data, and still bottlenecks on the same -requests. The technology changed — from rigid schema to flexible storage — but the -shape of the organization did not. +modeling and governance work, just without the warehouse's enforcement mechanisms. +And the organizational problem doesn't go away: a central team still owns the lake, +still can't understand every domain's data, still bottlenecks on the same requests. +The technology changed, from rigid schema to flexible storage, but the shape of the +organization didn't. -{% include excalidraw.html file="01-data-lake-architecture" alt="A data lake with raw, curated, and refined zones, showing heterogeneous data formats flowing in and the risk of the lake becoming a swamp without active governance" caption="Figure 1.3 — The data lake: schema-on-read and format flexibility, but without governance, the lake becomes a swamp — and the central-team bottleneck remains." %} +{% include excalidraw.html file="01-data-lake-architecture" alt="A data lake with raw, curated, and refined zones, showing heterogeneous data formats flowing in and the risk of the lake becoming a swamp without active governance" caption="Figure 1.3: The data lake: schema-on-read and format flexibility, but without governance, the lake becomes a swamp, and the central-team bottleneck remains." %} ## How data mesh differs -All three patterns — pipelines, warehouses, lakes — centralize data and hand -ownership to a single team. The technology improves with each generation: from -bespoke point-to-point plumbing, to a governed analytical store, to a flexible +Three patterns so far (pipelines, warehouses, lakes), and all three centralize data and +hand ownership to a single team. The technology improves with each generation: bespoke +point-to-point plumbing, then a governed analytical store, then a flexible format-agnostic lake. -But the organizational shape stays the same in every case: one team at the center, -receiving requests from every domain, owning data it did not produce, modeling -concepts it does not deeply understand, and scaling its capacity linearly while the -demands on it grow combinatorially. +But the organizational shape stays the same every time. One team at the center, +receiving requests from every domain, owning data it didn't produce, modeling concepts +it doesn't deeply understand, and scaling its capacity linearly while the demands on it +grow combinatorially. -The **data mesh** proposes a different axis of change. Instead of building a better -center, it decentralizes ownership to the domain teams that produce the data. Each -domain owns its data as a product — discoverable, addressable, trustworthy, -self-describing — and publishes it for other domains to consume. +The **data mesh** proposes a different axis of change entirely. Instead of building a +better center, it decentralizes ownership to the domain teams that produce the data. +Each domain owns its data as a product: discoverable, addressable, trustworthy, +self-describing, and published for other domains to consume. -A shared **self-serve data platform** provides the infrastructure that every domain -needs (streaming, storage, observability, schema registries) so that domain teams -don't each build their own. And **federated computational governance** keeps the -independently-owned products interoperable — not through review boards and policy -documents, but through standards enforced automatically by the platform itself. +A shared **self-serve data platform** provides the infrastructure every domain needs +(streaming, storage, observability, schema registries) so domain teams don't each build +their own. And **federated computational governance** keeps the independently-owned +products interoperable, not through review boards and policy documents but through +standards the platform enforces automatically. -This is a different kind of answer because it changes the organizational shape, not +That's a different kind of answer because it changes the organizational shape, not just the technology. A domain team that understands its own data curates it, models -it, documents it, and stands behind it the way a product team stands behind a -product. The central team's role shifts from owning every dataset to providing the -platform and enforcing the standards — a fundamentally different scaling model. - -An important clarification: the mesh is not a replacement for warehouses or lakes. A -domain may still use a warehouse or a lake internally — the order-analytics domain -might store its refined data in a Parquet-based lakehouse, and that's fine. The -reorganization is about *who owns the data*, not about which storage technology to -use. The mesh is an answer to the question "how does this organization scale its data -architecture?", not "which database should this team pick?" - -The mesh's complexity pays off in organizations with many domains, many data consumers, -and the operational maturity to sustain federated ownership. For smaller teams, simpler -data flows, or organizations where the central team is not yet the bottleneck, a -well-run warehouse or a governed lake may be exactly the right answer. The mesh is not -an upgrade in a linear progression — it's a different tool for a different problem. - -{% include excalidraw.html file="01-data-mesh-decentralized" alt="A data mesh with domain teams owning their data as products, a shared self-serve platform underneath, and federated governance keeping the products interoperable" caption="Figure 1.4 — The data mesh: domain teams own their data as products, a shared platform provides infrastructure, and federated governance keeps the products interoperable." %} - -## The evolution — and when each pattern fits - -The progression from pipelines to warehouses to lakes to mesh is not a replacement -chain where each generation obsoletes the last. It's an expansion of the problem space. -Pipelines solved the **movement problem** — getting data from where it's produced to -where it's consumed. Warehouses solved the **truth problem** — creating a single -governed view of the business from the chaos of independent pipelines. Lakes solved -the **flexibility problem** — accommodating the formats and workloads that warehouses -were too rigid to handle. The mesh solves the **organizational scaling problem** — -restructuring ownership so that the central team is no longer the bottleneck on every -domain's data needs. +it, documents it, and stands behind it the way a product team stands behind a product. +The central team's role shifts from owning every dataset to providing the platform and +enforcing the standards: a fundamentally different scaling model. + +One clarification worth making explicit: the mesh isn't a replacement for warehouses +or lakes. A domain may still use a warehouse or a lake internally. The order-analytics +domain might store its refined data in a Parquet-based lakehouse, and that's fine. The +reorganization is about *who owns the data*, not which storage technology to use. The +mesh answers "how does this organization scale its data architecture?", not "which +database should this team pick?" + +The mesh's complexity pays off in organizations with many domains, many data +consumers, and the operational maturity to sustain federated ownership. For smaller +teams, simpler data flows, or organizations where the central team isn't yet the +bottleneck, a well-run warehouse or a governed lake may be exactly the right answer. +The mesh isn't an upgrade in a linear progression. It's a different tool for a +different problem. + +{% include excalidraw.html file="01-data-mesh-decentralized" alt="A data mesh with domain teams owning their data as products, a shared self-serve platform underneath, and federated governance keeping the products interoperable" caption="Figure 1.4: The data mesh: domain teams own their data as products, a shared platform provides infrastructure, and federated governance keeps the products interoperable." %} + +## The evolution, and when each pattern fits + +The progression from pipelines to warehouses to lakes to mesh isn't a replacement +chain where each generation obsoletes the last. It's an expansion of the problem +space. Pipelines solved the **movement problem**: getting data from where it's +produced to where it's consumed. Warehouses solved the **truth problem**: creating a +single governed view of the business from the chaos of independent pipelines. Lakes +solved the **flexibility problem**: accommodating the formats and workloads that +warehouses were too rigid to handle. The mesh solves the **organizational scaling +problem**: restructuring ownership so the central team is no longer the bottleneck on +every domain's data needs. Each pattern is still valid when its problem is the dominant one. The choice depends on the organization's scale, its data landscape, and where the bottleneck actually @@ -205,28 +208,28 @@ sits: hasn't started. - **A data warehouse** fits when the organization needs a reliable, governed - analytical view of the business — BI dashboards, executive reporting, structured - decision-making — and a central team has the capacity to curate and model the data. + analytical view of the business (BI dashboards, executive reporting, structured + decision-making) and a central team has the capacity to curate and model the data. The bottleneck is not yet the central team's bandwidth but the absence of a single source of truth. -- **A data lake** fits when the data is heterogeneous — logs, events, unstructured - content, ML training sets — and the scale or the workload patterns (exploratory +- **A data lake** fits when the data is heterogeneous (logs, events, unstructured + content, ML training sets) and the scale or the workload patterns (exploratory analysis, feature engineering, large-scale batch processing) exceed what a warehouse handles well. The bottleneck is format rigidity, not organizational ownership. - **A data mesh** fits when the organization has many domains, many data consumers, - and the central team has become the constraint. The bottleneck is organizational — - who owns what, who can ship what, who understands what — not technical. The + and the central team has become the constraint. The bottleneck is organizational + (who owns what, who can ship what, who understands what), not technical. The organization needs the maturity to operate federated ownership: domain teams that can treat data as a product, a platform team that can provide self-serve infrastructure, and a governance model that works through automation rather than approvals. -{% include excalidraw.html file="01-architecture-evolution" alt="The evolution from pipelines to warehouses to lakes to mesh, showing the problem each pattern solves and the bottleneck each one leaves behind" caption="Figure 1.5 — The evolution: each pattern solves a different problem, and each remains the right answer when that problem is the dominant one." %} +{% include excalidraw.html file="01-architecture-evolution" alt="The evolution from pipelines to warehouses to lakes to mesh, showing the problem each pattern solves and the bottleneck each one leaves behind" caption="Figure 1.5: The evolution: each pattern solves a different problem, and each remains the right answer when that problem is the dominant one." %} -With this landscape in place, the next page defines data mesh precisely — what +With this landscape in place, the next page defines data mesh precisely: what Dehghani's four principles are, how they interlock, and why implementing one without the others produces a distributed mess rather than a mesh. @@ -234,13 +237,13 @@ the others produces a distributed mess rather than a mesh. ## For deeper coverage -- *Data Mesh* by Zhamak Dehghani — the full argument for why pipelines, warehouses, +- *Data Mesh* by Zhamak Dehghani: the full argument for why pipelines, warehouses, and lakes hit an organizational ceiling, and what the mesh changes. - *Designing Data-Intensive Applications, 2nd ed.* by Martin Kleppmann & Chris - Riccomini — the systems-level treatment of batch and stream processing, ETL/ELT, + Riccomini: the systems-level treatment of batch and stream processing, ETL/ELT, and the storage engines behind warehouses and lakes. -- *Cloud Native Anti-Patterns* by Gerald Bachlmayr et al. — common organizational and +- *Cloud Native Anti-Patterns* by Gerald Bachlmayr et al.: common organizational and architectural failure modes, including the centralized-ownership trap this chapter describes. -- *Cloud Native Patterns* by Cornelia Davis — the platform-level patterns a self-serve +- *Cloud Native Patterns* by Cornelia Davis: the platform-level patterns a self-serve data platform draws on once the organizational shift is underway. diff --git a/_docs/02-kubernetes-substrate.md b/_docs/02-kubernetes-substrate.md index b820c46..68d5787 100644 --- a/_docs/02-kubernetes-substrate.md +++ b/_docs/02-kubernetes-substrate.md @@ -21,75 +21,82 @@ By the end of this chapter you can: ## Why this matters -The [previous page]({{ '/docs/01-concepts/' | relative_url }}) ended on a -claim: a data mesh is a pattern, not a tool, and the tools are -expressions of it. So why build the final project on Kubernetes at all? Because the four -principles map onto Kubernetes primitives unusually cleanly — cleanly enough that -"implement a data mesh on Kubernetes" stops feeling like a translation exercise and -starts feeling like the primitives were waiting for it. This page makes that mapping -explicit, then shows the shape of the system you'll build on top of it. +A domain team that needs Kafka in this build doesn't learn to operate Kafka. It asks +the platform's Kafka operator for a cluster and gets one. That's one of four principles +this page maps onto a concrete Kubernetes primitive, and it's a good preview of the +whole argument. + +The [previous page]({{ '/docs/01-concepts/' | relative_url }}) ended on a claim: a data +mesh is a pattern, not a tool, and the tools are expressions of it. So why build the +final project on Kubernetes at all? Because the four principles map onto Kubernetes +primitives unusually cleanly, cleanly enough that "implement a data mesh on Kubernetes" +stops feeling like a translation exercise and starts feeling like the primitives were +waiting for it. This page makes that mapping explicit, then shows the shape of the +system you'll build on top of it. ## Why the alignment is so good Kubernetes was designed around multi-tenancy, declarative resources, an extensible -type system, and operators that turn operational knowledge into software. Those are -exactly the capabilities a data mesh needs — a place for each domain to own its slice, -a way to express a data product as a deployable artifact with a contract, a shared -platform layer domains consume without building it themselves, and a boundary where -standards get enforced automatically. You *can* build a data mesh without Kubernetes, -and you can certainly run Kubernetes without building a mesh. But the alignment is -strong enough that each principle has a natural home in Kubernetes primitives you -already know. +type system, and operators that turn operational knowledge into software. Those happen +to be exactly the capabilities a data mesh needs: a place for each domain to own its +slice, a way to express a data product as a deployable artifact with a contract, a +shared platform layer domains consume without building it themselves, and a boundary +where standards get enforced automatically. You *can* build a data mesh without +Kubernetes, and you can certainly run Kubernetes without building a mesh. But the +alignment is strong enough that each principle has a natural home in Kubernetes +primitives you already know. ## The four principles, mapped to primitives -**Domain ownership → namespaces, ServiceAccounts, RBAC, quotas.** The unit of tenancy -in Kubernetes is the namespace, and it carries its own identities (ServiceAccounts), -its own permissions (Roles and RoleBindings), and its own resource budget (quotas). -That's precisely the boundary a domain needs: a place it owns, with access it controls -and a budget it lives within, isolated from other domains by default. A domain team -gets a namespace the way a microservice team gets a repository. - -**Data as a product → Deployments, Services, and CRDs.** A data product is a deployable -artifact that exposes a contract — which is exactly what a Deployment plus a Service -*is*. The Deployment runs the product; the Service is its stable address. And because -Kubernetes lets you extend its own type system with Custom Resource Definitions, the -platform can offer domain-specific types — a Kafka `Topic`, a Postgres `Cluster`, a -schema registration — that a domain declares the same way it declares a Deployment. -The data product becomes a first-class, declarable thing rather than an informal -collection of scripts. - -**Self-serve data platform → operators and shared cluster infrastructure.** This is -where Kubernetes operators deliver the most value for a self-serve platform. An *operator* packages the knowledge of -how to run a complex stateful system — Kafka, Postgres, autoscaling, a service mesh — +**Domain ownership maps to namespaces, ServiceAccounts, RBAC, quotas.** The unit of +tenancy in Kubernetes is the namespace. It carries identities through ServiceAccounts, +permissions through Roles and RoleBindings, and a resource budget through quotas, all +scoped to that one namespace. That's precisely the boundary a domain needs: a place it +owns, with access it controls and a budget it lives within, isolated from other domains +by default. A domain team gets a namespace the way a microservice team gets a +repository. + +**Data as a product maps to Deployments, Services, and CRDs.** A data product is a +deployable artifact that exposes a contract, which is exactly what a Deployment plus a +Service *is*. The Deployment runs the product; the Service is its stable address. And +because Kubernetes lets you extend its own type system with Custom Resource +Definitions, the platform can offer domain-specific types (a Kafka `Topic`, a Postgres +`Cluster`, a schema registration) that a domain declares the same way it declares a +Deployment. The data product becomes a first-class, declarable thing rather than an +informal collection of scripts. + +**Self-serve data platform maps to operators and shared cluster infrastructure.** A +domain team that needs Kafka doesn't learn to operate Kafka here. It asks the +platform's Kafka operator for a cluster and gets one. That's where Kubernetes operators +deliver the most value for a self-serve platform: an *operator* packages the knowledge +of how to run a complex stateful system (Kafka, Postgres, autoscaling, a service mesh) into a controller that reconciles a simple declarative request into a running system. -A domain team that needs Kafka doesn't learn to operate Kafka; it asks the platform's -Kafka operator for a cluster and gets one. In this build the event backbone (via its -operator), the database (via its operator), autoscaling, the service mesh, and the -observability stack are all shared platform infrastructure the domains consume by -declaration. That's the self-serve principle made literal. +In this build the event backbone (via its operator), the database (via its operator), +autoscaling, the service mesh, and the observability stack are all shared platform +infrastructure the domains consume by declaration. That's the self-serve principle made +literal. ![The three planes of the self-serve data platform]({{ '/assets/diagrams/17-platform-planes.svg' | relative_url }}) -**Federated computational governance → admission control, policy engines, mesh +**Federated computational governance maps to admission control, policy engines, mesh policy, CRD validation.** Governance in a mesh is supposed to be enforced by the -platform, automatically, at the boundary — not by review meetings after the fact. +platform, automatically, at the boundary, not by review meetings after the fact. Kubernetes has several boundaries where that enforcement lives: admission controllers and policy engines that accept or reject resources as they're created, service-mesh authorization and mutual-TLS policies that govern traffic between products, and the schema validation built into every CRD. The rules become code that runs at the edge of the system, which is exactly what "computational governance" means. -![Each of the four principles and the OpenShift/Kubernetes pieces that deliver it — one picture of the mapping]({{ '/assets/diagrams/17-principles-to-pieces.svg' | relative_url }}) +![Each of the four principles and the OpenShift/Kubernetes pieces that deliver it: one picture of the mapping]({{ '/assets/diagrams/17-principles-to-pieces.svg' | relative_url }}) ## The shape of the system -With the mapping in hand, here's the system this final project builds. It has three +With the mapping in hand, here's the system this final project builds. Three horizontal tiers: **external clients** at the top, the **service mesh** running the domain services in the middle, and the **self-serve platform** providing shared infrastructure underneath. -![The final project's architecture — data mesh on minikube]({{ '/assets/diagrams/17-final-project-data-mesh.svg' | relative_url }}) +![The final project's architecture: data mesh on minikube]({{ '/assets/diagrams/17-final-project-data-mesh.svg' | relative_url }}) The diagram doubles as a map of the protocol decisions, which the [data planes page]({{ '/docs/05-data-planes/' | relative_url }}) develops @@ -100,23 +107,23 @@ producers to consumers. Telemetry flows out of every service to the observabilit stack, the subject of the [observability page]({{ '/docs/08-observability/' | relative_url }}). -Read top to bottom, the diagram is the four principles again: external clients consume -products through stable contracts (data as a product), the domain services each own -their slice of the mesh (domain ownership), the platform tier underneath is shared and -consumed by declaration (self-serve platform), and the mesh and policy layers enforce -the rules on the traffic between everything (federated governance). It's the picture to -return to as the rest of the set works through the parts. +Read top to bottom, and the diagram is the four principles again: external clients +consume products through stable contracts (data as a product), the domain services each +own their slice of the mesh (domain ownership), the platform tier underneath is shared +and consumed by declaration (self-serve platform), and the mesh and policy layers +enforce the rules on the traffic between everything (federated governance). It's the +picture to return to as the rest of the set works through the parts. ## A note on minikube specifically -This final project runs on minikube — a single-node Kubernetes cluster — which is the right -choice for *learning* the pattern and wrong for running it in production. A single node -means every tier shares one machine's resources, which keeps the whole mesh runnable on -a laptop but also concentrates failure modes that a real multi-node cluster would -spread out. +This final project runs on minikube, a single-node Kubernetes cluster, which is the +right choice for *learning* the pattern and wrong for running it in production. A +single node means every tier shares one machine's resources. That keeps the whole mesh +runnable on a laptop, but it also concentrates failure modes that a real multi-node +cluster would spread out. -Where those single-node realities bite — resource ceilings, node-level decay, the -operational care a long-lived single-node cluster needs — they're collected as +Where those single-node realities bite (resource ceilings, node-level decay, the +operational care a long-lived single-node cluster needs), they're collected as operational gotchas rather than mixed into the conceptual material here, because they're particular to this deployment choice rather than to data mesh as a pattern. @@ -139,15 +146,15 @@ is published and reachable. ## For deeper coverage -- *Kubernetes Patterns* by Bilgin Ibryam & Roland Huß — the primitive-by-primitive +- *Kubernetes Patterns* by Bilgin Ibryam & Roland Huß: the primitive-by-primitive treatment (namespaces, operators, CRDs) this chapter's mapping draws on. -- *Platform Engineering on Kubernetes* by Mauricio Salatino — the self-serve platform +- *Platform Engineering on Kubernetes* by Mauricio Salatino: the self-serve platform principle built out as an operating model, not just a set of operators. -- *Cloud Native Patterns* by Cornelia Davis — the broader cloud-native platform +- *Cloud Native Patterns* by Cornelia Davis: the broader cloud-native platform patterns this final project's shared infrastructure tier follows. -- [Kubernetes documentation](https://kubernetes.io/docs/) — namespaces, RBAC, CRDs, +- [Kubernetes documentation](https://kubernetes.io/docs/): namespaces, RBAC, CRDs, and admission control, straight from the source. -- [minikube documentation](https://minikube.sigs.k8s.io/docs/) — driver and runtime +- [minikube documentation](https://minikube.sigs.k8s.io/docs/): driver and runtime options, including the docker driver and containerd runtime this final project uses. Next, the services themselves: what a data product looks like in this build, the diff --git a/_docs/03-services-and-data-products.md b/_docs/03-services-and-data-products.md index 587628c..96914e2 100644 --- a/_docs/03-services-and-data-products.md +++ b/_docs/03-services-and-data-products.md @@ -1,7 +1,7 @@ --- title: "Services & data products" order: 4 -description: The services, the order-service template, and the anatomy of a data product — its ports, its internal transformation, and the container image that ships it. +description: The services, the order-service template, and the anatomy of a data product; its ports, its internal transformation, and the container image that ships it. duration: 30 min --- @@ -19,35 +19,42 @@ By the end of this chapter you can: With the [principles]({{ '/docs/01-concepts/' | relative_url }}) and the [Kubernetes mapping]({{ '/docs/02-kubernetes-substrate/' | relative_url }}) -in place, this page gets concrete: what the data products in this final project actually -are, the one service we build end-to-end as a template for the rest, and how a data -product is packaged and shipped as a container image. This is the first -implementation-heavy page — the conceptual scaffolding is behind us. +in place, this page gets concrete. What do the data products in this final project +actually look like? What's the one service we build end to end as a template for the +rest, and how does a data product get packaged and shipped as a container image? This +is the first implementation-heavy page; the conceptual scaffolding is behind us. ## What a data product looks like here -A data product, in the abstract, is the *architectural quantum* of a data mesh: the -smallest unit you can independently deploy and operate, carrying everything it needs -to do its job. It has input ports (where data comes in), output ports (where it serves -data out), the transformation logic between them, and the metadata and policies that -make it discoverable and governed. +Open `order-service` and you can point at everything it does in one running process. It +owns a slice of Postgres that no other service reads directly. It serves orders over +REST, checks stock over gRPC before it writes anything, and publishes an event when an +order lands. Point at any one of those and you're pointing at part of a data product: an +input port, an output port, the transformation between them, or the metadata that +describes the whole thing. -![Anatomy of a data product — ports in, ports out, transformation and governance inside]({{ '/assets/diagrams/17-data-product-anatomy.svg' | relative_url }}) +That's also the formal definition. A data product, in Zhamak Dehghani's data mesh model, +is the *architectural quantum*: the smallest unit you can independently deploy and +operate, carrying everything it needs to do its job. It has input ports (where data +comes in), output ports (where it serves data out), the transformation logic between +them, and the metadata and policies that make it discoverable and governed. + +![Anatomy of a data product: ports in, ports out, transformation and governance inside]({{ '/assets/diagrams/17-data-product-anatomy.svg' | relative_url }}) In this final project, that abstraction is concrete: **each domain service *is* a data product.** It owns a slice of the database (its input/internal state), it serves data through its APIs (output ports), it emits events as other domains' input, and it publishes a contract and metadata so it can be discovered and depended on. The service -boundary and the data-product boundary are the same boundary — which is the cleanest -way to make domain ownership real rather than aspirational. +boundary and the data-product boundary are the same boundary. That's the cleanest way +to make domain ownership real instead of aspirational. ## The domain -The domain is order-placement-through-fulfillment, modeled small so the -architecture stays legible. There are **five domain services**, each a bounded context -owning its data and its contract, plus **one gateway** that composes reads across them -(the gateway is a read-layer convenience, not a domain data product — six images in -all). The five domains: +The domain is order-placement-through-fulfillment, modeled small so the architecture +stays legible. There are **five domain services**, each a bounded context owning its +data and its contract, plus **one gateway** that composes reads across them (the +gateway is a read-layer convenience, not a domain data product, so six images in all). +The five domains: | Service | Domain | Owns | Talks via | |---|---|---|---| @@ -55,87 +62,164 @@ all). The five domains: | inventory-service | Stock levels | the `inventory` schema | gRPC server, publishes `inventory.updated`, consumes `orders.placed` | | payment-service | Payments | the `payments` schema | gRPC server, publishes `payments.processed`, consumes `orders.placed` | | shipping-service | Shipments | the `shipments` schema | gRPC server, publishes `shipments.dispatched`, consumes `payments.processed` | -| notification-service | Notifications | the `notifications` schema | Kafka consumer only — reacts to events, emits notifications | +| notification-service | Notifications | the `notifications` schema | Kafka consumer only: reacts to events, emits notifications | The variation is deliberate: **not every service exposes every protocol.** Each -exposes the protocols that fit its role, not a uniform surface. notification-service -is event-only because its job is to react, not to be called synchronously; the gateway -exists to compose reads so clients don't have to fan out across five services. The -reasoning behind which protocol goes where is the subject of the -[data planes page]({{ '/docs/05-data-planes/' | relative_url }}); here the -point is just that the surface follows the role. - -Each service owns its own schema in a shared Postgres cluster — one schema per domain, +exposes the protocols that fit its role, not a uniform surface. notification-service is +event-only because its job is to react, not to be called synchronously, and the +gateway exists to compose reads so clients don't have to fan out across five services. +The reasoning behind which protocol goes where is the subject of the +[data planes page]({{ '/docs/05-data-planes/' | relative_url }}); here the point is +just that the surface follows the role. + +Each service owns its own schema in a shared Postgres cluster, one schema per domain, so the database is partitioned by ownership even though it's one managed cluster. That "one cluster, one schema per service" choice is what keeps per-domain data ownership real without running five separate databases on a single learning node. ## Build one service end to end first -Rather than build all six services a layer at a time, the final project takes a single -service all the way through first — a *walking skeleton*. The point is to prove the -entire spine works before widening: build the image, get it to the cluster, deploy via -helm, have the operator-managed Postgres come up, the service connect, and data +Rather than build all six services a layer at a time, the final project takes a +single service all the way through first: a *walking skeleton*. The point is to prove +the entire spine works before widening. Build the image, get it to the cluster, deploy +via Helm, have the operator-managed Postgres come up, the service connect, and data round-trip through a real API call. Once that path is verified on real hardware, the remaining services are mechanical repetition of the same pattern. **order-service is that template.** It's a Python service that owns the `orders` schema. It starts speaking only REST, and gRPC, GraphQL, and event publishing get -layered on in later steps — but the deployment spine is proven first with the simplest +layered on in later steps, but the deployment spine is proven first with the simplest possible surface. +### Check before you persist, persist before you publish + +Once gRPC and events land on top of that REST skeleton, `POST /orders` has three +things to do in sequence: confirm there's stock, write the order, tell the rest of the +mesh it happened. Get the order of those three steps wrong and you get a service that +lies to somebody. Here's the wrong order first, because it's the one most people reach +for without thinking about it: + +```python +# Naive: persist first, validate second. +order = Order( + id=str(uuid.uuid4()), + customer_id=payload.customer_id, + item_sku=payload.item_sku, + quantity=payload.quantity, + amount=payload.amount, +) +async with SessionLocal() as session: + session.add(order) + await session.commit() # A + +await check_stock(payload.item_sku, payload.quantity) # B +await publish_order_placed(order) +``` + +**#A** The row is durable before anyone has confirmed the warehouse can fill it. If the +stock check below fails, you're now deleting an order you already told Postgres +happened, or leaving it there if the delete fails too. + +**#B** A rejected order from this check still has a row sitting in the `orders` table. +Publish the event before you've even reached this line and a downstream consumer has +already reacted to an order that was never valid. + +`order-service` runs the steps in the opposite order, and a failure at any step stops +everything after it cleanly: + +```python +try: + await check_stock(payload.item_sku, payload.quantity) # A +except StockUnavailable as exc: + raise HTTPException( + status_code=status.HTTP_409_CONFLICT, + detail=f"insufficient stock for {exc.sku} ({exc.quantity_on_hand} on hand)", + ) + +order = Order( + id=str(uuid.uuid4()), + customer_id=payload.customer_id, + item_sku=payload.item_sku, + quantity=payload.quantity, + amount=payload.amount, +) +async with SessionLocal() as session: + session.add(order) + await session.commit() # B + +try: + await publish_order_placed(order) # C +except Exception as exc: + logger.warning("failed to publish order.placed for %s: %s", order.id, exc) +``` + +**#A** The gRPC call to inventory-service happens before anything touches Postgres. A +rejected order never gets a row, so there's nothing to clean up. + +**#B** The order only becomes durable once inventory has said yes. By the time this +line finishes, the order is real no matter what happens next. + +**#C** Publishing is best effort and happens after the commit. The +order already occurred; a Kafka hiccup here gets logged, not treated as a request +failure. That does leave a dual-write gap between "committed to Postgres" and +"published to Kafka," which is exactly what the outbox pattern closes in production +(see `app/events.py`). + +That's nearly the actual body of `place_order` in `app/main.py`. Three steps, one +order, each one a gate for the next: check, then persist, then publish. + ### Packaging the image A couple of packaging choices carry across all six images. Dependencies are managed -with a lockfile so builds are reproducible, and the lockfile is exported into the image -rather than carrying the dependency manager into the runtime. +with a lockfile so builds are reproducible, and the lockfile gets exported into the +image rather than carrying the dependency manager into the runtime. The image itself is multi-stage: a builder stage resolves dependencies, and a slim runtime stage copies only the resolved environment and the application code, runs as a -non-root user, and serves the app. Standard production hygiene — the final project just -applies it from the start rather than retrofitting it. +non-root user, and serves the app. That's standard production hygiene; the final +project just applies it from the start instead of retrofitting it later. ## The one part that fights back: getting images to the kubelet -This is worth its own section because it's the single part of the final project that -reliably trips people up. The profile is a Docker container (the docker driver) running -containerd, and that container has its own image store, separate from the Docker Engine -on your host. An image you build locally is invisible to the kubelet until you put it -there, and a pod that can't find its image fails in ways that look like a broken -cluster. +This earns its own section because it's the single part of the final project that +reliably trips people up. The profile is a Docker container (the docker driver) +running containerd, and that container has its own image store, separate from the +Docker Engine on your host. An image you build locally is invisible to the kubelet +until you put it there, and a pod that can't find its image fails in ways that look +like a broken cluster. ### Build with Docker Engine, then load the image The final project's answer is simple: **build with Docker Engine, then load the -image into the profile with `minikube image load`**. There is no registry to run and no -addresses to reconcile, and it works the same on a native engine and on a VM-based one -(Docker Desktop, for example). That portability is the reason for the choice: -pushing to a registry on the host's loopback only works when the engine's daemon shares -the host's network, which a VM-based engine's does not. +image into the profile with `minikube image load`**. There's no registry to run and +no addresses to reconcile, and it works the same on a native engine and on a VM-based +one (Docker Desktop, for example). That portability is the reason for the choice: +pushing to a registry on the host's loopback only works when the engine's daemon +shares the host's network, and a VM-based engine's doesn't. ### Tags and pull policy Images are tagged `datamesh/:v1` and the charts set `imagePullPolicy: Never`. The policy matters in both directions. If an image was never loaded, the pod fails at once with `ErrImageNeverPull` instead of quietly trying Docker Hub for a name that exists -nowhere. And because `:v1` is a mutable tag, a rebuild does not change what running pods -use, so `scripts/build-image.sh` restarts the Deployments that use the image it just -loaded. +nowhere. And because `:v1` is a mutable tag, a rebuild doesn't change what running pods +use on its own, so `scripts/build-image.sh` restarts the Deployments that use the +image it just loaded. -None of this is unique to the final project — any local cluster has some version of the gap between the host's images and the node's — but -the final project is where it bites, because it's the first place you build and deploy your -*own* images at scale. Get the image workflow right once here, and every service -afterward is the same three commands. (The deeper operational sharp edges of running -all this on a single node are collected as gotchas, separate from this build-level -friction.) +None of this is unique to the final project; any local cluster has some version of the +gap between the host's images and the node's. But the final project is where it bites, +because it's the first place you build and deploy your *own* images at scale. Get the +image workflow right once here, and every service afterward is the same three +commands. (The deeper operational sharp edges of running all this on a single node are +collected as gotchas, separate from this build-level friction.) With a service shipped and running, the next question is how products describe -themselves so others can find and trust them — contracts and the catalog. +themselves so others can find and trust them: contracts and the catalog. ## For deeper coverage -- *Data Mesh* by Zhamak Dehghani — the architectural quantum and why a data product's boundary should match a service's boundary. -- *Cloud Native Patterns* by Cornelia Davis — packaging and runtime patterns for services that double as data products. -- *Kubernetes Patterns* by Bilgin Ibryam & Roland Huß — container image and workload patterns behind the build-load-deploy sequence. -- [Kubernetes documentation](https://kubernetes.io/docs/) — image pull policies and the kubelet's image lifecycle. -- [FastAPI](https://fastapi.tiangolo.com/) — the framework order-service and its siblings are built on. +- *Data Mesh* by Zhamak Dehghani: the architectural quantum and why a data product's boundary should match a service's boundary. +- *Cloud Native Patterns* by Cornelia Davis: packaging and runtime patterns for services that double as data products. +- *Kubernetes Patterns* by Bilgin Ibryam & Roland Huß: container image and workload patterns behind the build-load-deploy sequence. +- [Kubernetes documentation](https://kubernetes.io/docs/): image pull policies and the kubelet's image lifecycle. +- [FastAPI](https://fastapi.tiangolo.com/): the framework order-service and its siblings are built on. diff --git a/_docs/04-contracts-and-catalog.md b/_docs/04-contracts-and-catalog.md index 1b574e0..c94d662 100644 --- a/_docs/04-contracts-and-catalog.md +++ b/_docs/04-contracts-and-catalog.md @@ -1,7 +1,7 @@ --- title: "Contracts & the catalog" order: 5 -description: Versioned contracts in Apicurio and discovery plus lineage in OpenMetadata — and why a catalog is a mesh requirement, not an optional add-on. +description: Versioned contracts in Apicurio and discovery plus lineage in OpenMetadata, and why a catalog is a mesh requirement, not an optional add-on. duration: 25 min --- @@ -20,25 +20,26 @@ The [previous page]({{ '/docs/03-services-and-data-products/' | relative_url }}) established that each service is a data product. But a data product is only useful to *other* domains if they can find it, understand its shape, trust that it won't change out from under them, and trace where its data came from. That's what this page is -about: the registry that holds every product's contract, and the catalog that makes -the products discoverable and their lineage visible. This is the -**data-as-a-product** and **federated-governance** principles made operational — and -the place where "catalog" stops being an add-on and becomes a requirement. +about: the registry that holds every product's contract, and the catalog that makes the +products discoverable and their lineage visible. This is the **data-as-a-product** and +**federated-governance** principles made operational, and the place where "catalog" +stops being an add-on and becomes a requirement. ## Every protocol is a contract -The mesh speaks four protocols, and each one *is* a contract: the REST API has an -OpenAPI description, the gRPC services have Protobuf definitions, the events have a -schema, and the gateway has a GraphQL SDL. Left informal, these contracts drift — -a field gets renamed, a type changes, and a consumer three domains away breaks with no -warning. The mesh's answer is to give every contract a versioned home in a single -registry, regardless of its format. +order-service alone speaks four different shapes: a REST API described by OpenAPI, a +gRPC surface described by Protobuf, an `orders.placed` event described by a schema, +and (through the gateway) a slice of GraphQL SDL. Every protocol the mesh speaks *is* +a contract. Left informal, these contracts drift: a field gets renamed, a type +changes, and a consumer three domains away breaks with no warning. The mesh's answer +is to give every contract a versioned home in a single registry, regardless of its +format. -![Each service's contract surface — OpenAPI, Protobuf, schema, SDL]({{ '/assets/diagrams/17-final-project-contracts.svg' | relative_url }}) +![Each service's contract surface: OpenAPI, Protobuf, schema, SDL]({{ '/assets/diagrams/17-final-project-contracts.svg' | relative_url }}) -This final project uses **Apicurio** as that registry, holding all four contract types as -their native artifacts: the event schemas, the gRPC Protobuf definitions, the REST -OpenAPI documents, and the gateway's GraphQL SDL. One registry, every contract — so +This final project uses **Apicurio** as that registry, holding all four contract types +as their native artifacts: the event schemas, the gRPC Protobuf definitions, the REST +OpenAPI documents, and the gateway's GraphQL SDL. One registry, every contract, so there's a single place to ask "what is this product's shape, and what version am I depending on?" @@ -47,88 +48,88 @@ depending on?" Here's the distinction that makes the registry more than a documentation folder, and it's worth getting clear because the two jobs have very different coupling. -![How a contract flows — the runtime path versus the discovery path]({{ '/assets/diagrams/17-final-project-contract-flow.svg' | relative_url }}) +![How a contract flows: the runtime path versus the discovery path]({{ '/assets/diagrams/17-final-project-contract-flow.svg' | relative_url }}) ### Runtime contracts -A **runtime contract** is essential on the hot path. The event schema is the -example: the producer serializes an event against the registered schema and the -consumer deserializes against it, so the event literally won't encode or decode -correctly if the contract is wrong or missing. The registry is in the live path of -every event. (The gRPC Protobuf definitions are similar in spirit, though compiled -ahead of time rather than fetched at runtime.) Because a runtime contract is -essential to execution, the registry can enforce real governance on it: reject a +A **runtime contract** is essential on the hot path. The event schema is the example: +the producer serializes an event against the registered schema and the consumer +deserializes against it, so the event literally won't encode or decode correctly if +the contract is wrong or missing. The registry sits in the live path of every event. +(The gRPC Protobuf definitions work the same way in spirit, though they're compiled +ahead of time rather than fetched at runtime.) Because a runtime contract is essential +to execution, the registry can enforce real governance on it: reject a schema-incompatible change at publish time, before it can break a single consumer. ### Discovery contracts -A **discovery contract** is different. The OpenAPI document and the GraphQL SDL are +A **discovery contract** is different. The OpenAPI document and the GraphQL SDL get published as the source of truth for a product's shape, but nothing fails at runtime -if they're absent — they exist for humans reading the API, for breaking-change checks +if they're absent. They exist for humans reading the API, for breaking-change checks in CI, and for the catalog to ingest. They describe; they don't serialize. -The same registry does both jobs, but the coupling differs: the -runtime path is tightly coupled and online, the discovery path is loosely coupled and -offline. Conflating them — treating a discovery contract as if it were enforced at -runtime, or a runtime contract as if it were just documentation — is how teams get -surprised in both directions. +The same registry does both jobs, but the coupling differs. The runtime path is +tightly coupled and online; the discovery path is loosely coupled and offline. +Conflate them, treat a discovery contract as if it were enforced at runtime, or a +runtime contract as if it were just documentation, and that's how teams get surprised +in both directions. ## The catalog: discovery and lineage -A registry tells you the *shape* of each contract. It doesn't, on its own, tell you -which products exist, who owns them, who consumes whom, or where a given piece of data -ultimately came from. That's the catalog's job. +When an order is placed, an event flows to inventory, payment, and notification. If +the order schema changes, what breaks downstream? A registry can't answer that on its +own: it tells you the *shape* of each contract, but not which products exist, who owns +them, who consumes whom, or where a given piece of data ultimately came from. That's +the catalog's job. ![Contracts in the registry, discovery and lineage in the catalog]({{ '/assets/diagrams/17-contracts-registry-catalog.svg' | relative_url }}) This final project layers **OpenMetadata** on top as the catalog. It ingests from the registry (the contracts), from the database (the schemas each domain owns), and from the event backbone (the topics), and assembles the picture a consumer actually needs: -the products that exist, their schemas, their owners, and — crucially — their -**lineage**, the who-produces-and-who-consumes graph across domains. When an order is -placed, an event flows to inventory, payment, and notification; the catalog records -that path, so you can answer "if the order schema changes, what's downstream?" without -reverse-engineering it from code. +the products that exist, their schemas, their owners, and their **lineage**, the +who-produces-and-who-consumes graph across domains. The catalog records the +order-to-inventory-to-payment-to-notification path directly, so you can answer that +question without reverse-engineering it from code. The sequencing matters and reflects a real dependency: the registry comes before the -catalog, because a catalog with nothing to catalog is empty — the contracts have to +catalog, because a catalog with nothing to catalog is empty. The contracts have to exist before lineage over them can be built. ## Why this is a requirement, not an add-on It's tempting to treat a catalog as a nice-to-have you bolt on once the "real" system -works. In a data mesh it's the opposite, and this is the central -point of the page. The entire premise of a mesh is that domains own their data -independently and other domains consume it without a central team brokering access. -That premise *only works* if products are discoverable and their contracts are -trustworthy. - -Without a registry, contracts drift and consumers break silently. Without -a catalog, nobody can find products or understand lineage, so in practice they fall -back to asking the central team — which reintroduces exactly the bottleneck the mesh -exists to remove. - -So discovery infrastructure isn't decoration on top of a mesh; it's structural to -it. A mesh without a usable catalog degrades into the -[proxy anti-pattern]({{ '/docs/09-anti-patterns/' | relative_url }}) — a -central team back in the middle of every cross-domain question — even if every other -piece is in place. The registry and catalog are what let federated governance be -*computational*: contracts enforced at publish time, lineage recorded automatically as -part of deploying, rather than standards administered by meeting. - -One operational note that belongs to building this rather than to the concept: when the -catalog ingests by running jobs in the cluster, those jobs have to be kept *out* of the +works. In a data mesh it's the opposite, and this is the central point of the page. +The entire premise of a mesh is that domains own their data independently and other +domains consume it without a central team brokering access. That premise *only works* +if products are discoverable and their contracts are trustworthy. + +Without a registry, contracts drift and consumers break silently. Without a catalog, +nobody can find products or understand lineage, so in practice they fall back to +asking the central team, which reintroduces exactly the bottleneck the mesh exists to +remove. + +So discovery infrastructure isn't decoration on top of a mesh; it's structural to it. +A mesh without a usable catalog degrades into the +[proxy anti-pattern]({{ '/docs/09-anti-patterns/' | relative_url }}): a central team +back in the middle of every cross-domain question, even if every other piece is in +place. The registry and catalog are what let federated governance be *computational*: +contracts enforced at publish time, lineage recorded automatically as part of +deploying, rather than standards administered by meeting. + +One operational note belongs to building this rather than to the concept: when the +catalog ingests by running jobs in the cluster, those jobs have to stay *out* of the service mesh, or they can't complete. That's a real sharp edge we hit, and it lives -with the operational gotchas rather than here — but it's worth knowing the catalog's -ingestion has a deployment subtlety when a mesh is in play. +with the operational gotchas rather than here. But it's worth knowing that the +catalog's ingestion has a deployment subtlety when a mesh is in play. -With products that describe themselves and a catalog that makes them discoverable, the -next question is how data actually moves between them — the synchronous read paths and -the asynchronous event backbone. +With products that describe themselves and a catalog that makes them discoverable, +the next question is how data actually moves between them: the synchronous read paths +and the asynchronous event backbone. ## For deeper coverage -- *Data Mesh* by Zhamak Dehghani — data-as-a-product and federated computational governance, the two principles this chapter makes operational. -- *Designing Data-Intensive Applications, 2nd ed.* by Martin Kleppmann & Chris Riccomini — schema evolution and the compatibility guarantees a registry enforces. -- *Cloud Native Anti-Patterns* by Gerald Bachlmayr et al. — the proxy anti-pattern a mesh degrades into without a usable catalog. -- Apicurio Registry documentation — contract versioning and compatibility rules across schemas, Protobuf, OpenAPI, and GraphQL SDL. +- *Data Mesh* by Zhamak Dehghani: data-as-a-product and federated computational governance, the two principles this chapter makes operational. +- *Designing Data-Intensive Applications, 2nd ed.* by Martin Kleppmann & Chris Riccomini: schema evolution and the compatibility guarantees a registry enforces. +- *Cloud Native Anti-Patterns* by Gerald Bachlmayr et al.: the proxy anti-pattern a mesh degrades into without a usable catalog. +- Apicurio Registry documentation: contract versioning and compatibility rules across schemas, Protobuf, OpenAPI, and GraphQL SDL. diff --git a/_docs/05-data-planes.md b/_docs/05-data-planes.md index ea75bb9..4a8cf94 100644 --- a/_docs/05-data-planes.md +++ b/_docs/05-data-planes.md @@ -1,7 +1,7 @@ --- title: "The data planes" order: 6 -description: The async backbone of Kafka events and the read layer of GraphQL composing REST and gRPC — when to reach for each, and why the final project uses all of them. +description: The async backbone of Kafka events and the read layer of GraphQL composing REST and gRPC; when to reach for each, and why the final project uses all of them. duration: 25 min --- @@ -9,28 +9,28 @@ duration: 25 min By the end of this chapter you can: -- Match each of the four protocols — REST, gRPC, GraphQL, and events — to the job it's best suited for, and explain the "fitness, not hierarchy" framing. +- Match each of the four protocols (REST, gRPC, GraphQL, and events) to the job it's best suited for, and explain the "fitness, not hierarchy" framing. - Distinguish gateway orchestration from true subgraph federation and state the trade-off this final project accepts by choosing orchestration. - Trace the `order.placed` event flow from producer to consumers and explain why the producer doesn't block on any of them. - Explain why the synchronous read layer and the asynchronous event backbone are complementary planes rather than competing ones. ## Why this matters -The [services]({{ '/docs/03-services-and-data-products/' | relative_url }}) -each expose a contract, and the -[catalog]({{ '/docs/04-contracts-and-catalog/' | relative_url }}) makes -them discoverable. This page is about how data actually *moves* between them — the -planes the mesh runs on. There are two: a **synchronous read layer** (REST, gRPC, and -a GraphQL gateway composing across them) and an **asynchronous event backbone** (the -streaming spine that lets domains react to each other). The final project uses all of them -because each one fits a different kind of data movement. +The [services]({{ '/docs/03-services-and-data-products/' | relative_url }}) each +expose a contract, and the +[catalog]({{ '/docs/04-contracts-and-catalog/' | relative_url }}) makes them +discoverable. This page is about how data actually *moves* between them: the planes +the mesh runs on. There are two: a **synchronous read layer** (REST, gRPC, and a +GraphQL gateway composing across them) and an **asynchronous event backbone** (the +streaming spine that lets domains react to each other). The final project uses all of +them because each one fits a different kind of data movement. ## Protocols by fitness, not by hierarchy It would be simpler to pick one protocol and use it everywhere. The final project -doesn't, because each protocol is *best at a -different job*, and a data mesh exercises all those jobs. This isn't a ranking where -one protocol wins; it's a matter of fitness for context: +doesn't, because each protocol is *best at a different job*, and a data mesh exercises +all those jobs. This isn't a ranking where one protocol wins; it's a matter of fitness +for context: - **REST at the edge.** External clients meet the system over REST, because it's the universal, cacheable, tooling-rich lingua franca for crossing a trust boundary. @@ -38,16 +38,16 @@ one protocol wins; it's a matter of fitness for context: mesh use gRPC, because it's fast, strongly typed from its Protobuf contract, and built for exactly that low-latency internal RPC. - **GraphQL for composing reads.** When a client needs data assembled from several - domains in one shaped response, GraphQL composes it — one query, multiple backends, + domains in one shaped response, GraphQL composes it: one query, multiple backends, the response shaped by the caller rather than by each service. - **Events for everything asynchronous.** When something *happens* and other domains need to react without the producer waiting on them, the event backbone carries it. -The final project makes this concrete in one place: the gateway's resolvers literally call -REST and gRPC side by side, and GraphQL stitches the result — so you can see all three -synchronous protocols cooperating rather than competing. +The final project makes this concrete in one place: the gateway's resolvers literally +call REST and gRPC side by side, and GraphQL stitches the result. You can watch all +three synchronous protocols cooperate rather than compete. -![Four protocols and where each one earns its place — REST at the edge, gRPC between services, GraphQL composing reads, events on the async backbone]({{ '/assets/diagrams/17-api-implementations.svg' | relative_url }}) +![Four protocols and where each one fits: REST at the edge, gRPC between services, GraphQL composing reads, events on the async backbone]({{ '/assets/diagrams/17-api-implementations.svg' | relative_url }}) ## The read layer: a gateway that composes @@ -55,86 +55,87 @@ A consumer that wants an order *and* its current stock level *and* its payment s shouldn't have to call three services and join the results by hand. The read layer solves that with a GraphQL gateway: the client sends one query, and the gateway's resolvers fetch each piece from the owning service over that service's existing -interface — the order from order-service over REST, the nested stock field from -inventory-service over gRPC — and GraphQL assembles the shaped response. +interface. The order comes from order-service over REST; the nested stock field comes +from inventory-service over gRPC. GraphQL assembles the shaped response. ![How analytical data is composed across domains into a served product]({{ '/assets/diagrams/17-analytical-data-composition.svg' | relative_url }}) -There's an important precision here about *how* the gateway composes, because there are -two ways to build a unified GraphQL graph and they're often conflated. In **true +There's an important precision here about *how* the gateway composes, because there +are two ways to build a unified GraphQL graph and they're often conflated. In **true subgraph federation**, each service exposes its own GraphQL subgraph and a gateway -plans queries across them into a supergraph — each domain owns and evolves its slice of -the graph independently. That's the production-scale pattern. This final project uses the -simpler **gateway orchestration** approach instead: one stateless gateway exposes the -unified graph, and its resolvers fetch from the services over their *existing* REST and -gRPC interfaces, with no GraphQL added to the domain services at all. +plans queries across them into a supergraph, with each domain owning and evolving its +slice of the graph independently. That's the production-scale pattern. This final +project uses the simpler **gateway orchestration** approach instead: one stateless +gateway exposes the unified graph, and its resolvers fetch from the services over +their *existing* REST and gRPC interfaces, with no GraphQL added to the domain +services at all. ### Why orchestration, not federation, here -The final project is a learning implementation. Gateway -orchestration demonstrates the *value* GraphQL adds (one client query, multiple -backends, client-shaped response) with one new service and zero changes to the five -domain services, and it makes the protocol comparison vivid because the resolvers call -REST and gRPC right next to each other. +The final project is a learning implementation. Gateway orchestration demonstrates the +*value* GraphQL adds (one client query, multiple backends, client-shaped response) +with one new service and zero changes to the five domain services, and it makes the +protocol comparison vivid because the resolvers call REST and gRPC right next to each +other. -The trade-off: true federation is -what you'd reach for in production, because it preserves domain ownership of the graph -— with orchestration, the gateway has some knowledge of how to reach each domain. For -teaching the shape, orchestration is the right call; for a production mesh, federation -keeps the ownership boundary cleaner. (This is also why the gateway is *not* itself a -domain data product — it's a read-layer convenience that composes products it doesn't -own.) +The trade-off is real: true federation is what you'd reach for in production, because +it preserves domain ownership of the graph. With orchestration, the gateway carries +some knowledge of how to reach each domain. For teaching the shape, orchestration is +the right call; for a production mesh, federation keeps the ownership boundary +cleaner. (This is also why the gateway is *not* itself a domain data product: it's a +read-layer convenience that composes products it doesn't own.) ## The async backbone: domains reacting to events -The read layer is request-and-response: a consumer asks, a service answers, the caller -waits. The other plane is the opposite shape, and it's what actually keeps a mesh's -operational/analytical loop closed. When an order is placed, several domains need to -react — inventory adjusts stock, payment processes, notification informs the customer — -but the order service shouldn't block waiting on any of them, and it certainly -shouldn't call them in a synchronous chain that fails if one is down. +When an order is placed, several domains need to react: inventory adjusts stock, +payment processes, notification informs the customer. But order-service shouldn't +block waiting on any of them, and it certainly shouldn't call them in a synchronous +chain that fails if one is down. So it doesn't. It emits an event, and the interested +domains consume it on their own schedule. -Instead it emits an event, and the interested domains consume it on their own schedule. +That's the opposite shape from the read layer, which is request-and-response: a +consumer asks, a service answers, the caller waits. The event backbone is what +actually keeps a mesh's operational/analytical loop closed. ![Domain events and change-data-capture feeding the analytical side]({{ '/assets/diagrams/17-ingestion-streaming-sourcing.svg' | relative_url }}) -This final project builds that spine on a Kafka cluster managed by an operator, and models -the canonical pattern with the smallest real flow: order-service emits an -`order.placed` event, keyed by order id, and notification-service consumes it. That one -flow is the whole "something happened, downstreams react" shape in miniature — the +This final project builds that spine on a Kafka cluster managed by an operator, and +models the canonical pattern with the smallest real flow: order-service emits an +`order.placed` event, keyed by order id, and notification-service consumes it. That +one flow is the whole "something happened, downstreams react" shape in miniature. The producer doesn't know or care who's listening, and consumers can be added without touching the producer. -Widen it and you get the full choreography: inventory and -payment also react to `order.placed`, shipping reacts to `payments.processed`, and -notification listens across all of them. +Widen it and you get the full choreography: inventory and payment also react to +`order.placed`, shipping reacts to `payments.processed`, and notification listens +across all of them. This is the plane that distinguishes a data mesh from a pile of services behind a -gateway. The synchronous read layer composes *current* state on demand; the event +gateway. The synchronous read layer composes *current* state on demand. The event backbone propagates *change* as it happens, so analytical consumers attach to the live operational flow rather than to a stale nightly extract. That's the closed loop the -[concepts page]({{ '/docs/01-concepts/' | relative_url }}) introduced — -and its absence is the -[open-loop anti-pattern]({{ '/docs/09-anti-patterns/' | relative_url }}), -where downstream products are disconnected from the operational systems that feed them. +[concepts page]({{ '/docs/01-concepts/' | relative_url }}) introduced, and its absence +is the +[open-loop anti-pattern]({{ '/docs/09-anti-patterns/' | relative_url }}), where +downstream products are disconnected from the operational systems that feed them. ## Two planes, one mesh -The synchronous and asynchronous planes aren't competitors; they're complementary, and +The synchronous and asynchronous planes aren't competitors. They're complementary, and a mesh needs both. Reads compose current state when a consumer asks for it. Events propagate change when something happens. REST crosses the boundary, gRPC moves between services, GraphQL shapes the composite, and the event backbone carries the reactions. -Each protocol is best at its job — which is the whole argument -for using all of them rather than forcing one to do everything. +Each protocol is best at its job, which is the whole argument for using all of them +rather than forcing one to do everything. With data moving across both planes, the next concern is how the mesh evolves safely -while it's running: shipping a new version of a product's contract without breaking the -consumers depending on it. +while it's running: shipping a new version of a product's contract without breaking +the consumers depending on it. ## For deeper coverage -- *Data Mesh* by Zhamak Dehghani — polyglot access and the synchronous/asynchronous planes a mesh runs on. -- *Designing Data-Intensive Applications, 2nd ed.* by Martin Kleppmann & Chris Riccomini — event-driven architecture and the derived-data reasoning behind the backbone. -- *Cloud Native Patterns* by Cornelia Davis — composing services at the edge versus internally. -- [grpc.io documentation](https://grpc.io/docs/) — gRPC for internal, low-latency service-to-service calls. -- [Strawberry GraphQL](https://strawberry.rocks/) — the library behind the gateway's composed graph. +- *Data Mesh* by Zhamak Dehghani: polyglot access and the synchronous/asynchronous planes a mesh runs on. +- *Designing Data-Intensive Applications, 2nd ed.* by Martin Kleppmann & Chris Riccomini: event-driven architecture and the derived-data reasoning behind the backbone. +- *Cloud Native Patterns* by Cornelia Davis: composing services at the edge versus internally. +- [grpc.io documentation](https://grpc.io/docs/): gRPC for internal, low-latency service-to-service calls. +- [Strawberry GraphQL](https://strawberry.rocks/): the library behind the gateway's composed graph. diff --git a/_docs/06-progressive-delivery-mtls.md b/_docs/06-progressive-delivery-mtls.md index 3f3bee8..1537091 100644 --- a/_docs/06-progressive-delivery-mtls.md +++ b/_docs/06-progressive-delivery-mtls.md @@ -17,130 +17,128 @@ By the end of this chapter you can: ## Why this matters -Data products evolve. A product that can never change its contract is a product nobody -will build on for long — but a product that changes its contract *carelessly* breaks -every consumer downstream. This page is about evolving safely: shifting traffic +Data products evolve. A product that can never change its contract is a product +nobody will build on for long, but a product that changes its contract *carelessly* +breaks every consumer downstream. This page is about evolving safely: shifting traffic gradually between an old and a new version of a product's contract, with the service -mesh both routing the split and securing the traffic. It's also where the final project -makes an architectural decision that the rest of the set has been alluding to — -to mesh **selectively** rather than meshing everything — and explains why, from +mesh both routing the split and securing the traffic. It's also where the final +project makes an architectural decision that the rest of the set has been alluding to, +to mesh **selectively** rather than meshing everything, and explains why, from experience. -![The service mesh — sidecars, mTLS, and traffic management]({{ '/assets/diagrams/11-istio-mesh.svg' | relative_url }}) +![The service mesh: sidecars, mTLS, and traffic management]({{ '/assets/diagrams/11-istio-mesh.svg' | relative_url }}) ## Canarying a contract, not just a binary -The interesting thing to canary in a data mesh isn't a new build of the same service — -it's a new version of the *contract*. This final project takes order-service through a -v1→v2 change: v2 adds a `currency` field to the order response, the canonical -"evolve the contract without breaking clients" move. v1 keeps serving the old shape; v2 -serves the new one; and Istio splits live traffic between them by weight, so you can -move from all-v1 to all-v2 gradually, watching for trouble at each step rather than -flipping a switch. +The interesting thing to canary in a data mesh isn't a new build of the same service. +It's a new version of the *contract*. This final project takes order-service through a +v1→v2 change: v2 adds a `currency` field to the order response, the canonical "evolve +the contract without breaking clients" move. v1 keeps serving the old shape; v2 serves +the new one; and Istio splits live traffic between them by weight, so you can move from +all-v1 to all-v2 gradually, watching for trouble at each step rather than flipping a +switch. The mechanism is the standard Istio trio: a `DestinationRule` defines the v1 and v2 -*subsets* (by a `version` pod label), a `VirtualService` routes a weighted split across -them, and a `Gateway` admits the traffic. Shifting the canary is just re-applying the -`VirtualService` with new weights — 90/10, then 50/50, then 0/100 — so the -progressive rollout is a sequence of one-line weight changes, and rolling back is the -same operation in reverse. - -Two simplifications here: v1 and v2 run the *same image* with an environment toggle -and a different `version` label, rather than two images from two commits — that -keeps the demo focused on the traffic-management mechanism rather than an image -pipeline, and the Istio mechanics are identical either way. - -And traffic +*subsets* (by a `version` pod label), a `VirtualService` routes a weighted split +across them, and a `Gateway` admits the traffic. Shifting the canary is just +re-applying the `VirtualService` with new weights (90/10, then 50/50, then 0/100), so +the progressive rollout is a sequence of one-line weight changes, and rolling back is +the same operation in reverse. + +Two things are simplified here. v1 and v2 run the *same image* with an +environment toggle and a different `version` label, rather than two images from two +commits. That keeps the demo focused on the traffic-management mechanism rather than +an image pipeline, and the Istio mechanics are identical either way. And traffic enters through the ingress gateway (reached on its NodePort, published to -127.0.0.1:8088 when the profile is created, DRA-017) rather than from a meshed client, so no client needs -to join the mesh for the canary to work. +127.0.0.1:8088 when the profile is created, DRA-017) rather than from a meshed client, +so no client needs to join the mesh for the canary to work. -This was verified end to end: with the split set to 90/10, a run of 100 requests landed -95 on v1 and 5 on v2; shifted to 50/50, it landed 46 on v1 and 54 on v2 — both within -the expected band, both subsets healthy in the mesh. The contract evolved in the open, -under controlled traffic, exactly as the pattern intends. +This was verified end to end: with the split set to 90/10, a run of 100 requests +landed 95 on v1 and 5 on v2; shifted to 50/50, it landed 46 on v1 and 54 on v2, both +within the expected band, both subsets healthy in the mesh. The contract evolved in +the open, under controlled traffic, exactly as the pattern intends. -![The final project's mesh in canary mode — sidecars beside each product, mTLS between them, weights moving traffic from v1 to v2]({{ '/assets/diagrams/17-service-mesh.svg' | relative_url }}) +![The final project's mesh in canary mode: sidecars beside each product, mTLS between them, weights moving traffic from v1 to v2]({{ '/assets/diagrams/17-service-mesh.svg' | relative_url }}) ## mTLS without code changes -The same mesh that routes the canary also secures it. When two services are in the -mesh, Istio establishes mutual TLS between their sidecars automatically — each service -proves its identity to the other and the traffic between them is encrypted, with no -change to the application code. That's the **federated computational governance** -principle in action: "traffic between products is authenticated and encrypted" becomes -a property the platform enforces, not a checklist item each team implements. The -service author writes no TLS code; the mesh provides it at the boundary. +The same mesh that routes the canary also secures it. Take order-service calling +inventory-service: once both are in the mesh, Istio establishes mutual TLS between +their sidecars automatically. Each service proves its identity to the other and the +traffic between them is encrypted, with no change to the application code. That's the +**federated computational governance** principle in action: "traffic between products +is authenticated and encrypted" becomes a property the platform enforces, not a +checklist item each team implements. The service author writes no TLS code; the mesh +provides it at the boundary. ## The decision: mesh selectively, not namespace-wide Here's the architectural choice this page exists to make, and it runs against the -convenient default. Istio offers a one-label switch — mark a namespace +convenient default. Istio offers a one-label switch: mark a namespace `istio-injection=enabled` and *every* pod created in it gets a sidecar automatically. -It's tempting: one label, whole-namespace mTLS, nothing to think about per workload. -The final project's position, learned the hard way, is that namespace-wide injection is the -wrong default for a mesh that contains more than just mesh-appropriate services — and -that you should instead opt specific workloads *into* the mesh. +It's tempting. One label, whole-namespace mTLS, nothing to think about per workload. +The final project's position, learned the hard way, is that namespace-wide injection +is the wrong default for a mesh that contains more than just mesh-appropriate +services, and that you should instead opt specific workloads *into* the mesh. -The reasoning is that a sidecar is not free, and it's not appropriate for every kind of -workload. Three categories in particular do not belong in the mesh, and meshing them +The reasoning is that a sidecar isn't free, and it isn't appropriate for every kind of +workload. Three categories in particular don't belong in the mesh, and meshing them ranges from wasteful to actively broken: - **Batch jobs that are supposed to finish.** A meshed Job gets a sidecar that never - exits — the proxy keeps running after the job's work is done, so the pod never reaches - a completed state and the Job hangs forever. The catalog's ingestion jobs hit exactly - this. A workload that's meant to run-and-terminate cannot carry a sidecar designed to - run-and-stay. -- **Operator-managed infrastructure with its own TLS.** A managed database that runs its - own TLS on its internal ports collides with an injected sidecar that re-wraps those - connections — the database's own encrypted channels break, and it crash-loops. Stateful - infrastructure that already secures itself doesn't want a second TLS layer wrapped - around it. + exits: the proxy keeps running after the job's work is done, so the pod never + reaches a completed state and the Job hangs forever. The catalog's ingestion jobs + hit exactly this. A workload that's meant to run-and-terminate can't carry a + sidecar designed to run-and-stay. +- **Operator-managed infrastructure with its own TLS.** A managed database that runs + its own TLS on its internal ports collides with an injected sidecar that re-wraps + those connections. The database's own encrypted channels break, and it + crash-loops. Stateful infrastructure that already secures itself doesn't want a + second TLS layer wrapped around it. - **Anything where the sidecar's overhead or coupling buys nothing.** Each sidecar - consumes resources and couples the workload's startup to the mesh control plane being - reachable. For a workload that isn't doing mesh-managed service-to-service traffic, - that's cost with no benefit. + consumes resources and couples the workload's startup to the mesh control plane + being reachable. For a workload that isn't doing mesh-managed service-to-service + traffic, that's cost with no benefit. Beyond those specific breakages, there's a systemic reason: with namespace-wide -injection, *every* pod creation in the namespace depends on the injection webhook being -reachable. If the mesh control plane has a bad moment, you can't create a database pod, -a job, or anything else — workloads that have nothing to do with the mesh are now -coupled to its health. - -Selective injection contains that blast radius: only the -workloads that actually need the mesh depend on it. So the final project meshes the services -that participate in mesh-managed traffic — the ones doing the canary, the ones that -benefit from mTLS and traffic management — and keeps the database, the batch jobs, and -other operator-managed infrastructure *out* of the mesh by opting them out explicitly. - -The trade-off is real: -namespace-wide injection is simpler to reason about and gives blanket mTLS -with one label, while selective injection is more work to configure and means you have -to decide, per workload, whether it belongs in the mesh. The final project takes that -configuration cost in exchange for resilience and correctness — because the failures -above aren't hypothetical, they're what happens otherwise. - -> The specific incidents behind this decision — the hung ingestion job, the +injection, *every* pod creation in the namespace depends on the injection webhook +being reachable. If the mesh control plane has a bad moment, you can't create a +database pod, a job, or anything else. Workloads that have nothing to do with the +mesh are now coupled to its health. + +Selective injection contains that blast radius: only the workloads that actually need +the mesh depend on it. So the final project meshes the services that participate in +mesh-managed traffic, the ones doing the canary, the ones that benefit from mTLS and +traffic management, and keeps the database, the batch jobs, and other +operator-managed infrastructure *out* of the mesh by opting them out explicitly. + +The trade-off is real: namespace-wide injection is simpler to reason about and gives +blanket mTLS with one label, while selective injection is more work to configure and +means you have to decide, per workload, whether it belongs in the mesh. The final +project takes that configuration cost in exchange for resilience and correctness, +because the failures above aren't hypothetical. They're what happens otherwise. + +> The specific incidents behind this decision, the hung ingestion job, the > crash-looping database, the control-plane dependency that bit during a node -> recovery — are documented as operational gotchas rather than retold here. This +> recovery, are documented as operational gotchas rather than retold here. This > page is the *decision*; the gotchas are the *evidence*, in detail. ## Why this belongs to the mesh chapter -Progressive delivery and mTLS are two sides of one capability: a service mesh that sits -between products, routing their traffic and securing it. The canary shows the routing -side — moving consumers from one contract version to the next without a flag day. The -mTLS shows the security side — authenticated, encrypted traffic as a platform property. -And the selective-injection decision is what keeps the mesh an asset rather than a -liability: applied where it helps, kept away from where it hurts. +Progressive delivery and mTLS are two sides of one capability: a service mesh that +sits between products, routing their traffic and securing it. The canary shows the +routing side: moving consumers from one contract version to the next without a flag +day. The mTLS shows the security side: authenticated, encrypted traffic as a platform +property. And the selective-injection decision is what keeps the mesh an asset rather +than a liability, applied where it helps, kept away from where it hurts. Next, the other half of operating a mesh under real conditions: scaling products to -match demand — including down to zero — and recovering when things fail. +match demand, including down to zero, and recovering when things fail. ## For deeper coverage -- [Istio documentation](https://istio.io/latest/docs/) — `DestinationRule`, `VirtualService`, `Gateway`, and automatic mutual TLS between sidecars. -- *Kubernetes Patterns* by Bilgin Ibryam & Roland Huß — the Sidecar pattern and the trade-offs of attaching one to every workload. -- *Platform Engineering on Kubernetes* by Mauricio Salatino — treating mesh injection as a platform policy decision rather than a blanket default. -- *GitOps and Kubernetes* by Billy Yuen et al. — progressive delivery workflows for shifting traffic between versions. -- *Cloud Native Anti-Patterns* by Gerald Bachlmayr et al. — the failure modes of meshing workloads that don't belong in the mesh. +- [Istio documentation](https://istio.io/latest/docs/): `DestinationRule`, `VirtualService`, `Gateway`, and automatic mutual TLS between sidecars. +- *Kubernetes Patterns* by Bilgin Ibryam & Roland Huß: the Sidecar pattern and the trade-offs of attaching one to every workload. +- *Platform Engineering on Kubernetes* by Mauricio Salatino: treating mesh injection as a platform policy decision rather than a blanket default. +- *GitOps and Kubernetes* by Billy Yuen et al.: progressive delivery workflows for shifting traffic between versions. +- *Cloud Native Anti-Patterns* by Gerald Bachlmayr et al.: the failure modes of meshing workloads that don't belong in the mesh. diff --git a/_docs/07-elastic-and-resilient.md b/_docs/07-elastic-and-resilient.md index 822a3c7..de4ef27 100644 --- a/_docs/07-elastic-and-resilient.md +++ b/_docs/07-elastic-and-resilient.md @@ -17,46 +17,46 @@ By the end of this chapter you can: ## Why this matters -A data product's demand isn't constant. A read gateway is busy when consumers are -querying and idle otherwise; an event consumer has work only when events are flowing. -Provisioning every product for its peak, all the time, wastes resources; provisioning -for the average means falling over at the peak. +Look at notification-service and the gateway and you can see two different shapes of +demand. The gateway is busy when consumers are querying and idle otherwise. Notification- +service, an event consumer, has work only when events are flowing. Provisioning either +one for its peak, all the time, wastes resources. Provisioning for the average means +falling over at the peak. -This page is about the platform handling that automatically — scaling products to -match demand, including down to *zero* when there's none — and about the other half of -operating under real conditions: recovering when something fails. Both are **self-serve -platform** capabilities: a domain team gets elasticity and resilience from the platform -rather than building them. +This page is about the platform handling that automatically: scaling products to match +demand, including down to *zero* when there's none, and about the other half of +operating under real conditions, recovering when something fails. Both are **self-serve +platform** capabilities. A domain team gets elasticity and resilience from the platform +instead of building them itself. -## Scaling to demand — and to zero +## Scaling to demand, and to zero The standard Kubernetes autoscaler scales on CPU and memory, which is a poor proxy for -the demand a data product actually faces. A read gateway's load is *requests*; an event -consumer's load is *messages waiting to be processed*. KEDA — Kubernetes Event-Driven -Autoscaling — scales on those real signals instead, and crucially it can scale a -workload all the way to **zero** when the signal is absent, then back up when it -returns. +the demand a data product actually faces. A read gateway's load is *requests*. An event +consumer's load is *messages waiting to be processed*. KEDA (Kubernetes Event-Driven +Autoscaling) scales on those real signals instead, and it can scale a workload all the +way to **zero** when the signal is absent, then back up when it returns. ![Standard CPU/memory autoscaling versus event-driven scaling]({{ '/assets/diagrams/12-hpa-vs-keda.svg' | relative_url }}) Scale-to-zero is what makes "elastic data product" more than a slogan. A product that -costs nothing while idle can exist without justifying its standing footprint — you can +costs nothing while idle can exist without justifying its standing footprint. You can have many such products, each materializing only when something actually asks for it. That's a different economics from "every service holds a baseline of resources forever," -and it's particularly apt for a mesh of many domain products with uneven, bursty demand. +and it fits a mesh of many domain products with uneven, bursty demand particularly well. This final project wires up **two** scalers of different kinds, on two different products: - **Consumer-lag scaling on notification-service.** notification-service is an - event consumer — its real workload is the backlog of events waiting in its consumer + event consumer: its real workload is the backlog of events waiting in its consumer group. A KEDA scaler watches that consumer-group lag and scales the service on it, down to zero when the backlog is empty and there's nothing to do. When orders start flowing again, the lag rises and KEDA scales it back up. -![KEDA scaling notification-service on Kafka consumer-group lag — scaling to zero when the backlog is empty, back up as events arrive]({{ '/assets/diagrams/17-keda-lag.svg' | relative_url }}) +![KEDA scaling notification-service on Kafka consumer-group lag: scaling to zero when the backlog is empty, back up as events arrive]({{ '/assets/diagrams/17-keda-lag.svg' | relative_url }}) - **HTTP-request scaling on the gateway.** The read gateway's load is incoming queries, - so it scales on HTTP request volume via KEDA's HTTP add-on — including to zero when no + so it scales on HTTP request volume via KEDA's HTTP add-on, including to zero when no one is querying. ![Scaling on HTTP request volume with the KEDA HTTP add-on]({{ '/assets/diagrams/12-keda-http-addon.svg' | relative_url }}) @@ -66,29 +66,29 @@ gateway, *not* on order-service, because order-service is where the [canary]({{ '/docs/06-progressive-delivery-mtls/' | relative_url }}) lives, and HTTP-scaling a service whose traffic is being split by weight would have the two mechanisms fighting over the same pods. The gateway is a natural synchronous-read load -target with no such conflict. Matching each scaler to the right product — lag-based for -the consumer, request-based for the gateway, and neither on the canaried service — is +target with no such conflict. Matching each scaler to the right product (lag-based for +the consumer, request-based for the gateway, neither on the canaried service) is the kind of fitting-the-mechanism-to-the-workload judgment the whole final project keeps returning to. -![Why the HTTP scaler goes on the gateway and not on the canaried order-service — matching the scaler to the workload]({{ '/assets/diagrams/17-keda-http.svg' | relative_url }}) +![Why the HTTP scaler goes on the gateway and not on the canaried order-service: matching the scaler to the workload]({{ '/assets/diagrams/17-keda-http.svg' | relative_url }}) One harmless quirk you'll see at rest: with the gateway scaled to zero, its HTTP scaler reports an "unknown" status until traffic first arrives and the add-on has something to -measure. That's expected — a scaled-to-zero workload has no current metric — and it -resolves the moment a request hits the gateway. +measure. That's expected. A scaled-to-zero workload has no current metric, and the +status resolves the moment a request hits the gateway. ## Resilience: recoverability as a platform property -The other half of operating under real conditions is what happens when things break — +The other half of operating under real conditions is what happens when things break, and on a real system, things break. The cloud-native answer isn't "prevent all -failure"; it's "make recovery cheap and automatic." Kubernetes provides a great deal of +failure." It's "make recovery cheap and automatic." Kubernetes provides a great deal of this without extra application code, and a data mesh inherits it: a crashed pod is restarted, a failed deployment is rolled back, a node's workloads are rescheduled, and declarative state means the system continuously reconciles back toward what you asked for rather than drifting away from it. -A data product deployed this way is *recoverable by construction* — you can kill it +A data product deployed this way is *recoverable by construction*. You can kill it and the platform brings it back. That property is worth designing *toward*, not just relying on. A product that recovers @@ -101,7 +101,7 @@ declarative deployment means re-applying the desired state is always a valid rec move. The final project runs on a single node, which makes recoverability both more visible and -more demanding than a multi-node cluster would — a single node concentrates the failure +more demanding than a multi-node cluster would: a single node concentrates the failure modes that several nodes would spread out and absorb. Operating a long-lived single-node cluster well turns out to require real care: knowing when to let the platform self-heal versus when to cycle a node, recognizing the difference between a @@ -118,19 +118,19 @@ than each building their own. Elasticity and recoverability are the same kind of thing from a domain team's point of view: capabilities the self-serve platform provides so that individual products don't -each have to solve them. A product scales to its demand because the platform scales it; -a product survives a crash because the platform reconciles it back. The domain declares -what it wants and the platform delivers the operational behavior — which is exactly the +each have to solve them. A product scales to its demand because the platform scales it. +A product survives a crash because the platform reconciles it back. The domain declares +what it wants and the platform delivers the operational behavior, which is exactly the self-serve principle, now applied to the *runtime* properties of a product rather than just its deployment. Last, the capability that makes all of this observable: seeing what the mesh is -actually doing — its metrics, its traces, and the live view of traffic moving between +actually doing, its metrics, its traces, and the live view of traffic moving between products. ## For deeper coverage -- *Kubernetes Patterns* by Bilgin Ibryam & Roland Huß — autoscaling and self-healing patterns on Kubernetes. -- *Cloud Native Patterns* by Cornelia Davis — elasticity and recoverability as platform-provided properties. -- *Platform Engineering on Kubernetes* by Mauricio Salatino — self-serve platform capabilities and what domains should and shouldn't operate themselves. -- [KEDA documentation](https://keda.sh/docs/) — scalers, `ScaledObject`s, and the HTTP add-on used in this chapter. +- *Kubernetes Patterns* by Bilgin Ibryam & Roland Huß: autoscaling and self-healing patterns on Kubernetes. +- *Cloud Native Patterns* by Cornelia Davis: elasticity and recoverability as platform-provided properties. +- *Platform Engineering on Kubernetes* by Mauricio Salatino: self-serve platform capabilities and what domains should and shouldn't operate themselves. +- [KEDA documentation](https://keda.sh/docs/): scalers, `ScaledObject`s, and the HTTP add-on used in this chapter. diff --git a/_docs/08-observability.md b/_docs/08-observability.md index 90950bb..4aac032 100644 --- a/_docs/08-observability.md +++ b/_docs/08-observability.md @@ -1,7 +1,7 @@ --- title: "Observability" order: 9 -description: Seeing what the mesh is doing — metrics, distributed traces across products, and the live view of traffic between them. +description: Seeing what the mesh is doing, metrics, distributed traces across products, and the live view of traffic between them. duration: 20 min --- @@ -11,13 +11,13 @@ By the end of this chapter you can: - Name the three observability signals (metrics, traces, logs) and the question each answers about a request crossing products. - Explain why the service mesh's sidecars provide inter-product metrics without any application instrumentation. -- Trace a single GraphQL query as a multi-span trace crossing three protocols — HTTP, REST, and gRPC — through the gateway. +- Trace a single GraphQL query as a multi-span trace crossing three protocols (HTTP, REST, and gRPC) through the gateway. - Explain what a live mesh-visualization tool like Kiali adds beyond metrics and traces. ## Why this matters The [previous page]({{ '/docs/07-elastic-and-resilient/' | relative_url }}) -ended on scaling and recovery — the platform doing things automatically in response to +ended on scaling and recovery: the platform doing things automatically in response to demand and failure. But "the platform does things automatically" is only reassuring if you can *see* it happening. @@ -25,88 +25,91 @@ This page is about observability: the metrics that show a product's load and the autoscaler's response, the distributed traces that follow a single request across several products, and the live view of traffic moving through the mesh. In a system of independently-owned products talking to each other, observability -isn't a dashboard you add at the end — it's how anyone understands behavior that no +isn't a dashboard you add at the end. It's how anyone understands behavior that no single product fully contains. ## The three signals, and why a mesh needs all of them -Observability conventionally rests on three kinds of signal, and a data mesh has a -distinct use for each. **Metrics** are aggregate numbers over time — request rates, -error rates, consumer lag, replica counts — and they're what show you that the +Watch a GraphQL query fan out from the gateway to order-service and inventory-service, +and you need three different kinds of evidence to understand what happened. +**Metrics** are aggregate numbers over time: request rates, error rates, consumer lag, +replica counts. They're what show you that the [autoscaler]({{ '/docs/07-elastic-and-resilient/' | relative_url }}) woke the gateway, or that one product's error rate is climbing. **Traces** follow a single request as it crosses product boundaries, so you can see that a slow GraphQL query was -slow because the gRPC call it made downstream was slow — something no single product's +slow because the gRPC call it made downstream was slow, something no single product's own logs would reveal. **Logs** are the detailed per-event record you reach for once metrics and traces have told you *where* to look. -The reason a mesh needs all three, rather than just per-service logs, is that the -interesting behavior lives *between* products. A request that touches the gateway, an +A mesh needs all three, not just per-service logs, because the interesting behavior +lives *between* products. A request that touches the gateway, an order service, and an inventory service is a single user-facing operation spread across -three independently-owned products; understanding it means correlating signals that no +three independently-owned products. Understanding it means correlating signals that no one product owns in full. -![The three signals — metrics, traces, and logs — and the question each answers about a request crossing products]({{ '/assets/diagrams/17-three-signals.svg' | relative_url }}) +![The three signals: metrics, traces, and logs, and the question each answers about a request crossing products]({{ '/assets/diagrams/17-three-signals.svg' | relative_url }}) ## Metrics from the mesh, no instrumentation needed -The first pleasant surprise is how much you get without instrumenting anything. Because +Open order-service's code and you won't find a line of metrics instrumentation in it, +yet Grafana still shows its request rate climbing under load. That's the first pleasant +surprise: how much you get without instrumenting anything. Because the services are in the [service mesh]({{ '/docs/06-progressive-delivery-mtls/' | relative_url }}), -the sidecars already emit request-level metrics — counts, latencies, error rates for -the traffic flowing between products — and a Prometheus that scrapes them gets a +the sidecars already emit request-level metrics (counts, latencies, error rates for +the traffic flowing between products), and a Prometheus that scrapes them gets a detailed picture of inter-product traffic without a single line of metrics code in the services. -This final project leans on exactly that: the metrics half of observability adds +This final project leans on exactly that. The metrics half of observability adds *nothing* to the application code, because the mesh is already measuring the traffic it -carries. Grafana then turns those metrics into the dashboards you'd actually watch — -the gateway's request volume rising, KEDA spinning up replicas in response, the canary's +carries. Grafana then turns those metrics into the dashboards you'd actually watch: the +gateway's request volume rising, KEDA spinning up replicas in response, the canary's traffic split landing where the weights say it should. -That property is the self-serve platform principle showing up again: a domain +That property is the self-serve platform principle showing up again. A domain team gets request metrics for its product because the platform's mesh provides them, not because the team instrumented anything. -![The observability stack — sidecars emit, Prometheus scrapes metrics, Tempo stores spans, Grafana correlates them, Kiali shows the live topology]({{ '/assets/diagrams/17-observability-stack.svg' | relative_url }}) +![The observability stack: sidecars emit, Prometheus scrapes metrics, Tempo stores spans, Grafana correlates them, Kiali shows the live topology]({{ '/assets/diagrams/17-observability-stack.svg' | relative_url }}) ## Traces across products Metrics tell you *that* something is slow or failing; traces tell you *where*. Distributed tracing assigns a request an identity that propagates as it crosses product boundaries, so the work each product does on its behalf becomes a connected tree of -**spans** — one request, many products, one trace. +**spans**: one request, many products, one trace. This final project collects traces in Tempo and instruments the **gateway** as the trace source. The gateway is the entry point for the federated read path, so a single GraphQL query there produces the most instructive trace there is in this system: an HTTP server span for the query itself, a REST client span for the -call out to order-service, and a gRPC client span for the call to inventory-service — -all three protocols from +call out to order-service, and a gRPC client span for the call to inventory-service. +All three protocols from [the data planes page]({{ '/docs/05-data-planes/' | relative_url }}) -appearing as nested spans in one trace. +show up as nested spans in one trace. You can *see* the composition happen: the gateway fanning out to two products over two different protocols and assembling the result. One well-chosen instrumentation point illuminates the whole read path. Instrumenting the entry point first, and leaving deeper per-service instrumentation as a -follow-on, is the same incremental discipline the rest of the build follows — one new +follow-on, is the same incremental discipline the rest of the build follows: one new capability at a time, each verifiable before the next. ## The live mesh view Metrics and traces are recorded and queried after the fact. The other thing you want -with a mesh is a *live* picture of the topology — which products are talking to which, +with a mesh is a *live* picture of the topology: which products are talking to which, right now, with traffic rates and health on the edges between them. That's what a mesh-visualization layer such as Kiali provides: a real-time graph of the products and the traffic between them, drawn from the same mesh telemetry the metrics come from. -During a canary it shows the v1/v2 split as live traffic on the graph; when a product is +During a canary it shows the v1/v2 split as live traffic on the graph. When a product is struggling, the unhealthy edges are visible in the topology. For a system whose defining behavior is products talking to each other, a live map of exactly that is the single most legible view available. -![A reference data-mesh architecture — products, planes, and the observability that spans them]({{ '/assets/diagrams/17-reference-data-mesh-architecture.svg' | relative_url }}) +![A reference data-mesh architecture: products, planes, and the observability that spans them]({{ '/assets/diagrams/17-reference-data-mesh-architecture.svg' | relative_url }}) ## What it all adds up to @@ -116,10 +119,10 @@ show the gateway's load and the autoscaler's response to it. The catalog records lineage of which products feed which. The mesh view shows the live topology, canary split and all. -None of these is the system — they're the *instruments* through which you +None of these is the system. They're the *instruments* through which you understand a system too distributed for any single vantage point to capture. That's the acceptance bar the final project is built toward: not just that the mesh runs, but that you -can watch it run — see a request thread across products, see a product scale to meet +can watch it run. See a request thread across products, see a product scale to meet demand, see a contract roll out version by version, and see where every product's data came from. @@ -127,13 +130,13 @@ That completes the implementation. The build is a working data mesh: services th their data as products, contracts and a catalog that make them discoverable, two data planes moving current state and propagating change, progressive delivery with mTLS, elastic scaling with recovery, and the observability to see all of it. The one thing -left is to step back from *how* and look at *what goes wrong* — the conceptual and +left is to step back from *how* and look at *what goes wrong*: the conceptual and organizational failure modes that derail data-mesh efforts even when the technology is sound. ## For deeper coverage -- *Designing Data-Intensive Applications, 2nd ed.* by Martin Kleppmann & Chris Riccomini — reasoning about the behavior of distributed systems built from independently-owned parts. -- [OpenTelemetry documentation](https://opentelemetry.io/docs/) — the metrics/traces/logs signal model this chapter applies to a mesh. -- [Grafana documentation](https://grafana.com/docs/) — Tempo for traces, Mimir/Prometheus for metrics, and correlating the two. -- *Cloud Native Patterns* by Cornelia Davis — observability as a platform-provided concern rather than a per-service add-on. +- *Designing Data-Intensive Applications, 2nd ed.* by Martin Kleppmann & Chris Riccomini: reasoning about the behavior of distributed systems built from independently-owned parts. +- [OpenTelemetry documentation](https://opentelemetry.io/docs/): the metrics/traces/logs signal model this chapter applies to a mesh. +- [Grafana documentation](https://grafana.com/docs/): Tempo for traces, Mimir/Prometheus for metrics, and correlating the two. +- *Cloud Native Patterns* by Cornelia Davis: observability as a platform-provided concern rather than a per-service add-on. diff --git a/_docs/09-anti-patterns.md b/_docs/09-anti-patterns.md index e1a0fd2..c41b96b 100644 --- a/_docs/09-anti-patterns.md +++ b/_docs/09-anti-patterns.md @@ -1,7 +1,7 @@ --- title: "Anti-patterns" order: 10 -description: The conceptual and organizational ways data-mesh efforts go wrong, drawn from the literature — so you can recognize them early. +description: The conceptual and organizational ways data-mesh efforts go wrong, drawn from the literature, so you can recognize them early. duration: 15 min --- @@ -17,14 +17,14 @@ By the end of this chapter you can: ## Why this matters Everything else in this set is about what to build and how. This page is about -what goes wrong — not the implementation potholes we hit wiring up the demo (those +what goes wrong. Not the implementation potholes we hit wiring up the demo (those are operational, and collected separately), but the *conceptual and organizational* failure modes that show up again and again in real data-mesh efforts. They're worth knowing before you start, because most of them are invisible at the architecture- diagram level and only become obvious months in, once they've calcified. A useful framing first: a data mesh is a **socio-technical** system. The technical -pieces — a registry, a catalog, event streams, autoscaling — are the easy part, and +pieces (a registry, a catalog, event streams, autoscaling) are the easy part, and this tutorial spends most of its pages there because they're concrete and runnable. But the literature is nearly unanimous that data-mesh efforts fail for *organizational* reasons far more often than technical ones. The anti-patterns below are mostly @@ -34,24 +34,23 @@ note on where it touched our own build. ## The tool will solve it -The most common trap is treating data mesh as something you buy or install. A team -adopts a catalog product, relabels the data lake's tables as "data products," and -declares the mesh delivered. But the principles a mesh rests on — domain ownership, -product thinking, self-serve infrastructure, federated governance — are mostly about -*how people and teams work*, not which software is running. +Open a vendor's catalog product, point it at the data lake's existing tables, relabel +them "data products," and declare the mesh delivered. That's the most common trap: +treating data mesh as something you buy or install. The principles a mesh rests on +(domain ownership, product thinking, self-serve infrastructure, federated governance) +are mostly about *how people and teams work*, not which software is running. -Practitioners writing -about failed adoptions consistently put this first: a data mesh is a shift in -operating model, and no tool delivers an operating-model change on its own. The -software can *support* the principles or quietly undermine them, but it can't -substitute for the cultural and structural change underneath. +Practitioners writing about failed adoptions consistently put this first: a data mesh +is a shift in operating model, and no tool delivers an operating-model change on its +own. The software can *support* the principles or quietly undermine them, but it +can't substitute for the cultural and structural change underneath. The tell is a project plan that's entirely a tooling rollout with no mention of team boundaries, ownership, or incentives. If the only thing changing is the software, what you'll have at the end is the old centralized model with a new dashboard. *In this build:* every component we chose (Apicurio, OpenMetadata, KEDA, Istio) is -a *substrate* the domains build on, not a turnkey mesh — the +a *substrate* the domains build on, not a turnkey mesh. The [concepts page]({{ '/docs/01-concepts/' | relative_url }}) frames each tool as serving a principle rather than being the point. @@ -61,17 +60,16 @@ Data mesh exists to break the bottleneck of a single central data team that owns the data but none of the domains. The failure mode is recreating that bottleneck under new vocabulary. It takes a few recognizable shapes. -A central team stays the -*proxy* for every domain — still the only group that can actually ship a data +A central team stays the *proxy* for every domain, still the only group that can actually ship a data product, just now called a "platform team." Or governance re-centralizes as a manual approval gate: every schema change, every deployment, every contract waits on a review board, and the lead times that data mesh was supposed to eliminate quietly return. Or domains, lacking support, spin up *shadow* data teams of their own, and -you end up with silos again — just more of them. +you end up with silos again, just more of them. The common thread is that decentralization is the whole point, and any structure that -funnels decisions back through one team — for control, for "consistency," for -governance — reintroduces the original problem. The fix the literature points to is +funnels decisions back through one team (for control, for "consistency," for +governance) reintroduces the original problem. The fix the literature points to is the same in every case: push real ownership and the ability to ship to the domains, and make governance *automated and policy-driven* rather than approval-driven. @@ -82,31 +80,31 @@ automatically rather than gating changes through review. ## Data products that are "dumb" This is the failure mode that the principle's own author warns about most sharply. A -*data product* is supposed to be an autonomous unit — it serves its data, governs it, +*data product* is supposed to be an autonomous unit. It serves its data, governs it, describes itself, makes itself discoverable, and carries everything it needs to do its job. The reduction is to strip all of that away and call a renamed table, a view in a warehouse, or a row in a catalog a "data product." Those are static datasets with a label. They don't serve themselves, can't enforce their own policies, and have -no lifecycle — and because there's nowhere to embed governance *in* them, federated +no lifecycle. And because there's nowhere to embed governance *in* them, federated computational governance has no home and gets dropped too. The downgrade of data products cascades: lose autonomy, and you lose the governance model with it. -The recognizable symptom is a "data product" you can't deploy, version, or call — -you can only query the table it points at. A real data product has ports, a contract, -a version, and an owner; a dumb one has a name in a catalog. +The recognizable symptom is a "data product" you can't deploy, version, or call; you +can only query the table it points at. A real data product has ports, a contract, +a version, and an owner. A dumb one has a name in a catalog. -*In this build:* each service *is* its data product — it owns its schema, publishes a +*In this build:* each service *is* its data product. It owns its schema, publishes a versioned contract, emits its own events, and exposes its own API. The [services page]({{ '/docs/03-services-and-data-products/' | relative_url }}) and [contracts page]({{ '/docs/04-contracts-and-catalog/' | relative_url }}) are about exactly this autonomy. -## Governance as an afterthought — or as bureaucracy +## Governance as an afterthought, or as bureaucracy -Governance fails in two opposite directions. Bolt it on from outside — a separate -team, a separate process, run after products are already built — and it never quite +Governance fails in two opposite directions. Bolt it on from outside (a separate +team, a separate process, run after products are already built) and it never quite fits; quality, lineage, and access rules become things that happen *to* a product rather than properties *of* it. Over-correct, and governance becomes a heavyweight approval bureaucracy that bottlenecks everything, which is just centralization again. @@ -114,13 +112,13 @@ approval bureaucracy that bottlenecks everything, which is just centralization a The target the principles name is *federated computational governance*: a small set of global rules that the platform enforces automatically, embedded in each product rather than administered by a committee. The interesting design question for any mesh -is which rules are global (so products can interoperate — shared identifiers, contract +is which rules are global (so products can interoperate: shared identifiers, contract formats, lineage conventions) versus which are left to domains. Get that line wrong in either direction and you get either a free-for-all where nothing joins up, or a bottleneck where nothing ships. *In this build:* contracts live in a registry and lineage is recorded in the catalog -automatically as part of deploying, not as a separate review step — +automatically as part of deploying, not as a separate review step: [progressive delivery & mTLS]({{ '/docs/06-progressive-delivery-mtls/' | relative_url }}) shows version evolution governed by the platform rather than by sign-off. @@ -131,40 +129,39 @@ here. The first is the ownership vacuum: a dataset that no individual or team is actually responsible for, so its quality drifts, nobody fields questions about it, and trust erodes until consumers route around it. -The second is fuzzy domain boundaries: -domains that overlap or are ill-defined, so the same concept is modeled three -different ways by three teams, changes ripple across services that should have been -independent, and consumers get conflicting versions of what's nominally the same data. +The second is fuzzy domain boundaries. Domains that overlap or are ill-defined, so the +same concept is modeled three different ways by three teams, changes ripple across +services that should have been independent, and consumers get conflicting versions of +what's nominally the same data. -Both come down to bounded contexts — the same discipline that makes microservices +Both come down to bounded contexts, the same discipline that makes microservices work. Without clear, agreed domain boundaries and an explicit owner per data product, the mesh degrades into the distributed mess the principles were meant to prevent. The remedy is unglamorous: actually map the domains, write down who owns each product, and revisit the boundaries as the organization changes. *In this build:* each domain is a service with a single clear responsibility and a -named owner-by-construction — the order domain owns orders, inventory owns stock, and +named owner-by-construction: the order domain owns orders, inventory owns stock, and the boundaries are the service boundaries. -## The open loop — no feedback +## The open loop, no feedback Data mesh is meant to close the loop between the operational systems that produce data and the analytical uses that consume it, organized by domain. The degraded version is an *open loop*: static analytical products built downstream from a lake or warehouse, disconnected from the operational systems and from their own consumers. -There's no -feedback path — neither the operational-to-analytical loop that keeps products current +There's no feedback path. Neither the operational-to-analytical loop that keeps products current with the running business, nor the consumer-feedback loop that tells a product owner whether the product is actually useful. Without feedback, products go stale, the long lead time between an application change and its analytical impact never shrinks, and the mesh delivers little more than the warehouse it replaced. The tell is a data product nobody monitors for use and nobody updates in response to -how it's consumed — published once, then frozen. +how it's consumed, published once, then frozen. *In this build:* the async spine means analytical consumers react to operational -events as they happen rather than to a nightly extract — the +events as they happen rather than to a nightly extract. The [data planes page]({{ '/docs/05-data-planes/' | relative_url }}) covers why the event backbone is what keeps the loop closed. @@ -176,19 +173,18 @@ organizations with many data domains, many consumers, and the organizational mat to operate products and standards across teams. For a small organization, the overhead of decentralization can cost more than the bottleneck it removes. -Related traps in this -family: chasing more data products as an end in itself (the right number is what the +Related traps in this family: chasing more data products as an end in itself (the right number is what the organization can actually consume, not the maximum it can produce); analysis paralysis, where teams plan the perfect mesh for months instead of standing up one real product and learning from it; and adopting the whole paradigm because it's fashionable when -adopting a single principle — say, self-serve platform infrastructure — would have +adopting a single principle, say, self-serve platform infrastructure, would have served better. Before committing, weigh data size, organizational complexity, existing tooling, and culture, and be willing to conclude that a full mesh isn't the -right fit — or that only some of its principles are. +right fit, or that only some of its principles are. -*In this build:* the final project is a *learning* implementation — a small, +*In this build:* the final project is a *learning* implementation, a small, runnable mesh that makes the principles concrete. It's sized to teach the shape, not to argue that every reader should run a mesh in production. @@ -198,18 +194,18 @@ None of these are exotic. They're the predictable result of taking a paradigm th fundamentally about ownership, autonomy, and feedback and implementing only its visible technical surface. The recurring lesson across everyone who's written about failed efforts is the same: the architecture diagram is the easy part, and the parts -that aren't on the diagram — who owns what, how governance is enforced, whether the -loop is closed, whether the organization needed a mesh at all — are where efforts +that aren't on the diagram (who owns what, how governance is enforced, whether the +loop is closed, whether the organization needed a mesh at all) are where efforts actually succeed or fail. -The implementation-level potholes we hit building *this* demo — the operational -gotchas, as opposed to these conceptual ones — are collected separately, since they're +The implementation-level potholes we hit building *this* demo, the operational +gotchas, as opposed to these conceptual ones, are collected separately, since they're a different kind of lesson: specific, technical, and particular to running a mesh on a single-node cluster rather than to data mesh as an idea. ## For deeper coverage -- *Data Mesh* by Zhamak Dehghani — the four principles and the failure modes that appear when each is skipped. -- *Cloud Native Anti-Patterns* by Gerald Bachlmayr et al. — recognizable failure shapes in cloud-native platform adoption generally. -- *Learning Domain-Driven Design* by Vlad Khononov — bounded contexts and why fuzzy domain boundaries cascade into mesh failures. -- *Implementing Domain-Driven Design* by Vaughn Vernon — the ownership and autonomy a real data product needs to avoid the "dumb data product" trap. +- *Data Mesh* by Zhamak Dehghani: the four principles and the failure modes that appear when each is skipped. +- *Cloud Native Anti-Patterns* by Gerald Bachlmayr et al.: recognizable failure shapes in cloud-native platform adoption generally. +- *Learning Domain-Driven Design* by Vlad Khononov: bounded contexts and why fuzzy domain boundaries cascade into mesh failures. +- *Implementing Domain-Driven Design* by Vaughn Vernon: the ownership and autonomy a real data product needs to avoid the "dumb data product" trap. diff --git a/_docs/10-summary.md b/_docs/10-summary.md index c4a0e20..7091f0d 100644 --- a/_docs/10-summary.md +++ b/_docs/10-summary.md @@ -1,7 +1,7 @@ --- -title: "Summary — the four principles, realized" +title: "Summary: the four principles, realized" order: 11 -description: A closing summary — for each of the four data-mesh principles, the value it delivers, the implementation pieces that realize it, and what happens without it. +description: A closing summary, for each of the four data-mesh principles, the value it delivers, the implementation pieces that realize it, and what happens without it. duration: 10 min --- @@ -17,19 +17,19 @@ By the end of this chapter you can: ## Why this matters The reading set has worked through the data mesh implementation by -*concern* — substrate, services, contracts, planes, mesh, scaling, +*concern*: substrate, services, contracts, planes, mesh, scaling, observability, failure modes. This closing page reorganizes the same material by *principle*. -For each of Dehghani's four principles, you'll -see the value the principle delivers when it's honored, the concrete -implementation pieces in this reference that realize it, and the failure -mode that shows up when the principle is missing. +For each of Dehghani's four principles, you'll see the value the +principle delivers when it's honored, the concrete implementation +pieces in this reference that realize it, and the failure mode that +shows up when the principle is missing. The diagrams on this page are reproduced from the implementation deck at `presentation/data-mesh-openshift/`. Because the deck pairs this reference with OpenShift specifically, the diagrams use OpenShift vocabulary in a few -places — *Project* for namespace, *SCCs* for PodSecurity, *OperatorHub / +places: *Project* for namespace, *SCCs* for PodSecurity, *OperatorHub / OLM* for the operator install path, *AMQ Streams* for Strimzi, *CNPG* for CloudNativePG. @@ -41,17 +41,18 @@ namespace, *SCCs* as PodSecurity, *OperatorHub* as `helm install` of the relevant operators, and so on. The reference's own runnable implementation lives in -`examples/lgtm-datamesh/` and runs on a minikube profile — verified -configuration for laptop reproducibility — but every choice it makes is +`examples/lgtm-datamesh/` and runs on a minikube profile, verified +configuration for laptop reproducibility, but every choice it makes is recognizable on any compliant Kubernetes cluster. -## Principle 01 — Domain ownership +## Principle 01: Domain ownership -Each domain ships its data product on its own, with a clear owner and an -enforced boundary. No central data team sits in the path between the -domain that produces the data and the domains that consume it. The -platform's job is to enforce the boundaries; the domain's job is to fill -them with a working product. +The order domain ships order-service on its own, with a clear owner and +an enforced boundary, and the same holds for inventory, payment, +shipping, and notification. No central data team sits in the path +between the domain that produces the data and the domains that consume +it. The platform's job is to enforce the boundaries; the domain's job is +to fill them with a working product. The value of getting this right is that ownership is unambiguous. There's one team responsible for the customer data product, one team responsible @@ -59,7 +60,7 @@ for orders, one team responsible for inventory. Decisions about what those products look like, how they evolve, and what guarantees they provide are made by the people closest to the work. -![Principle 01 realized — domain ownership and the implementation pieces that deliver it]({{ '/assets/diagrams/17-value-domain-ownership.svg' | relative_url }}) +![Principle 01 realized: domain ownership and the implementation pieces that deliver it]({{ '/assets/diagrams/17-value-domain-ownership.svg' | relative_url }}) In Kubernetes terms, the boundary is the *namespace*. Each domain gets a namespace (an OpenShift Project, on that runtime), and the platform @@ -73,20 +74,20 @@ of capacity another domain needs. The reference's order, inventory, payment, shipping, and notification domains each live in their own namespace. The reference uses a single shared `datamesh` namespace for the demo's clarity, but the production -shape is one namespace per domain — the reference's helm chart structure +shape is one namespace per domain: the reference's helm chart structure supports that directly. Without this principle: fuzzy boundaries and ownership vacuums. Data that nobody owns is modeled three different ways by three different teams. The mesh becomes a misnomer; it's a centralized data lake with extra steps. -## Principle 02 — Data as a product +## Principle 02: Data as a product A data product is *discoverable*, *addressable*, *trustworthy*, and *self-describing*. Consumers find it through a catalog rather than through a relationship with the producing team. They depend on it through a versioned contract rather than a phone call. They trust it -because the platform — not a tribal knowledge channel — enforces the +because the platform, not a tribal knowledge channel, enforces the guarantees. The value of getting this right is that consumption decouples from @@ -94,14 +95,14 @@ production. A consumer doesn't need to know which team owns a data product; they need to know what the product is, what it guarantees, and how to address it. -![Principle 02 realized — data as a product and the implementation pieces that deliver it]({{ '/assets/diagrams/17-value-data-product.svg' | relative_url }}) +![Principle 02 realized: data as a product and the implementation pieces that deliver it]({{ '/assets/diagrams/17-value-data-product.svg' | relative_url }}) The implementation pieces that realize this in the reference: every -service's Deployment + Service makes the product *addressable* — a +service's Deployment + Service makes the product *addressable*. A domain consumer reaches `order-service.datamesh.svc.cluster.local` and gets the order product. The Apicurio schema registry holds the -*versioned contracts* — OpenAPI for REST, Protobuf for gRPC, AsyncAPI -for the Kafka events — and rejects breaking changes at registration +*versioned contracts* (OpenAPI for REST, Protobuf for gRPC, AsyncAPI +for the Kafka events) and rejects breaking changes at registration time. OpenMetadata is the catalog of *discoverability*: every product in the mesh is listed, with its owner, its schema, its lineage, and its consumption pattern. CRDs (custom resource definitions) carry @@ -114,12 +115,12 @@ discover it (catalog), depend on its shape (contract), reach it (addressable Service), and verify it conforms to mesh standards (CRD + admission). -Without this principle: "dumb" data products — renamed tables with no +Without this principle: "dumb" data products, renamed tables with no contract that can't serve themselves, govern themselves, or describe themselves to consumers. Discovery becomes a Slack channel; trust becomes word-of-mouth; governance has nowhere to live. -## Principle 03 — Self-serve data platform +## Principle 03: Self-serve data platform Domains consume the platform's capabilities (streaming, databases, scaling, the mesh itself) *by declaration*. They don't operate the @@ -133,18 +134,18 @@ to provision a queue, stand up a database, configure scaling, or wire in observability. The declarative pattern means the platform team scales by *adding capabilities*, not by *processing tickets*. -![Principle 03 realized — self-serve platform and the implementation pieces that deliver it]({{ '/assets/diagrams/17-value-self-serve.svg' | relative_url }}) +![Principle 03 realized: self-serve platform and the implementation pieces that deliver it]({{ '/assets/diagrams/17-value-self-serve.svg' | relative_url }}) The implementation pieces: the operator pattern (OperatorHub / OLM on OpenShift; `helm install` of upstream operators on other runtimes) -makes platform capabilities *curated and lifecycle-managed* — install +makes platform capabilities *curated and lifecycle-managed*: install once, every domain consumes by declaration. Strimzi (AMQ Streams on OpenShift) makes Kafka available *as a custom resource*: a domain declares its consumer group; the operator handles the cluster. CloudNativePG (CNPG) does the same for Postgres: a domain declares its database; the operator handles the cluster, the backups, the failover. -KEDA makes elastic scaling — including scale-to-zero — available as a +KEDA makes elastic scaling, including scale-to-zero, available as a ScaledObject, the same way: declare the demand signal, the operator handles the scaling. GitOps closes the loop by making the whole mesh *reproducible from Git*: every namespace, every operator, every @@ -152,12 +153,12 @@ ScaledObject, every Istio policy lives in a repo and is the source of truth. Without this principle: every domain reinvents the same infrastructure -badly. Ownership fragments into shadow platform teams — each domain -builds its own Kafka cluster, its own database, its own scaling logic -— and the cost of the duplication is hidden in domain budgets where +badly. Ownership fragments into shadow platform teams: each domain +builds its own Kafka cluster, its own database, its own scaling logic, +and the cost of the duplication is hidden in domain budgets where nobody sees it as a platform-investment problem. -## Principle 04 — Federated computational governance +## Principle 04: Federated computational governance Global rules (security policy, contract-versioning rules, observability standards, data-classification rules) are enforced *by the platform*, @@ -170,13 +171,13 @@ The value of getting this right is that decentralization doesn't require giving up consistency. Every domain can ship independently *because* the platform is enforcing the shared rules automatically. -![Principle 04 realized — federated computational governance and the implementation pieces that deliver it]({{ '/assets/diagrams/17-value-governance.svg' | relative_url }}) +![Principle 04 realized: federated computational governance and the implementation pieces that deliver it]({{ '/assets/diagrams/17-value-governance.svg' | relative_url }}) The implementation pieces: service-mesh mTLS makes every product-to-product call authenticated and encrypted, without any application code knowing about TLS. Istio's traffic-management primitives (VirtualService, DestinationRule, the canary subset routing the reference uses for the -v1→v2 contract demo) make *progressive delivery* a platform capability — +v1→v2 contract demo) make *progressive delivery* a platform capability: a contract evolves under controlled traffic, weight-shifted live, with the application layer unchanged. @@ -188,10 +189,10 @@ resource requests set. Prometheus/Tempo/Grafana/Kiali make governance *observable*: the metrics show the mesh's behavior, the traces show cross-product dependencies, the topology shows who's calling whom in real time. That visibility is the feedback loop that makes governance -actually work — without it, governance is theatre. +actually work; without it, governance is theatre. Without this principle: governance bolted on from outside the platform -never fits, or worse, re-centralizes into an approval bottleneck — the +never fits, or worse, re-centralizes into an approval bottleneck: the mesh's own anti-pattern of "federated governance" implemented as a review board. The [anti-patterns page]({{ '/docs/09-anti-patterns/' | relative_url }}) @@ -203,10 +204,10 @@ One more piece, beyond the four principles: the supply chain. Every data product in the mesh ships as a container image, and the platform's job extends to giving every domain a *trusted base* to layer on. -![A trusted base for every data product — the supply chain the platform gives each domain]({{ '/assets/diagrams/17-trusted-supply-chain.svg' | relative_url }}) +![A trusted base for every data product: the supply chain the platform gives each domain]({{ '/assets/diagrams/17-trusted-supply-chain.svg' | relative_url }}) The deck pairs this reference with OpenShift's specific supply chain: -Red Hat Universal Base Image (UBI) — `ubi10/python-314-minimal` is the base for +Red Hat Universal Base Image (UBI): `ubi10/python-314-minimal` is the base for every Python service in the reference. UBI is freely redistributable and enterprise-maintained, which means a domain team gets a base image they can use without licensing constraints and that's patched on a @@ -216,11 +217,11 @@ Language dependencies come from Red Hat's verified channels or PyPI, with known provenance; the resulting images are signed and have an SBOM, and admission policies verify both before the image runs. The domain's data product layers its FastAPI / gRPC / -Strawberry code on top of the trusted base — the domain owns its +Strawberry code on top of the trusted base. The domain owns its application; the platform owns the base. On other runtimes, the supply-chain shape is the same: a curated base -image (Distroless, Chainguard, or a minimal distro base — choose by your +image (Distroless, Chainguard, or a minimal distro base: choose by your trust model), language dependencies from controlled channels, signed images, an SBOM at build time, and admission policies that verify both. The reference's bootstrap uses UBI because the implementation deck @@ -230,7 +231,7 @@ line in each service's Dockerfile. The reason the supply chain belongs in this summary is that it's the piece of the mesh that's most often missed in discussion. Domains can own their products, products can be discoverable and self-describing, -the platform can be self-serve, and governance can be federated — and +the platform can be self-serve, and governance can be federated, and none of that matters if the base images are unpatched. The mesh's weakest link is the layer underneath everyone's application code. @@ -243,7 +244,7 @@ your own data mesh on Kubernetes, the pattern is: 1. **Read this set straight through once** for the shape of the system, then once more for the trade-offs. -2. **Run the example tree** to see the cross-product behaviors live — +2. **Run the example tree** to see the cross-product behaviors live: the trace tree across three products, the canary contract evolution under live traffic, the autoscaler reacting to a Kafka backlog. 3. **Use the decision log** at `_plans/archive/final-project-decisions.md` @@ -258,12 +259,12 @@ your own data mesh on Kubernetes, the pattern is: The reference's most important property is that the principles and their implementation pieces are *visible together*. Conceptual material gives you the principles; vendor material gives you the pieces. This -reference's job is to be the bridge — concrete enough to be runnable, +reference's job is to be the bridge: concrete enough to be runnable, explicit enough about its choices to be portable. ## For deeper coverage -- *Data Mesh* by Zhamak Dehghani — the four principles this closing page reorganizes the reference by. -- *Cloud Native Patterns* by Cornelia Davis — the platform patterns (self-serve, federated governance) behind this summary's implementation pieces. -- *Kubernetes Patterns* by Bilgin Ibryam & Roland Huß — the operator and CRD shapes that make each principle self-serve. -- *GitOps and Kubernetes* by Billy Yuen et al. — reproducing the whole mesh from a Git repository, the practice Principle 03's GitOps piece rests on. +- *Data Mesh* by Zhamak Dehghani: the four principles this closing page reorganizes the reference by. +- *Cloud Native Patterns* by Cornelia Davis: the platform patterns (self-serve, federated governance) behind this summary's implementation pieces. +- *Kubernetes Patterns* by Bilgin Ibryam & Roland Huß: the operator and CRD shapes that make each principle self-serve. +- *GitOps and Kubernetes* by Billy Yuen et al.: reproducing the whole mesh from a Git repository, the practice Principle 03's GitOps piece rests on. diff --git a/_docs/11-running-on-openshift-crc.md b/_docs/11-running-on-openshift-crc.md index 0839d90..e05d545 100644 --- a/_docs/11-running-on-openshift-crc.md +++ b/_docs/11-running-on-openshift-crc.md @@ -9,7 +9,7 @@ duration: 40 min By the end of this chapter you can: -- Explain which Kubernetes assumptions break on OpenShift — SCC/UID handling, ingress, the image registry — and how this chart's design responds to each. +- Explain which Kubernetes assumptions break on OpenShift (SCC/UID handling, ingress, the image registry) and how this chart's design responds to each. - Size, set up, and start OpenShift Local (CRC), then build and push the seven service images to its integrated registry. - Deploy the Helm chart, verify the live core through its two Routes, and interpret the expected app-pod crash-loop-then-converge startup pattern. - Bring up the optional platform tier (service mesh, autoscaling, observability, orchestration, catalog) and read what's actually verified live for each layer. @@ -18,43 +18,43 @@ By the end of this chapter you can: ## Why this matters [Chapter 3]({{ '/docs/02-kubernetes-substrate/' | relative_url }}) built the case for -Kubernetes as the mesh's substrate and brought the final project up on minikube via Helm — +Kubernetes as the mesh's substrate and brought the final project up on minikube via Helm: NodePorts published to loopback, a local image registry. That's a real deployment, and everything in the reading set up to the [summary]({{ '/docs/10-summary/' | relative_url }}) assumes it. But "deploy it to Kubernetes" and "deploy it to OpenShift" are not the same -sentence. OpenShift adds opinions on top of plain Kubernetes — Security Context +sentence. OpenShift adds opinions on top of plain Kubernetes: Security Context Constraints that reject a pod demanding a specific UID, Routes instead of hand-rolled -ingress, an integrated image registry with no external account — and those opinions are +ingress, an integrated image registry with no external account. Those opinions are exactly what a chart has to be built around if it's going to run there at all. -This appendix runs the same seven-service system — the GraphQL gateway, the six -data-product services, Postgres, Kafka, and Apicurio — on **OpenShift Local (CRC)**, a +This appendix runs the same seven-service system (the GraphQL gateway, the six +data-product services, Postgres, Kafka, and Apicurio) on **OpenShift Local (CRC)**, a real single-node OpenShift cluster you run on a laptop. The implementation deck at `presentation/data-mesh-openshift/` already speaks OpenShift's vocabulary in its -diagrams — *Project* for namespace, *SCCs* for pod security, *OperatorHub/OLM* for the -operator install path, *AMQ Streams* for Strimzi, *CNPG* for CloudNativePG — because the -deck was written to pair this reference with OpenShift specifically. This appendix is +diagrams: *Project* for namespace, *SCCs* for pod security, *OperatorHub/OLM* for the +operator install path, *AMQ Streams* for Strimzi, *CNPG* for CloudNativePG. The +deck was written to pair this reference with OpenShift specifically, and this appendix is what makes that pairing runnable: a Helm chart under [`examples/lgtm-datamesh/openshift/helm/datamesh/`](https://github.com/patterncatalyst/datamesh-reference-arch-python/tree/main/examples/lgtm-datamesh/openshift/helm/datamesh), built to the three constraints OpenShift imposes, plus the build/push/deploy sequence that gets it running. One scoping note up front, visible in the chart's own `Chart.yaml`: this is the **live -core** of the final project — the seven services, Postgres, Kafka, and Apicurio — not the +core** of the final project (the seven services, Postgres, Kafka, and Apicurio), not the full minikube stack. OpenMetadata, Prefect, the LGTM observability stack, KEDA autoscaling, and the Istio mesh are out of this chart *by default*. A later section of this appendix, "The full platform tier, live," brings all five of -those layers up on top of this same core as an **opt-in** addition — applied via +those layers up on top of this same core as an **opt-in** addition, applied via `openshift/platform/install-platform.sh` plus two chart flags (`mesh.enabled`/`observability.otlp.enabled`) rather than baked into a plain `helm -install` — and reports what's actually verified live for each, rather than just +install`, and reports what's actually verified live for each, rather than just documenting the OpenShift-native counterpart. -![The chart's topology on CRC — app pods under restricted-v2, infra pods under nonroot-v2, the integrated registry, and two edge-TLS Routes]({{ '/assets/diagrams/18-crc-openshift-topology.svg' | relative_url }}) +![The chart's topology on CRC: app pods under restricted-v2, infra pods under nonroot-v2, the integrated registry, and two edge-TLS Routes]({{ '/assets/diagrams/18-crc-openshift-topology.svg' | relative_url }}) ## Prerequisites -You don't need a paid subscription or a cloud account. **OpenShift Local** (CRC — +You don't need a paid subscription or a cloud account. **OpenShift Local** (CRC, "CodeReady Containers") runs a real, single-node OpenShift cluster in a local VM. This appendix uses podman (and `oc`) on the host to build and push to the integrated registry; the minikube path uses Docker Engine. Podman is needed only here. @@ -69,11 +69,11 @@ This appendix uses podman (and `oc`) on the host to build and push to the integr 3. **Install `crc`.** Extract the archive and put the `crc` binary on your `PATH`. This appendix was authored against **CRC 2.64.0**, which bundles **OpenShift 4.22.14**. 4. **Size the host.** CRC's defaults are not enough for seven Python services plus - Postgres, Kafka, and Apicurio plus OpenShift's own control plane — the infra pods will + Postgres, Kafka, and Apicurio plus OpenShift's own control plane. The infra pods will sit `Pending` with `Insufficient memory` on the defaults, and the default disk will run into `DiskPressure` once the nine images in this appendix are pushed to the integrated registry (see "What broke on live apply" below). Give it more of all three, while the - instance is still stopped — CRC can only grow disk size before the first `crc start`: + instance is still stopped: CRC can only grow disk size before the first `crc start`: ```sh crc config set memory 32768 # 32 GB @@ -81,17 +81,17 @@ This appendix uses podman (and `oc`) on the host to build and push to the integr crc config set disk-size 100 # 100 GB ``` - This is the sizing the **full platform tier** actually needs — Istio sidecars on + This is the sizing the **full platform tier** actually needs: Istio sidecars on every app pod, the otel-lgtm observability backend, KEDA's operands, and Prefect's and OpenMetadata's dependencies, all on top of the live core (see "The full platform tier, live" below, and DRA-008 in `_plans/decisions.md`). If you only want the **live - core** — the seven services, Postgres, Kafka, and Apicurio, with none of the platform - tier applied — the smaller footprint this chart originally shipped with still works: + core** (the seven services, Postgres, Kafka, and Apicurio, with none of the platform + tier applied), the smaller footprint this chart originally shipped with still works: ```sh - crc config set memory 20480 # 20 GB — core only + crc config set memory 20480 # 20 GB, core only crc config set cpus 8 - crc config set disk-size 80 # 80 GB — the default ~32 GB fills once 9 images are pushed + crc config set disk-size 80 # 80 GB; the default ~32 GB fills once 9 images are pushed ``` 5. **Set it up and start it.** @@ -103,7 +103,7 @@ This appendix uses podman (and `oc`) on the host to build and push to the integr `crc setup` needs root for a couple of steps (a small setuid helper, the libvirt network), so run it where `sudo` can prompt you. The first `crc start` provisions the - VM and waits for the cluster to settle — budget 10–15 minutes. When it finishes it + VM and waits for the cluster to settle: budget 10-15 minutes. When it finishes it prints the `kubeadmin` console URL and credentials. 6. **Get `oc` and log in.** CRC ships a version-matched `oc`; `eval $(crc oc-env)` puts @@ -115,9 +115,9 @@ This appendix uses podman (and `oc`) on the host to build and push to the integr ``` Right after `crc start` returns, the API server can briefly reset connections while - its operators roll out — if `oc login` fails with `EOF`, wait a minute and retry. + its operators roll out. If `oc login` fails with `EOF`, wait a minute and retry. -7. **Install `helm`** (v3 or later). The chart has no other external dependencies — no +7. **Install `helm`** (v3 or later). The chart has no other external dependencies: no operators are assumed to be pre-installed on the target cluster. ## What changes when the target is OpenShift @@ -134,20 +134,20 @@ default, `restricted-v2`, *assigns* each pod a high UID from a per-namespace ran OpenShift. The seven Python services are built for exactly this. Their `services//Containerfile` -images already declare `USER 1001:0` — an arbitrary UID in group `0`, with the -application's files group-readable and group-writable — which is precisely what +images already declare `USER 1001:0`, an arbitrary UID in group `0`, with the +application's files group-readable and group-writable. That's precisely what `restricted-v2` expects: it doesn't care *which* UID a pod runs as, only that the pod isn't demanding one it assigned instead. So the fix for the app Deployments is to **drop `runAsUser` entirely** and let OpenShift pick. `templates/app-deployment.yaml` sets -`runAsNonRoot: true` and nothing else UID-related — no `runAsUser` key appears anywhere +`runAsNonRoot: true` and nothing else UID-related. No `runAsUser` key appears anywhere in the pod's `securityContext`. Whatever UID `restricted-v2` assigns, the image already works under it. Postgres and Kafka are the exception, and the chart treats them as one. Their upstream -images hardcode specific non-root UIDs that own their data directories — Postgres as -`70`, the Apache Kafka image as `1000` — so dropping `runAsUser` for them would be wrong: +images hardcode specific non-root UIDs that own their data directories, Postgres as +`70`, the Apache Kafka image as `1000`, so dropping `runAsUser` for them would be wrong: whatever UID `restricted-v2` assigned instead wouldn't own `/var/lib/postgresql/data` or `/var/lib/kafka/data`, and the containers would fail to start. @@ -161,8 +161,8 @@ respectively, pinned explicitly, now legally. The binding is the one piece of OpenShift RBAC worth internalizing on its own: OpenShift auto-generates a ClusterRole named `system:openshift:scc:` for every SCC on the cluster, whose single permission is "use this SCC." Binding a ServiceAccount to that -ClusterRole via a RoleBinding is what grants the SCC — there's no separate "SCC -assignment" API, it's ordinary RBAC pointed at a cluster-generated role: +ClusterRole via a RoleBinding is what grants the SCC. There's no separate "SCC +assignment" API; it's ordinary RBAC pointed at a cluster-generated role: {% raw %} ```yaml @@ -183,7 +183,7 @@ subjects: The CLI shorthand for the same grant is `oc adm policy add-scc-to-user nonroot-v2 -z datamesh-infra`, which is worth knowing when debugging a pod stuck in -`CreateContainerConfigError` with an SCC-related event — it's the same RoleBinding under +`CreateContainerConfigError` with an SCC-related event. It's the same RoleBinding under the hood. ### 2. Routes, not published NodePorts @@ -192,7 +192,7 @@ The minikube path this final project uses elsewhere in the reading set exposes s NodePorts published to `127.0.0.1` when the profile is created, because minikube has no cluster-native router. OpenShift has one built in: a `Route` object hands a Service a real external hostname, served by the cluster's router, with nothing to publish on the -host. The chart defines two — +host. The chart defines two: `templates/route.yaml` ranges over the same `services:` map and emits a Route for any service with `route: true` set (currently just `graphql-gateway`), and `apicurio.yaml` wires its own Route directly since Apicurio isn't part of that map: @@ -203,7 +203,7 @@ apicurio apicurio-datamesh.apps-crc.testing edge/Redirect ``` `edge/Redirect` means TLS terminates at the router and plain HTTP is redirected to -HTTPS — no certificate handling in the application at all. Every other service +HTTPS, no certificate handling in the application at all. Every other service (inventory-service, order-service, the rest) stays `ClusterIP`-only, reachable in-cluster but with no Route, exactly as it would on minikube without a NodePort. @@ -212,7 +212,7 @@ but with no Route, exactly as it would on minikube without a NodePort. OpenShift runs its own image registry inside the cluster, addressed in-cluster by its Service DNS: `image-registry.openshift-image-registry.svc:5000`. `values.yaml`'s `image.registry` points there by default. No external registry account, and no -Docker-Hub-style push credentials, are required for a local cluster — the only +Docker-Hub-style push credentials, are required for a local cluster. The only credential involved is an `oc whoami -t` token used to `podman login` to the registry's *external* route when you need to push from outside the cluster, which the build step below does. @@ -257,7 +257,7 @@ on `:8080`), whether it exposes a second gRPC port (inventory-service's `:50051` whether it owns a Postgres schema, whether it's a Kafka participant needing a graceful shutdown window, whether it needs an Alembic migration init container ahead of the app container, and its resource tier. `templates/app-deployment.yaml` then ranges over that -map once, and the interesting part is the conditionals built from it — this is where +map once, and the interesting part is the conditionals built from it. This is where "the template is driven by data" pays off: {% raw %} @@ -283,7 +283,7 @@ map once, and the interesting part is the conditionals built from it — this is tcpSocket: { port: http } {{- end }} securityContext: - runAsNonRoot: true # NOTE: no runAsUser — restricted-v2 assigns it + runAsNonRoot: true # NOTE: no runAsUser, restricted-v2 assigns it allowPrivilegeEscalation: false capabilities: { drop: ["ALL"] } seccompProfile: { type: RuntimeDefault } @@ -292,18 +292,18 @@ map once, and the interesting part is the conditionals built from it — this is {% endraw %} Every service in this system ships `/health` as its liveness check and `/healthz` as its -readiness check — all seven are `probe: http` in practice, but the conditional exists +readiness check. All seven are `probe: http` in practice, but the conditional exists because the chart is carried over from a sibling project where some services shipped without a health-check library and needed a bare TCP probe instead. It costs nothing to keep the branch here even though every current service takes the `http` path. `notification-service` adds one more wrinkle: `migrate: true` adds an `initContainer` running `alembic upgrade head` against the same database before the -`uvicorn` container starts, so the schema exists before the app issues a single query — -no `CREATE TABLE` happens on the hot path. +`uvicorn` container starts, so the schema exists before the app issues a single query. +No `CREATE TABLE` happens on the hot path. -The shared environment contract — the Kafka bootstrap address, the Apicurio URL, the -gRPC coordinate for `inventory-service`, the gateway's downstream REST URL — is one +The shared environment contract (the Kafka bootstrap address, the Apicurio URL, the +gRPC coordinate for `inventory-service`, the gateway's downstream REST URL) is one ConfigMap, `templates/configmap-app.yaml`. Every in-cluster address it builds is namespace-relative: @@ -316,22 +316,21 @@ data: ``` {% endraw %} -so the chart installs cleanly into any namespace — `datamesh`, as this appendix does, or -anything else — without a single hardcoded string to edit. +so the chart installs cleanly into any namespace, `datamesh`, as this appendix does, or +anything else, without a single hardcoded string to edit. -Postgres and Kafka are plain StatefulSets, not operator-managed CRs — `templates/postgres.yaml` +Postgres and Kafka are plain StatefulSets, not operator-managed CRs. `templates/postgres.yaml` and `templates/kafka.yaml` don't assume the CloudNativePG or Strimzi operators are installed on the target cluster, since this is meant to run on a bare CRC instance with nothing pre-installed beyond OpenShift itself. Kafka runs single-broker KRaft (no ZooKeeper), self-referencing its own pod-stable DNS (`kafka-0.kafka..svc...:9093`) as -its quorum voter, with `publishNotReadyAddresses: true` on its headless Service — a +its quorum voter, with `publishNotReadyAddresses: true` on its headless Service. A single-node KRaft broker has to resolve its own controller address to become ready in the first place, so the Service has to advertise it before it's marked ready. -Both -StatefulSets run under `datamesh-infra` with `fsGroup` set to match their UID, so their +Both StatefulSets run under `datamesh-infra` with `fsGroup` set to match their UID, so their PersistentVolumeClaims are writable from the first mount. ## Building and pushing the images @@ -351,13 +350,13 @@ oc new-project datamesh ``` The script loops `podman build` over each `services//Containerfile`, then tags and -pushes to `$REG/datamesh/:v1` — exactly the reference `values.yaml`'s +pushes to `$REG/datamesh/:v1`, exactly the reference `values.yaml`'s `image.registry`/`image.tag` compose, so a plain `helm upgrade --install` finds the images with no further wiring. Pushing auto-creates an ImageStream per image in the namespace, which is how the internal registry tracks what it's holding. **One real snag worth planning for.** The CRC VM's outbound networking does not -necessarily reach Docker Hub — a direct `podman pull docker.io/library/postgres:18.6-alpine` +necessarily reach Docker Hub. A direct `podman pull docker.io/library/postgres:18.6-alpine` run *inside* the cluster's image-pull path can fail with something like `dial tcp registry-1.docker.io:443: i/o timeout`, which shows up as `postgres-0`/`kafka-0` stuck in `ImagePullBackOff` after `helm install`. The *host* running `podman`/`oc`, by contrast, @@ -371,7 +370,7 @@ directly: which pulls `docker.io/library/postgres:18.6-alpine` and `docker.io/apache/kafka:4.3.1` on the host, retags them into `$REG/datamesh/postgres:18.6-alpine` and -`$REG/datamesh/kafka:4.3.1`, and pushes — exactly the refs `values.yaml`'s +`$REG/datamesh/kafka:4.3.1`, and pushes, exactly the refs `values.yaml`'s `postgres.image`/`kafka.image` default to. On a cluster whose CRC VM *does* have Docker Hub egress, skip `--mirror-infra` and uncomment the commented upstream `docker.io/...` lines in `values.yaml` instead; both forms are left in the file, one active. @@ -405,22 +404,22 @@ shipping-service-c8c94685b-vzs8l 1/1 Running 3 (2m58s ago) 12m ``` That's the live run, twelve minutes after `helm upgrade --install`: all ten workloads -`1/1`. The 2–3 restarts on `inventory-service`, `payment-service`, `review-service`, and -`shipping-service` are exactly the convergence described above, not a defect — -those four happened to lose the race against `postgres-0` reaching `Ready` on this +`1/1`. The 2-3 restarts on `inventory-service`, `payment-service`, `review-service`, and +`shipping-service` are exactly the convergence described above, not a defect. +Those four happened to lose the race against `postgres-0` reaching `Ready` on this particular run; `graphql-gateway`, `order-service`, and `notification-service` happened to win it. Re-running the same `helm install` against a fresh `crc start` will likely -distribute the restarts across a different subset of services — which four crash-loop +distribute the restarts across a different subset of services. Which four crash-loop is a function of scheduling order and Postgres's own startup time, not something the chart controls or needs to. Expect the app pods to **crash-loop before the infrastructure is ready**, and expect that to be fine. `notification-service`'s migrate init container, and every service's database connection on first boot, depend on Postgres and Kafka already being reachable. -A pod that starts before its StatefulSet dependency is `Ready` fails fast — a DNS lookup -with no ready endpoints behind it, then connection refused — and restarts on the normal +A pod that starts before its StatefulSet dependency is `Ready` fails fast: a DNS lookup +with no ready endpoints behind it, then connection refused, and restarts on the normal back-off schedule. As `postgres-0` and `kafka-0` reach `1/1`, each service recovers on @@ -430,11 +429,11 @@ of how many restarts it took to get there. (If you want a clean start with no restart churn, install the infra StatefulSets' templates first and wait for them before letting -the app Deployments roll out — but the self-healing path above is what a plain `helm +the app Deployments roll out, but the self-healing path above is what a plain `helm install` does, and it converges on its own.) If you skipped the memory bump in the prerequisites, the infra pods sit `Pending` -instead — `oc describe pod postgres-0` reports `Insufficient memory`, and `oc describe +instead. `oc describe pod postgres-0` reports `Insufficient memory`, and `oc describe node`'s "Allocated resources" section shows memory requests near 100%. Stop the cluster, raise `crc config set memory`, and `crc start` again; the pods schedule once there's room. @@ -452,10 +451,10 @@ $ curl -sk https://graphql-gateway-datamesh.apps-crc.testing/healthz {"status":"ready","service":"graphql-gateway"} ``` -And a real query through the gateway spanning two or more of the underlying products — +Run a real query through the gateway spanning two or more of the underlying products: the point of a GraphQL gateway is composing reads across services the caller never talks to directly. So the query worth running is one that can only succeed if that composition -actually works — for example, an order placed through `order-service` with its live +actually works. For example, an order placed through `order-service` with its live stock level attached from `inventory-service`: ```sh @@ -468,11 +467,11 @@ curl -sk https://$(oc get route graphql-gateway -n datamesh -o jsonpath='{.spec. {"data": {"order": {"id": "6c5a7fb2-a9d6-44dd-bbe1-eef95c5bdb5e", "customerId": "cust-openshift", "itemSku": "WIDGET-001", "quantity": 2, "status": "placed", "amount": "19.98", "stock": {"sku": "WIDGET-001", "quantityOnHand": 50, "available": true}}}} ``` -The top-level `order` fields — `customerId`, `itemSku`, `quantity`, `status`, `amount` — +The top-level `order` fields (`customerId`, `itemSku`, `quantity`, `status`, `amount`) come straight from `order-service`'s own Postgres row. The nested `stock` object is a second, independent hop the gateway makes over gRPC to `inventory-service`'s `:50051` port to resolve `quantityOnHand`/`available` for that same SKU. A 200 with both halves -populated — the order half and the `stock` half — confirms the Route, the SCC-assigned +populated, the order half and the `stock` half, confirms the Route, the SCC-assigned UID, the ConfigMap wiring (`INVENTORY_GRPC_ADDR`), and the gateway's REST-plus-gRPC fan-out to two independent services are all working together, not just that one pod answers its own health check. @@ -488,31 +487,31 @@ $ curl notification-service:8080/received [{"order_id": "6c5a7fb2-a9d6-44dd-bbe1-eef95c5bdb5e", "event_type": "order.placed", "customer_id": "cust-openshift", "item_sku": "WIDGET-001", "quantity": 2, "amount": "19.98", "status": "OrderStatus.placed", "created_at": "2026-10-07T16:20:12.074293+00:00"}] ``` -Same order ID, same SKU, same quantity — one `order.placed` write landed through two -independent consumers of the mesh: the GraphQL gateway's synchronous read path above, +Same order ID, same SKU, same quantity: one `order.placed` write landed through two +independent consumers of the mesh, the GraphQL gateway's synchronous read path above, and Kafka's asynchronous one. Together the two checks exercise everything the live-core -chart wires up — Route → REST → gRPC on one side, Route-independent Kafka -produce/consume with schema-registry-backed Avro on the other — independent of the +chart wires up, Route → REST → gRPC on one side, Route-independent Kafka +produce/consume with schema-registry-backed Avro on the other, independent of the platform tier described below. ### What broke on live apply -Two things went wrong on the actual run against CRC, neither in the chart's design — +Two things went wrong on the actual run against CRC, neither in the chart's design, both in the tooling around it. One thing did *not* break: the SCC assignment this appendix spent most of its length justifying. **1. All ten pods stuck `Pending` behind a `disk-pressure` taint.** Partway through `helm upgrade --install`, every pod sat `Pending` with a `node.kubernetes.io/disk-pressure:NoSchedule` taint and no obvious resource request -problem — `oc describe node` wasn't short on memory or CPU. The cause was the registry +problem. `oc describe node` wasn't short on memory or CPU. The cause was the registry push that happens just before deployment: `build-and-push.sh` had already pushed all nine images (seven app images plus the mirrored `postgres`/`kafka` infra images) into the integrated registry, and that registry's storage lives on the CRC node's own disk. CRC's *default* VM disk is a modest ~32 GB, and nine images pushed into it was enough to cross the kubelet's disk-pressure eviction threshold (27 GB used on a 32 GB disk). -Once a node is under `DiskPressure`, the scheduler taints it and nothing new schedules — -not a Kubernetes misconfiguration, just the VM running out of room for what this chart +Once a node is under `DiskPressure`, the scheduler taints it and nothing new schedules. +That's not a Kubernetes misconfiguration, just the VM running out of room for what this chart actually ships. The fix is the same host-sizing move as the memory and CPU bump in the prerequisites, just for disk: @@ -523,7 +522,7 @@ crc start --pull-secret-file ~/Downloads/pull-secret.txt ``` After the restart the taint cleared and all ten pods scheduled and converged to `1/1` -on their own — this appendix's prerequisites section above already reflects the `80` +on their own. This appendix's prerequisites section above already reflects the `80` value for exactly this reason; a reader following it from a stopped state should never hit the taint in the first place. (This is also why `crc config set disk-size` is worth setting *before* the first `crc start` rather than discovering it mid-deploy: CRC disk @@ -532,7 +531,7 @@ size can only be grown while the instance is stopped.) **2. `build-and-push.sh` exited `2` after every image had already pushed.** The script's final step prints a one-line hint for the `--mirror-infra` flag described earlier in this appendix. That hint line is passed to `printf` with a leading `--mirror-infra ...` -string, which `printf` parses as an option flag rather than a format string — an exit +string, which `printf` parses as an option flag rather than a format string. That produces an exit code `2` with no images actually affected, since the build-and-push loop itself had already completed and returned before the hint line ran. @@ -543,9 +542,9 @@ exits non-zero, the registry already has all nine images, which `oc get istag -n datamesh` confirms independent of the script's own exit status. **What didn't break: the SCC assignment.** The entire premise of the "what changes on -OpenShift" section above — drop `runAsUser` for the seven Python app images and let +OpenShift" section above (drop `runAsUser` for the seven Python app images and let `restricted-v2` assign a UID, pin `runAsUser` for Postgres and Kafka under a -`datamesh-infra` ServiceAccount bound to `nonroot-v2` — held exactly as designed on the +`datamesh-infra` ServiceAccount bound to `nonroot-v2`) held exactly as designed on the first deploy, no iteration needed. The live SCC/UID table earlier in this appendix is @@ -553,26 +552,26 @@ the proof: all seven app pods landed on `restricted-v2` with the same OpenShift- UID, `1000650000` (the images' `USER 1001:0` made them agnostic to whichever UID that turned out to be), while `postgres-0` kept its required `70` and `kafka-0` its required `1000`, both legally, both under `nonroot-v2` via the `datamesh-infra` RoleBinding. The -two breakages above were the registry-storage footprint and a shell quoting bug — not +two breakages above were the registry-storage footprint and a shell quoting bug, not the admission model this appendix is actually about. ## The full platform tier, live -The live core above is the Helm chart's whole job — but this appendix also brings up +The live core above is the Helm chart's whole job, but this appendix also brings up the five layers that chart leaves out, as an **opt-in platform tier** on top of that same core. None of this changes the core's default behavior: a plain `helm upgrade --install datamesh openshift/helm/datamesh` still renders exactly the chart -described above, with no mesh labels and no OTEL environment variables anywhere in it — -both `mesh.enabled` and `observability.otlp.enabled` default to `false` in +described above, with no mesh labels and no OTEL environment variables anywhere in it. +Both `mesh.enabled` and `observability.otlp.enabled` default to `false` in `values.yaml`. The platform tier only appears once you additionally run [`openshift/platform/install-platform.sh`](https://github.com/patterncatalyst/datamesh-reference-arch-python/blob/main/examples/lgtm-datamesh/openshift/platform/install-platform.sh) and flip those two flags on (`helm upgrade --set mesh.enabled=true --set observability.otlp.enabled=true`). Four of the five layers below are verified live against the resized CRC instance from -the prerequisites. The fifth, OpenMetadata, is best-effort and partial — read that +the prerequisites. The fifth, OpenMetadata, is best-effort and partial; read that subsection carefully before treating it as done. The full capture for each layer is committed under [`examples/lgtm-datamesh/openshift/evidence/`](https://github.com/patterncatalyst/datamesh-reference-arch-python/tree/main/examples/lgtm-datamesh/openshift/evidence/). @@ -585,7 +584,7 @@ directly on minikube, labeling the whole namespace for injection. OpenShift's me `openshift/platform/mesh/istio.yaml` and `istio-cni.yaml` install it as an `Istio` CR plus a matching `IstioCNI` CR, not a raw `istioctl install`. Injection is a pod-template **LABEL**, `istio.io/rev`, set on the seven app Deployments (and the -order-service canary) behind the chart's `mesh.enabled` flag — Sail's revisioned +order-service canary) behind the chart's `mesh.enabled` flag. Sail's revisioned injection webhook doesn't reliably fire on the classic `sidecar.istio.io/inject` *annotation* the minikube overlay uses. @@ -593,15 +592,14 @@ Postgres, Kafka, and Apicurio stay unlabeled, unmeshed infrastructure, which is exactly what the live sidecar counts show: the seven app pods carry `2/2` (app container plus Envoy), the three infra pods stay `1/1`. -Namespace-wide mTLS is `STRICT` — a `PeerAuthentication` named `default` — with one +Namespace-wide mTLS is `STRICT` (a `PeerAuthentication` named `default`), with one scoped exception: a second `PeerAuthentication`, `graphql-gateway-permissive`, sets `PERMISSIVE` for just the gateway. The OpenShift Route fronting the gateway is not mesh-aware and arrives as plaintext, and `STRICT` namespace-wide rejected that edge traffic outright (502s from the Route) before the gateway-scoped exception was carved -out — the live lesson behind this appendix's newest decision entry. +out. That's the live lesson behind this appendix's newest decision entry. -Every other hop in -the mesh — service to service, all mesh-internal — stays `STRICT`. +Every other hop in the mesh, service to service, all mesh-internal, stays `STRICT`. The v1/v2 canary itself runs as a **mesh-internal** `VirtualService`/`DestinationRule` rather than through an Istio ingress gateway: OSSM3 provisions no default ingress @@ -625,8 +623,8 @@ The full scale cycle was observed live: idle at zero replicas (`minReplicaCount: a burst of 12 orders drove the `ScaledObject` to `Active=True` and scaled 0→1 within ~15 seconds → the backlog drained, `Active` returned to `False` → a roughly 90-second cooldown scaled it back 1→0. The KEDA HTTP add-on that scales `graphql-gateway` on -minikube is **not** part of the Custom Metrics Autoscaler — CMA ships core KEDA only — -so gateway HTTP-request scale-to-zero was dropped on OpenShift; there's no drop-in CMA +minikube is **not** part of the Custom Metrics Autoscaler; CMA ships core KEDA only, +so gateway HTTP-request scale-to-zero was dropped on OpenShift. There's no drop-in CMA equivalent for it. (Evidence: `examples/lgtm-datamesh/openshift/evidence/keda-verification.txt`.) @@ -636,24 +634,24 @@ equivalent for it. [Chapter 8]({{ '/docs/08-observability/' | relative_url }}) stands up the full split Loki/Grafana/Tempo/Mimir stack on minikube. This appendix instead deploys the single-container `grafana/otel-lgtm` all-in-one image -(`openshift/platform/observability/lgtm-deployment.yaml`) — Collector, Tempo, Loki, -Mimir, and Grafana in one Deployment — and re-enables OTLP export from the services via +(`openshift/platform/observability/lgtm-deployment.yaml`), Collector, Tempo, Loki, +Mimir, and Grafana in one Deployment, and re-enables OTLP export from the services via the chart's `observability.otlp.enabled` flag, scoped to traces only (`OTEL_METRICS_EXPORTER=none` keeps metrics load off a single-replica backend). The real lesson here was admission, not the image. The otel-lgtm image owns its data directories as UID 0, so its pod needs `runAsUser: 0`, which forces it onto the -`anyuid` SCC via a dedicated `lgtm` ServiceAccount — `restricted-v2` rejects root +`anyuid` SCC via a dedicated `lgtm` ServiceAccount: `restricted-v2` rejects root outright. The non-obvious part: adding a `seccompProfile` or `capabilities` block to -that same `securityContext` — the kind of change that reads as pure hardening — makes +that same `securityContext`, the kind of change that reads as pure hardening, makes `anyuid` **reject** the pod instead of admitting it, because its allowed seccomp-profile list is empty. That rejection doesn't surface as a loud SCC-denial -event; it silently drops the pod back onto `restricted-v2`'s randomly assigned UID, +event. It silently drops the pod back onto `restricted-v2`'s randomly assigned UID, where Grafana can't write `/data` and the pod hangs forever at "Waiting for Grafana to -start up...", with nothing pointing at the real cause. `runAsUser: 0` alone — no -`seccompProfile`, no `capabilities` — is what `anyuid` actually wants. +start up...", with nothing pointing at the real cause. `runAsUser: 0` alone, no +`seccompProfile`, no `capabilities`, is what `anyuid` actually wants. Live, Grafana reports healthy through its Route (`"database": "ok"`), and Tempo is ingesting per-service traces: `graphql-gateway`, `order-service`, `inventory-service`, @@ -663,7 +661,7 @@ and `notification-service` each produced recent traces during the capture window **Limit:** those are per-service traces, not one stitched cross-service trace. The gateway's downstream HTTP/gRPC calls don't propagate a W3C `traceparent` header, so each service roots its own trace instead of continuing the -caller's — an application-instrumentation gap, not something OpenShift or the mesh +caller's. That's an application-instrumentation gap, not something OpenShift or the mesh fails to do. Envoy's own mesh spans hit the same ceiling: they can't correlate across hops without the application propagating that header either. @@ -673,8 +671,8 @@ hops without the application propagating that header either. Prefect's server and worker (`openshift/platform/prefect/server-deployment.yaml`, `worker-deployment.yaml`) reuse the core `postgres` StatefulSet via a dedicated -`prefect` database and login role, rather than bundling a second Postgres instance — -this chart's established one-Postgres-per-cluster convention. +`prefect` database and login role, rather than bundling a second Postgres instance, +following this chart's established one-Postgres-per-cluster convention. Live, the server's `/api/health` returned `true` through its Route, and the example two-task flow (`datamesh-example-flow`) ran end to end against the server API: both the @@ -683,11 +681,11 @@ two-task flow (`datamesh-example-flow`) ran end to end against the server API: b (Evidence: `examples/lgtm-datamesh/openshift/evidence/prefect-verification.txt`.) -### Catalog: OpenMetadata — best-effort, partial +### Catalog: OpenMetadata, best-effort, partial [Chapter 4]({{ '/docs/04-contracts-and-catalog/' | relative_url }}) uses OpenMetadata on minikube. The OpenShift port is the same tool, retargeted to this chart's Postgres for -its backend database and a single-node OpenSearch for search — and it is the one layer +its backend database and a single-node OpenSearch for search, and it is the one layer in this platform tier that is **not** fully verified live. **What's live:** OpenSearch came up `green` (single node, confirmed against its own @@ -698,14 +696,14 @@ fsGroup admission gate was the hard part of this whole layer, and it's solved: t Secrets/SCC wiring the OM server needs is in place. **What's blocked, and why it isn't a deployment defect:** the OpenMetadata server image -(`docker.getcollate.io/openmetadata/server:1.12.8`) fails to pull — -`ImagePullBackOff`, `toomanyrequests: unauthenticated pull rate limit` — against Docker +(`docker.getcollate.io/openmetadata/server:1.12.8`) fails to pull: +`ImagePullBackOff`, `toomanyrequests: unauthenticated pull rate limit`, against Docker Hub, from both the CRC VM and the host directly. That's an **external quota**, not a -problem with the chart, the database wiring, or the SCC grant: the two unblock paths are +problem with the chart, the database wiring, or the SCC grant. The two unblock paths are authenticating to Docker Hub (`podman login docker.io`) before mirroring the image into the internal registry, or waiting out the rate-limit reset window (roughly six hours); neither was exercised for this verification pass. The distinction matters if you -reproduce this yourself — the hard problem (SCC admission for a fixed-UID, +reproduce this yourself: the hard problem (SCC admission for a fixed-UID, fsGroup-dependent dependency) is solved; an unrelated external rate limit is what's still standing between this layer and a running OM server. @@ -715,7 +713,7 @@ still standing between this layer and a running OM server. Everything above is imperative: you ran `helm upgrade --install` by hand. The OpenShift-native way to *keep* a cluster matching the chart is **OpenShift GitOps** -(Argo CD) — an `Application` object points at the chart in this repo, and Argo +(Argo CD): an `Application` object points at the chart in this repo, and Argo continuously reconciles the cluster to it, pruning drift and self-healing. A starter manifest is committed at [`openshift/gitops/application.yaml`](https://github.com/patterncatalyst/datamesh-reference-arch-python/blob/main/examples/lgtm-datamesh/openshift/gitops/application.yaml): @@ -744,7 +742,7 @@ Its symmetric partner on the build side is **OpenShift Pipelines** (Tekton): the `podman build`/`push` loop in `build-and-push.sh` becomes a `Pipeline` triggered on push, so a commit to a service's source produces a new image in the internal registry without a human running the script. Both are left as starting points rather -than part of the verified deploy in this appendix — each needs its operator installed +than part of the verified deploy in this appendix: each needs its operator installed (OpenShift GitOps, OpenShift Pipelines), and GitOps additionally needs this repo reachable from the cluster. Neither was applied for this appendix's verification pass. @@ -756,8 +754,8 @@ sizings in the prerequisites' "Size the host" step. **Live core** (20 GB RAM / 8 vCPU / 80 GB disk; `helm upgrade --install datamesh openshift/helm/datamesh --namespace datamesh`, both platform flags left at their -`false` default). All ten workloads — the seven application Deployments, Apicurio, and -the Postgres and Kafka StatefulSets — reached `1/1 Running` in Project `datamesh`. SCC +`false` default). All ten workloads, the seven application Deployments, Apicurio, and +the Postgres and Kafka StatefulSets, reached `1/1 Running` in Project `datamesh`. SCC behavior was confirmed by reading each pod's effective SCC and runtime UID: the seven app pods under `restricted-v2` with OpenShift-assigned UID `1000650000`, `postgres-0` under `nonroot-v2` with UID `70`, and `kafka-0` under `nonroot-v2` with UID `1000`, both @@ -765,7 +763,7 @@ infra pods running under the `datamesh-infra` ServiceAccount. The GraphQL gatewa Route returned `{"status":"ready","service":"graphql-gateway"}` from `/healthz`, and a cross-service `/graphql` query through that same Route returned a populated `order` object from `order-service` joined to a populated `stock` object resolved over gRPC -from `inventory-service` — the Route, the SCC-assigned UID, the ConfigMap wiring, and +from `inventory-service`: the Route, the SCC-assigned UID, the ConfigMap wiring, and the gateway's REST-plus-gRPC fan-out, all exercised together. The Kafka path was confirmed independently: the same order's `order.placed` event, Avro-encoded against the Apicurio schema registry, was observed decoded on `notification-service`'s own @@ -776,30 +774,30 @@ endpoint. The full capture is committed at `openshift/platform/install-platform.sh` and `mesh.enabled=true` / `observability.otlp.enabled=true`): -- **Service mesh** — OSSM3/Sail, Istio v1.30: the seven app pods report `2/2` (app +- **Service mesh**: OSSM3/Sail, Istio v1.30: the seven app pods report `2/2` (app container plus Envoy sidecar) against the three infra pods' `1/1`; namespace-wide `PeerAuthentication` `STRICT` with a `graphql-gateway-permissive` `PERMISSIVE` exception so the non-mesh OpenShift Route still reaches the edge; the order-service v1/v2 canary (mesh-internal `VirtualService`, no ingress gateway) split 26/4 over 30 requests against a 90/10 configured weight. (`examples/lgtm-datamesh/openshift/evidence/mesh-verification.txt`) -- **Autoscaling** — Custom Metrics Autoscaler: the `notification-service-scaler` - `ScaledObject` demonstrated the full cycle live — idle at zero replicas, scaled 0→1 +- **Autoscaling**: Custom Metrics Autoscaler: the `notification-service-scaler` + `ScaledObject` demonstrated the full cycle live: idle at zero replicas, scaled 0→1 within ~15s of a 12-order burst raising Kafka consumer lag, back to 0 after a ~90s cooldown once the backlog cleared. (`examples/lgtm-datamesh/openshift/evidence/keda-verification.txt`) -- **Observability** — grafana/otel-lgtm all-in-one under the `anyuid` SCC, root with no +- **Observability**: grafana/otel-lgtm all-in-one under the `anyuid` SCC, root with no seccomp profile or capabilities (see DRA-016): Grafana reports `"database": "ok"` through its Route, and Tempo is ingesting per-service traces from `graphql-gateway`, `order-service`, `inventory-service`, and `notification-service`. (`examples/lgtm-datamesh/openshift/evidence/observability-verification.txt`) -- **Orchestration** — Prefect 3.x server and worker on the core Postgres: `/api/health` +- **Orchestration**: Prefect 3.x server and worker on the core Postgres: `/api/health` returned `true` through its Route, and the example flow run (`zircon-waxbill`, `datamesh-example-flow`) reached `COMPLETED`. (`examples/lgtm-datamesh/openshift/evidence/prefect-verification.txt`) -- **Catalog (OpenMetadata) — partial/best-effort, not fully verified:** OpenSearch - reached `green` under a dedicated `anyuid`-bound ServiceAccount — the fixed-UID/ - fsGroup SCC gate, the hard part, is solved — while the OpenMetadata server itself is +- **Catalog (OpenMetadata), partial/best-effort, not fully verified:** OpenSearch + reached `green` under a dedicated `anyuid`-bound ServiceAccount (the fixed-UID/ + fsGroup SCC gate, the hard part, is solved), while the OpenMetadata server itself is blocked by an external Docker Hub unauthenticated pull-rate limit (`toomanyrequests`), not a deployment defect: the chart, database wiring, and SCC grant are all in place and simply unproven because the server image never pulled. @@ -808,7 +806,7 @@ endpoint. The full capture is committed at **Limits carried forward:** - Observability's per-service traces are **not** stitched into one cross-service - trace — the services don't propagate a W3C `traceparent` header across their + trace. The services don't propagate a W3C `traceparent` header across their HTTP/gRPC calls, an application-instrumentation gap, not an OpenShift or mesh limitation. - OpenMetadata's server component remains unverified live, blocked on an external @@ -824,21 +822,21 @@ endpoint. The full capture is committed at Metrics Autoscaler) and the OSSM3 Istio version can't be resolved without a cluster; they are re-checked at the next CRC run. - **OpenShift GitOps** (the `Application` at `openshift/gitops/application.yaml`) and - **OpenShift Pipelines** (Tekton) remain authored, not applied — no operators for + **OpenShift Pipelines** (Tekton) remain authored, not applied. No operators for either were installed for either verification pass. Re-confirm the live core by re-running `helm upgrade --install` against a fresh `crc -start`, then re-driving the two Route `curl`s and re-reading each pod's SCC annotation — -the assigned app UID will differ per cluster, but the SCC names, the infra UIDs, and the +start`, then re-driving the two Route `curl`s and re-reading each pod's SCC annotation. +The assigned app UID will differ per cluster, but the SCC names, the infra UIDs, and the 200s should not. Re-confirm the platform tier by re-running `install-platform.sh` on top of a resized instance and re-capturing the same evidence files under `examples/lgtm-datamesh/openshift/evidence/`.* ## For deeper coverage -- [OpenShift documentation](https://docs.openshift.com/) — Security Context Constraints, Routes, and the integrated image registry this appendix is built around. -- [OpenShift Local (CRC) documentation](https://crc.dev/) — sizing, setup, and the `crc` CLI. -- [Strimzi documentation](https://strimzi.io/documentation/) — the Kafka-on-Kubernetes operator model that AMQ Streams, referenced throughout this appendix, is built on. -- [Istio documentation](https://istio.io/latest/docs/) — mTLS and the `VirtualService`/`DestinationRule` traffic shaping the platform tier's mesh section uses. -- [KEDA documentation](https://keda.sh/docs/) — the upstream API the Custom Metrics Autoscaler operator packages. -- *Platform Engineering on Kubernetes* by Mauricio Salatino — operator-based platform capabilities (OperatorHub/OLM) and SCCs as policy, the pattern this appendix's chart is built around. +- [OpenShift documentation](https://docs.openshift.com/): Security Context Constraints, Routes, and the integrated image registry this appendix is built around. +- [OpenShift Local (CRC) documentation](https://crc.dev/): sizing, setup, and the `crc` CLI. +- [Strimzi documentation](https://strimzi.io/documentation/): the Kafka-on-Kubernetes operator model that AMQ Streams, referenced throughout this appendix, is built on. +- [Istio documentation](https://istio.io/latest/docs/): mTLS and the `VirtualService`/`DestinationRule` traffic shaping the platform tier's mesh section uses. +- [KEDA documentation](https://keda.sh/docs/): the upstream API the Custom Metrics Autoscaler operator packages. +- *Platform Engineering on Kubernetes* by Mauricio Salatino: operator-based platform capabilities (OperatorHub/OLM) and SCCs as policy, the pattern this appendix's chart is built around.