From e15150cc215f5e740741990cba687f57fb032927 Mon Sep 17 00:00:00 2001 From: PatternCatalyst Date: Thu, 22 Oct 2026 19:00:06 -0400 Subject: [PATCH] docs: add chapter-template sections + readability/voice pass across all chapters Bring the DataMesh Python site up to the API-patterns house chapter template and make it read as follow-along material for engineers, matching the Quarkus pilot (PR #62 on the Quarkus repo). - Add `## Learning objectives` (top), `## Why this matters` (over the existing intro), and `## For deeper coverage` (final section) to all 13 _docs chapters. - Moderate readability restructure: split walls of text, add signposting subheadings so steps read in build order. All code, commands, `{% include %}` figures, captions, and verified output preserved verbatim. - Extra follow-along pass on the minikube substrate (ch02) and the OpenShift/CRC appendix (ch11); commands cross-checked against the real scripts. - Professional-voice pass: cleared all non-"capstone" ban-tier tics (honest/deliberately/worth-noting/load-bearing/for-free/etc.); forbidden-syntax, cross-references, liquid-collisions all clean; jekyll build green. Deeper-coverage citations use real hub reading-list titles and canonical docs only; no invented ISBNs or URLs. The word "capstone" is intentionally left for a separate docs+code rename follow-up. --- _docs/00-index.md | 30 ++++++- _docs/01-concepts.md | 32 +++++++- _docs/01-data-architectures.md | 93 +++++++++++++++------- _docs/02-kubernetes-substrate.md | 62 ++++++++++++--- _docs/03-services-and-data-products.md | 41 ++++++++-- _docs/04-contracts-and-catalog.md | 38 +++++++-- _docs/05-data-planes.md | 42 ++++++++-- _docs/06-progressive-delivery-mtls.md | 52 ++++++++---- _docs/07-elastic-and-resilient.md | 55 +++++++++---- _docs/08-observability.md | 41 ++++++++-- _docs/09-anti-patterns.md | 52 +++++++++--- _docs/10-summary.md | 35 +++++++- _docs/11-running-on-openshift-crc.md | 106 ++++++++++++++++++------- 13 files changed, 531 insertions(+), 148 deletions(-) diff --git a/_docs/00-index.md b/_docs/00-index.md index 2940260..abed3e9 100644 --- a/_docs/00-index.md +++ b/_docs/00-index.md @@ -5,13 +5,27 @@ description: "The map — what this reference builds, the reading order, and a l duration: 5 min --- +## Learning objectives + +By the end of this chapter you can: + +- Name the four movements the reference set is organized into and which pages fall + into each one. +- Pick the reading order that matches how you want to use the set — straight through, + or jumping to one topic. +- Identify the three pages that cover the pattern's shape if you only have time for a + few. +- Locate the page that answers a specific question without re-reading the whole set. + +## Why this matters + This is the reading guide for the data-mesh reference — the map of what the set covers and the order to read it in. The reference builds one working system end to end: a data mesh of services that own their data as products, talk over a deliberate mix of protocols, evolve their contracts safely, scale to demand, and stay observable throughout. -The set is written to be read straight through the first time — each page picks up where +The set is written to be read straight through the first time. Each page picks up where the last left off, and the cross-references assume you've seen the earlier material. If you're returning to find one thing, the descriptions below will point you at the right page. @@ -50,7 +64,7 @@ the value it delivers and the implementation pieces that realize it. The synchronous read layer (REST, gRPC, a GraphQL gateway) and the asynchronous event backbone — and why the capstone uses all of them. - [**7 · Progressive delivery & mTLS**]({{ '/docs/06-progressive-delivery-mtls/' | relative_url }}) — - Evolving a contract in the open with a v1→v2 canary, mTLS for free, and the decision to + Evolving a contract in the open with a v1→v2 canary, mTLS with no application changes, and the decision to mesh selectively rather than namespace-wide. - [**8 · Elastic & resilient**]({{ '/docs/07-elastic-and-resilient/' | relative_url }}) — Scaling to demand and to zero with KEDA, and the cloud-native recoverability the @@ -75,3 +89,15 @@ to avoid. Those three cover the shape of the thing without the full implementati depth. Start with [concepts & principles]({{ '/docs/01-concepts/' | relative_url }}). + +## For deeper coverage + +- *Data Mesh* by Zhamak Dehghani — the full argument this reference implements, including + the history and the operating-model shifts this set only summarizes. +- *Designing Data-Intensive Applications, 2nd ed.* by Martin Kleppmann & Chris + Riccomini — the data-systems foundations (storage, replication, consistency) that sit + underneath every data product in this build. +- *Cloud Native Patterns* by Cornelia Davis — the platform patterns this reference's + self-serve tier draws on. +- *Kubernetes Patterns* by Bilgin Ibryam & Roland Huß — the Kubernetes-native building + blocks the implementation pages rely on throughout. diff --git a/_docs/01-concepts.md b/_docs/01-concepts.md index b08b261..e4a069a 100644 --- a/_docs/01-concepts.md +++ b/_docs/01-concepts.md @@ -5,6 +5,20 @@ description: What a data mesh is, operational vs. analytical data, and Dehghani' duration: 20 min --- +## Learning objectives + +By the end of this chapter you can: + +- State what a data mesh is and the failure it responds to, in your own words. +- Distinguish operational data from analytical data and explain how a mesh + reorganizes that boundary by domain rather than by technology layer. +- Name Dehghani's four principles and give a one-line example of each from this + build. +- Recognize the difference between the data-mesh pattern and the specific tools this + reference uses to express it. + +## Why this matters + The [previous page]({{ '/docs/01-data-architectures/' | relative_url }}) covered the landscape of data architectures — pipelines, warehouses, and lakes — and where the mesh sits in relation to all of them. This page goes deeper into what a data mesh actually @@ -13,7 +27,7 @@ is and the four principles it rests on. Before any commands, it's worth being precise about what a data mesh actually is — because the term gets attached to a lot of things it isn't. This page is a working grounding in the idea and its four principles, enough to make the rest of the set -make sense. It's deliberately brief: if you want the full conceptual treatment — +make sense. It's brief: if you want the full conceptual treatment — the history, the operating-model shifts, the worked argument for decentralization — that's what the *Data Mesh 101* material in the presentation is for. Here we cover just enough to build on. @@ -41,7 +55,7 @@ let them interoperate. ## Operational vs. analytical data -One distinction underlies everything and is worth stating plainly, because blurring it +One distinction underlies everything, and blurring it is itself a common mistake. **Operational data** is the current-state data behind a domain's running services — the rows a microservice reads and writes to do its job, transactional and live. **Analytical data** is the historical, aggregated view used to @@ -117,3 +131,17 @@ common way these efforts fail is mistaking the tooling for the transformation. With the vocabulary in place, the next page looks at why Kubernetes is a natural substrate for all of this — and how each of the four principles maps onto concrete Kubernetes primitives. + +## For deeper coverage + +- *Data Mesh* by Zhamak Dehghani — the primary source for the four principles and the + domain-ownership argument this page summarizes. +- *Learning Domain-Driven Design* by Vlad Khononov — the bounded-context thinking + behind domain ownership and why a data product maps to a domain, not a table. +- *Implementing Domain-Driven Design* by Vaughn Vernon — a deeper, implementation-level + treatment of the same domain boundaries. +- *Designing Data-Intensive Applications, 2nd ed.* by Martin Kleppmann & Chris + Riccomini — the operational-versus-analytical data distinction from the systems + side. +- *Cloud Native Patterns* by Cornelia Davis — the self-serve platform principle made + concrete in cloud-native infrastructure terms. diff --git a/_docs/01-data-architectures.md b/_docs/01-data-architectures.md index adf28a9..cf8d2d5 100644 --- a/_docs/01-data-architectures.md +++ b/_docs/01-data-architectures.md @@ -5,12 +5,28 @@ description: "The landscape: data pipelines, data warehouses, data lakes, and da duration: 25 min --- +## Learning objectives + +By the end of this chapter you can: + +- Describe data pipelines, data warehouses, and data lakes — the problem each pattern + solves and the limitation that bounds it. +- Explain why pipeline count grows as the product of sources and destinations, not + the sum, and why that growth is called pipeline sprawl. +- Contrast schema-on-write and schema-on-read, and say which pattern uses each. +- Match an organization's bottleneck (movement, truth, flexibility, or organizational + scaling) to the pattern built to solve it. + +## Why this matters + Before defining data mesh precisely, it helps to see the landscape it lives in. The mesh did not appear out of nowhere — it's a response to specific organizational scaling -problems that earlier data architectures left unsolved. Each of those earlier patterns -solved a real problem, and each is still the right answer when that problem is the -dominant one. Understanding what came before, and what each pattern does well, makes -the mesh's value proposition concrete rather than abstract. +problems that earlier data architectures left unsolved. + +Each of those earlier patterns solved a real problem, and each is still the right +answer when that problem is the dominant one. Understanding what came before, and what +each pattern does well, makes the mesh's value proposition concrete rather than +abstract. ## Data pipelines @@ -69,15 +85,16 @@ The limitation is organizational, not technical. The warehouse is owned by a cen data team — sometimes called the BI team, the analytics engineering team, or the data platform team — and that team becomes a bottleneck in direct proportion to the organization's growth. The central team owns all the data but understands none of the -domains that produced it. When the logistics domain needs a new dimension added to the -warehouse, the request goes to the central team, who must learn enough about logistics -to model it correctly, prioritize it against requests from every other domain, and -coordinate the schema change without breaking downstream consumers. The analytical -view is always hours or days behind the operational truth, because the ingestion -pipelines run on a schedule, and the central team's capacity to model new data is the -constraint on how fast the warehouse can grow. As the organization adds domains and -use cases, the central team processes more requests than it can understand, and the -queue becomes the bottleneck. +domains that produced it. + +When the logistics domain needs a new dimension added to the warehouse, the request +goes to the central team, who must learn enough about logistics to model it correctly, +prioritize it against requests from every other domain, and coordinate the schema +change without breaking downstream consumers. The analytical view is always hours or +days behind the operational truth, because the ingestion pipelines run on a schedule, +and the central team's capacity to model new data is the constraint on how fast the +warehouse can grow. As the organization adds domains and use cases, the central team +processes more requests than it can understand, and the queue becomes the bottleneck. {% include excalidraw.html file="01-data-warehouse-architecture" alt="A centralized data warehouse receiving data from multiple operational systems through ingestion pipelines, with a central team owning the schema and all analytical consumers querying one governed store" caption="Figure 1.2 — The data warehouse: a centralized, schema-on-write analytical store that solves the truth problem but concentrates ownership in a single team." %} @@ -111,12 +128,14 @@ The limitation is governance. Without active curation, the lake becomes the **da swamp** — a term so common in the field that it's practically a synonym for a poorly-run lake. Undocumented datasets accumulate. Nobody knows which version of the customer table is authoritative, which datasets are stale, which ones duplicate each -other under different names. The curation effort needed to keep a lake useful -re-creates much of the warehouse's modeling and governance work, just without the -warehouse's enforcement mechanisms. And critically, the organizational problem is -unchanged: a central team still owns the lake, still cannot understand every domain's -data, and still bottlenecks on the same requests. The technology changed — from rigid -schema to flexible storage — but the shape of the organization did not. +other under different names. + +The curation effort needed to keep a lake useful re-creates much of the warehouse's +modeling and governance work, just without the warehouse's enforcement mechanisms. And +critically, the organizational problem is unchanged: a central team still owns the +lake, still cannot understand every domain's data, and still bottlenecks on the same +requests. The technology changed — from rigid schema to flexible storage — but the +shape of the organization did not. {% include excalidraw.html file="01-data-lake-architecture" alt="A data lake with raw, curated, and refined zones, showing heterogeneous data formats flowing in and the risk of the lake becoming a swamp without active governance" caption="Figure 1.3 — The data lake: schema-on-read and format flexibility, but without governance, the lake becomes a swamp — and the central-team bottleneck remains." %} @@ -125,18 +144,21 @@ schema to flexible storage — but the shape of the organization did not. All three patterns — pipelines, warehouses, lakes — centralize data and hand ownership to a single team. The technology improves with each generation: from bespoke point-to-point plumbing, to a governed analytical store, to a flexible -format-agnostic lake. But the organizational shape stays the same in every case: one -team at the center, receiving requests from every domain, owning data it did not -produce, modeling concepts it does not deeply understand, and scaling its capacity -linearly while the demands on it grow combinatorially. +format-agnostic lake. + +But the organizational shape stays the same in every case: one team at the center, +receiving requests from every domain, owning data it did not produce, modeling +concepts it does not deeply understand, and scaling its capacity linearly while the +demands on it grow combinatorially. The **data mesh** proposes a different axis of change. Instead of building a better center, it decentralizes ownership to the domain teams that produce the data. Each domain owns its data as a product — discoverable, addressable, trustworthy, -self-describing — and publishes it for other domains to consume. A shared -**self-serve data platform** provides the infrastructure that every domain needs -(streaming, storage, observability, schema registries) so that domain teams don't -each build their own. And **federated computational governance** keeps the +self-describing — and publishes it for other domains to consume. + +A shared **self-serve data platform** provides the infrastructure that every domain +needs (streaming, storage, observability, schema registries) so that domain teams +don't each build their own. And **federated computational governance** keeps the independently-owned products interoperable — not through review boards and policy documents, but through standards enforced automatically by the platform itself. @@ -151,9 +173,9 @@ domain may still use a warehouse or a lake internally — the order-analytics do might store its refined data in a Parquet-based lakehouse, and that's fine. The reorganization is about *who owns the data*, not about which storage technology to use. The mesh is an answer to the question "how does this organization scale its data -architecture?" — not "which database should this team pick?" +architecture?", not "which database should this team pick?" -The mesh earns its complexity in organizations with many domains, many data consumers, +The mesh's complexity pays off in organizations with many domains, many data consumers, and the operational maturity to sustain federated ownership. For smaller teams, simpler data flows, or organizations where the central team is not yet the bottleneck, a well-run warehouse or a governed lake may be exactly the right answer. The mesh is not @@ -209,3 +231,16 @@ Dehghani's four principles are, how they interlock, and why implementing one wit the others produces a distributed mess rather than a mesh. [Concepts & principles]({{ '/docs/01-concepts/' | relative_url }}). + +## For deeper coverage + +- *Data Mesh* by Zhamak Dehghani — the full argument for why pipelines, warehouses, + and lakes hit an organizational ceiling, and what the mesh changes. +- *Designing Data-Intensive Applications, 2nd ed.* by Martin Kleppmann & Chris + Riccomini — the systems-level treatment of batch and stream processing, ETL/ELT, + and the storage engines behind warehouses and lakes. +- *Cloud Native Anti-Patterns* by Gerald Bachlmayr et al. — common organizational and + architectural failure modes, including the centralized-ownership trap this chapter + describes. +- *Cloud Native Patterns* by Cornelia Davis — the platform-level patterns a self-serve + data platform draws on once the organizational shift is underway. diff --git a/_docs/02-kubernetes-substrate.md b/_docs/02-kubernetes-substrate.md index cb892ab..6e0b05c 100644 --- a/_docs/02-kubernetes-substrate.md +++ b/_docs/02-kubernetes-substrate.md @@ -5,8 +5,24 @@ description: Why Kubernetes is a natural substrate for a data mesh, and how the duration: 15 min --- +## Learning objectives + +By the end of this chapter you can: + +- Map each of Dehghani's four principles to the specific Kubernetes primitive that + realizes it (namespaces/RBAC, Deployments/CRDs, operators, admission control). +- Describe the three horizontal tiers of this capstone's architecture and which + protocol crosses each boundary. +- Explain why this capstone runs on minikube with the docker driver and containerd + runtime, and what trade-off a single-node cluster makes. +- State how the capstone exposes host access (NodePorts published on `127.0.0.1` at + profile creation) and what command recreates the profile if the published ports + need to change. + +## Why this matters + The [previous page]({{ '/docs/01-concepts/' | relative_url }}) ended on a -claim worth taking seriously: a data mesh is a pattern, not a tool, and the tools are +claim: a data mesh is a pattern, not a tool, and the tools are expressions of it. So why build the capstone on Kubernetes at all? Because the four principles map onto Kubernetes primitives unusually cleanly — cleanly enough that "implement a data mesh on Kubernetes" stops feeling like a translation exercise and @@ -44,7 +60,7 @@ The data product becomes a first-class, declarable thing rather than an informal collection of scripts. **Self-serve data platform → operators and shared cluster infrastructure.** This is -where Kubernetes earns its place most clearly. An *operator* packages the knowledge of +where Kubernetes operators deliver the most value for a self-serve platform. An *operator* packages the knowledge of how to run a complex stateful system — Kafka, Postgres, autoscaling, a service mesh — into a controller that reconciles a simple declarative request into a running system. A domain team that needs Kafka doesn't learn to operate Kafka; it asks the platform's @@ -97,20 +113,42 @@ This capstone runs on minikube — a single-node Kubernetes cluster — which is choice for *learning* the pattern and wrong for running it in production. A single node means every tier shares one machine's resources, which keeps the whole mesh runnable on a laptop but also concentrates failure modes that a real multi-node cluster would -spread out. Where those single-node realities bite — resource ceilings, node-level -decay, the operational care a long-lived single-node cluster needs — they're collected -as operational gotchas rather than mixed into the conceptual material here, because -they're particular to this deployment choice rather than to data mesh as a pattern. The -profile is a Docker container (the docker driver) running containerd with the runc -runtime, sized by `MINIKUBE_MEMORY`, `MINIKUBE_CPUS` and `MINIKUBE_DISK`. +spread out. + +Where those single-node realities bite — resource ceilings, node-level decay, the +operational care a long-lived single-node cluster needs — they're collected as +operational gotchas rather than mixed into the conceptual material here, because +they're particular to this deployment choice rather than to data mesh as a pattern. + +The profile itself is a Docker container (the docker driver) running containerd with +the runc runtime, sized by `MINIKUBE_MEMORY`, `MINIKUBE_CPUS` and `MINIKUBE_DISK`. + +### Host access Host access follows the same single-node logic. The capstone's UIs and service endpoints are NodePorts published on `127.0.0.1` when the minikube profile is created (`--ports=127.0.0.1::`), never SSH tunnels or port-forwarding. -The ports are fixed at creation, so changing the set means recreating the profile with -`./scripts/setup-capstone-profile.sh --replace`. `--replace` deletes and recreates the cluster, so run `./scripts/bootstrap-capstone.sh` again afterwards. Run the workshop in isolation: shut down -CRC, other minikube profiles, and other workloads first, so the host ports are free. -`./scripts/show-endpoints.sh` prints what is published and reachable. + +The ports are fixed at creation, so changing the published set means recreating the +profile with `./scripts/setup-capstone-profile.sh --replace`. `--replace` deletes and +recreates the cluster, so run `./scripts/bootstrap-capstone.sh` again afterwards. + +Run the workshop in isolation: shut down CRC, other minikube profiles, and other +workloads first, so the host ports are free. `./scripts/show-endpoints.sh` prints what +is published and reachable. + +## For deeper coverage + +- *Kubernetes Patterns* by Bilgin Ibryam & Roland Huß — the primitive-by-primitive + treatment (namespaces, operators, CRDs) this chapter's mapping draws on. +- *Platform Engineering on Kubernetes* by Mauricio Salatino — the self-serve platform + principle built out as an operating model, not just a set of operators. +- *Cloud Native Patterns* by Cornelia Davis — the broader cloud-native platform + patterns this capstone's shared infrastructure tier follows. +- [Kubernetes documentation](https://kubernetes.io/docs/) — namespaces, RBAC, CRDs, + and admission control, straight from the source. +- [minikube documentation](https://minikube.sigs.k8s.io/docs/) — driver and runtime + options, including the docker driver and containerd runtime this capstone uses. Next, the services themselves: what a data product looks like in this build, the order-service template the others follow, and how each one is packaged and shipped. diff --git a/_docs/03-services-and-data-products.md b/_docs/03-services-and-data-products.md index f585d9a..bb2f661 100644 --- a/_docs/03-services-and-data-products.md +++ b/_docs/03-services-and-data-products.md @@ -5,6 +5,18 @@ description: The services, the order-service template, and the anatomy of a data duration: 30 min --- +## Learning objectives + +By the end of this chapter you can: + +- Map a data product's ports, transformation, and metadata onto a concrete domain-service boundary. +- Enumerate the five domain services plus the gateway and the protocol each one exposes. +- Trace the walking-skeleton build order for order-service, from image to a running pod with a real API round trip. +- Apply the multi-stage, non-root container packaging pattern shared by all six images. +- Load a locally built image into the minikube profile's containerd store and explain why `imagePullPolicy: Never` matters there. + +## Why this matters + With the [principles]({{ '/docs/01-concepts/' | relative_url }}) and the [Kubernetes mapping]({{ '/docs/02-kubernetes-substrate/' | relative_url }}) in place, this page gets concrete: what the data products in this capstone actually @@ -31,7 +43,7 @@ way to make domain ownership real rather than aspirational. ## The domain -The domain is order-placement-through-fulfillment, modeled deliberately small so the +The domain is order-placement-through-fulfillment, modeled small so the architecture stays legible. There are **five domain services**, each a bounded context owning its data and its contract, plus **one gateway** that composes reads across them (the gateway is a read-layer convenience, not a domain data product — six images in @@ -72,13 +84,16 @@ schema. It starts speaking only REST, and gRPC, GraphQL, and event publishing ge layered on in later steps — but the deployment spine is proven first with the simplest possible surface. +### Packaging the image + A couple of packaging choices carry across all six images. Dependencies are managed with a lockfile so builds are reproducible, and the lockfile is exported into the image -rather than carrying the dependency manager into the runtime. The image itself is -multi-stage: a builder stage resolves dependencies, and a slim runtime stage copies -only the resolved environment and the application code, runs as a non-root user, and -serves the app. Standard production hygiene — the capstone just applies it from the -start rather than retrofitting it. +rather than carrying the dependency manager into the runtime. + +The image itself is multi-stage: a builder stage resolves dependencies, and a slim +runtime stage copies only the resolved environment and the application code, runs as a +non-root user, and serves the app. Standard production hygiene — the capstone just +applies it from the start rather than retrofitting it. ## The one part that fights back: getting images to the kubelet @@ -89,13 +104,17 @@ on your host. An image you build locally is invisible to the kubelet until you p there, and a pod that can't find its image fails in ways that look like a broken cluster. -The capstone's answer is deliberately simple: **build with Docker Engine, then load the +### Build with Docker Engine, then load the image + +The capstone's answer is simple: **build with Docker Engine, then load the image into the profile with `minikube image load`**. There is no registry to run and no addresses to reconcile, and it works the same on a native engine and on a VM-based one (Docker Desktop, for example). That portability is the reason for the choice: pushing to a registry on the host's loopback only works when the engine's daemon shares the host's network, which a VM-based engine's does not. +### Tags and pull policy + Images are tagged `capstone/:v1` and the charts set `imagePullPolicy: Never`. The policy matters in both directions. If an image was never loaded, the pod fails at once with `ErrImageNeverPull` instead of quietly trying Docker Hub for a name that exists @@ -112,3 +131,11 @@ friction.) With a service shipped and running, the next question is how products describe themselves so others can find and trust them — contracts and the catalog. + +## For deeper coverage + +- *Data Mesh* by Zhamak Dehghani — the architectural quantum and why a data product's boundary should match a service's boundary. +- *Cloud Native Patterns* by Cornelia Davis — packaging and runtime patterns for services that double as data products. +- *Kubernetes Patterns* by Bilgin Ibryam & Roland Huß — container image and workload patterns behind the build-load-deploy sequence. +- [Kubernetes documentation](https://kubernetes.io/docs/) — image pull policies and the kubelet's image lifecycle. +- [FastAPI](https://fastapi.tiangolo.com/) — the framework order-service and its siblings are built on. diff --git a/_docs/04-contracts-and-catalog.md b/_docs/04-contracts-and-catalog.md index 056e850..810e4e7 100644 --- a/_docs/04-contracts-and-catalog.md +++ b/_docs/04-contracts-and-catalog.md @@ -5,6 +5,17 @@ description: Versioned contracts in Apicurio and discovery plus lineage in OpenM duration: 25 min --- +## Learning objectives + +By the end of this chapter you can: + +- Classify a given contract as a runtime contract or a discovery contract, and explain why the coupling differs between the two. +- Name the four contract types each service produces and where each one lives in Apicurio. +- Trace how OpenMetadata ingests from the registry, the database, and Kafka to assemble a catalog with lineage. +- Explain why a catalog is a requirement in a data mesh rather than optional tooling, and what a mesh degrades into without one. + +## Why this matters + The [previous page]({{ '/docs/03-services-and-data-products/' | relative_url }}) established that each service is a data product. But a data product is only useful to *other* domains if they can find it, understand its shape, trust that it won't change @@ -38,21 +49,25 @@ it's worth getting clear because the two jobs have very different coupling. ![How a contract flows — the runtime path versus the discovery path]({{ '/assets/diagrams/17-capstone-contract-flow.svg' | relative_url }}) -A **runtime contract** is load-bearing on the hot path. The event schema is the +### Runtime contracts + +A **runtime contract** is essential on the hot path. The event schema is the example: the producer serializes an event against the registered schema and the consumer deserializes against it, so the event literally won't encode or decode correctly if the contract is wrong or missing. The registry is in the live path of every event. (The gRPC Protobuf definitions are similar in spirit, though compiled ahead of time rather than fetched at runtime.) Because a runtime contract is -load-bearing, the registry can enforce real governance on it: reject a +essential to execution, the registry can enforce real governance on it: reject a schema-incompatible change at publish time, before it can break a single consumer. +### Discovery contracts + A **discovery contract** is different. The OpenAPI document and the GraphQL SDL are published as the source of truth for a product's shape, but nothing fails at runtime if they're absent — they exist for humans reading the API, for breaking-change checks in CI, and for the catalog to ingest. They describe; they don't serialize. -The same registry does both jobs, but the coupling is the thing to internalize: the +The same registry does both jobs, but the coupling differs: the runtime path is tightly coupled and online, the discovery path is loosely coupled and offline. Conflating them — treating a discovery contract as if it were enforced at runtime, or a runtime contract as if it were just documentation — is how teams get @@ -82,17 +97,19 @@ exist before lineage over them can be built. ## Why this is a requirement, not an add-on It's tempting to treat a catalog as a nice-to-have you bolt on once the "real" system -works. In a data mesh it's the opposite, and this is the conceptually load-bearing +works. In a data mesh it's the opposite, and this is the central point of the page. The entire premise of a mesh is that domains own their data independently and other domains consume it without a central team brokering access. That premise *only works* if products are discoverable and their contracts are -trustworthy. Without a registry, contracts drift and consumers break silently. Without +trustworthy. + +Without a registry, contracts drift and consumers break silently. Without a catalog, nobody can find products or understand lineage, so in practice they fall back to asking the central team — which reintroduces exactly the bottleneck the mesh exists to remove. -So discovery infrastructure isn't decoration on top of a mesh; it's load-bearing -structure. A mesh without a usable catalog degrades into the +So discovery infrastructure isn't decoration on top of a mesh; it's structural to +it. A mesh without a usable catalog degrades into the [proxy anti-pattern]({{ '/docs/09-anti-patterns/' | relative_url }}) — a central team back in the middle of every cross-domain question — even if every other piece is in place. The registry and catalog are what let federated governance be @@ -108,3 +125,10 @@ ingestion has a deployment subtlety when a mesh is in play. With products that describe themselves and a catalog that makes them discoverable, the next question is how data actually moves between them — the synchronous read paths and the asynchronous event backbone. + +## For deeper coverage + +- *Data Mesh* by Zhamak Dehghani — data-as-a-product and federated computational governance, the two principles this chapter makes operational. +- *Designing Data-Intensive Applications, 2nd ed.* by Martin Kleppmann & Chris Riccomini — schema evolution and the compatibility guarantees a registry enforces. +- *Cloud Native Anti-Patterns* by Gerald Bachlmayr et al. — the proxy anti-pattern a mesh degrades into without a usable catalog. +- Apicurio Registry documentation — contract versioning and compatibility rules across schemas, Protobuf, OpenAPI, and GraphQL SDL. diff --git a/_docs/05-data-planes.md b/_docs/05-data-planes.md index e189f85..5b0daf5 100644 --- a/_docs/05-data-planes.md +++ b/_docs/05-data-planes.md @@ -5,6 +5,17 @@ description: The async backbone of Kafka events and the read layer of GraphQL co duration: 25 min --- +## Learning objectives + +By the end of this chapter you can: + +- Match each of the four protocols — REST, gRPC, GraphQL, and events — to the job it's best suited for, and explain the "fitness, not hierarchy" framing. +- Distinguish gateway orchestration from true subgraph federation and state the trade-off this capstone accepts by choosing orchestration. +- Trace the `order.placed` event flow from producer to consumers and explain why the producer doesn't block on any of them. +- Explain why the synchronous read layer and the asynchronous event backbone are complementary planes rather than competing ones. + +## Why this matters + The [services]({{ '/docs/03-services-and-data-products/' | relative_url }}) each expose a contract, and the [catalog]({{ '/docs/04-contracts-and-catalog/' | relative_url }}) makes @@ -12,12 +23,12 @@ them discoverable. This page is about how data actually *moves* between them — planes the mesh runs on. There are two: a **synchronous read layer** (REST, gRPC, and a GraphQL gateway composing across them) and an **asynchronous event backbone** (the streaming spine that lets domains react to each other). The capstone uses all of them -on purpose, and the purpose is the point. +because each one fits a different kind of data movement. ## Protocols by fitness, not by hierarchy It would be simpler to pick one protocol and use it everywhere. The capstone -deliberately doesn't, because the honest lesson is that each protocol is *best at a +doesn't, because each protocol is *best at a different job*, and a data mesh exercises all those jobs. This isn't a ranking where one protocol wins; it's a matter of fitness for context: @@ -58,11 +69,15 @@ simpler **gateway orchestration** approach instead: one stateless gateway expose unified graph, and its resolvers fetch from the services over their *existing* REST and gRPC interfaces, with no GraphQL added to the domain services at all. -The reason is honesty about what the capstone is — a learning implementation. Gateway +### Why orchestration, not federation, here + +The capstone is a learning implementation. Gateway orchestration demonstrates the *value* GraphQL adds (one client query, multiple backends, client-shaped response) with one new service and zero changes to the five domain services, and it makes the protocol comparison vivid because the resolvers call -REST and gRPC right next to each other. The trade-off worth stating: true federation is +REST and gRPC right next to each other. + +The trade-off: true federation is what you'd reach for in production, because it preserves domain ownership of the graph — with orchestration, the gateway has some knowledge of how to reach each domain. For teaching the shape, orchestration is the right call; for a production mesh, federation @@ -77,8 +92,9 @@ waits. The other plane is the opposite shape, and it's what actually keeps a mes operational/analytical loop closed. When an order is placed, several domains need to react — inventory adjusts stock, payment processes, notification informs the customer — but the order service shouldn't block waiting on any of them, and it certainly -shouldn't call them in a synchronous chain that fails if one is down. Instead it emits -an event, and the interested domains consume it on their own schedule. +shouldn't call them in a synchronous chain that fails if one is down. + +Instead it emits an event, and the interested domains consume it on their own schedule. ![Domain events and change-data-capture feeding the analytical side]({{ '/assets/diagrams/17-ingestion-streaming-sourcing.svg' | relative_url }}) @@ -87,7 +103,9 @@ the canonical pattern with the smallest real flow: order-service emits an `order.placed` event, keyed by order id, and notification-service consumes it. That one flow is the whole "something happened, downstreams react" shape in miniature — the producer doesn't know or care who's listening, and consumers can be added without -touching the producer. Widen it and you get the full choreography: inventory and +touching the producer. + +Widen it and you get the full choreography: inventory and payment also react to `order.placed`, shipping reacts to `payments.processed`, and notification listens across all of them. @@ -106,9 +124,17 @@ The synchronous and asynchronous planes aren't competitors; they're complementar a mesh needs both. Reads compose current state when a consumer asks for it. Events propagate change when something happens. REST crosses the boundary, gRPC moves between services, GraphQL shapes the composite, and the event backbone carries the reactions. -Each protocol earns its place by being best at its job — which is the whole argument +Each protocol is best at its job — which is the whole argument for using all of them rather than forcing one to do everything. With data moving across both planes, the next concern is how the mesh evolves safely while it's running: shipping a new version of a product's contract without breaking the consumers depending on it. + +## For deeper coverage + +- *Data Mesh* by Zhamak Dehghani — polyglot access and the synchronous/asynchronous planes a mesh runs on. +- *Designing Data-Intensive Applications, 2nd ed.* by Martin Kleppmann & Chris Riccomini — event-driven architecture and the derived-data reasoning behind the backbone. +- *Cloud Native Patterns* by Cornelia Davis — composing services at the edge versus internally. +- [grpc.io documentation](https://grpc.io/docs/) — gRPC for internal, low-latency service-to-service calls. +- [Strawberry GraphQL](https://strawberry.rocks/) — the library behind the gateway's composed graph. diff --git a/_docs/06-progressive-delivery-mtls.md b/_docs/06-progressive-delivery-mtls.md index 51b8e51..4ba3d34 100644 --- a/_docs/06-progressive-delivery-mtls.md +++ b/_docs/06-progressive-delivery-mtls.md @@ -5,12 +5,24 @@ description: Evolving a contract in the open with an Istio v1→v2 canary, and t duration: 25 min --- +## Learning objectives + +By the end of this chapter you can: + +- Explain how a `DestinationRule`, a `VirtualService`, and a `Gateway` work together to canary a contract version rather than just a binary. +- Read a weighted traffic-split result and judge whether it falls within the expected band. +- Explain how Istio provides mTLS between meshed sidecars with no change to application code. +- Identify the three workload categories that should opt out of mesh injection and why each one breaks under it. +- Articulate the trade-off between namespace-wide injection and selective injection. + +## Why this matters + Data products evolve. A product that can never change its contract is a product nobody will build on for long — but a product that changes its contract *carelessly* breaks every consumer downstream. This page is about evolving safely: shifting traffic gradually between an old and a new version of a product's contract, with the service mesh both routing the split and securing the traffic. It's also where the capstone -makes a genuine architectural decision that the rest of the set has been alluding to — +makes an architectural decision that the rest of the set has been alluding to — to mesh **selectively** rather than meshing everything — and explains why, from experience. @@ -33,10 +45,12 @@ them, and a `Gateway` admits the traffic. Shifting the canary is just re-applyin progressive rollout is a sequence of one-line weight changes, and rolling back is the same operation in reverse. -A couple of honest simplifications worth naming. v1 and v2 here run the *same image* -with an environment toggle and a different `version` label, rather than two images from -two commits — that keeps the demo focused on the traffic-management mechanism rather -than an image pipeline, and the Istio mechanics are identical either way. And traffic +Two simplifications here: v1 and v2 run the *same image* with an environment toggle +and a different `version` label, rather than two images from two commits — that +keeps the demo focused on the traffic-management mechanism rather than an image +pipeline, and the Istio mechanics are identical either way. + +And traffic enters through the ingress gateway (reached on its NodePort, published to 127.0.0.1:8088 when the profile is created, DRA-017) rather than from a meshed client, so no client needs to join the mesh for the canary to work. @@ -48,7 +62,7 @@ under controlled traffic, exactly as the pattern intends. ![The capstone's mesh in canary mode — sidecars beside each product, mTLS between them, weights moving traffic from v1 to v2]({{ '/assets/diagrams/17-service-mesh.svg' | relative_url }}) -## mTLS for free +## mTLS without code changes The same mesh that routes the canary also secures it. When two services are in the mesh, Istio establishes mutual TLS between their sidecars automatically — each service @@ -91,14 +105,16 @@ Beyond those specific breakages, there's a systemic reason: with namespace-wide injection, *every* pod creation in the namespace depends on the injection webhook being reachable. If the mesh control plane has a bad moment, you can't create a database pod, a job, or anything else — workloads that have nothing to do with the mesh are now -coupled to its health. Selective injection contains that blast radius: only the -workloads that actually need the mesh depend on it. - -So the capstone meshes the services that participate in mesh-managed traffic — the ones -doing the canary, the ones that benefit from mTLS and traffic management — and keeps the -database, the batch jobs, and other operator-managed infrastructure *out* of the mesh by -opting them out explicitly. The trade-off is real and worth stating plainly: -namespace-wide injection is genuinely simpler to reason about and gives blanket mTLS +coupled to its health. + +Selective injection contains that blast radius: only the +workloads that actually need the mesh depend on it. So the capstone meshes the services +that participate in mesh-managed traffic — the ones doing the canary, the ones that +benefit from mTLS and traffic management — and keeps the database, the batch jobs, and +other operator-managed infrastructure *out* of the mesh by opting them out explicitly. + +The trade-off is real: +namespace-wide injection is simpler to reason about and gives blanket mTLS with one label, while selective injection is more work to configure and means you have to decide, per workload, whether it belongs in the mesh. The capstone takes that configuration cost in exchange for resilience and correctness — because the failures @@ -120,3 +136,11 @@ liability: applied where it helps, kept away from where it hurts. Next, the other half of operating a mesh under real conditions: scaling products to match demand — including down to zero — and recovering when things fail. + +## For deeper coverage + +- [Istio documentation](https://istio.io/latest/docs/) — `DestinationRule`, `VirtualService`, `Gateway`, and automatic mutual TLS between sidecars. +- *Kubernetes Patterns* by Bilgin Ibryam & Roland Huß — the Sidecar pattern and the trade-offs of attaching one to every workload. +- *Platform Engineering on Kubernetes* by Mauricio Salatino — treating mesh injection as a platform policy decision rather than a blanket default. +- *GitOps and Kubernetes* by Billy Yuen et al. — progressive delivery workflows for shifting traffic between versions. +- *Cloud Native Anti-Patterns* by Gerald Bachlmayr et al. — the failure modes of meshing workloads that don't belong in the mesh. diff --git a/_docs/07-elastic-and-resilient.md b/_docs/07-elastic-and-resilient.md index 1bf25aa..2afff2a 100644 --- a/_docs/07-elastic-and-resilient.md +++ b/_docs/07-elastic-and-resilient.md @@ -5,14 +5,28 @@ description: Scaling to demand and to zero with KEDA, and the cloud-native recov duration: 20 min --- +## Learning objectives + +By the end of this chapter you can: + +- Explain why CPU/memory autoscaling is a poor proxy for data-product demand and what KEDA scales on instead. +- Distinguish consumer-lag scaling (notification-service) from HTTP-request scaling (the gateway), and say why each scaler fits its product. +- Explain why the HTTP scaler sits on the gateway rather than on the canaried order-service. +- Describe what "recoverable by construction" means for a data product running on Kubernetes. +- Explain why elasticity and resilience are self-serve platform capabilities rather than something each domain builds itself. + +## Why this matters + A data product's demand isn't constant. A read gateway is busy when consumers are querying and idle otherwise; an event consumer has work only when events are flowing. Provisioning every product for its peak, all the time, wastes resources; provisioning -for the average means falling over at the peak. This page is about the platform -handling that automatically — scaling products to match demand, including down to -*zero* when there's none — and about the other half of operating under real conditions: -recovering when something fails. Both are **self-serve platform** capabilities: a -domain team gets elasticity and resilience from the platform rather than building them. +for the average means falling over at the peak. + +This page is about the platform handling that automatically — scaling products to +match demand, including down to *zero* when there's none — and about the other half of +operating under real conditions: recovering when something fails. Both are **self-serve +platform** capabilities: a domain team gets elasticity and resilience from the platform +rather than building them. ## Scaling to demand — and to zero @@ -31,8 +45,7 @@ have many such products, each materializing only when something actually asks fo That's a different economics from "every service holds a baseline of resources forever," and it's particularly apt for a mesh of many domain products with uneven, bursty demand. -This capstone wires up **two** scalers, deliberately of different kinds, on two -different products: +This capstone wires up **two** scalers of different kinds, on two different products: - **Consumer-lag scaling on notification-service.** notification-service is an event consumer — its real workload is the backlog of events waiting in its consumer @@ -48,7 +61,7 @@ different products: ![Scaling on HTTP request volume with the KEDA HTTP add-on]({{ '/assets/diagrams/12-keda-http-addon.svg' | relative_url }}) -There's a deliberate placement decision worth noting: the HTTP scaler goes on the +The HTTP scaler goes on the gateway, *not* on order-service, because order-service is where the [canary]({{ '/docs/06-progressive-delivery-mtls/' | relative_url }}) lives, and HTTP-scaling a service whose traffic is being split by weight would have the two @@ -70,11 +83,13 @@ resolves the moment a request hits the gateway. The other half of operating under real conditions is what happens when things break — and on a real system, things break. The cloud-native answer isn't "prevent all failure"; it's "make recovery cheap and automatic." Kubernetes provides a great deal of -this for free, and a data mesh inherits it: a crashed pod is restarted, a failed +this without extra application code, and a data mesh inherits it: a crashed pod is restarted, a failed deployment is rolled back, a node's workloads are rescheduled, and declarative state means the system continuously reconciles back toward what you asked for rather than -drifting away from it. A data product deployed this way is *recoverable by -construction* — you can kill it and the platform brings it back. +drifting away from it. + +A data product deployed this way is *recoverable by construction* — you can kill it +and the platform brings it back. That property is worth designing *toward*, not just relying on. A product that recovers cleanly is one that doesn't hold critical state only in memory, that can be restarted @@ -92,11 +107,12 @@ single-node cluster well turns out to require real care: knowing when to let the platform self-heal versus when to cycle a node, recognizing the difference between a workload failure and an infrastructure failure beneath it, and giving stateful workloads enough resources to recover rather than just to run. Those operational lessons -are specific and hard-won, and they're collected as gotchas rather than mixed in here — -but the conceptual point stands: a mesh built on a platform that reconciles toward -declared state is far more resilient than one held together by manual operations, and -that resilience is something the domains get from the platform rather than each -building their own. +are specific and hard-won, and they're collected as gotchas rather than mixed in here. + +The conceptual point stands regardless: a mesh built on a platform that reconciles +toward declared state is far more resilient than one held together by manual +operations, and that resilience is something the domains get from the platform rather +than each building their own. ## Elastic and resilient, from the platform @@ -111,3 +127,10 @@ just its deployment. Last, the capability that makes all of this observable: seeing what the mesh is actually doing — its metrics, its traces, and the live view of traffic moving between products. + +## For deeper coverage + +- *Kubernetes Patterns* by Bilgin Ibryam & Roland Huß — autoscaling and self-healing patterns on Kubernetes. +- *Cloud Native Patterns* by Cornelia Davis — elasticity and recoverability as platform-provided properties. +- *Platform Engineering on Kubernetes* by Mauricio Salatino — self-serve platform capabilities and what domains should and shouldn't operate themselves. +- [KEDA documentation](https://keda.sh/docs/) — scalers, `ScaledObject`s, and the HTTP add-on used in this chapter. diff --git a/_docs/08-observability.md b/_docs/08-observability.md index 95e6add..05a760f 100644 --- a/_docs/08-observability.md +++ b/_docs/08-observability.md @@ -5,10 +5,23 @@ description: Seeing what the mesh is doing — metrics, distributed traces acros duration: 20 min --- +## Learning objectives + +By the end of this chapter you can: + +- Name the three observability signals (metrics, traces, logs) and the question each answers about a request crossing products. +- Explain why the service mesh's sidecars provide inter-product metrics without any application instrumentation. +- Trace a single GraphQL query as a multi-span trace crossing three protocols — HTTP, REST, and gRPC — through the gateway. +- Explain what a live mesh-visualization tool like Kiali adds beyond metrics and traces. + +## Why this matters + The [previous page]({{ '/docs/07-elastic-and-resilient/' | relative_url }}) ended on scaling and recovery — the platform doing things automatically in response to demand and failure. But "the platform does things automatically" is only reassuring if -you can *see* it happening. This page is about observability: the metrics that show a +you can *see* it happening. + +This page is about observability: the metrics that show a product's load and the autoscaler's response, the distributed traces that follow a single request across several products, and the live view of traffic moving through the mesh. In a system of independently-owned products talking to each other, observability @@ -35,20 +48,22 @@ one product owns in full. ![The three signals — metrics, traces, and logs — and the question each answers about a request crossing products]({{ '/assets/diagrams/17-three-signals.svg' | relative_url }}) -## Metrics, for free from the mesh +## Metrics from the mesh, no instrumentation needed The first pleasant surprise is how much you get without instrumenting anything. Because the services are in the [service mesh]({{ '/docs/06-progressive-delivery-mtls/' | relative_url }}), the sidecars already emit request-level metrics — counts, latencies, error rates for the traffic flowing between products — and a Prometheus that scrapes them gets a detailed picture of inter-product traffic without a single line of metrics code in the -services. This capstone leans on exactly that: the metrics half of observability adds +services. + +This capstone leans on exactly that: the metrics half of observability adds *nothing* to the application code, because the mesh is already measuring the traffic it carries. Grafana then turns those metrics into the dashboards you'd actually watch — the gateway's request volume rising, KEDA spinning up replicas in response, the canary's traffic split landing where the weights say it should. -That "for free" property is the self-serve platform principle showing up again: a domain +That property is the self-serve platform principle showing up again: a domain team gets request metrics for its product because the platform's mesh provides them, not because the team instrumented anything. @@ -62,13 +77,15 @@ boundaries, so the work each product does on its behalf becomes a connected tree **spans** — one request, many products, one trace. This capstone collects traces in Tempo and instruments the **gateway** as the trace -source, and that choice is deliberate. The gateway is the entry point for the federated +source. The gateway is the entry point for the federated read path, so a single GraphQL query there produces the most instructive trace there is in this system: an HTTP server span for the query itself, a REST client span for the call out to order-service, and a gRPC client span for the call to inventory-service — all three protocols from [the data planes page]({{ '/docs/05-data-planes/' | relative_url }}) -appearing as nested spans in one trace. You can *see* the composition happen: the +appearing as nested spans in one trace. + +You can *see* the composition happen: the gateway fanning out to two products over two different protocols and assembling the result. One well-chosen instrumentation point illuminates the whole read path. @@ -83,6 +100,7 @@ with a mesh is a *live* picture of the topology — which products are talking t right now, with traffic rates and health on the edges between them. That's what a mesh-visualization layer such as Kiali provides: a real-time graph of the products and the traffic between them, drawn from the same mesh telemetry the metrics come from. + During a canary it shows the v1/v2 split as live traffic on the graph; when a product is struggling, the unhealthy edges are visible in the topology. For a system whose defining behavior is products talking to each other, a live map of exactly that is the @@ -96,7 +114,9 @@ Put the pieces together and the mesh becomes legible end to end. A single GraphQ appears as a trace whose spans cross three products and three protocols. The metrics show the gateway's load and the autoscaler's response to it. The catalog records the lineage of which products feed which. The mesh view shows the live topology, canary -split and all. None of these is the system — they're the *instruments* through which you +split and all. + +None of these is the system — they're the *instruments* through which you understand a system too distributed for any single vantage point to capture. That's the acceptance bar the capstone is built toward: not just that the mesh runs, but that you can watch it run — see a request thread across products, see a product scale to meet @@ -110,3 +130,10 @@ elastic scaling with recovery, and the observability to see all of it. The one t left is to step back from *how* and look at *what goes wrong* — the conceptual and organizational failure modes that derail data-mesh efforts even when the technology is sound. + +## For deeper coverage + +- *Designing Data-Intensive Applications, 2nd ed.* by Martin Kleppmann & Chris Riccomini — reasoning about the behavior of distributed systems built from independently-owned parts. +- [OpenTelemetry documentation](https://opentelemetry.io/docs/) — the metrics/traces/logs signal model this chapter applies to a mesh. +- [Grafana documentation](https://grafana.com/docs/) — Tempo for traces, Mimir/Prometheus for metrics, and correlating the two. +- *Cloud Native Patterns* by Cornelia Davis — observability as a platform-provided concern rather than a per-service add-on. diff --git a/_docs/09-anti-patterns.md b/_docs/09-anti-patterns.md index 610e618..be4a6c6 100644 --- a/_docs/09-anti-patterns.md +++ b/_docs/09-anti-patterns.md @@ -5,6 +5,17 @@ description: The conceptual and organizational ways data-mesh efforts go wrong, duration: 15 min --- +## Learning objectives + +By the end of this chapter you can: + +- Explain why data-mesh efforts fail for organizational reasons far more often than technical ones. +- Recognize the tells of at least four of the anti-patterns covered: tool-as-solution, recentralization, dumb data products, governance imbalance, ownership vacuums, the open loop, and hype-driven adoption. +- Map each anti-pattern to the specific implementation choice in this reference that avoids it. +- Weigh whether a full data-mesh adoption fits an organization's size and maturity, or whether a single principle would serve better. + +## Why this matters + Everything else in this set is about what to build and how. This page is about what goes wrong — not the implementation potholes we hit wiring up the demo (those are operational, and collected separately), but the *conceptual and organizational* @@ -27,7 +38,9 @@ The most common trap is treating data mesh as something you buy or install. A te adopts a catalog product, relabels the data lake's tables as "data products," and declares the mesh delivered. But the principles a mesh rests on — domain ownership, product thinking, self-serve infrastructure, federated governance — are mostly about -*how people and teams work*, not which software is running. Practitioners writing +*how people and teams work*, not which software is running. + +Practitioners writing about failed adoptions consistently put this first: a data mesh is a shift in operating model, and no tool delivers an operating-model change on its own. The software can *support* the principles or quietly undermine them, but it can't @@ -38,7 +51,7 @@ boundaries, ownership, or incentives. If the only thing changing is the software what you'll have at the end is the old centralized model with a new dashboard. *In this build:* every component we chose (Apicurio, OpenMetadata, KEDA, Istio) is -deliberately a *substrate* the domains build on, not a turnkey mesh — the +a *substrate* the domains build on, not a turnkey mesh — the [concepts page]({{ '/docs/01-concepts/' | relative_url }}) frames each tool as serving a principle rather than being the point. @@ -46,7 +59,9 @@ tool as serving a principle rather than being the point. Data mesh exists to break the bottleneck of a single central data team that owns all the data but none of the domains. The failure mode is recreating that bottleneck -under new vocabulary. It takes a few recognizable shapes. A central team stays the +under new vocabulary. It takes a few recognizable shapes. + +A central team stays the *proxy* for every domain — still the only group that can actually ship a data product, just now called a "platform team." Or governance re-centralizes as a manual approval gate: every schema change, every deployment, every contract waits on a @@ -69,7 +84,9 @@ automatically rather than gating changes through review. This is the failure mode that the principle's own author warns about most sharply. A *data product* is supposed to be an autonomous unit — it serves its data, governs it, describes itself, makes itself discoverable, and carries everything it needs to do -its job. The reduction is to strip all of that away and call a renamed table, a view +its job. + +The reduction is to strip all of that away and call a renamed table, a view in a warehouse, or a row in a catalog a "data product." Those are static datasets with a label. They don't serve themselves, can't enforce their own policies, and have no lifecycle — and because there's nowhere to embed governance *in* them, federated @@ -112,7 +129,9 @@ shows version evolution governed by the platform rather than by sign-off. A mesh is only as good as the clarity of who owns what. Two related failures show up here. The first is the ownership vacuum: a dataset that no individual or team is actually responsible for, so its quality drifts, nobody fields questions about it, and -trust erodes until consumers route around it. The second is fuzzy domain boundaries: +trust erodes until consumers route around it. + +The second is fuzzy domain boundaries: domains that overlap or are ill-defined, so the same concept is modeled three different ways by three teams, changes ripple across services that should have been independent, and consumers get conflicting versions of what's nominally the same data. @@ -132,7 +151,9 @@ the boundaries are the service boundaries. Data mesh is meant to close the loop between the operational systems that produce data and the analytical uses that consume it, organized by domain. The degraded version is an *open loop*: static analytical products built downstream from a lake or warehouse, -disconnected from the operational systems and from their own consumers. There's no +disconnected from the operational systems and from their own consumers. + +There's no feedback path — neither the operational-to-analytical loop that keeps products current with the running business, nor the consumer-feedback loop that tells a product owner whether the product is actually useful. Without feedback, products go stale, the long @@ -150,10 +171,12 @@ covers why the event backbone is what keeps the loop closed. ## Hype-driven and wrong-fit adoption Finally, a category that's less about *how* you build a mesh and more about *whether* -you should. Data mesh is not for every organization. It earns its complexity in large +you should. Data mesh is not for every organization. Its complexity pays off in large organizations with many data domains, many consumers, and the organizational maturity to operate products and standards across teams. For a small organization, the overhead -of decentralization can cost more than the bottleneck it removes. Related traps in this +of decentralization can cost more than the bottleneck it removes. + +Related traps in this family: chasing more data products as an end in itself (the right number is what the organization can actually consume, not the maximum it can produce); analysis paralysis, where teams plan the perfect mesh for months instead of standing up one real product @@ -161,11 +184,11 @@ and learning from it; and adopting the whole paradigm because it's fashionable w adopting a single principle — say, self-serve platform infrastructure — would have served better. -The honest move before committing is to weigh data size, organizational complexity, -existing tooling, and culture, and to be willing to conclude that a full mesh isn't the +Before committing, weigh data size, organizational complexity, +existing tooling, and culture, and be willing to conclude that a full mesh isn't the right fit — or that only some of its principles are. -*In this build:* the capstone is deliberately a *learning* implementation — a small, +*In this build:* the capstone is a *learning* implementation — a small, runnable mesh that makes the principles concrete. It's sized to teach the shape, not to argue that every reader should run a mesh in production. @@ -183,3 +206,10 @@ The implementation-level potholes we hit building *this* demo — the operationa gotchas, as opposed to these conceptual ones — are collected separately, since they're a different kind of lesson: specific, technical, and particular to running a mesh on a single-node cluster rather than to data mesh as an idea. + +## For deeper coverage + +- *Data Mesh* by Zhamak Dehghani — the four principles and the failure modes that appear when each is skipped. +- *Cloud Native Anti-Patterns* by Gerald Bachlmayr et al. — recognizable failure shapes in cloud-native platform adoption generally. +- *Learning Domain-Driven Design* by Vlad Khononov — bounded contexts and why fuzzy domain boundaries cascade into mesh failures. +- *Implementing Domain-Driven Design* by Vaughn Vernon — the ownership and autonomy a real data product needs to avoid the "dumb data product" trap. diff --git a/_docs/10-summary.md b/_docs/10-summary.md index 4f6b84a..9dc1caa 100644 --- a/_docs/10-summary.md +++ b/_docs/10-summary.md @@ -5,10 +5,23 @@ description: A closing summary — for each of the four data-mesh principles, th duration: 10 min --- +## Learning objectives + +By the end of this chapter you can: + +- Restate each of Dehghani's four data-mesh principles in terms of the value it delivers. +- Map each principle to the concrete Kubernetes-native implementation pieces in this reference (namespaces, Apicurio, OpenMetadata, the operator pattern, Istio, admission policies). +- Identify the failure mode that appears when each principle is missing. +- Explain why a trusted base image (UBI) underlies all four principles rather than standing apart as a separate concern. + +## Why this matters + The reading set has worked through the data mesh implementation by *concern* — substrate, services, contracts, planes, mesh, scaling, observability, failure modes. This closing page reorganizes the same -material by *principle*. For each of Dehghani's four principles, you'll +material by *principle*. + +For each of Dehghani's four principles, you'll see the value the principle delivers when it's honored, the concrete implementation pieces in this reference that realize it, and the failure mode that shows up when the principle is missing. @@ -18,7 +31,9 @@ The diagrams on this page are reproduced from the implementation deck at with OpenShift specifically, the diagrams use OpenShift vocabulary in a few places — *Project* for namespace, *SCCs* for PodSecurity, *OperatorHub / OLM* for the operator install path, *AMQ Streams* for Strimzi, *CNPG* for -CloudNativePG. The principles and the implementation pieces themselves are +CloudNativePG. + +The principles and the implementation pieces themselves are runtime-agnostic; the OpenShift framing is one concrete instance of the same shape. If your runtime is something else (vanilla Kubernetes, EKS, GKE, AKS, minikube), the substitutions are direct: read *Project* as @@ -128,6 +143,7 @@ OpenShift) makes Kafka available *as a custom resource*: a domain declares its consumer group; the operator handles the cluster. CloudNativePG (CNPG) does the same for Postgres: a domain declares its database; the operator handles the cluster, the backups, the failover. + KEDA makes elastic scaling — including scale-to-zero — available as a ScaledObject, the same way: declare the demand signal, the operator handles the scaling. GitOps closes the loop by making the whole mesh @@ -162,7 +178,9 @@ about TLS. Istio's traffic-management primitives (VirtualService, DestinationRule, the canary subset routing the reference uses for the v1→v2 contract demo) make *progressive delivery* a platform capability — a contract evolves under controlled traffic, weight-shifted live, with -the application layer unchanged. Kubernetes admission policies (ValidatingAdmissionPolicy +the application layer unchanged. + +Kubernetes admission policies (ValidatingAdmissionPolicy on modern K8s, OPA/Gatekeeper or Kyverno on older clusters) enforce mesh-wide invariants: every data product carries an owner annotation, every namespace carries a domain label, every Deployment has its @@ -192,7 +210,9 @@ Red Hat Universal Base Image (UBI) — `ubi10/python-314-minimal` is the base fo every Python service in the reference. UBI is freely redistributable and enterprise-maintained, which means a domain team gets a base image they can use without licensing constraints and that's patched on a -regular cadence by Red Hat. Language dependencies come from Red Hat's +regular cadence by Red Hat. + +Language dependencies come from Red Hat's verified channels or PyPI, with known provenance; the resulting images are signed and have an SBOM, and admission policies verify both before the image runs. The domain's data product layers its FastAPI / gRPC / @@ -240,3 +260,10 @@ their implementation pieces are *visible together*. Conceptual material gives you the principles; vendor material gives you the pieces. This reference's job is to be the bridge — concrete enough to be runnable, explicit enough about its choices to be portable. + +## For deeper coverage + +- *Data Mesh* by Zhamak Dehghani — the four principles this closing page reorganizes the reference by. +- *Cloud Native Patterns* by Cornelia Davis — the platform patterns (self-serve, federated governance) behind this summary's implementation pieces. +- *Kubernetes Patterns* by Bilgin Ibryam & Roland Huß — the operator and CRD shapes that make each principle self-serve. +- *GitOps and Kubernetes* by Billy Yuen et al. — reproducing the whole mesh from a Git repository, the practice Principle 03's GitOps piece rests on. diff --git a/_docs/11-running-on-openshift-crc.md b/_docs/11-running-on-openshift-crc.md index 062169a..90bd742 100644 --- a/_docs/11-running-on-openshift-crc.md +++ b/_docs/11-running-on-openshift-crc.md @@ -5,6 +5,18 @@ description: "The same seven-service data mesh, redeployed to OpenShift Local: a duration: 40 min --- +## Learning objectives + +By the end of this chapter you can: + +- Explain which Kubernetes assumptions break on OpenShift — SCC/UID handling, ingress, the image registry — and how this chart's design responds to each. +- Size, set up, and start OpenShift Local (CRC), then build and push the seven service images to its integrated registry. +- Deploy the Helm chart, verify the live core through its two Routes, and interpret the expected app-pod crash-loop-then-converge startup pattern. +- Bring up the optional platform tier (service mesh, autoscaling, observability, orchestration, catalog) and read what's actually verified live for each layer. +- Distinguish the two real deploy-time snags this appendix hit (disk pressure, a `printf` quoting bug) from the SCC design, which held on the first try. + +## Why this matters + [Chapter 3]({{ '/docs/02-kubernetes-substrate/' | relative_url }}) built the case for Kubernetes as the mesh's substrate and brought the capstone up on minikube via Helm — NodePorts published to loopback, a local image registry. That's a real deployment, and @@ -30,7 +42,7 @@ that gets it running. One scoping note up front, visible in the chart's own `Chart.yaml`: this is the **live core** of the capstone — the seven services, Postgres, Kafka, and Apicurio — not the full minikube stack. OpenMetadata, Prefect, the LGTM observability stack, KEDA -autoscaling, and the Istio mesh are intentionally out of this chart *by default*. A +autoscaling, and the Istio mesh are out of this chart *by default*. A later section of this appendix, "The full platform tier, live," brings all five of those layers up on top of this same core as an **opt-in** addition — applied via `openshift/platform/install-platform.sh` plus two chart flags @@ -125,7 +137,9 @@ The seven Python services are built for exactly this. Their `services//Cont images already declare `USER 1001:0` — an arbitrary UID in group `0`, with the application's files group-readable and group-writable — which is precisely what `restricted-v2` expects: it doesn't care *which* UID a pod runs as, only that the pod -isn't demanding one it assigned instead. So the fix for the app Deployments is to **drop +isn't demanding one it assigned instead. + +So the fix for the app Deployments is to **drop `runAsUser` entirely** and let OpenShift pick. `templates/app-deployment.yaml` sets `runAsNonRoot: true` and nothing else UID-related — no `runAsUser` key appears anywhere in the pod's `securityContext`. Whatever UID `restricted-v2` assigns, the image already @@ -135,7 +149,9 @@ Postgres and Kafka are the exception, and the chart treats them as one. Their up images hardcode specific non-root UIDs that own their data directories — Postgres as `70`, the Apache Kafka image as `1000` — so dropping `runAsUser` for them would be wrong: whatever UID `restricted-v2` assigned instead wouldn't own `/var/lib/postgresql/data` or -`/var/lib/kafka/data`, and the containers would fail to start. For these two, the chart +`/var/lib/kafka/data`, and the containers would fail to start. + +For these two, the chart grants an SCC that *allows* their UID instead of letting the default one assign a different one. `templates/serviceaccount-infra.yaml` creates a ServiceAccount, `datamesh-infra`, and binds it to the `nonroot-v2` SCC; `postgres.yaml` and `kafka.yaml` @@ -242,7 +258,7 @@ whether it owns a Postgres schema, whether it's a Kafka participant needing a gr shutdown window, whether it needs an Alembic migration init container ahead of the app container, and its resource tier. `templates/app-deployment.yaml` then ranges over that map once, and the interesting part is the conditionals built from it — this is where -"the template is driven by data" earns its keep: +"the template is driven by data" pays off: {% raw %} ```yaml @@ -278,18 +294,18 @@ map once, and the interesting part is the conditionals built from it — this is Every service in this system ships `/health` as its liveness check and `/healthz` as its readiness check — all seven are `probe: http` in practice, but the conditional exists because the chart is carried over from a sibling project where some services shipped -without a health-check library and needed a bare TCP probe instead; it costs nothing to +without a health-check library and needed a bare TCP probe instead. It costs nothing to keep the branch here even though every current service takes the `http` path. -`notification-service` adds one more wrinkle worth calling out: `migrate: true` adds an +`notification-service` adds one more wrinkle: `migrate: true` adds an `initContainer` running `alembic upgrade head` against the same database before the `uvicorn` container starts, so the schema exists before the app issues a single query — no `CREATE TABLE` happens on the hot path. The shared environment contract — the Kafka bootstrap address, the Apicurio URL, the gRPC coordinate for `inventory-service`, the gateway's downstream REST URL — is one -ConfigMap, `templates/configmap-app.yaml`. The one detail worth noting is that every -in-cluster address it builds is namespace-relative: +ConfigMap, `templates/configmap-app.yaml`. Every in-cluster address it builds is +namespace-relative: {% raw %} ```yaml @@ -306,11 +322,15 @@ anything else — without a single hardcoded string to edit. Postgres and Kafka are plain StatefulSets, not operator-managed CRs — `templates/postgres.yaml` and `templates/kafka.yaml` don't assume the CloudNativePG or Strimzi operators are installed on the target cluster, since this is meant to run on a bare CRC instance with -nothing pre-installed beyond OpenShift itself. Kafka runs single-broker KRaft (no +nothing pre-installed beyond OpenShift itself. + +Kafka runs single-broker KRaft (no ZooKeeper), self-referencing its own pod-stable DNS (`kafka-0.kafka..svc...:9093`) as its quorum voter, with `publishNotReadyAddresses: true` on its headless Service — a single-node KRaft broker has to resolve its own controller address to become ready in -the first place, so the Service has to advertise it before it's marked ready. Both +the first place, so the Service has to advertise it before it's marked ready. + +Both StatefulSets run under `datamesh-infra` with `fsGroup` set to match their UID, so their PersistentVolumeClaims are writable from the first mount. @@ -389,7 +409,9 @@ That's the live run, twelve minutes after `helm upgrade --install`: all ten work `shipping-service` are exactly the convergence described above, not a defect — those four happened to lose the race against `postgres-0` reaching `Ready` on this particular run; `graphql-gateway`, `order-service`, and `notification-service` happened -to win it. Re-running the same `helm install` against a fresh `crc start` will likely +to win it. + +Re-running the same `helm install` against a fresh `crc start` will likely distribute the restarts across a different subset of services — which four crash-loop is a function of scheduling order and Postgres's own startup time, not something the chart controls or needs to. @@ -399,10 +421,14 @@ that to be fine. `notification-service`'s migrate init container, and every serv database connection on first boot, depend on Postgres and Kafka already being reachable. A pod that starts before its StatefulSet dependency is `Ready` fails fast — a DNS lookup with no ready endpoints behind it, then connection refused — and restarts on the normal -back-off schedule. As `postgres-0` and `kafka-0` reach `1/1`, each service recovers on +back-off schedule. + +As `postgres-0` and `kafka-0` reach `1/1`, each service recovers on its next attempt. The restart counts you see in `oc get pods` are the scar tissue of that ordering, not a sign anything is actually broken; the end state is all green regardless -of how many restarts it took to get there. (If you want a clean start with no restart +of how many restarts it took to get there. + +(If you want a clean start with no restart churn, install the infra StatefulSets' templates first and wait for them before letting the app Deployments roll out — but the self-healing path above is what a plain `helm install` does, and it converges on its own.) @@ -428,9 +454,9 @@ $ curl -sk https://graphql-gateway-datamesh.apps-crc.testing/healthz And a real query through the gateway spanning two or more of the underlying products — the point of a GraphQL gateway is composing reads across services the caller never talks -to directly, so the query worth running is one that can only succeed if that composition -actually works, for example an order placed through `order-service` with its live stock -level attached from `inventory-service`: +to directly. So the query worth running is one that can only succeed if that composition +actually works — for example, an order placed through `order-service` with its live +stock level attached from `inventory-service`: ```sh curl -sk https://$(oc get route graphql-gateway -n datamesh -o jsonpath='{.spec.host}')/graphql \ @@ -443,7 +469,7 @@ curl -sk https://$(oc get route graphql-gateway -n datamesh -o jsonpath='{.spec. ``` The top-level `order` fields — `customerId`, `itemSku`, `quantity`, `status`, `amount` — -come straight from `order-service`'s own Postgres row; the nested `stock` object is a +come straight from `order-service`'s own Postgres row. The nested `stock` object is a second, independent hop the gateway makes over gRPC to `inventory-service`'s `:50051` port to resolve `quantityOnHand`/`available` for that same SKU. A 200 with both halves populated — the order half and the `stock` half — confirms the Route, the SCC-assigned @@ -472,8 +498,8 @@ platform tier described below. ### What broke on live apply Two things went wrong on the actual run against CRC, neither in the chart's design — -both in the tooling around it — plus one thing worth stating plainly because it did -*not* break: the SCC assignment this appendix spent most of its length justifying. +both in the tooling around it. One thing did *not* break: the SCC assignment this +appendix spent most of its length justifying. **1. All ten pods stuck `Pending` behind a `disk-pressure` taint.** Partway through `helm upgrade --install`, every pod sat `Pending` with a @@ -482,6 +508,7 @@ problem — `oc describe node` wasn't short on memory or CPU. The cause was the push that happens just before deployment: `build-and-push.sh` had already pushed all nine images (seven app images plus the mirrored `postgres`/`kafka` infra images) into the integrated registry, and that registry's storage lives on the CRC node's own disk. + CRC's *default* VM disk is a modest ~32 GB, and nine images pushed into it was enough to cross the kubelet's disk-pressure eviction threshold (27 GB used on a 32 GB disk). Once a node is under `DiskPressure`, the scheduler taints it and nothing new schedules — @@ -507,7 +534,9 @@ final step prints a one-line hint for the `--mirror-infra` flag described earlie this appendix. That hint line is passed to `printf` with a leading `--mirror-infra ...` string, which `printf` parses as an option flag rather than a format string — an exit code `2` with no images actually affected, since the build-and-push loop itself had -already completed and returned before the hint line ran. The fix is the standard +already completed and returned before the hint line ran. + +The fix is the standard `printf`-no-longer-takes-flags-after-this idiom, `printf -- '%s\n' "..."`, which forces everything after `--` to be treated as data. Purely cosmetic: by the time the script exits non-zero, the registry already has all nine images, which `oc get istag -n @@ -517,7 +546,9 @@ datamesh` confirms independent of the script's own exit status. OpenShift" section above — drop `runAsUser` for the seven Python app images and let `restricted-v2` assign a UID, pin `runAsUser` for Postgres and Kafka under a `datamesh-infra` ServiceAccount bound to `nonroot-v2` — held exactly as designed on the -first deploy, no iteration needed. The live SCC/UID table earlier in this appendix is +first deploy, no iteration needed. + +The live SCC/UID table earlier in this appendix is the proof: all seven app pods landed on `restricted-v2` with the same OpenShift-assigned UID, `1000650000` (the images' `USER 1001:0` made them agnostic to whichever UID that turned out to be), while `postgres-0` kept its required `70` and `kafka-0` its required @@ -528,8 +559,10 @@ the admission model this appendix is actually about. ## The full platform tier, live The live core above is the Helm chart's whole job — but this appendix also brings up -the five layers that chart deliberately leaves out, as an **opt-in platform tier** on -top of that same core. None of this changes the core's default behavior: a plain `helm +the five layers that chart leaves out, as an **opt-in platform tier** on +top of that same core. + +None of this changes the core's default behavior: a plain `helm upgrade --install datamesh openshift/helm/datamesh` still renders exactly the chart described above, with no mesh labels and no OTEL environment variables anywhere in it — both `mesh.enabled` and `observability.otlp.enabled` default to `false` in @@ -554,7 +587,9 @@ plus a matching `IstioCNI` CR, not a raw `istioctl install`. Injection is a pod-template **LABEL**, `istio.io/rev`, set on the seven app Deployments (and the order-service canary) behind the chart's `mesh.enabled` flag — Sail's revisioned injection webhook doesn't reliably fire on the classic `sidecar.istio.io/inject` -*annotation* the minikube overlay uses. Postgres, Kafka, and Apicurio stay unlabeled, +*annotation* the minikube overlay uses. + +Postgres, Kafka, and Apicurio stay unlabeled, unmeshed infrastructure, which is exactly what the live sidecar counts show: the seven app pods carry `2/2` (app container plus Envoy), the three infra pods stay `1/1`. @@ -563,7 +598,9 @@ scoped exception: a second `PeerAuthentication`, `graphql-gateway-permissive`, s `PERMISSIVE` for just the gateway. The OpenShift Route fronting the gateway is not mesh-aware and arrives as plaintext, and `STRICT` namespace-wide rejected that edge traffic outright (502s from the Route) before the gateway-scoped exception was carved -out — the live lesson behind this appendix's newest decision entry. Every other hop in +out — the live lesson behind this appendix's newest decision entry. + +Every other hop in the mesh — service to service, all mesh-internal — stays `STRICT`. The v1/v2 canary itself runs as a **mesh-internal** `VirtualService`/`DestinationRule` @@ -610,7 +647,9 @@ directories as UID 0, so its pod needs `runAsUser: 0`, which forces it onto the outright. The non-obvious part: adding a `seccompProfile` or `capabilities` block to that same `securityContext` — the kind of change that reads as pure hardening — makes `anyuid` **reject** the pod instead of admitting it, because its allowed -seccomp-profile list is empty. That rejection doesn't surface as a loud SCC-denial +seccomp-profile list is empty. + +That rejection doesn't surface as a loud SCC-denial event; it silently drops the pod back onto `restricted-v2`'s randomly assigned UID, where Grafana can't write `/data` and the pod hangs forever at "Waiting for Grafana to start up...", with nothing pointing at the real cause. `runAsUser: 0` alone — no @@ -621,7 +660,7 @@ ingesting per-service traces: `graphql-gateway`, `order-service`, `inventory-ser and `notification-service` each produced recent traces during the capture window (`payment-service` produced none in that window). -**Honest limit, stated plainly:** those are per-service traces, not one stitched +**Limit:** those are per-service traces, not one stitched cross-service trace. The gateway's downstream HTTP/gRPC calls don't propagate a W3C `traceparent` header, so each service roots its own trace instead of continuing the caller's — an application-instrumentation gap, not something OpenShift or the mesh @@ -704,7 +743,7 @@ spec: Its symmetric partner on the build side is **OpenShift Pipelines** (Tekton): the `podman build`/`push` loop in `build-and-push.sh` becomes a `Pipeline` triggered on push, so a commit to a service's source produces a new image in the internal registry -without a human running the script. Both are deliberately left as starting points rather +without a human running the script. Both are left as starting points rather than part of the verified deploy in this appendix — each needs its operator installed (OpenShift GitOps, OpenShift Pipelines), and GitOps additionally needs this repo reachable from the cluster. Neither was applied for this appendix's verification pass. @@ -766,7 +805,7 @@ endpoint. The full capture is committed at grant are all in place and simply unproven because the server image never pulled. (`examples/lgtm-datamesh/openshift/evidence/openmetadata-verification.txt`) -**Honest limits, carried forward:** +**Limits carried forward:** - Observability's per-service traces are **not** stitched into one cross-service trace — the services don't propagate a W3C `traceparent` header across their @@ -794,3 +833,12 @@ the assigned app UID will differ per cluster, but the SCC names, the infra UIDs, 200s should not. Re-confirm the platform tier by re-running `install-platform.sh` on top of a resized instance and re-capturing the same evidence files under `examples/lgtm-datamesh/openshift/evidence/`.* + +## For deeper coverage + +- [OpenShift documentation](https://docs.openshift.com/) — Security Context Constraints, Routes, and the integrated image registry this appendix is built around. +- [OpenShift Local (CRC) documentation](https://crc.dev/) — sizing, setup, and the `crc` CLI. +- [Strimzi documentation](https://strimzi.io/documentation/) — the Kafka-on-Kubernetes operator model that AMQ Streams, referenced throughout this appendix, is built on. +- [Istio documentation](https://istio.io/latest/docs/) — mTLS and the `VirtualService`/`DestinationRule` traffic shaping the platform tier's mesh section uses. +- [KEDA documentation](https://keda.sh/docs/) — the upstream API the Custom Metrics Autoscaler operator packages. +- *Platform Engineering on Kubernetes* by Mauricio Salatino — operator-based platform capabilities (OperatorHub/OLM) and SCCs as policy, the pattern this appendix's chart is built around.