Skip to content

feat(component): emit per-resource apply metrics keyed by a stable id - #182

Merged
sourcehawk merged 9 commits into
mainfrom
feat/resource-apply-metrics
Aug 23, 2026
Merged

feat(component): emit per-resource apply metrics keyed by a stable id#182
sourcehawk merged 9 commits into
mainfrom
feat/resource-apply-metrics

Conversation

@sourcehawk

@sourcehawk sourcehawk commented Aug 16, 2026

Copy link
Copy Markdown
Owner

Description

Closes #181

A resource that keeps failing to apply, or is rewritten on every reconcile, has no metric signal today: the only trace
is the owner's event stream, which client-go's spam filter truncates within seconds under exactly those conditions.
#179 was invisible in a running cluster for that reason and surfaced only through envtest event counts. This adds two
per-resource counters, ocf_resource_apply_total and ocf_resource_apply_errors_total, a stable low-cardinality
identifier to key them by, and a pkg/metrics implementation so consumers get working metrics without writing
collector plumbing.

This is a breaking change: component.MetricsRecorder grew two methods, so a bare ocm.ConditionMetricRecorder no
longer satisfies it. The fix is one line at each call site, swapping in metrics.NewRecorder(controller, gauge, collectors), or adding the two methods to a hand-written recorder.

Changes

  • ocf_resource_apply_total{controller, owner_kind, component, resource, kind, operation} counts every framework apply
    of a managed resource, classified created / updated / none. A resource whose updated rate never settles to
    zero is the applyResource reports Updated on every reconcile when components are rebuilt per reconcile #179 smell.
  • ocf_resource_apply_errors_total{controller, owner_kind, component, resource, kind} counts failed applies. Exactly
    one of the two counters moves per attempt.
  • WithMetricsIdentifier on every primitive builder (21 typed, 4 unstructured), the generic builders, and the scaffold
    templates. It sets the resource label; when unset the framework labels the resource by its lowercased kind, so the
    counters work with no configuration. Build rejects a blank identifier.
  • pkg/metrics implements component.MetricsRecorder: shared Collectors registered once per process, plus a
    Recorder per controller. Both halves are independently optional, so a nil gauge or nil collectors disables that
    family rather than panicking mid-reconcile. RemoveConditionsFor is delegated so condition cleanup is unaffected.
  • MetricsRecorder gained RecordResourceApply and RecordResourceApplyError, taking a ResourceMetricLabels struct
    rather than four positional strings, so adding a label later is not a signature break.
  • Emission covers the reconcile path and the suspension path, both of which share applyResource. Read-only resources,
    deletions and orphans record nothing.

Challenges

The series carry no owner name or namespace, so their count is bounded by the operator's static topology rather than by
how many custom resources exist. That is what makes the identifier's cardinality contract load-bearing, and why there
is deliberately no resource-metric counterpart to RemoveConditionsFor: nothing accumulates per owner to reap, and
deleting a counter series mid-flight reads downstream as a counter reset that corrupts rate().

Making the object's kind available for the labels meant resolving the GVK at the top of applyResource instead of just
before the patch. That silently dropped a guarantee the old call site provided: a Mutate that assigns the whole
struct (*current = *desired) clears TypeMeta, and SSA then fails with "unstructured object has no kind". The call
is restored at its original position too, with a test that fails without it.

Related

Testing

make all passes, including lint, the scaffold golden and end-to-end suites, and the examples.

Unit tests in pkg/component cover the emission points against a spy recorder: one apply per pass with the right
operation and labels, the kind default, an explicit identifier, the fallback when a resource returns an empty one, the
error path via a client that fails every patch, silence when Metrics is nil, and silence for a read-only resource.
pkg/metrics asserts the rendered series through prometheus/testutil, including operation lowercasing and two
controllers sharing one set of collectors.

The envtest spec is the one that matters most: it reconciles a rebuilt desired object five times against a real API
server and asserts none keeps climbing while updated stays at its post-change value of 1. That is #179 expressed as
a metric instead of an event count.

The GVK regression above was verified by removing the fix and confirming the new test fails with the SSA error, then
restoring it.

go run ./examples/custom-resource prints the counters after two reconciles, showing operation="created" once and
operation="none" once.

sourcehawk and others added 8 commits August 16, 2026 19:56
@sourcehawk @claude
feat(generic): add a stable metrics identifier to resource builders
4295000 
Resources had no low-cardinality identity. Identity() embeds the Kubernetes
name, so any per-resource metric keyed by it is unbounded for generated or
per-CR names, and useless for asking which resource type is misbehaving.

WithMetricsIdentifier records a stable label value on the resource, read by
the framework through the new concepts.MetricsIdentifiable capability. The
value is a Prometheus label, not a Kubernetes name: Build rejects a blank one,
and omitting the call leaves the framework free to apply its own default.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@sourcehawk @claude
feat(metrics): add resource-level recorders and a Prometheus implemen…
619cc28 
…tation

MetricsRecorder now covers resource applies as well as conditions, carrying
the label set in a struct so a later label is not a signature break and four
same-typed arguments cannot be transposed at a call site.

The new metrics package implements it: ocf_resource_apply_total and
ocf_resource_apply_errors_total, keyed only by the operator's static topology
so no series accumulates per owner. Both halves are independently optional,
and the condition recorder is held rather than embedded so a missing gauge
skips recording instead of panicking mid-reconcile.

BREAKING CHANGE: a bare ocm.ConditionMetricRecorder no longer satisfies
component.MetricsRecorder. Use metrics.NewRecorder, or add the two methods to
a hand-written recorder.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@sourcehawk @claude
feat(component): record apply and apply-error metrics per resource
92a9eb9 
applyResource now emits one apply counter per pass, carrying the same
converging operation as the apply event, and one error counter on every
failure. Events were the only trace of a churning or failing resource, and
client-go's spam filter truncates those within seconds under exactly the
conditions worth watching.

The GVK is resolved at the top of the function rather than just before the
patch, so the object's kind labels every path. Errors that happen before it,
where the framework never worked out what to apply, emit nothing rather than
an unlabelled series.

Closes #181

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@sourcehawk @claude
feat(primitives): expose WithMetricsIdentifier on every builder
94c1bde 
Each primitive builder re-exposes the generic option in the fluent style, and
each primitive resource delegates MetricsIdentifier so the framework's
capability assertion finds it. Without the delegation the identifier would be
accepted at build time and silently ignored at apply time.

The scaffolding templates carry both, so a generated wrapper is labelled the
same way a built-in primitive is.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@sourcehawk @claude
test(component): assert apply metrics settle for a rebuilt resource
4fd24fe 
Reconciles a rebuilt desired object against envtest and asserts that the none
counter keeps growing while updated stays at its post-change value. This is
the #179 regression expressed as a metric rather than an event count, which is
the point of the metric: events are gone from a real cluster within seconds
under exactly these conditions.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@sourcehawk @claude
docs(component): document per-resource apply metrics
96a97bb 
Adds a Metrics section covering both metric families, the series and their
labels, the reading that catches a churning resource, and the cardinality
contract the resource identifier has to honour. The primitives, custom-resource
and CLI pages pick up the builder option and the delegation it needs.

The custom-resource example wires the collectors and prints the counters after
its two reconciles. Its CertificateRequest is named after the owner, which
makes it a worked case for why the identifier is a constant rather than the
object's name.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@sourcehawk @claude
fix(component): re-assert the GVK after mutating the desired object
8c8cd48 
Hoisting ensureGVK to the top of applyResource, so the object's kind could
label the metrics, removed the guarantee it used to provide. A Mutate that
assigns the whole struct (*current = *desired) drops the TypeMeta, and
Server-Side Apply then fails with "unstructured object has no kind".

The call is restored at its original position as well. It is a no-op whenever
the kind is still present, and the test covering it fails without it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@sourcehawk @claude
docs(component): state exactly what the apply error counter covers
b0c64e0 
Exactly one of the two counters moves per apply attempt. Spelling that out,
along with the one case where neither does, saves the reader inferring the rule
from the emission points.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings August 16, 2026 19:07
Copilot AI reviewed Aug 16, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds first-class Prometheus resource-apply metrics to the Operator Component Framework, including a stable low-cardinality identifier to key per-resource series, and ships a default pkg/metrics implementation so consumers can enable metrics without writing custom collector plumbing. This extends the public component.MetricsRecorder interface (breaking change) and threads metric emission through the shared applyResource path.

Changes:

  • Extend component.MetricsRecorder with resource-apply counters (RecordResourceApply / RecordResourceApplyError) using a ResourceMetricLabels struct.
  • Add WithMetricsIdentifier plumbing across generic + primitive builders/resources (and scaffolding/templates) via concepts.MetricsIdentifiable.
  • Introduce pkg/metrics (Prometheus recorder + collectors), update docs/plugin references/examples/e2e to wire it, and add unit/envtest coverage for emission and identifier defaults.

Reviewed changes

Copilot reviewed 105 out of 105 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
plugin/skills/using-primitives/references/primitives.md Document WithMetricsIdentifier for primitives (plugin reference copy).
plugin/skills/custom-resource-wrappers/references/custom-resource.md Document forwarding MetricsIdentifier in custom wrappers (plugin reference copy).
plugin/skills/building-components/references/component.md Document new metrics families + wiring pkg/metrics (plugin reference copy).
pkg/primitives/unstructured/workload/resource.go Expose MetricsIdentifier on unstructured workload resource.
pkg/primitives/unstructured/workload/builder.go Add WithMetricsIdentifier to unstructured workload builder.
pkg/primitives/unstructured/task/resource.go Expose MetricsIdentifier on unstructured task resource.
pkg/primitives/unstructured/task/builder.go Add WithMetricsIdentifier to unstructured task builder.
pkg/primitives/unstructured/static/resource.go Expose MetricsIdentifier on unstructured static resource.
pkg/primitives/unstructured/static/builder.go Add WithMetricsIdentifier to unstructured static builder.
pkg/primitives/unstructured/integration/resource.go Expose MetricsIdentifier on unstructured integration resource.
pkg/primitives/unstructured/integration/metrics_identifier_test.go Test identifier exposure/validation for unstructured integration builder.
pkg/primitives/unstructured/integration/builder.go Add WithMetricsIdentifier to unstructured integration builder.
pkg/primitives/statefulset/resource.go Expose MetricsIdentifier on StatefulSet primitive resource.
pkg/primitives/statefulset/builder.go Add WithMetricsIdentifier to StatefulSet primitive builder.
pkg/primitives/serviceaccount/resource.go Expose MetricsIdentifier on ServiceAccount primitive resource.
pkg/primitives/serviceaccount/builder.go Add WithMetricsIdentifier to ServiceAccount primitive builder.
pkg/primitives/service/resource.go Expose MetricsIdentifier on Service primitive resource.
pkg/primitives/service/builder.go Add WithMetricsIdentifier to Service primitive builder.
pkg/primitives/secret/resource.go Expose MetricsIdentifier on Secret primitive resource.
pkg/primitives/secret/builder.go Add WithMetricsIdentifier to Secret primitive builder.
pkg/primitives/rolebinding/resource.go Expose MetricsIdentifier on RoleBinding primitive resource.
pkg/primitives/rolebinding/builder.go Add WithMetricsIdentifier to RoleBinding primitive builder.
pkg/primitives/role/resource.go Expose MetricsIdentifier on Role primitive resource.
pkg/primitives/role/builder.go Add WithMetricsIdentifier to Role primitive builder.
pkg/primitives/replicaset/resource.go Expose MetricsIdentifier on ReplicaSet primitive resource.
pkg/primitives/replicaset/builder.go Add WithMetricsIdentifier to ReplicaSet primitive builder.
pkg/primitives/pvc/resource.go Expose MetricsIdentifier on PVC primitive resource.
pkg/primitives/pvc/builder.go Add WithMetricsIdentifier to PVC primitive builder.
pkg/primitives/pv/resource.go Expose MetricsIdentifier on PV primitive resource.
pkg/primitives/pv/builder.go Add WithMetricsIdentifier to PV primitive builder.
pkg/primitives/pod/resource.go Expose MetricsIdentifier on Pod primitive resource.
pkg/primitives/pod/builder.go Add WithMetricsIdentifier to Pod primitive builder.
pkg/primitives/pdb/resource.go Expose MetricsIdentifier on PDB primitive resource.
pkg/primitives/pdb/builder.go Add WithMetricsIdentifier to PDB primitive builder.
pkg/primitives/networkpolicy/resource.go Expose MetricsIdentifier on NetworkPolicy primitive resource.
pkg/primitives/networkpolicy/builder.go Add WithMetricsIdentifier to NetworkPolicy primitive builder.
pkg/primitives/job/resource.go Expose MetricsIdentifier on Job primitive resource.
pkg/primitives/job/metrics_identifier_test.go Test identifier exposure/validation for task primitives (Job example).
pkg/primitives/job/builder.go Add WithMetricsIdentifier to Job primitive builder.
pkg/primitives/ingress/resource.go Expose MetricsIdentifier on Ingress primitive resource.
pkg/primitives/ingress/builder.go Add WithMetricsIdentifier to Ingress primitive builder.
pkg/primitives/hpa/resource.go Expose MetricsIdentifier on HPA primitive resource.
pkg/primitives/hpa/builder.go Add WithMetricsIdentifier to HPA primitive builder.
pkg/primitives/deployment/resource.go Expose MetricsIdentifier on Deployment primitive resource.
pkg/primitives/deployment/metrics_identifier_test.go Test identifier exposure/validation for workload primitives (Deployment example).
pkg/primitives/deployment/builder.go Add WithMetricsIdentifier to Deployment primitive builder.
pkg/primitives/daemonset/resource.go Expose MetricsIdentifier on DaemonSet primitive resource.
pkg/primitives/daemonset/builder.go Add WithMetricsIdentifier to DaemonSet primitive builder.
pkg/primitives/cronjob/resource.go Expose MetricsIdentifier on CronJob primitive resource.
pkg/primitives/cronjob/builder.go Add WithMetricsIdentifier to CronJob primitive builder.
pkg/primitives/configmap/resource.go Expose MetricsIdentifier on ConfigMap primitive resource.
pkg/primitives/configmap/builder.go Add WithMetricsIdentifier to ConfigMap primitive builder.
pkg/primitives/configmap/builder_test.go Add ConfigMap test coverage for identifier exposure/validation.
pkg/primitives/clusterrolebinding/resource.go Expose MetricsIdentifier on ClusterRoleBinding primitive resource.
pkg/primitives/clusterrolebinding/builder.go Add WithMetricsIdentifier to ClusterRoleBinding primitive builder.
pkg/primitives/clusterrole/resource.go Expose MetricsIdentifier on ClusterRole primitive resource.
pkg/primitives/clusterrole/builder.go Add WithMetricsIdentifier to ClusterRole primitive builder.
pkg/metrics/metrics.go Add Prometheus Collectors + per-controller Recorder implementation.
pkg/metrics/metrics_test.go Validate rendered series and shared-collectors behavior via testutil.
pkg/generic/resource_base.go Add metrics identifier storage + MetricsIdentifier() to generic base resource.
pkg/generic/metrics_identifier_test.go Test default/override/blank-rejection for generic builder/base resource.
pkg/generic/builder_workload.go Forward WithMetricsIdentifier on generic workload builder.
pkg/generic/builder_task.go Forward WithMetricsIdentifier on generic task builder.
pkg/generic/builder_static.go Forward WithMetricsIdentifier on generic static builder.
pkg/generic/builder_integration.go Forward WithMetricsIdentifier on generic integration builder.
pkg/generic/builder_base.go Implement WithMetricsIdentifier + validation for blank identifiers.
pkg/component/zz_setup_test.go Add spyMetrics recorder for resource-metrics assertions in tests.
pkg/component/suite_test.go Switch test reconcile contexts to use spyMetrics.
pkg/component/create.go Emit apply metrics, add label builder, and record error counts on failures.
pkg/component/create_test.go Add regression test ensuring GVK reassertion after mutators clear TypeMeta.
pkg/component/create_metrics_test.go Add unit tests asserting correct metric emission points/labels/error paths.
pkg/component/conditions_test.go Update mocks to satisfy extended MetricsRecorder interface.
pkg/component/concepts/metrics.go Introduce concepts.MetricsIdentifiable interface.
pkg/component/component.go Extend metrics interface and define ResourceMetricLabels.
pkg/component/component_test.go Add envtest asserting updated stops increasing and labels are correct.
internal/scaffold/testdata/golden/workload/resource.go.golden Scaffold golden: include MetricsIdentifier plumbing (workload).
internal/scaffold/testdata/golden/workload/builder.go.golden Scaffold golden: include WithMetricsIdentifier (workload).
internal/scaffold/testdata/golden/task/resource.go.golden Scaffold golden: include MetricsIdentifier plumbing (task).
internal/scaffold/testdata/golden/task/builder.go.golden Scaffold golden: include WithMetricsIdentifier (task).
internal/scaffold/testdata/golden/static/resource.go.golden Scaffold golden: include MetricsIdentifier plumbing (static).
internal/scaffold/testdata/golden/static/builder.go.golden Scaffold golden: include WithMetricsIdentifier (static).
internal/scaffold/testdata/golden/static-cluster-scoped/resource.go.golden Scaffold golden: include MetricsIdentifier plumbing (cluster-scoped static).
internal/scaffold/testdata/golden/static-cluster-scoped/builder.go.golden Scaffold golden: include WithMetricsIdentifier (cluster-scoped static).
internal/scaffold/testdata/golden/integration/resource.go.golden Scaffold golden: include MetricsIdentifier plumbing (integration).
internal/scaffold/testdata/golden/integration/builder.go.golden Scaffold golden: include WithMetricsIdentifier (integration).
internal/scaffold/templates/resource.go.tmpl Scaffold template: add MetricsIdentifier method + interface assertion.
internal/scaffold/templates/builder.go.tmpl Scaffold template: add WithMetricsIdentifier method.
internal/scaffold/data.go Add derived LowercaseKind used by templates/docs.
go.mod Promote prometheus/client_golang to direct dependency.
examples/mutations-and-gating/main.go Wire metrics.NewRecorder in example controller.
examples/grace-inconsistency/main.go Wire metrics.NewRecorder in example controller.
examples/extraction-and-guards/main.go Wire metrics.NewRecorder in example controller.
examples/custom-resource/resources/certificate.go Demonstrate constant WithMetricsIdentifier for per-owner-named resource.
examples/custom-resource/README.md Document resource counters + why identifier must not be per-owner.
examples/custom-resource/main.go Register collectors and print apply counters from a local registry.
examples/component-prerequisites/main.go Wire metrics.NewRecorder in example controller.
e2e/primitives/suite_test.go Use metrics.NewRecorder in E2E reconcilers.
e2e/component/suite_test.go Use metrics.NewRecorder in E2E reconcilers.
docs/primitives.md Document WithMetricsIdentifier for primitives.
docs/getting-started.md Update getting-started table to reflect condition + resource metrics.
docs/custom-resource.md Document wrapper forwarding of MetricsIdentifier + builder API.
docs/component.md Add full Metrics section including series/labels/identifier contract.
docs/cli.md Update CLI docs to include WithMetricsIdentifier + MetricsIdentifier.
.github/copilot-instructions.md Add pkg/metrics to the “Source to read” list.
.ai/base.md Add pkg/metrics to the “Source to read” list (AI base doc).

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread pkg/generic/resource_base.go Outdated
Comment thread pkg/component/create.go
@sourcehawk @claude
fix(generic): unexport the metrics identifier field and reject blank …
bcbfdd5 
…ones

Two review findings. The BaseResource field was exported and abbreviated,
which reads badly on a public struct; nothing outside pkg/generic touches it,
and WithMetricsIdentifier is the only supported way to set it, so it is now
unexported. Renaming it to MetricsIdentifier was not an option: Go forbids a
field and a method sharing a name, and that name belongs to the accessor.

The label builder also treated any non-empty identifier as configured, so a
hand-written concepts.MetricsIdentifiable returning " " reached the series as
its resource label. Blank now falls back to the kind, matching what the
builders reject at build time. Non-blank values are still taken verbatim,
since rewriting one would silently split or merge series.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings August 16, 2026 19:13
Copilot AI reviewed Aug 16, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 105 out of 105 changed files in this pull request and generated no new comments.

@sourcehawk
sourcehawk merged commit bb30997 into main Aug 23, 2026
7 checks passed
@sourcehawk
sourcehawk deleted the feat/resource-apply-metrics branch August 23, 2026 18:05
This was referenced Aug 23, 2026
Closed
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Emit per-resource apply and status metrics keyed by a stable resource identifier

2 participants

@sourcehawk