[DylanTinianov]

Adds resourcemanager.UsageFaultInjection, a deterministic sampler applied inside the resource manager's usage emit path so the billing reconcilers can measure their own recall against known faults. The zero value is off; this is for test environments only.

Behaviour

• Drop and perturb run together on two independent samples (DropRate, PerturbRate), keyed apart on the capability event id (drop|<id>, perturb|<id>); an event in both is dropped. FaultSelected(rate, key) hashes the key with FNV-1a 64, applies the murmur3 fmix64 finalizer so near-identical ids spread out, and takes the top 53 bits as a fraction below rate. Every node of a DON therefore selects the same events, and a reconciler that knows rate can recompute the sample without any shared table. Only records of the orgs in Orgs (our canaries) are ever faulted, an unresolved org is never faulted, and a rate without an allowlist fails validation, so the injector can run in production without touching a customer.
• drop: the selected usage record is not emitted on any node. The producer's own "Emitted capability usage meter record" line still fires, which is what lets the emit reconciler score the record as emitted-but-unbilled.
• perturb: Delta is added to the value on the nodes selected by FaultSelected(nodeRate, nodeID+"|"+eventID); NodeRate 1 (or 0) means every node. Delta must be negative and the result is clamped at zero, so a leak past the quorum median can only under-bill. A node rate below 1 produces node disagreement for the data reconciler; a node rate of 1 produces a wrong settled value that only the on-chain check can see.
• Every applied fault logs Injected usage fault with mode, resourceType, eventID, resourceId, workflowID, executionID, orgID, nodeID, value and (perturb) originalValue. Durable delta and snapshot records are never faulted.

Plumbing

ResourceManagerConfig.FaultInjection carries the settings. loop.EnvConfig gains MeterFaultOrgs, MeterFaultDropRate, MeterFaultPerturbRate, MeterFaultDelta, MeterFaultNodeRate (env CL_METER_FAULT_*), round-tripped by AsCmdEnv/parse and mapped by EnvConfig.MeteringConfig, so LOOP capability producers that build their resource manager from the metering config get the fault with no code change. UsageFaultInjection.Validate is exposed for the host config layer.

Tested locally

Beyond the unit tests: a Local CRE stack whose node image carried this change and the host TOML plumbing, with perturb on half the workflow DON. Compute records from the engine and gas records from the EVM write capability (configured through loop.EnvConfig.MeteringConfig) carried the delta on exactly the nodes the sampler selected, each applied fault logged Injected usage fault, and a reconciler recomputing the sample from the same rates identified the same records.

[github-actions]

⚠️ API Diff Results - github.com/smartcontractkit/chainlink-common

⚠️ Breaking Changes (2)

pkg/resourcemanager (2)

• Config — Old is comparable, new is not

• ResourceManagerConfig — Old is comparable, new is not

✅ Compatible Changes (11)

pkg/loop.EnvConfig (5)

• MeterFaultDelta — ➕ Added

• MeterFaultDropRate — ➕ Added

• MeterFaultNodeRate — ➕ Added

• MeterFaultOrgs — ➕ Added

• MeterFaultPerturbRate — ➕ Added

pkg/resourcemanager (4)

• FaultModeDrop — ➕ Added

• FaultModePerturb — ➕ Added

• FaultSelected — ➕ Added

• UsageFaultInjection — ➕ Added

pkg/resourcemanager.Config (1)

• FaultInjection — ➕ Added

pkg/resourcemanager.ResourceManagerConfig (1)

• FaultInjection — ➕ Added

---

📄 View full apidiff report

[Copilot]

🟡 Changes recommended

The LOOP configuration initializer does not compile, and rate validation and parsing can silently produce unintended fault selection.

2 open issues

• Fix composite literal for embedded ResourceManagerConfig · New
• Reject NaN values in fault injection rate validation · New

What changed in this PR

Adds deterministic, test-only usage fault injection so billing reconcilers can measure recall against reproducible dropped or altered records.

Changes:

• Adds event and node sampling, drop/perturb modes, validation, and fault logging.
• Connects fault settings to the usage emit path and LOOP environment configuration.
• Adds tests for sampling, validation, perturbations, drops, and durable-delta exclusion.

File	Description
pkg/​resourcemanager/​resourcemanager.go	Stores fault settings and applies them to usage emission.
pkg/​resourcemanager/​fault_injection.go	Implements deterministic selection, validation, and fault application.
pkg/​resourcemanager/​fault_injection_test.go	Tests sampling and fault behavior.
pkg/​loop/​metering_config.go	Maps LOOP fault settings into resource manager configuration.
pkg/​loop/​config.go	Adds fault environment variables, serialization, and parsing.

_{Review effort: Balanced}