HDDS-16288. Replace MutableRate with lock-free ConcurrentMutableRate on OM hot-path metrics

[yandrey321]

What changes were proposed in this pull request?

Problem

OM request-latency metrics are recorded as Hadoop @Metric MutableRate counters. MutableRate extends MutableStat, and MutableStat.add(long) is synchronized(this). On the OM hot path many handler threads record a measurement on the same metric instance, so they serialize on that per-counter monitor — a lock convoy that appears under concurrent load. A single getKeyInfo touches 6–7 of these counters in sequence, so the effect compounds on the read path.

HDDS-9377 (already merged) introduced the lock-free ConcurrentMutableStat (striped LongAdder/LongAccumulator, drained lazily at snapshot) and applied it to OMLockMetrics and the PerformanceMetrics wrapper. The remaining MutableRate counters in OMPerformanceMetrics and KeyLifecycleServiceMetrics are the gap this PR closes, scoped deliberately to OM hot-path classes only.

Approach

New ConcurrentMutableRate (hadoop-hdds/common) — a lock-free counterpart of Hadoop's MutableRate, extending ConcurrentMutableStat with a public constructor that fixes sampleName="Ops" / valueName="Time". Hadoop's MutableRate constructor is package-private and only instantiable reflectively by the @Metric factory; this public ctor lets OM metric sources build it directly. Semantics and emitted names (NumOps / AvgTime) are identical.

OMPerformanceMetrics — converted all 44 @Metric MutableRate latency counters to ConcurrentMutableRate. Because @Metric fields are instantiated reflectively (the factory always builds a real MutableRate, so a field cannot simply be retyped), the class is reworked into a hand-rolled MetricsSource following the OMLockMetrics / S3GatewayMetrics template: stats built in the constructor, getMetrics() snapshots all 44 stats plus the 6 surviving @Metric gauges. The single-writer gauges (listKeysOpsPerSec + 5 *ServiceLatencyMs) stay @Metric.

KeyLifecycleServiceMetrics — taskLatencyMs (written concurrently by the lifecycle BackgroundService pool, per bucket) converted to ConcurrentMutableRate; the class becomes a hand-rolled MetricsSource. Its 13 MutableGaugeLong gauges remain @Metric.

MetricUtil.captureLatencyNs — the two overload parameters widened from MutableRate to its superclass MutableStat. The body already only calls add(long). This is source-compatible (MutableRate and ConcurrentMutableStat both extend MutableStat) and lets all existing captureLatencyNs(getter(), block) call sites in ozone-manager compile unchanged.

Metric-name compatibility

All emitted metric names are preserved byte-identically: the capitalized field name plus extended=false reproduces exactly the NumOps / AvgTime pairs the @Metric factory generated. No dashboard or alert names change. Caveat: standard deviation is slightly underestimated under concurrent batched adds (documented on ConcurrentMutableStat, inherited here) — acceptable for these latency stats.

Not converted (deliberately)

OzoneManagerDoubleBufferMetrics (flushTime, queueSize) — written only by the single OMDoubleBufferFlushThread daemon, so there is no cross-thread contention and lock-free would only add churn. The MutableGaugeLong/Float gauges throughout are single-writer and left as @Metric.

The genuinely-contended per-RPC MutableRate counters in the shared ipc_ fork (RpcMetrics) are out of OM scope and tracked separately (HDDS-16304).

Generated-by: Claude Code (Opus 4.8)

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16288

How was this patch tested?

CI: https://github.com/yandrey321/ozone/actions/runs/33043941610/job/98425998600

Unit tests
Integration tests