Repository navigation
fix(monitoring): follow decdn v0.1.0 buyer metrics, quiet the registry-resync alert (#125) - #162
Merged
Merged
Conversation
…y-resync alert (#125) decdn v0.1.0 (decdn/decdn#2406) removed decdn_voucher_nonce_gaps_total and widened the pool-open and lane-seed counters: - chain dashboard: drop the nonce-gap series (the panel is now "Lane and receipt-log writes"); name the pool-open aggregate's real causes (store, open-task, capability signing, lane seed) and the four lane-seed causes. - delivery dashboard: pool open failures are gas, RPC, buyer store or lane seed, not an expired pool. - DecdnBuyerLaneSeedFailing: "Buyer lane seeds refused", all four causes. Alerts and panels that misled on the live fleet: - DecdnRpcErrorRatioHigh and "RPC failures by outcome": rpc_error on eth_getLogs is mostly a provider's transient error the poller retries in place, not a range cap (that has its own counter). Text only. - DecdnCapacityBondRegistryResyncFailing fires on two failed attempts in the hour (>= 2), not one transient eth_call error; promtool tests added. - Node drilldown "Peers known" gates on max_over_time of pull attempts, so a restart no longer blanks it. Closes #125 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟢 Approval recommended
The focused changes have no blocking findings and include regression coverage for the revised alert threshold.
1 open finding
What changed in this PR
Aligns deCDN monitoring with v0.1.0 buyer metrics and reduces transient registry-resync alert noise.
Changes:
- Removes the obsolete nonce-gap series and updates buyer/RPC explanations.
- Requires two registry-resync failures per hour, with regression tests.
- Makes the peer-count display tolerate counter resets after restarts.
| File | Description |
|---|---|
| monitoring/README.md | Documents expanded alert tests. |
| monitoring/decdn-node/prometheus-alerts.yml | Adjusts resync threshold and alert explanations. |
| monitoring/decdn-node/prometheus-alerts_test.yml | Tests resync firing and recovery. |
| monitoring/decdn-node/dashboard-node.json | Adds historical gating for peer counts. |
| monitoring/decdn-node/dashboard-delivery.json | Updates pool-failure explanations. |
| monitoring/decdn-node/dashboard-chain.json | Removes obsolete metric and updates panel text. |
| .claude/skills/grafana-dashboards/SKILL.md | Documents resync test coverage. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…not a recent pull Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
thiras
force-pushed
the
fix/monitoring-v0.1
branch
from
October 14, 2026 22:20
1568d53 to
93ea508
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Checked against decdn v0.1.0 (which carries decdn/decdn#2406) and the live testnet fleet.
#125
decdn_voucher_nonce_gaps_total, which v0.1.0 no longer exports. I dropped that series and renamed the panel "Lane and receipt-log writes". "Pool-open failures by cause" now names the aggregate's real extra causes (store, open-task, capability signing, lane seed). "Buyer ledger integrity" names the four lane-seed causes.DecdnBuyerLaneSeedFailingsummary is now "Buyer lane seeds refused on …", and the description lists all four causes.Firing on noise in the fleet
DecdnRpcErrorRatioHigh(firing on all 8 nodes): the provider answers about a third ofeth_getLogscalls with "error code 19: Temporary internal error". The poller retries these in place by design. The annotation and the "RPC failures by outcome" panel said rpc_error "usually means a range cap", which is wrong: a range cap has its own counter, and it is at 0 fleet-wide. This PR changes the text only; the expression is unchanged.DecdnCapacityBondRegistryResyncFailing(firing on nbg1): it fired on one failed attempt of a 15-minute resync. It now fires atincrease(...[1h]) >= 2. I added promtool tests: one failure does not fire, two do. The test fails against the old> 0rule. The slash-resync and blacklist re-enumeration rules stay at> 0because those are slash exposure.max_over_time(...[1d]).The other 395 panel queries ran clean against the fleet. After
make lint-helmandmake lint, the metric-name check against v0.1.0 finds no unexported name.After merge: republish the dashboards and Grafana-managed rules in the deCDN folder. Those copies still predate #128.
Closes #125
🤖 Generated with Claude Code