Repository navigation
fix(monitoring): probe timeout, dial timeouts, no-hash causes and maybe-broadcast alert - #172
Merged
Merged
Conversation
…be-broadcast alert - dashboard-node: the probe-collection text names the 1 s probe timeout (decdn#2533); the DHT panel adds decdn_dht_lookup_dial_timeouts_total beside the round timeouts (decdn#2532). - Buyer-pool unresolved panel and DecdnBuyerTopupUnresolved comment name the already known / nonce too low / underpriced no-hash answers. - New DecdnRedemptionMaybeBroadcast on decdn_onchain_tx_maybe_broadcast_total (decdn#2574), with a promtool test; the outcome panel adds the series and DecdnRedemptionFailing points at the new rule. Closes #163, closes #164, closes #169, closes #170 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟢 Approval recommended
No blocking issues were identified, and the new alert remains silent when its metric is absent on older nodes.
No open issues 🎉
What changed in this PR
Updates deCDN monitoring to expose additional transaction and DHT failure signals and clarify operator guidance.
Changes:
- Adds a maybe-broadcast redemption alert, dashboard series, and threshold tests.
- Adds DHT dial timeouts and corrects probe-timeout guidance.
- Documents additional no-hash causes for unresolved buyer transactions.
| File | Description |
|---|---|
| monitoring/decdn-node/prometheus-alerts.yml | Adds redemption alert and updates explanatory text. |
| monitoring/decdn-node/prometheus-alerts_test.yml | Tests firing and below-threshold behavior. |
| monitoring/decdn-node/dashboard-node.json | Adds dial timeouts and corrects probe guidance. |
| monitoring/decdn-node/dashboard-chain.json | Adds maybe-broadcast outcomes and clarifies unresolved transactions. |
| .claude/skills/grafana-dashboards/SKILL.md | Updates the documented alert count. |
Review effort: Balanced
- Buyer-pool texts and the topUp rule comment: the three no-hash answers apply from the first decdn release after v0.2.0 (decdn#2560/#2563/#2571 are not in v0.2.0); the panel names the seller-redemption nonce case. - Drop the duplicate dial-timeout series from "DHT republish" and describe it in "DHT discovery and publishing". - Redemption failures panel, outcome panel and the grace-close alert point at maybe broadcast / DecdnRedemptionMaybeBroadcast; the rule's description names the seller/buyer collision and the extrapolated threshold. - Stronger promtool case: pins the for, the threshold on a running counter and a one-off burst; tested-rule lists updated; SKILL.md rule 4 carves out the unreleased counter until the pin leaves 0.2. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Batch fix for the open node-monitoring issues.
decdn_dht_lookup_dial_timeouts_totalnext to the round timeouts (in v0.2.0). It usesratelike the panel's other targets.DecdnBuyerTopupUnresolvedcomment list thealready known/nonce too low/replacement transaction underpricedno-hash answers, following the ADR appendix rows.DecdnRedemptionMaybeBroadcastrule:increase(...[30m]) > 2,for: 30m, labelledpayments/warning. It has norunbook_urlbecause the upstream runbook has no section for it.maybe broadcast, so the six-counter family adds up again.DecdnRedemptionFailingnow names the new rule.The
maybe_broadcastcounter (decdn#2574) is merged but not released: v0.2.0 does not export it. Until a release does, the rule and the panel series are silent, which #170 allows.Checks run:
make lint-helm(53 rules, promtool unit tests pass) andmake lint. The live Grafana copies are not republished.Closes #163, closes #164, closes #169, closes #170
🤖 Generated with Claude Code