Skip to content

fix(monitoring): probe timeout, dial timeouts, no-hash causes and maybe-broadcast alert #172

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
thiras merged 2 commits into from
Oct 21, 2026
Merged

fix(monitoring): probe timeout, dial timeouts, no-hash causes and maybe-broadcast alert #172

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
96134c7
fix(monitoring): probe timeout, dial timeouts, no-hash causes and may…
thiras Oct 21, 2026
d693a92
fix(monitoring): address review of the batch fix
thiras Oct 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 9 additions & 4 deletions .claude/skills/grafana-dashboards/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,9 +39,9 @@ the iroh relay's in `monitoring/iroh-relay/`):
process, iroh transport, DHT and probe, logs, traces.
- `prometheus-alerts.yml` — three groups: `decdn-slash-safety`, `decdn-liveness`,
`decdn-delivery`. Every rule carries a `component` label. Only rules with a matching
section in upstream's `docs/runbook.md` (20 of 51) also carry a `runbook_url`.
`DecdnNodeDown` and `DecdnCapacityBondRegistryResyncFailing` have promtool unit
tests in `prometheus-alerts_test.yml` (the chart's `.helmignore` keeps that file
section in upstream's `docs/runbook.md` (20 of 53) also carry a `runbook_url`.
`DecdnNodeDown`, `DecdnCapacityBondRegistryResyncFailing`, `DecdnBuyerTopupUnresolved`
and `DecdnRedemptionMaybeBroadcast` have promtool unit tests in `prometheus-alerts_test.yml` (the chart's `.helmignore` keeps that file
out of the package): keep them in step.

- `monitoring/sponsord/` — the onboarding sponsor's own pair: `dashboard-sponsord.json` (uid
Expand Down Expand Up @@ -123,7 +123,12 @@ region) emits nothing — read a stray hit with that in mind. `decdn_health` is
Do not write a `by (reason)` query against a series that has no such label.
3. A name that resolves can still sit at a permanent zero because nothing increments it.
4. A panel or alert that names a *new* metric needs the metric in upstream
`crates/node/src/metrics.rs` first, in a released or pinned `decdn-node`.
`crates/node/src/metrics.rs` first, in a released or pinned `decdn-node`. The one
exception: `decdn_onchain_tx_maybe_broadcast_total` (decdn/decdn#2574) lands ahead of
the release after v0.2.0, because its absence keeps `DecdnRedemptionMaybeBroadcast`
silent and the hole it covers would otherwise go unalerted (#170). The name check
prints it against a v0.2 node; drop this exception once `decdn_node_version` moves
past 0.2.
5. The two "Unattributed stream failures" panels (overview and delivery) subtract every
inbound failure reason from `decdn_streams_failed_total{direction="inbound"}`. When
upstream adds a reason to `INBOUND_FAILURE_REASONS` in `crates/node/src/metrics.rs`,
Expand Down
2 changes: 1 addition & 1 deletion monitoring/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ this repo's MIT license. sponsord's were written here.
| `decdn-node/dashboard-chain.json` | Chain, payments and slash safety (`uid: decdn-chain`): watcher liveness, chain RPC, registries, payments. |
| `decdn-node/dashboard-node.json` | Single-node drilldown (`uid: decdn-node`): host, process, iroh transport, DHT and probe, logs, traces. |
| `decdn-node/prometheus-alerts.yml` | Rule groups `decdn-slash-safety`, `decdn-liveness`, `decdn-delivery`. A rule with a matching runbook section carries a `runbook_url` into upstream's `docs/runbook.md`. |
| `decdn-node/prometheus-alerts_test.yml` | promtool unit tests for `DecdnNodeDown` and `DecdnCapacityBondRegistryResyncFailing` (run by `make lint-helm`; the chart's `.helmignore` keeps them out of the package). |
| `decdn-node/prometheus-alerts_test.yml` | promtool unit tests for `DecdnNodeDown`, `DecdnCapacityBondRegistryResyncFailing`, `DecdnBuyerTopupUnresolved` and `DecdnRedemptionMaybeBroadcast` (run by `make lint-helm`; the chart's `.helmignore` keeps them out of the package). |
| `sponsord/` | The onboarding sponsor's dashboard and alert rules ([below](#sponsord)). Not rendered by the chart. |
| `iroh-relay/` | The iroh relay's dashboard and alert rules ([below](#iroh-relay)). Not rendered by the chart. |

Expand Down
21 changes: 15 additions & 6 deletions monitoring/decdn-node/dashboard-chain.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2087,7 +2087,7 @@
},
"title": "Redemption failures",
"type": "timeseries",
"description": "Redemption steps that failed: a redeemMany that reverted or was refused at send, or a lane the redeemer could not plan. An unconfirmed receipt does not count here; the On-chain transaction outcomes panel shows it as receipt failed or timeout.",
"description": "Redemption steps that failed: a redeemMany that reverted or was refused at send, or a lane the redeemer could not plan or load. An unconfirmed receipt does not count here; the On-chain transaction outcomes panel shows it as receipt failed or timeout. From the first decdn release after v0.2.0, neither does a send that failed with no tx hash; that panel shows it as maybe broadcast.",
"targets": [
{
"datasource": {
Expand Down Expand Up @@ -2438,7 +2438,7 @@
},
"title": "On-chain transaction outcomes",
"type": "timeseries",
"description": "Every node transaction counts once by outcome. A receipt wait that fails or times out falls back to fetching the receipt by hash, and a receipt found there counts as landed or reverted. receipt failed and timeout are the transactions still unconfirmed after that fetch; they can still mine, and their tx hash is in the warn! line. receipt recovered is not an outcome: it counts the waits the fetch by hash resolved, which points at a lagging RPC provider.",
"description": "Every settlement redeemMany counts once by outcome (openPool, topUp and reclaim do not count here). A receipt wait that fails or times out falls back to fetching the receipt by hash, and a receipt found there counts as landed or reverted. receipt failed and timeout are the transactions still unconfirmed after that fetch; they can still mine, and their tx hash is in the warn! line. maybe broadcast is a send that failed with no tx hash (a timeout, a reset, an HTTP 5xx, or an `already known`, `nonce too low` or `replacement transaction underpriced` answer): it can be in a mempool or have mined, and the claims stay for the next sweep; or it collided at one nonce with this node's buyer service, which shares the key, and broadcast nothing. It is counted from the first decdn release after v0.2.0, and DecdnRedemptionMaybeBroadcast alerts on it. receipt recovered is not an outcome: it counts the waits the fetch by hash resolved, which points at a lagging RPC provider.",
"targets": [
{
"datasource": {
Expand Down Expand Up @@ -2472,8 +2472,17 @@
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "sum(rate(decdn_onchain_tx_receipt_failed_total{deployment_environment=\"$env\",region=~\"$region\",instance=~\"$instance\"}[$__rate_interval]))",
"expr": "sum(rate(decdn_onchain_tx_maybe_broadcast_total{deployment_environment=\"$env\",region=~\"$region\",instance=~\"$instance\"}[$__rate_interval]))",
"refId": "D",
"legendFormat": "maybe broadcast"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "sum(rate(decdn_onchain_tx_receipt_failed_total{deployment_environment=\"$env\",region=~\"$region\",instance=~\"$instance\"}[$__rate_interval]))",
"refId": "E",
"legendFormat": "receipt failed"
},
{
Expand All @@ -2482,7 +2491,7 @@
"uid": "${DS_PROMETHEUS}"
},
"expr": "sum(rate(decdn_onchain_tx_timeout_total{deployment_environment=\"$env\",region=~\"$region\",instance=~\"$instance\"}[$__rate_interval]))",
"refId": "E",
"refId": "F",
"legendFormat": "timeout"
},
{
Expand All @@ -2491,7 +2500,7 @@
"uid": "${DS_PROMETHEUS}"
},
"expr": "sum(rate(decdn_onchain_tx_receipt_recovered_total{deployment_environment=\"$env\",region=~\"$region\",instance=~\"$instance\"}[$__rate_interval]))",
"refId": "F",
"refId": "G",
"legendFormat": "receipt recovered"
}
],
Expand Down Expand Up @@ -3166,7 +3175,7 @@
"y": 83
},
"id": 38,
"description": "1 while this node holds its slot for an openPool or a topUp whose outcome the chain has not shown (the topUp hold is decdn v0.2): the submit failed in transport or the receipt could not be read, and the transaction may still mine, so the node sends no other transaction of that kind until a receipt arrives, the account's nonce passes it, or a re-send shows the pool closed. A hold that stays means the RPC cannot answer, a pending transaction holds the nonce or an earlier one, or every re-send at that nonce fails (the wallet cannot fund the amount or the gas); the daemon's repeating warning names the last failure in its `last` field. Do not restart a node holding a topUp slot: the restart forgets the hold, and the node may escrow a second topUp into its own pool. DecdnBuyerTopupUnresolved fires after 15m."
"description": "1 while this node holds its slot for an openPool or a topUp whose outcome the chain has not shown (the topUp hold is decdn v0.2): the submit got no tx hash or the receipt could not be read, and the transaction may still mine. A submit gets no tx hash when the transport fails and, from the first decdn release after v0.2.0, also when the RPC node already holds the transaction (`already known`), counts its nonce as spent (`nonce too low`) or holds another transaction of this key at that nonce (`replacement transaction underpriced`). Each of those can come back when a gateway retries a send that another backend already accepted, or when this node's own seller redemption took that nonce; the node then settles the nonce against the chain, and the hold ends once the redemption mines. Meanwhile the node sends no other transaction of that kind until a receipt arrives, the account's nonce passes it, or a re-send shows the pool closed. A hold that stays means the RPC cannot answer, a pending transaction holds the nonce or an earlier one, or every re-send at that nonce fails (the wallet cannot fund the amount or the gas); the daemon's repeating warning names the last failure in its `last` field. Do not restart a node holding a topUp slot: the restart forgets the hold, and the node may escrow a second topUp into its own pool. DecdnBuyerTopupUnresolved fires after 15m."
},
{
"datasource": {
Expand Down
26 changes: 13 additions & 13 deletions monitoring/decdn-node/dashboard-node.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2776,15 +2776,6 @@
"expr": "sum(rate(decdn_dht_lookup_round_ceiling_total{instance=~\"$node\"}[$__rate_interval]))",
"refId": "H",
"legendFormat": "lookup round ceiling"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "sum(rate(decdn_dht_lookup_dial_timeouts_total{instance=~\"$node\"}[$__rate_interval]))",
"refId": "I",
"legendFormat": "lookup dial timeouts"
}
],
"fieldConfig": {
Expand Down Expand Up @@ -3421,8 +3412,17 @@
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "sum(rate(decdn_dht_bucket_refresh_failures_total{instance=~\"$node\"}[$__rate_interval]))",
"expr": "sum(rate(decdn_dht_lookup_dial_timeouts_total{instance=~\"$node\"}[$__rate_interval]))",
"refId": "D",
"legendFormat": "lookup dial timeouts"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "sum(rate(decdn_dht_bucket_refresh_failures_total{instance=~\"$node\"}[$__rate_interval]))",
"refId": "E",
"legendFormat": "bucket refresh failures"
},
{
Expand All @@ -3431,7 +3431,7 @@
"uid": "${DS_PROMETHEUS}"
},
"expr": "sum(rate(decdn_dht_bootstrap_find_node_failures_total{instance=~\"$node\"}[$__rate_interval]))",
"refId": "E",
"refId": "F",
"legendFormat": "bootstrap failures"
}
],
Expand Down Expand Up @@ -3516,7 +3516,7 @@
"y": 85
},
"id": 45,
"description": "Lookups this node ran and STORE records peers accepted from it. A round timeout is a lookup round whose peers did not all answer in time. A refresh failure is a FindNode error, an invalid peer key or a panicked refresh task; a bootstrap failure is a FindNode against a seed that failed."
"description": "Lookups this node ran and STORE records peers accepted from it. A round timeout is a lookup round whose peers did not all answer in time. A dial timeout is a lookup FIND_VALUE RPC whose QUIC dial missed the 2 s dial budget; the lookup then skips that peer for a cooldown, so a steady rate of about one per cooldown names an offline bonded peer, and round timeouts should stay near zero (ADR 022 § Routing Table). A refresh failure is a FindNode error, an invalid peer key or a panicked refresh task; a bootstrap failure is a FindNode against a seed that failed."
},
{
"datasource": {
Expand Down Expand Up @@ -3730,7 +3730,7 @@
"y": 92
},
"id": 48,
"description": "p50, p95 and p99 duration of one probe collection window on the cache-miss path, from the start of the concurrent probes (dials included) to the end of collection (ADR 001 §Probe response collection). Collection stops early once enough holders answer, or when every probe resolves; the 500 ms probe timeout bounds each probe. A p95 near 0.5 s means rounds wait on a slow or unreachable peer before enough holders answer. A probe-cache hit sends no probe and is not in these series."
"description": "p50, p95 and p99 duration of one probe collection window on the cache-miss path, from the start of the concurrent probes (dials included) to the end of collection (ADR 001 §Probe response collection). Collection stops early once enough holders answer, or when every probe resolves; the 1 s probe timeout bounds each probe. A p95 near 1 s means rounds wait on a slow or unreachable peer before enough holders answer. A probe-cache hit sends no probe and is not in these series."
},
{
"collapsed": false,
Expand Down
49 changes: 43 additions & 6 deletions monitoring/decdn-node/prometheus-alerts.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -846,12 +846,42 @@ groups:
Redemption steps keep failing: a redeemMany that reverted or was
refused at send, or a lane the redeemer could not plan or load.
An unconfirmed receipt does not count here; DecdnOnchainTxUnconfirmed
covers it.
covers it. From the first decdn release after v0.2.0, nor does a send
that failed with no tx hash; DecdnRedemptionMaybeBroadcast covers it.
The claims stay and retry, but unredeemed revenue accrues and a
closing pool's grace window can run out. Read the redeemMany warn!
lines (each carries the tx hash when one was issued) and check the
RPC provider and the operator key's gas balance.

# The counter ships in the first decdn release after v0.2.0; on an older
# node the series is absent and the rule stays silent. Not folded into
# DecdnOnchainTxUnconfirmed (for: 0m): a node's seller and buyer share one
# key, so a harmless nonce collision between them gives these same answers.
# As with DecdnRedemptionFailing, increase() extrapolation reads two events
# in the window at about 2.05, so two failed sends per ~30m, sustained, fire.
- alert: DecdnRedemptionMaybeBroadcast
expr: increase(decdn_onchain_tx_maybe_broadcast_total[30m]) > 2
for: 30m
labels:
component: payments
severity: warning
annotations:
summary: "redeemMany sends failing with no tx hash on {{ $labels.instance }}"
description: |
redeemMany sends keep failing with no tx hash: the request timed
out, was reset or got an HTTP 5xx, or the RPC answered `already
known`, `nonce too low` or `replacement transaction underpriced`.
Each may still have broadcast; the claims stay, and the next sweep
skips lanes the chain shows paid. Or this node's seller and buyer
services, which share one key, collided at a nonce, and nothing
was broadcast. If decdn_onchain_tx_landed_total is flat and
decdn_unredeemed_usdc keeps rising, redemptions are not landing and
a closing pool's grace window can run out. Check the RPC's
eth_sendRawTransaction health (reads can still work), compare
`cast nonce <operator> --block pending` with `--block latest` for
a lagging backend, and read the "redeemMany send failed with no tx
hash" warn lines.

- alert: DecdnWatermarkReconcileFailing
expr: increase(decdn_redemption_reconcile_failures_total[30m]) > 2
for: 30m
Expand Down Expand Up @@ -971,7 +1001,8 @@ groups:
vouchers on it. The redeemer must land them before the grace window
ends or the revenue is lost. Check that redeemMany is landing
(decdn_onchain_tx_landed_total, DecdnRedemptionFailing,
DecdnOnchainTxUnconfirmed) and that the operator key has gas.
DecdnRedemptionMaybeBroadcast, DecdnOnchainTxUnconfirmed) and that
the operator key has gas.

- alert: DecdnBuyerWalletUnfunded
# Scoped on `> 0`, not on the series existing: every metric is registered
Expand Down Expand Up @@ -1027,10 +1058,16 @@ groups:
operator address owns on chain.

# The gauge (decdn v0.2) is 1 while the node holds its top-up slot for a
# topUp whose outcome the chain has not shown: the submit failed in
# transport or the receipt could not be read, and the transaction may
# still mine, so no other topUp goes out meanwhile. A healthy settle
# clears it within a few blocks; 15m of hold is a stuck chain path.
# topUp whose outcome the chain has not shown: the submit got no tx hash
# or the receipt could not be read, and the transaction may still mine,
# so no other topUp goes out meanwhile. A submit gets no tx hash when the
# transport fails and, from the first decdn release after v0.2.0, also
# when the RPC node already holds the transaction (`already known`),
# counts its nonce as spent (`nonce too low`) or holds another
# transaction of this key at that nonce (`replacement transaction
# underpriced`), as when a seller redemption took the topUp's nonce. A
# healthy settle clears it within a few blocks; 15m of hold is a stuck
# chain path.
- alert: DecdnBuyerTopupUnresolved
expr: decdn_buyer_topup_unresolved == 1
for: 15m
Expand Down
Loading
Loading