[aw-failures] [aw] Failure Investigator Report — 2026-08-19 07:05 UTC (6h)

[github-actions]

Fix the sandbox VM network bring-up first — it hit 2 engines in 22 minutes and left zero egress on either run. That's the highest-severity, least-covered problem in this window.

Cluster table

Sev	Signature	Workflows (runs)	Tracking
P0	Cloud Hypervisor guest network never initializes (lo interface DOWN, connectivity probe exit 4) → zero egress, not a proxy refusal	Daily AgentRx Trace Optimizer §32220130823, Daily Cli Tools Tester §32218752360	New — sub-issue below
P1	safe_outputs "Process Safe Outputs" hard-fails the whole job on one non-retryable error	Contribution Check §32217761657, Designer Drift Audit §32213811962	Matches open #53263 — updated with today's occurrences
P1	check_skip_if_match.cjs pre_activation gate has retries: 0 — any transient GitHub Search 500 hard-kills the run	Issue Monster §32210356710	New — folded into roadmap, no dedicated GitHub issue (create-issue budget)
P2	Copilot headless server internal readiness timeout (ETIMEDOUT 127.0.0.1:3002) — distinct from the tracked port-10002 proxy-refusal bug	Daily GitHub Docs SEO Optimizer §32217479194	Not #52253 (see correction below) — isolated, watch only
P2	Copilot harness failureClass=model_not_supported	Metrics Collector §32209589795	Isolated — watch only
P2	Container image gate correctly failing on real CVEs — working as designed, not a gh-aw bug	Daily Container Image Security Scan §32220139949	No action

Evidence

P0 — Cloud Hypervisor guest network failure (click for audit-diff)

Both failing runs show the identical fatal line before any proxy/CLI even starts:

[WARN] [cloud-hypervisor] stage=guest-connectivity status=failed: Cloud Hypervisor guest connectivity probe failed with exit code 4
(stderr: Connection to 172.30.0.10 3128 port [tcp/*] succeeded!; guest network state: 1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN ...)
[ERROR] Fatal error: Error: Cloud Hypervisor guest connectivity probe failed with exit code 4

audit-diff of §32220130823 (failed) against §32103562899 (last good AgentRx run, 2026-08-18) confirms the blast radius: every domain the good run touched — github.com, api.anthropic.com, o205451.ingest.us.sentry.io, otlp-gateway-prod-eu-west-2.grafana.net, both Fastly IPs — drops to 0 allowed / 0 blocked in the failing run. That's a complete egress blackout at the guest-network layer, not a proxy or engine rejecting connections. Same signature independently reproduced 22 minutes earlier on Daily Cli Tools Tester (Copilot engine), so this is engine-agnostic and sits below both the Anthropic-proxy (#52459) and Copilot-proxy (#52253) code paths.

No commits have touched pkg/workflow sandbox/cloud-hypervisor code or either affected workflow file since 2026-08-13 — this is a runner/hypervisor-layer regression, not a workflow-config drift.

P1 — safe_outputs hard-fail (2 new occurrences today, matches #53263)

Contribution Check §32217761657 and Designer Drift Audit §32213811962 both fail in the safe_outputs job at "Process Safe Outputs" — same job/step signature already tracked in #53263. No fix has landed (no matching commits since the issue opened); still reproducing today.

P1 — skip-if-match gate has zero retry tolerance

Issue Monster §32210356710, pre_activation job, check_skip_if_match.cjs:

GET /search/issues?q=...repo:github%2Fgh-aw&per_page=1 - 500 with id UNKNOWN in 10021ms
##[error]ERR_API: Failed to execute search query: Connect Timeout Error

The github-script step is configured with retries: 0. Any single transient 500/timeout from the Search API is guaranteed to hard-fail the entire workflow with no retry — this isn't a one-off, it's a design gap that will keep recurring under normal GitHub API flakiness. No existing issue covers check_skip_if_match/pre_activation.

P2 — two more "Execute Copilot CLI" failures with unrelated root causes (do not merge with #52253)

• Daily GitHub Docs SEO Optimizer §32217479194: the outbound api-proxy on port 10002 accepted connections fine; the failure is the local copilot-sdk headless server never reporting ready — unexpected error: copilot-sdk headless server did not become ready on 127.0.0.1:3002 within 60000ms (connect ETIMEDOUT 127.0.0.1:3002). This is internal-to-the-guest, unrelated to [aw-failures] [P1] Copilot engine proxy (port 10002) refuses connections despite passing key validation — Daily Assign Issue To [Content truncated due to length] #52253's external ECONNREFUSED-on-10002 signature.
• Metrics Collector §32209589795: failureClass=model_not_supported in the copilot-harness attempt log — a model-selection bug, not a connectivity issue.

Both are single occurrences this window — watching, not filing separately yet.

Existing issue correlation

• [aw-failures] safe_outputs job hard-fails entire batch on one non-retryable error #53263 (safe_outputs hard-fail) — confirmed still broken, updated with today's 2 runs.
• [aw-failures] [P1] Anthropic proxy connection-refused/retry-exhaustion now hitting 3 workflows after ~10 retries #52459 (Anthropic proxy ConnectionRefused) — does not cover today's AgentRx failure; corrected with a note pointing to the new P0 sub-issue below.
• [aw-failures] [P1] Copilot engine proxy (port 10002) refuses connections despite passing key validation — Daily Assign Issue To [Content truncated due to length] #52253 (Copilot proxy ECONNREFUSED port 10002) — does not cover any of today's 3 Copilot CLI failures; corrected with a note — none of today's occurrences show ECONNREFUSED to the proxy.
• [aw-failures] [P0] Sentry/Grafana MCP servers silently omitted when secrets missing — blocks Daily AIC Consumption Report #52726, [aw-failures] Smoke Copilot Sub Agents: 'evals' artifact not found — full run failure #53262, [aw-failures] [P1] PR Sous Chef: Start DIFC Proxy step fails with no captured root cause #52502, [aw-failures] [P1] Two new job-failure root causes: AWF binary curl reset in Daily SPDD evals + MCP Gateway serena digest-pin sc [Content truncated due to length] #53191 — no new evidence this window; left open, no action.

Fix roadmap

P0 — Add a hard pre-flight gate that fails the job fast (with a clear ##[error]) when the Cloud Hypervisor guest-connectivity probe reports lo: DOWN, instead of letting it cascade into an opaque engine-specific timeout/startup error. Root-cause the network bring-up race in the runner/hypervisor image itself. Tracked in the new sub-issue.

P1 — (a) File a fix for check_skip_if_match.cjs: add retries (e.g. 2 with backoff) around the Search API call before treating it as a hard gate failure. (b) Keep #53263 open until a landed commit actually retries/isolates a single bad safe-output instead of failing the whole batch.

P2 — No action needed; watch for recurrence of the copilot-sdk 3002 timeout and model_not_supported before filing.

Sub-issues created

• New P0 sub-issue: Cloud Hypervisor guest network failure (linked below).

References:

• §32220130823
• §32218752360
• §32217761657

Generated by 🔍 [aw] Failure Investigator (6h) · agent · 167.2 AIC · ⌖ 10.4 AIC · ⊞ 5.9K · ◷

• expires on Aug 25, 2026, 11:17 PM UTC-08:00