Repository navigation
[Incident] Hourly Product Development blocks its own GitHub API egress #22
Description
Incident summary
The original failure was a Harden Runner egress parsing defect in the hourly product-development workflow, not an NVIDIA credential failure. Protected-main repairs in PRs #23 and #24 removed the observed GitHub API connection-refused failure while preserving exact egress allowlists, NVIDIA_NIM_API_KEY isolation, credential-free reverification, publisher separation, and the one-proposal authority boundary.
Protected-main evidence already satisfied
Scheduled Hourly Product Development run 31343380043 executed on protected main SHA fb7dab5698ffd24b1a6db0943f1e387f0eda4d31 and completed successfully. The deterministic gh api pull-request gate passed through Harden Runner and exited cleanly with reason=open_pull_request; model, reverification, and publication jobs were correctly skipped because the repository had open PRs.
Therefore closure criteria 1–4 are satisfied:
- exact-head repair CI/actionlint/security/review gates passed;
- repair reached protected
main; - a protected-main scheduled execution passed Harden Runner + GitHub API access;
- with an open PR queue, the scheduler exited cleanly at the documented
open_pull_requestgate.
The canonical documentation graph subsequently reached protected main through PR #25 merge fe9b46f5404f368b311de205c4e647f47db89ab3; PR #29 post-integration audit merged as 59eb6d609a6cbf0699e427e913067d8b0c3beb41.
Remaining closure criterion
Keep this issue open for criterion 5 only:
- after the PR queue is genuinely drained and release policy permits product development, a protected-main scheduled/manual run must reach the bounded OpenCode/NVIDIA path and either:
- produce one sealed proposal that passes independent credential-free reverification and the existing publisher boundary, or
- exit at a documented product/release gate without weakening the deterministic trust boundary.
Do not manufacture an empty PR queue, close valid work prematurely, bypass release blocker #17, broaden egress, expose the NVIDIA key to repository-controlled code, or use COPILOT_GITHUB_TOKEN merely to obtain this proof.
Current dependency order — updated 2026-08-23 (originally 2026-08-10)
PR fix(release): fail before side effects when Trusted Publishing is unready #30 must finish its exact-head CI/security/review/merge cycle.Done — merged 2026-08-11 (2559425084389176870eac9d1a855d219bc12ce3); adds the repository-side Trusted Publishing release-readiness preflight and ADR-0008.- Release blocker Release operations: publish and verify ThreadWeave 0.2.0 #17 must then complete the external GitHub
pypienvironment + PyPI Trusted Publisher + public0.2.0acceptance path. Re-verified 2026-08-23:gh api repos/ContextualWisdomLab/ThreadWeave/environmentsstill returnstotal_count: 0; PyPI still lists only0.1.0. This step is the current binding blocker — external account-side action, not repository-side. - Draft PR feat: add incremental mailbox threading with stable identity handoff #20 remains intentionally non-mergeable until Release operations: publish and verify ThreadWeave 0.2.0 #17 closes; after publication it must be refreshed onto the released protected main and fully revalidated.
- Only after the truthful product/documentation PR queue is drained and release policy allows model-backed development can criterion 5 execute without violating the product freeze or independent review boundaries. As of 2026-08-23 the queue is PR fix(operations): audit orphaned Actions workflow identities #32 and PR docs: add product/technical gap baseline and LineageWeave consumer boundary ADR #34, both CI-green with no unresolved review threads but stuck on the org-wide review-dispatch outage tracked in
ContextualWisdomLab/.github#624(GitHub Models retirement + provider credit exhaustion causingOPENCODE_MODEL_POOL_OUTCOME: exhaustedacross every pool candidate, not a per-repo issue) — the queue isn't "genuinely drained," it's externally stalled the same way Release operations: publish and verify ThreadWeave 0.2.0 #17 is.
Queued CI/reviewer/provider time on one item is not an incident blocker for other safe repository work; criterion 5 itself is dependency-gated by the truthful empty-queue/release state above.
Closure evidence required
Close this issue only after criterion 5 records:
- exact protected source SHA;
- workflow run/job identities;
- Harden Runner and model-provider boundary outcome;
- bounded OpenCode proposal/defer outcome;
- independent credential-free reverification result when a patch is proposed;
- publisher/defer result;
- confirmation that no credential, egress, review-authority, or one-proposal boundary was weakened to obtain the proof.
Activity
seonghobae commentedon Aug 8, 2026
Operational revalidation after #23 reached protected
mainfound the incident still open. Scheduled Hourly Product Development run31250354848at protected-main commite9e52e4cae47c1c0164a37ed907719f5e899caf5failed before the deterministic open-PR exit. Harden Runner's workflow input contained bothapi.github.com:443andcafe.github.com:443, but its installed agent retainedapi.github.comwith port0plus the dynamically space-appended cache host;gh apithen resolvedapi.github.comto140.82.116.5, the agent dropped that IP, and the request failed with connection refused. This is still not an NVIDIA credential failure.Follow-up PR #24 is the bounded test-first repair. It preserves
egress-policy: blockand the exact endpoint sets, but changes the threeallowed-endpointsscalars from literal|to folded>-so the action receives a space-delimited runtime string consistent with the pinned action's own dynamic endpoint append behavior and the observed agent parser. Exact-head CI/actionlint, SAST, and Security Scan are currently green on71a6d7e151ec6a9d53f9761ad65d4f22d3951c5f; independent formal review remains separate. Keep this issue open through protected-main operational reruns required by the existing closure criteria.seonghobae commentedon Aug 10, 2026
Protected-main operational acceptance checkpoint — 2026-08-10
Scheduled Hourly Product Development run
31343380043executed on protectedmainSHAfb7dab5698ffd24b1a6db0943f1e387f0eda4d31and completed successfully. The hardened runner accepted bothapi.github.com:443and the observedcafe.github.com:443alias, and the deterministicgh apipull-request gate completed without the prior connection-refused failure. Because open pull requests existed, the gate exited cleanly withreason=open_pull_request; the model, reverification, and publication jobs were correctly skipped.This satisfies closure criteria 1–4 for the incident. Keep the issue open for criterion 5 only: after the PR queue is genuinely drained (and release policy permits product development), a protected-main scheduled/manual run must reach the bounded OpenCode/NVIDIA path and either publish one independently verified proposal or exit at a documented product gate. Do not manufacture an empty PR queue or bypass release blocker #17 merely to satisfy that proof.
seonghobae commentedon Aug 12, 2026
Fresh causal-chain and feasibility checkpoint — 2026-08-12
The original Harden Runner incident remains repaired and operationally proven through the open-PR branch. Protected-main evidence already shows the folded endpoint scalar preserves port
443, GitHub API inventory succeeds under fail-closed egress, and the workflow exitsopen_pull_requestbefore touching the model credential.The remaining zero-PR acceptance condition is currently blocked by a distinct, legitimate release dependency rather than by egress or NVIDIA authentication:
mainat2559425084389176870eac9d1a855d219bc12ce3;pypi(total_count: 0);Feasibility decision
Creating another token, removing the release blocker, force-merging #20, weakening the environment preflight, or claiming a model-path run while the queue is intentionally nonzero would not repair the causal chain. The realistic next action is the account-side
pypienvironment and PyPI Pending Trusted Publisher configuration documented on #17, followed by the protected0.2.0release, #20 reconciliation/merge under normal policy, and then a fresh zero-PR scheduled/manual Hourly Product Development run.Issue #22 therefore stays open only for criterion 5. No further local egress patch is justified by current runtime evidence.
seonghobae commentedon Aug 23, 2026
Updated the dependency-order section: PR #30 merged 2026-08-11 (was still listed as pending). Re-verified today that #17's external blocker is unchanged (no
pypienvironment, PyPI still at 0.1.0), and added the concrete reason PR #32/#34 haven't drained the queue — it's the org-wide review-dispatch model-pool outage tracked in.github#624, not a per-repo scheduling delay. No repository-side action available on any of these right now.docs: refresh gap baseline with PR #32/#34 status and .github#624 roo…
seonghobae commentedon Sep 1, 2026
Release dependency correction — 2026-09-01
Issue #17 no longer requires PyPI Trusted Publishing as the only permitted publisher identity. The organization already has approved GitHub Secrets named exactly
PIPY_USERNAMEandPIPY_TOKEN; #17 now requires the existingrelease.ymlto support that protected secret-backed publisher path while retaining Trusted Publishing/OIDC as an optional preferred mode.Therefore references in this issue describing #17 as a purely external GitHub-environment/PyPI-Trusted-Publisher blocker are stale. The current dependency is repository-owned release-workflow repair → exact-head validation →
threadweave==0.2.0publication/public-artifact proof. Do not expose the secret values and do not weaken any existing digest/provenance/review/security gates.seonghobae commentedon Sep 30, 2026
2026-09-30 dependency refresh: criteria 1–4 remain satisfied and criterion 5 remains open. The former Trusted Publisher account setup text is superseded by issue #17's approved organization
PIPY_TOKENpublisher path on Draft PR #35. The live shortest path is PR #44 (protected-ref CI foundation) → PR #35 → issue #17 immutable 0.2.0 publication proof → truthful queue drain → this issue's criterion-5 bounded model-path run. No early closure or manufactured empty queue.