Merge-result verification only builds the merged tree, so combined-change test failures land on main

docs/concepts/base-broken.md

@@ -0,0 +1,60 @@
+# Base-broken containment (build-red and test-red)
+
+## The gap
+
+Merge-result verification builds the combined tree before landing, but two
+changes that each passed their full audit on their own base can still combine
+into a tree whose **tests** fail — and that tree lands on main. The build
+check passes because the combined tree compiles.
+
+## Design choice
+
+Of the available closes, this codebase implements **post-merge red detection
++ test attribution**, and deliberately defers **pre-landing affected-test
+runs** on the merged tree:
+
+- **Post-merge CI watch** (`PostMergeCiWatchService`): every landed merge
+  schedules a bounded poll of the merge commit's forge CI status
+  (`IUpstreamRemote.GetCheckResultsAsync`). A red tip immediately records a
+  test-kind condition and files one top-priority fix item naming the failing
+  checks and the suspect commits. This reuses the verdict CI already
+  computes — one status poll per interval per recent merge.
+- **Test attribution** (`BaseBrokenTestClassifier`): an audit iteration whose
+  blocking findings are test-only, with every failure genuinely reproducing
+  on the base, is adopted by an active condition (main is recorded red) or
+  recorded against a forge-red tip — the item parks under the hold, never
+  fails, and is never blamed.
+- **Deferred: pre-landing affected-test runs.** Running the affected subset
+  on the merged tree before every landing duplicates CI cost on the hottest
+  path, and a flaky selected test would block landings that CI would clear
+  on retry. The test-selection project graph / coverage baseline stays
+  available if a future pre-landing gate is wanted; the watch + attribution
+  pair closes the correctness gap (red main is detected, held, and repaired)
+  without the landing-latency and flake-blocking costs.
+
+## Semantics
+
+- Conditions have a `FailureKind`: `Build` (tip fails the required build)
+  or `Test` (tip builds but its tests fail). The dispatcher hold, SHA-keyed
+  dedupe, and one-fix-item-per-SHA rule are identical for both kinds.
+- Clear signals differ on purpose: a build-kind condition clears when a
+  moved tip builds; a test-kind condition clears only when the tip's forge
+  CI passes. A green build alone must never clear a test-red hold — that is
+  exactly the gap above.
+- Fix items are parked for operator approval (`NeedsOperatorInput`, never
+  auto-enqueued). Test names and check names are untrusted test-runner/forge
+  output: they travel sanitized, bounded, and instruction-scrubbed; the full
+  list lives on the operator-held note, with counts and opaque SHAs in the
+  agent prompt. The agent identifies targets by running the suite on the
+  base itself.
+- Unknown forges (or SHAs the forge never saw, e.g. local-only squash SHAs)
+  resolve CI to "unknown": the watch drops and detection falls back to the
+  audit-time check. Unknown never records and never clears.
+
+## Knobs (`CodeyBox:PipelineTuning`)
+
+`BaseBrokenDetectionEnabled` (master switch),
+`BaseBrokenFixItemPriority`, `BaseBrokenRecheckInterval`,
+`PostMergeCiWatchEnabled`, `PostMergeCiWatchInterval`,
+`PostMergeCiWatchMaxAttempts`, `PostMergeCiWatchMaxWatches` — all
+hot-reloadable. See `docs/reference/configuration.md`.

docs/operating/agent-turn-checkpoints.md

@@ -102,16 +102,25 @@ a no-op resume uses the normal initial-work/rework no-diff failure path.
 When the base moves under the checkpoint after it was written — a
 pickup-time rebase onto a fresh base or a restart-recovery recomposition — the
 base ref changes across the pickup's upstream refresh and the checkpoint tree
-no longer applies, but that says nothing about the work. The turn preparation
+no longer applies, but that says nothing about the work. A checkpoint that
+descends from neither the current branch tip nor the current base tip is
+stale even when the base ref is unchanged across the refresh (the branch was
+recomposed at an already-staged fresh base). When no pre-refresh base
+snapshot exists, the same reachability test alone decides; a
+checkpoint still reachable from the base tip (a plain branch advance past the
+interrupted turn's base) is resumed, not discarded. The turn preparation
 discards the stale checkpoint (logged as `Discarding durable agent-turn resume
 checkpoint`, the same record as a phase-boundary discard) and runs a normal
 fresh turn from the current branch tip with the branch preserved. This is not a
 failure and does not count toward terminal failures. A same-base advance of
 the work branch (an outside writer landing commits without moving the base)
 is not stale: the base ref is unchanged across the refresh, so the checkpoint
 still applies and resumes against the fork-point baseline. A checkpoint backed by a
-retained-sandbox recovery lease is never discarded this way: provider-owned
-state stays authoritative and the resume is refused explicitly.
+retained-sandbox recovery lease is never silently dropped: the stale Git
+preempt ref is cleared so the lease stays the single authoritative backing,
+the lease and its checkpoint metadata are preserved, and the resume is
+refused explicitly with the retry/cancel path instead of running a fresh
+turn over provider-owned state.
 
 Durable re-dispatches are bounded by
 `CodeyBox:PipelineTuning:AgentSessionResumeMaxAttempts`. The attempt is claimed

docs/reference/configuration.md

@@ -588,7 +588,11 @@ Hot-reloadable retry and recovery bounds used by pipeline execution.
 | `SuggestionDedupeMaxRecordedSources` | `25` | Cap on the source work-item id list recorded on a merged suggestion. Once full, repeats still bump the occurrence count but stop appending ids. Must be ≥ 1. Hot-reloadable. |
 | `BaseBrokenDetectionEnabled` | `true` | When a required-build failure's compiler errors point at files outside the item's diff, the orchestrator sandbox-builds the base tip (verdict cached per base SHA). A reproducing failure parks the item without a failure charge and records a project-level base-broken condition that holds build-dependent dispatch; `false` keeps every required-build failure item-attributed. Hot-reloadable. |
 | `BaseBrokenFixItemPriority` | `1000` | Dispatch priority for the auto-filed base-fix work item (one per broken base SHA). Must be within [-1000, 1000]. Hot-reloadable. |
-| `BaseBrokenRecheckInterval` | `00:02:00` (2 min) | Interval between base-tip re-checks for active base-broken conditions: the same SHA skips the rebuild, a moved tip is built once and the condition clears when it passes. Must be positive. Hot-reloadable. |
+| `BaseBrokenRecheckInterval` | `00:02:00` (2 min) | Interval between base-tip re-checks for active base-broken conditions: the same SHA skips the rebuild, a moved tip is built once and the condition clears when it passes. For test-kind conditions a passing build alone never clears — the new tip's forge CI must also pass. Must be positive. Hot-reloadable. |
+| `PostMergeCiWatchEnabled` | `true` | After every landed merge, watch the merge commit's forge CI status: a red tip records a test-kind base-broken condition (holding build-dependent dispatch) and files one top-priority fix item naming the failing checks and the suspect commits. Hot-reloadable. |
+| `PostMergeCiWatchInterval` | `00:01:00` (1 min) | Interval between post-merge CI status polls. Must be positive. Hot-reloadable. |
+| `PostMergeCiWatchMaxAttempts` | `30` | Maximum polls per watched merge commit before the watch is dropped (CI finished, went green, or never reported). Must be ≥ 1. Hot-reloadable. |
+| `PostMergeCiWatchMaxWatches` | `64` | Maximum concurrently watched merge commits; past the cap the oldest watch is evicted (newest merge wins). Must be ≥ 1. Hot-reloadable. |
 
 Durable agent-turn scratchpad archives have a non-configurable 32 MiB safety
 cap. They are stored as content-verified, host-private SQLite BLOBs and are

docs/reference/events.md

@@ -100,8 +100,8 @@ subscribe to.
 | `project.budget_warning` | 1.0 | Project crossed the cost warning threshold. |
 | `project.budget_exceeded` | 1.0 | Project crossed the cost hard cap. |
 | `project.budget_recovered` | 1.0 | Project spend dropped below the warning threshold. |
-| `project.base_broken_detected` | 1.7 | The project's base branch tip reproduces the required-build failure; build-dependent dispatch is held. Details include `projectId`, `baseBranch`, `baseSha`. |
-| `project.base_broken_cleared` | 1.7 | The base tip builds again; held items resume. |
+| `project.base_broken_detected` | 1.7 | The project's base branch tip is broken (build-kind: reproduces the required-build failure; test-kind: builds but its tests fail — forge CI red for the merge commit, or audit test failures reproducing on the base). Build-dependent dispatch is held. Details include `projectId`, `baseBranch`, `baseSha`. |
+| `project.base_broken_cleared` | 1.7 | The base is healthy again (build-kind: tip builds; test-kind: tip CI passes); held items resume. |
 | `work_item.planning` | 1.5 | Planning-only agent turn started. |
 | `work_item.plan_review` | 1.5 | Planning artifact entered review. |
 | `work_item.plan_approved` | 1.5 | Planning artifact was approved; implementation may start. |
@@ -126,7 +126,7 @@ subscribe to.
 | `work_item.waiting_for_quota_reset` | 1.0 | Work item parked until quota reset window. |
 | `work_item.waiting_for_agent_resume` | 1.3 | Work item parked because its only eligible agent is paused. |
 | `work_item.waiting_for_transient_retry` | 1.4 | Work item parked until the transient transport/network retry backoff expires. |
-| `work_item.base_broken_parked` | 1.7 | A required-build failure was attributed to the broken base branch rather than the item's diff; the item returned to a resumable state (branch preserved, no failure charge). Details include `baseBranch`, `baseSha`, `resumeState`, `fixWorkItemId`, `phase`. |
+| `work_item.base_broken_parked` | 1.7 | A required-build failure (or test-only audit failures) was attributed to the broken base branch rather than the item's diff; the item returned to a resumable state (branch preserved, no failure charge). Details include `baseBranch`, `baseSha`, `failureKind`, `resumeState`, `fixWorkItemId`, `phase`. |
 | `work_item.agent_stuck` | 1.0 | Stuck-agent probe killed a hung agent. |
 | `work_item.auto_retry` | 1.0 | Quota/transient auto-retry re-queued a failed item. |
 | `work_item.agent_restore_requeued` | 1.5 | Agent-restore infra sweep re-queued a failed item. |

src/CodeyBox.Api/AgentConfigHotReload.cs

@@ -1681,6 +1681,10 @@ internal static string SerializePipelineTuning(PipelineTuningOptions opts) =>
                 opts.BaseBrokenDetectionEnabled,
                 opts.BaseBrokenFixItemPriority,
                 BaseBrokenRecheckIntervalSeconds = opts.BaseBrokenRecheckInterval.TotalSeconds,
+                opts.PostMergeCiWatchEnabled,
+                PostMergeCiWatchIntervalSeconds = opts.PostMergeCiWatchInterval.TotalSeconds,
+                opts.PostMergeCiWatchMaxAttempts,
+                opts.PostMergeCiWatchMaxWatches,
             },
             JsonOpts);
 

src/CodeyBox.Api/Program.cs

@@ -4145,8 +4145,22 @@ static Func<TestRunOptions> DotnetTestRunOptionsAccessor(IServiceProvider sp)
     sp.GetRequiredService<IWorkItemStore>(),
     sp.GetRequiredService<PipelineTuningSnapshot>(),
     sp.GetService<TimeProvider>(),
-    sp.GetRequiredService<ILogger<BaseBrokenMonitorService>>()));
+    sp.GetRequiredService<ILogger<BaseBrokenMonitorService>>(),
+    sp.GetService<IUpstreamRemoteFactory>()));
 builder.Services.AddHostedService(sp => sp.GetRequiredService<BaseBrokenMonitorService>());
+// Post-merge CI watch: polls the forge CI status of landed merge commits
+// and records a test-kind base-broken condition (plus the top-priority fix
+// item) the moment a red main is detected. Wired into the pipeline for
+// scheduling; runs as a hosted service for polling.
+builder.Services.AddSingleton<PostMergeCiWatchService>(sp => new PostMergeCiWatchService(
+    sp.GetRequiredService<IProjectRepository>(),
+    sp.GetRequiredService<IUpstreamRemoteFactory>(),
+    sp.GetRequiredService<BaseBrokenConditionTracker>(),
+    sp.GetRequiredService<IWorkItemStore>(),
+    sp.GetRequiredService<PipelineTuningSnapshot>(),
+    sp.GetService<TimeProvider>(),
+    sp.GetRequiredService<ILogger<PostMergeCiWatchService>>()));
+builder.Services.AddHostedService(sp => sp.GetRequiredService<PostMergeCiWatchService>());
 
 builder.Services.AddSingleton<PipelineRunner>(sp => new PipelineRunner(
     sp.GetRequiredService<ISandboxProvider>(),
@@ -4260,6 +4274,7 @@ static Func<TestRunOptions> DotnetTestRunOptionsAccessor(IServiceProvider sp)
     providerTransientCorrelation: sp.GetRequiredService<ProviderTransientCorrelationTracker>(),
     baseBuildVerifier: sp.GetRequiredService<BaseBuildVerifier>(),
     baseBrokenConditions: sp.GetRequiredService<BaseBrokenConditionTracker>(),
+    postMergeCiWatch: sp.GetService<PostMergeCiWatchService>(),
     attributionPolicy: sp.GetRequiredService<CodeyBox.Core.CommitAttributionPolicy>(),
     branchOwnership: sp.GetRequiredService<CodeyBox.Core.UpstreamBranchOwnershipSnapshot>()));
 builder.Services.AddSingleton<IPipelineRunner>(sp => sp.GetRequiredService<PipelineRunner>());

src/CodeyBox.Core/AuditLog.cs

@@ -1390,10 +1390,12 @@ public static void ProjectQueueResumed(ProjectId projectId) =>
             .Information("Project {ProjectId} queue resumed", projectId.Value);
 
     /// <summary>
-    /// A required-build failure was attributed to the project's base branch
-    /// (the base tip reproduces it with no contributing error file inside the
-    /// item diff). While the condition is active the dispatcher holds
-    /// build-dependent phases for the project.
+    /// A failure was attributed to the project's base branch: a
+    /// required-build failure the base tip reproduces with no contributing
+    /// error file inside the item diff, or a test-red tip (forge CI red for
+    /// the merge commit, or audit test failures reproducing on the base).
+    /// While the condition is active the dispatcher holds build-dependent
+    /// phases for the project.
     /// </summary>
     public static void BaseBrokenDetected(
         ProjectId projectId,
@@ -1402,7 +1404,7 @@ public static void BaseBrokenDetected(
         string? errorSummary = null) =>
         Audit("project.base_broken_detected")
             .Warning(
-                "Project {ProjectId} base branch {BaseBranch} tip {BaseSha} fails the required build; build-dependent phases held: {ErrorSummary}",
+                "Project {ProjectId} base branch {BaseBranch} tip {BaseSha} is broken (build or test red); build-dependent phases held: {ErrorSummary}",
                 projectId.Value.Replace("\r", "").Replace("\n", ""),
                 AuditSingleLine(baseBranch, 128),
                 AuditSingleLine(baseSha, 64),

src/CodeyBox.Core/IBaseBrokenConditionStore.cs

@@ -1,23 +1,46 @@
 namespace CodeyBox.Core;
 
 /// <summary>
-/// A recorded "base branch does not build" condition for a project. Created
-/// when a required-build failure on a work item cannot be attributed to the
-/// item's diff and the base branch tip reproduces the same failure. While a
-/// row is active (<see cref="ClearedAt"/> is null) the dispatcher holds
-/// build-dependent phases (work / audit / merge) for the project — except
-/// the auto-filed fix item (<see cref="FixWorkItemId"/>), which must run to
-/// repair the base. Rows are keyed by base tip SHA: a base that moved to a
-/// different commit is a different condition.
+/// How the base branch tip proved broken: a failing required build, or
+/// failing tests on a tree that builds (a merged tree whose tests regress
+/// even though every parent branch was green on its own base).
+/// </summary>
+public enum BaseBrokenFailureKind
+{
+    /// <summary>The base tip fails the required build.</summary>
+    Build = 0,
+
+    /// <summary>
+    /// The base tip builds but its tests fail (forge CI red for the tip, or
+    /// audit test failures that genuinely reproduce on the base). The
+    /// dispatcher hold, fix-item dedupe, and monitor sweep treat both kinds
+    /// identically; only the clear signal differs (build pass vs CI pass).
+    /// </summary>
+    Test = 1,
+}
+
+/// <summary>
+/// A recorded "base branch is broken" condition for a project — either it
+/// does not build, or it builds but its tests fail. Build-kind rows are
+/// created when a required-build failure on a work item cannot be attributed
+/// to the item's diff and the base branch tip reproduces the same failure;
+/// test-kind rows are created when the post-merge CI watch reports the
+/// landed tip red, or when an item's test-only audit failures all reproduce
+/// on the base while the tip's CI is red. While a row is active
+/// (<see cref="ClearedAt"/> is null) the dispatcher holds build-dependent
+/// phases (work / audit / merge) for the project — except the auto-filed
+/// fix item (<see cref="FixWorkItemId"/>), which must run to repair the
+/// base. Rows are keyed by base tip SHA: a base that moved to a different
+/// commit is a different condition.
 /// </summary>
 public sealed record BaseBrokenCondition
 {
     public required ProjectId ProjectId { get; init; }
 
-    /// <summary>The base branch whose tip failed the required build.</summary>
+    /// <summary>The base branch whose tip proved broken.</summary>
     public required string BaseBranch { get; init; }
 
-    /// <summary>The base branch tip SHA that reproduced the build failure.</summary>
+    /// <summary>The base branch tip SHA that reproduced the failure.</summary>
     public required string BaseSha { get; init; }
 
     /// <summary>
@@ -28,15 +51,38 @@ public sealed record BaseBrokenCondition
     /// </summary>
     public required string RepositoryId { get; init; }
 
-    /// <summary>Bounded, sanitized summary of the failing base build output.</summary>
+    /// <summary>
+    /// Bounded, sanitized summary of the failing evidence: base build output
+    /// for <see cref="BaseBrokenFailureKind.Build"/>, failing test/check
+    /// names plus suspect commits for
+    /// <see cref="BaseBrokenFailureKind.Test"/>.
+    /// </summary>
     public string? ErrorSummary { get; init; }
 
+    /// <summary>
+    /// Which evidence proved the base broken. Pre-existing rows (written
+    /// before this field existed) read back as
+    /// <see cref="BaseBrokenFailureKind.Build"/>.
+    /// </summary>
+    public BaseBrokenFailureKind FailureKind { get; init; } = BaseBrokenFailureKind.Build;
+
+    /// <summary>
+    /// Failing test or CI-check names for
+    /// <see cref="BaseBrokenFailureKind.Test"/> (sanitized, bounded at the
+    /// write site). Empty for build-kind rows. Pre-existing rows read back
+    /// empty.
+    /// </summary>
+    public IReadOnlyList<string> FailingNames { get; init; } = [];
+
     /// <summary>The auto-filed fix item for this base SHA, once filed.</summary>
     public WorkItemId? FixWorkItemId { get; init; }
 
     public required DateTimeOffset DetectedAt { get; init; }
 
-    /// <summary>Set when the base tip builds again; null while the hold is active.</summary>
+    /// <summary>
+    /// Set when the base is healthy again (tip builds for build-kind, tip CI
+    /// passes for test-kind); null while the hold is active.
+    /// </summary>
     public DateTimeOffset? ClearedAt { get; init; }
 }
 

src/CodeyBox.Orchestrator/AgentInvolvementOutcomes.cs

@@ -40,7 +40,10 @@ internal static class AgentInvolvementOutcomes
         NoActionRequiredException => Success,
         // The build failure traces to the base branch, not to anything the
         // agent produced — the attempt is infrastructure, not agent fault.
+        // A test-red base parks the same way (the tests fail on the base
+        // tip, not because of the diff).
         BaseBuildBrokenException => FailureInfrastructure,
+        BaseTestBrokenException => FailureInfrastructure,
         OperationCanceledException => FailureCancelled,
         _ => FailureAgent,
     };