Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/concepts/work-items.md
Original file line number Diff line number Diff line change
Expand Up @@ -180,6 +180,8 @@ POST /workitems/{id}/redrive-upstream

Two recovery paths cover items the recorded-number join would miss. When the item has no recorded PR number, the open PR is discovered by head branch (`codeybox/<id>`), recorded on the item, and used; when no open PR exists at all, the push is re-driven alone and the pipeline opens a fresh PR. When the branch has no recorded push, the remote tip must prove itself: the re-drive (and the push path itself) accepts the tip only when every commit exclusive of the current base tip carries any accepted CodeyBox trailer (`CodeyBox:UpstreamBranchOwnership`) — commits already reachable from the base never count against the branch — otherwise it keeps refusing with a message that names the offending commit SHAs, this endpoint, and the exact flag. Pass `{"confirmOwnership": true}` in the `POST` body to assert the tip is CodeyBox's own history explicitly — the assertion is logged and audited.

Every successful work-branch push is recorded on the item immediately (latest sha plus a bounded history, `CodeyBox:UpstreamBranchOwnership:MaxPushHistoryEntries`), on all paths — normal push, re-drive retry, lease-guarded rewrite, and conflict-rework recomposition — including when the later PR/merge steps fail. Before a moved remote is treated as third-party work, the push path also accepts a tip equal to any recorded push, or a tip whose exclusive commits all carry this instance's CodeyBox trailers. A genuinely foreign tip (unrecorded sha, unmarked commits) is still refused and the branch is left untouched.

### AbandonedAfterRecoveryAttempts

When the recovery loop has retried an item more than `CodeyBox:DeadWorker:MaxRecoveryAttempts` times (default 10) without it ever completing the recovered phase, the item is transitioned to `AbandonedAfterRecoveryAttempts` with a descriptive `lastError`. Use `POST /workitems/{id}/retry` to resume manually after investigating the root cause.
Expand Down
4 changes: 3 additions & 1 deletion docs/reference/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -629,14 +629,16 @@ Proves a stale CodeyBox-owned branch tip is CodeyBox's own history when no prior
```json
"UpstreamBranchOwnership": {
"RequiredTrailerKeys": ["Co-Authored-By", "CodeyBox-WorkItem", "CodeyBox-Agent", "CodeyBox-Prompt-Revision", "CodeyBox-Mechanical-Fixer"],
"MaxCommitsToVerify": 100
"MaxCommitsToVerify": 100,
"MaxPushHistoryEntries": 20
}
```

| Key | Default | Description |
|-----|---------|-------------|
| `RequiredTrailerKeys` | `["Co-Authored-By", "CodeyBox-WorkItem", "CodeyBox-Agent", "CodeyBox-Prompt-Revision", "CodeyBox-Mechanical-Fixer"]` | Trailer keys accepted as ownership proof, any-of: each commit exclusive of the current base tip counts as CodeyBox's own history when its trailer block carries ANY one of these keys (matched exactly). Any-of because each writer stamps a different subset (agents write `CodeyBox-Prompt-Revision` + `Co-Authored-By`; the orchestrator adds `CodeyBox-WorkItem` + `CodeyBox-Agent` / `CodeyBox-Mechanical-Fixer`). `Co-Authored-By` additionally requires its value to equal the canonical CodeyBox identity exactly. Keys from a family disabled by `CodeyBox:CommitAttribution` cannot prove ownership; when attribution disables every family (or the list is empty), nothing proves ownership and the re-drive requires `{"confirmOwnership": true}`. A refusal names the offending commit SHAs. |
| `MaxCommitsToVerify` | `100` | Upper bound on the exclusive commits examined, enforced before buffering. A larger exclusive set is unverifiable and refused unless the operator passes `{"confirmOwnership": true}` on the re-drive. |
| `MaxPushHistoryEntries` | `20` | Upper bound on the per-item history of pushed work-branch tips. Every successful push is recorded (latest sha plus this bounded history); a diverged remote tip matching ANY retained entry is accepted as CodeyBox's own history without consulting commit trailers — the only proof available when trailers are disabled or the exclusive range is unverifiable. Oldest entries drop first; values below 1 behave as 1. |

## `WorkerPoolHealthWatchdog`

Expand Down
3 changes: 2 additions & 1 deletion src/CodeyBox.Api/Program.cs
Original file line number Diff line number Diff line change
Expand Up @@ -4260,7 +4260,8 @@ static Func<TestRunOptions> DotnetTestRunOptionsAccessor(IServiceProvider sp)
providerTransientCorrelation: sp.GetRequiredService<ProviderTransientCorrelationTracker>(),
baseBuildVerifier: sp.GetRequiredService<BaseBuildVerifier>(),
baseBrokenConditions: sp.GetRequiredService<BaseBrokenConditionTracker>(),
attributionPolicy: sp.GetRequiredService<CodeyBox.Core.CommitAttributionPolicy>()));
attributionPolicy: sp.GetRequiredService<CodeyBox.Core.CommitAttributionPolicy>(),
branchOwnership: sp.GetRequiredService<CodeyBox.Core.UpstreamBranchOwnershipSnapshot>()));
builder.Services.AddSingleton<IPipelineRunner>(sp => sp.GetRequiredService<PipelineRunner>());
// Isolated base-branch fix-item spawner for NotDiffAttributable audit test
// failures. Constructed lazily from the store/queue plus the hot-reloadable
Expand Down
29 changes: 27 additions & 2 deletions src/CodeyBox.Core/IUpstreamRemote.cs
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,16 @@ public interface IUpstreamRemote
/// <summary>Stable identifier for diagnostics ("noop", "github", "git-generic").</summary>
string Name { get; }

/// <summary>
/// True when <see cref="CompleteAsync"/> pushes the item's work branch to
/// the upstream (the common case). False for upstreams whose completion
/// pushes a different ref (git-generic lands the base branch) or nothing
/// at all (noop). The orchestrator records a pushed work-branch tip only
/// when this is true, so a recorded sha is always evidence of a real
/// work-branch push — never a guess.
/// </summary>
bool PushesWorkBranch => true;

/// <summary>
/// Pushes the named ref from the host bare repo to the upstream. The
/// repository identifier is opaque and must be understood by the host
Expand Down Expand Up @@ -757,12 +767,27 @@ public sealed record UpstreamCompletionRequest
/// as <c>LastPushedWorkBranchSha</c>), or an operator-authorized sha
/// observed on the remote during a re-drive. When the remote tip has
/// diverged from the new head, the implementation rewrites the owned
/// branch only under a <c>--force-with-lease</c> bound to this sha; a
/// mismatch means a third party pushed and the rewrite is refused.
/// branch only under a <c>--force-with-lease</c> bound to this sha. A
/// moved remote is first checked against <see cref="KnownWorkBranchShas"/>
/// and the commit-trailer ownership proof before it is refused as
/// third-party work; only a foreign or unverifiable tip is refused.
/// Null when CodeyBox has no record (first push, or legacy rows): a
/// diverged remote is then reported distinctly rather than rewritten.
/// </summary>
public string? ExpectedRemoteHeadSha { get; init; }

/// <summary>
/// Every work-branch tip sha CodeyBox has recorded pushing for this item
/// (bounded history, newest last; see
/// <c>WorkItem.PushedWorkBranchShaHistory</c>). When the remote tip
/// matches <see cref="ExpectedRemoteHeadSha"/> the lease proceeds as
/// usual; when it has moved, a tip equal to ANY entry here is still
/// accepted as CodeyBox's own history before the implementation falls
/// back to the commit-trailer ownership proof. Null or empty when no
/// history was recorded (first push, legacy rows): only the trailer
/// proof can then authorize a rewrite.
/// </summary>
public IReadOnlyList<string>? KnownWorkBranchShas { get; init; }
}

/// <summary>Result of <see cref="IUpstreamRemote.CompleteAsync"/>.</summary>
Expand Down
13 changes: 13 additions & 0 deletions src/CodeyBox.Core/UpstreamBranchOwnership.cs
Original file line number Diff line number Diff line change
Expand Up @@ -37,10 +37,23 @@ public sealed class UpstreamBranchOwnershipOptions
/// </summary>
public int MaxCommitsToVerify { get; set; } = 100;

/// <summary>
/// Upper bound on the per-item pushed-sha history retained for the lease
/// fallback (see <see cref="WorkItemPushHistory"/>). A diverged remote
/// tip matching any retained entry is accepted as CodeyBox's own history
/// without consulting commit trailers — the only proof available when
/// trailers are disabled by configuration or the exclusive range is
/// unverifiable. Enforced at record time (oldest entries drop first), so
/// the stored history never grows without bound. Default 20; must be at
/// least 1 (smaller values behave as 1).
/// </summary>
public int MaxPushHistoryEntries { get; set; } = 20;

public UpstreamBranchOwnershipOptions Clone() => new()
{
RequiredTrailerKeys = (string[])RequiredTrailerKeys.Clone(),
MaxCommitsToVerify = MaxCommitsToVerify,
MaxPushHistoryEntries = MaxPushHistoryEntries,
};
}

Expand Down
11 changes: 11 additions & 0 deletions src/CodeyBox.Core/WorkItem.cs
Original file line number Diff line number Diff line change
Expand Up @@ -735,6 +735,17 @@ AgentTurnResumeCheckpoint is not null
/// </summary>
public string? LastPushedWorkBranchSha { get; init; }

/// <summary>
/// Bounded history of work-branch tip shas CodeyBox has pushed to the
/// upstream for this item, oldest first (see
/// <see cref="WorkItemPushHistory"/>). <see cref="LastPushedWorkBranchSha"/>
/// is always the newest entry. A later retry accepts a diverged remote
/// tip that equals ANY entry here — not just the latest — before falling
/// back to the commit-trailer ownership proof. Empty for legacy rows
/// predating push tracking.
/// </summary>
public IReadOnlyList<string> PushedWorkBranchShaHistory { get; init; } = [];

/// <summary>
/// The release this work item belongs to. When set, the orchestrator targets the
/// release branch instead of the project's default base branch, and the release state
Expand Down
92 changes: 92 additions & 0 deletions src/CodeyBox.Core/WorkItemPushHistory.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
namespace CodeyBox.Core;

/// <summary>
/// Pure bookkeeping for the bounded per-item history of pushed work-branch
/// tip shas. Every successful work-branch push (plain or lease-guarded, on
/// any path) is recorded through <see cref="RecordPush"/> so a later retry
/// can prove a diverged remote tip is CodeyBox's own history even when the
/// latest record went stale (crash between push and persist, legacy rows).
/// The history also backs the lease check when commit trailers are disabled
/// by configuration and no trailer proof is possible.
/// </summary>
public static class WorkItemPushHistory
{
/// <summary>
/// Default cap on retained entries when no configured bound is
/// available. The configured
/// <see cref="UpstreamBranchOwnershipOptions.MaxPushHistoryEntries"/>
/// wins whenever options are present.
/// </summary>
public const int DefaultMaxEntries = 20;

/// <summary>
/// Guards the persisted-state sink: only plausible commit ids (hex,
/// 40–64 chars after trimming) qualify as pushed shas. Forge-reported
/// tips are untrusted input, and lease decisions branch on the recorded
/// history, so garbage must never enter it.
/// </summary>
public static bool IsPlausibleSha(string? sha)
{
if (string.IsNullOrWhiteSpace(sha))
return false;
var trimmed = sha.Trim();
if (trimmed.Length is < 40 or > 64)
return false;
foreach (var c in trimmed)
{
if (!Uri.IsHexDigit(c))
return false;
}
return true;
}

/// <summary>
/// Returns <paramref name="item"/> with <paramref name="sha"/> recorded
/// as the latest push: <see cref="WorkItem.LastPushedWorkBranchSha"/>
/// becomes <paramref name="sha"/> and the sha is appended to
/// <see cref="WorkItem.PushedWorkBranchShaHistory"/> (moved to the end
/// when already present), truncated to the newest
/// <paramref name="maxEntries"/> entries. Blank or implausible shas
/// (see <see cref="IsPlausibleSha"/>) leave the item unchanged — a guess
/// is never persisted. Pure: the input is never mutated.
/// </summary>
public static WorkItem RecordPush(WorkItem item, string? sha, int maxEntries = DefaultMaxEntries)
{
ArgumentNullException.ThrowIfNull(item);
if (!IsPlausibleSha(sha))
return item;
var bound = Math.Max(1, maxEntries);
var trimmed = sha!.Trim();
var history = item.PushedWorkBranchShaHistory
.Where(IsPlausibleSha)
.Select(s => s.Trim())
.Where(s => !string.Equals(s, trimmed, StringComparison.OrdinalIgnoreCase))
.ToList();
history.Add(trimmed);
while (history.Count > bound)
history.RemoveAt(0);
return item with
{
LastPushedWorkBranchSha = trimmed,
PushedWorkBranchShaHistory = history,
};
}

/// <summary>
/// True when <paramref name="sha"/> equals any recorded push for
/// <paramref name="item"/> (latest or history), compared ordinally
/// ignoring case. Used by the lease guard before treating a moved remote
/// as third-party work.
/// </summary>
public static bool Contains(WorkItem item, string? sha)
{
ArgumentNullException.ThrowIfNull(item);
if (string.IsNullOrWhiteSpace(sha))
return false;
var trimmed = sha.Trim();
if (string.Equals(item.LastPushedWorkBranchSha?.Trim(), trimmed, StringComparison.OrdinalIgnoreCase))
return true;
return item.PushedWorkBranchShaHistory.Any(s =>
string.Equals(s?.Trim(), trimmed, StringComparison.OrdinalIgnoreCase));
}
}
42 changes: 42 additions & 0 deletions src/CodeyBox.Orchestrator/PipelineRunner.AgentTurnResume.cs
Original file line number Diff line number Diff line change
Expand Up @@ -1270,6 +1270,48 @@ private async Task<bool> IsStaleAgentTurnCheckpointAsync(
StringComparison.Ordinal);
}

/// <summary>
/// True when the base branch moved past the checkpoint's source commit:
/// the current base tip is no longer contained in the checkpoint's
/// history, so the checkpoint tree was staged on an abandoned base and a
/// fresh turn from the current tip must replace it. This covers the
/// recompose/rebase case that the fork-point ancestry check above cannot
/// see: the pre-turn commit is the fork point of the tip and the source,
/// which is always an ancestor of the source itself, so that check alone
/// never reports a typed checkpoint stale. Applies only to typed refs
/// (the only shape carrying a source commit); anything unresolvable
/// answers false so an uncertain lineage resumes rather than discards.
/// </summary>
private async Task<bool> IsCheckpointBaseMovedAsync(
string repoId,
string baseBranch,
string checkpointSourceSha,
CancellationToken ct)
{
var source = checkpointSourceSha.Trim();
string baseTip;
try
{
baseTip = (await _gitHost.ResolveCommitAsync(repoId, baseBranch, ct).ConfigureAwait(false)).Trim();
}
catch (Exception ex) when (ex is not OperationCanceledException)
{
_log.LogDebug(ex, "Could not resolve base branch '{BaseBranch}' tip for checkpoint lineage check; assuming unmoved", baseBranch);
return false;
}
if (string.Equals(baseTip, source, StringComparison.OrdinalIgnoreCase))
return false;
try
{
return !await _gitHost.IsAncestorAsync(repoId, baseTip, source, ct).ConfigureAwait(false);
}
catch (Exception ex) when (ex is not OperationCanceledException)
{
_log.LogDebug(ex, "Could not check checkpoint lineage against base branch '{BaseBranch}'; assuming unmoved", baseBranch);
return false;
}
}

private const string StaleCheckpointLeaseRefusalMessage =
"Cannot discard a retained-sandbox recovery lease for a stale durable agent-turn checkpoint; retry the retained agent turn or cancel the item so provider cleanup remains authoritative.";

Expand Down
11 changes: 10 additions & 1 deletion src/CodeyBox.Orchestrator/PipelineRunner.State.cs
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,9 @@ public sealed partial class PipelineRunner
// a private snapshot (unchanging defaults) when DI does not supply one.
private readonly PipelineTuningSnapshot _pipelineTuning;
private readonly CommitAttributionPolicy _attribution;
// Hot-reloadable work-branch ownership knobs. Null when unwired (tests /
// legacy embedders): readers fall back to shipped defaults.
private readonly UpstreamBranchOwnershipSnapshot? _branchOwnership;
// Optional stale-base remediation router. When wired AND enabled, a
// stale-base conflict discovered at upstream-push time re-dispatches the
// item into conflict-rework instead of parking. Null preserves the
Expand Down Expand Up @@ -615,7 +618,12 @@ public PipelineRunner(
// condition, the fix-item dedupe, and the dispatcher hold.
BaseBuildVerifier? baseBuildVerifier = null,
BaseBrokenConditionTracker? baseBrokenConditions = null,
CommitAttributionPolicy? attributionPolicy = null)
CommitAttributionPolicy? attributionPolicy = null,
// Hot-reloadable work-branch ownership knobs (trailer proof, push
// history bound). Optional: when null the shipped defaults apply, so
// tests and embeddings that don't wire the snapshot keep the
// previous behaviour.
UpstreamBranchOwnershipSnapshot? branchOwnership = null)
{
_sandboxes = sandboxes;
_gitHost = gitHost;
Expand Down Expand Up @@ -740,6 +748,7 @@ public PipelineRunner(
_incrementalRebase = incrementalRebase;
_pipelineTuning = pipelineTuning ?? new PipelineTuningSnapshot(new PipelineTuningOptions());
_attribution = attributionPolicy ?? new CommitAttributionPolicy(new CommitAttributionSnapshot(new CommitAttributionOptions()));
_branchOwnership = branchOwnership;
_staleBaseReworkRouter = staleBaseReworkRouter;
_flakeEscalation = flakeEscalation;
_flakeEscalationOptions = flakeEscalationOptions;
Expand Down
Loading
Loading