docs(design): Phase 2 evidence-plane ADRs (proposed)

docs/design/0000-phase2-sequencing.md

@@ -0,0 +1,85 @@
+# ADR-0000: Phase 2 sequencing
+
+## Status
+
+Proposed
+
+## Date
+
+2026-10-18
+
+## Context
+
+The [v1 contract](../spec/RECEIPT-v1.md) is frozen.
+The [ledger](../../internal/receipt/ledger.go) and [signer](../../internal/signer/store.go) already coordinate group commits and rotation leases.
+Phase 2 proposes new protocols around those guarantees.
+These documents change no production code.
+
+## Decision Drivers
+
+- Preserve v1 evidence and offline verification.
+- Separate schema agreement from implementation.
+- Keep air-gapped deployments independent of cloud services.
+
+## Considered Options
+
+| Option | Pros | Cons |
+| --- | --- | --- |
+| Implement all proposals together | Few temporary interfaces | Hard failure attribution and broad review |
+| Sequence contracts, then independent components | Smaller tests and clearer dependencies | Requires agreed artifact fixtures first |
+
+## Decision
+
+Propose this dependency order:
+
+| Step | ADR | Prerequisites | Acceptance gate |
+| --- | --- | --- | --- |
+| 1 | [0001](0001-per-replica-chains.md) | None | Agree v2 chain identity and unchanged v1 fixtures |
+| 2 | [0005](0005-root-certified-keys.md) | 0001 | Agree certificate scopes and epoch rotation fixtures |
+| 2a | [0002](0002-two-phase-receipts.md) | 0001 | Verify durable intent ordering and crash recovery with pinned raw keys |
+| 3b | [0003](0003-durable-shipping.md) | 0001, 0005 | Recover and verify sealed segments offline |
+| 4a | [0004](0004-merkle-checkpoints.md) | 0003, 0005 | Verify registry-backed head vectors; gate later Merkle proofs |
+| 3c | [0006](0006-receipt-failure-modes.md) | 0001, 0002 | Verify 503 responses and authenticated control receipts |
+| 5 | [0007](0007-kubernetes-lifecycle.md) | 0001, 0002, 0003, 0005 | Pass enrolment, drain, volume fencing, and outage tests |
+
+0004 is not a prerequisite for Kubernetes ledger availability.
+Coverage-publication tests require 0004, independently of gateway readiness.
+0002 does not require 0005. Pinned raw keys are sufficient for its v2 lifecycle tests.
+0007 needs 0006's spool protocol only if the owner later enables the deferred spool.
+That deferred protocol also depends on 0003 for direct segment shipping.
+Multi-replica release remains blocked until authorization and vault provisioning are solved.
+
+## Consequences
+
+### Positive
+
+0002 and 0005 can proceed independently after the chain contract settles.
+0003 and the fail-closed portion of 0006 can then proceed independently.
+0004 publication does not block 0006.
+Tessera POSIX feasibility tests, benchmark reruns, and Kubernetes manifest drafts can start earlier.
+V1 golden-vector regression work is independent of every new protocol.
+
+### Negative
+
+Final integration waits for stable fixtures and key provisioning.
+Manifest-signer validity-window hardening remains separate work, not a prerequisite for preserving v1 bytes.
+
+### Risks and Mitigations
+
+Interface drift could invalidate parallel work. Freeze shared fixtures before merging dependent implementations.
+Additional commits could worsen latency. Re-run Phase 0 workloads before accepting performance claims.
+
+## Implementation Notes
+
+1. Review [RECEIPT-v2](../spec/RECEIPT-v2.md) before any producer emits v2.
+2. Land compatibility tests before new encoding or epoch storage.
+3. Require each ADR's acceptance tests before enabling its feature.
+4. Run integrated crash tests and disconnected verification before declaring Phase 2 usable.
+
+### Owner Decisions
+
+The [open decisions](README.md#open-decisions) list recommended defaults and remaining approval gates.
+
+## Related ADRs
+
+[Index](README.md) links all 7 proposed decisions.

docs/design/0001-per-replica-chains.md

@@ -0,0 +1,141 @@
+# ADR-0001: Per-replica chains and epochs
+
+## Status
+
+Proposed
+
+## Date
+
+2026-10-18
+
+## Context
+
+The [ledger](../../internal/receipt/ledger.go) reads 1 head per `BEGIN IMMEDIATE` batch.
+[Sequence](../../internal/db/migrations/002_receipts.sql) is global within its database.
+The [signer](../../internal/signer/store.go) supports only 1 gateway writer per database.
+
+[V1](../spec/RECEIPT-v1.md) authenticates neither a replica identifier nor an epoch.
+Unsigned metadata cannot add authenticated identity.
+
+## Decision Drivers
+
+- Avoid cross-replica SQLite locks.
+- Distinguish rotation and replacement.
+- Preserve v1 bytes and offline verification.
+
+## Considered Options
+
+| Option | Pros | Cons |
+| --- | --- | --- |
+| Shared global chain | Simple total order | Shared writer bottleneck and coordination dependency |
+| Per-replica chains without epochs | Local availability | Recovery and key replacement can reuse identities |
+| Per-replica epoch chains | Local commits and explicit boundaries | Auditors must inspect several chains |
+
+## Decision
+
+Propose a required authenticated v2 field: `chain_id = <replica-id>:<epoch>`.
+The [v2 outline](../spec/RECEIPT-v2.md) records encoding boundaries.
+`replica-id` is a provisioned UUID, not a reusable pod ordinal.
+Persist a positive, monotonically increasing `epoch` with replica identity.
+The signed epoch-open control record is exactly `seq = 1`, with a zero previous hash.
+The uniqueness key becomes `(chain_id, seq)`.
+
+Intact restarts resume their epoch unless a loss or takeover cutoff requires another.
+Rotation starts another epoch.
+Lost storage requires a new UUID, even for reused ordinals.
+There is no total cross-replica execution order.
+
+### Fencing and Replica Authority
+
+Propose an external epoch lease with a cached ownership view checked before admission and every writer batch.
+Use a Kubernetes Lease by default. Non-Kubernetes installations may use conditional object updates on an epoch-open lease.
+Renew in the background. Never call the API server per batch.
+Use a monotonic self-fencing deadline, approximately 10 seconds with 5-second renewals and a 15-second lease.
+The deadline must be shorter than lease duration minus maximum clock drift.
+Measure from the start of the last successful renewal attempt, not its delayed acknowledgment.
+Deadline expiry stops admission and signing, including batches still waiting for SQLite.
+The authoritative registry allocates monotonically increasing fencing tokens through compare-and-swap.
+Lease acquisition requires the current workload identity, not merely a copied receipt certificate.
+Include the token in epoch-open, subsequent records, and segment descriptors.
+Confirmed lease loss fences immediately; failed renewals never extend the cached deadline.
+Loss declarations and lease takeovers for the same UUID require an intermediate-signed `trust_status` cutoff at the last covered head.
+Bind chain ID, sequence, and head hash; without coverage, use sequence 0 and the zero hash.
+Verifiers reject every old-epoch record beyond that cutoff, including an unshipped local suffix.
+Persist the cutoff before activating takeover; continue under a newly certified epoch, never beyond the old cutoff.
+Registry handoffs bind that covered head and token boundary after storage fencing.
+Already-dispatched upstream calls cannot be recalled by a lease.
+
+The intermediate issuer refuses to certify the same `(replica UUID, epoch)` twice.
+That rule alone cannot stop a clone reusing an existing certificate and private key.
+The external lease and storage fencing are therefore required, not optional certificate features.
+Kubernetes Lease ownership alone does not physically fence a partitioned volume writer.
+Before takeover, establish storage fencing or refuse failover.
+
+Propose an authoritative replica registry backed by an append-only, signed issued-certificate log.
+Store it outside gateway PVCs, so replica loss cannot erase registration.
+The intermediate signs registrations, certificate digests, fencing generations, replacements, and loss declarations.
+Each entry binds organization, installation, replica UUID, epoch, and deployment slot.
+Inventories bind the registry revision and digest; offline bundles include its signed snapshot and required log entries.
+A registered chain with neither epoch-close nor inventory coverage is reported `lost`, even when no receipt survives.
+This reports missing evidence, not proof that the replica never executed work.
+Unclosed chains with partial coverage report the last authenticated head and an unknown suffix.
+Missing registry history prevents a complete installation-level coverage claim.
+Loss detection is relative to the supplied registry revision.
+Without independently trusted freshness, unseen registrations cannot be ruled out.
+
+Epoch-open references the predecessor's terminal hash and final segment digest when available.
+Epoch-close is the final record and fixes the epoch's terminal sequence.
+A missing predecessor is explicit. Restoring an old backup requires enrolment with a new UUID.
+
+Propose dual-version verification with frozen v1 preimages, signatures, and accepted semantics.
+V1 receipts are never re-encoded during migration or shipping.
+Keep original JSONL bytes and vectors.
+Existing v1-only binaries will still reject v2.
+
+Migration closes the v1 ledger at a durable head.
+A migration descriptor binds head, original exports, legacy keys, and new v2 chain.
+The organization root and final v1 signing key must co-sign the same descriptor preimage.
+Neither intermediate signatures nor embedded keys replace those approvals.
+That descriptor does not add fields to historical receipts.
+The legacy export retains its original trust requirements.
+
+## Consequences
+
+### Positive
+
+Local group commit remains.
+Epochs separate signing histories.
+Offline tooling finds missing registered chains, not just surviving files.
+
+### Negative
+
+Storage must separate epoch heads.
+Leases and registry add local dependencies.
+
+### Risks and Mitigations
+
+Replica cloning can create forks. Require external fencing and reject duplicate positions with different hashes.
+An unpublished chain can disappear. Compare the issued-certificate log with signed inventory coverage.
+Epoch resets could hide missing history. Require explicit predecessor references and report unverifiable gaps.
+
+## Implementation Notes
+
+1. Freeze v2 hash fields and chain-ID grammar with cross-language fixtures before implementing storage.
+2. Add dual-version dispatch without changing the v1 parser or golden encodings.
+   Acceptance: all existing v1 vectors retain their results.
+3. Introduce durable replica identity, epoch allocation, and separate chain heads.
+   Acceptance: concurrent replicas can both emit sequence 1 without collision.
+4. Test concurrent lease acquisition, stale tokens, renewal loss, and copied certificates.
+   Acceptance: only the fenced owner admits work; duplicate certificate issuance is refused.
+   Acceptance: cached checks make no per-batch API calls; monotonic deadline expiry fences despite wall-clock changes.
+5. Remove every segment of a registered chain.
+   Acceptance: offline coverage reports it lost, rather than silently omitting it.
+6. Interrupt migration and verify both co-signatures.
+   Acceptance: either missing signature fails; v1 bytes remain identical; epoch-open occupies sequence 1.
+
+## Related ADRs
+
+- [0002: Lifecycle references](0002-two-phase-receipts.md).
+- [0003: Segment boundaries](0003-durable-shipping.md).
+- [0005: Issuance](0005-root-certified-keys.md).
+- [0007: Identity and fencing](0007-kubernetes-lifecycle.md).

docs/design/0002-two-phase-receipts.md

@@ -0,0 +1,134 @@
+# ADR-0002: Durable intent and outcome receipts
+
+## Status
+
+Proposed
+
+## Date
+
+2026-10-18
+
+## Context
+
+The [gateway](../../internal/gateway/gateway.go) executes upstream work, commits its v1 receipt, then responds.
+The [ledger](../../internal/receipt/ledger.go) documents the crash gap: an action can succeed before its receipt commits.
+
+[SSE](../../internal/gateway/stream.go) starts only after a header-time receipt commits.
+Digests reach logs, not receipts. The [stream-digest proposal](../spec/proposals/receipt-stream-digest.md) is unimplemented.
+
+## Decision Drivers
+
+- Record a durable obligation before dispatch.
+- Report uncertainty without replaying external side effects.
+- Preserve receipt-before-response, including SSE headers.
+- Measure added commit latency.
+
+## Considered Options
+
+| Option | Pros | Cons |
+| --- | --- | --- |
+| Outcome only | Current latency and simple schema | Crash can erase evidence of an action |
+| Intent only | Records attempted dispatch | Cannot attest observed outcomes |
+| Durable intent, then outcome | Exposes missing outcomes | Adds latency and recovery state |
+
+## Decision
+
+Propose v2 lifecycle records without changing v1.
+After authentication, validation, and policy evaluation, allocate an attempt ID.
+Persist a signed `intent` before any upstream call, including credential refresh.
+Bind identity, delegation, params digest, policy, action, and any validated client request ID.
+Denied attempts get a terminal decision record without an upstream intent.
+
+An acknowledged intent permits 1 logical execution under existing safe retries.
+Append timeout can still leave a committed record. Resolve ambiguity before dispatch, never by retrying upstream.
+
+Append a signed `outcome` after observing the upstream result.
+It references the intent's `chain_id`, sequence, entry hash, and attempt ID.
+Include `upstream_attempt_count`, counting actual action dispatches, including transport retries.
+Exclude credential refresh calls.
+Use `observation = not_dispatched` only when the running process knows no action dispatch occurred.
+That observation requires a known zero count, not an invented HTTP status.
+Crash recovery still writes `outcome_unknown`, with unknown attempt count unless durable evidence establishes it.
+For buffered responses, commit the outcome before returning status or body.
+Use a bounded commit context detached from client cancellation, as today.
+
+For SSE, commit an `outcome` with `observation = headers_accepted` before forwarding headers.
+That outcome does not claim stream completion.
+Append a supplemental `stream_result` at termination.
+Bind outcome hash, written-byte SHA-256, byte count, and termination reason.
+The digest covers downstream writes, not proven client delivery.
+A missing terminal record means unknown completion, never success.
+Keep terminal artifacts out of forwarded SSE bytes.
+This proposed lifecycle replaces the earlier stream proposal's single-action-receipt rule for v2 only.
+
+Before admitting traffic after restart, scan committed intents without outcomes.
+Append recovery outcomes marked `outcome_unknown`. Never call upstream during recovery.
+For header outcomes without terminal stream records, append `stream_result` with unknown termination and no fabricated digest.
+Fenced recovery and unique references must prevent duplicate terminal records.
+Recovery records can reference an earlier epoch.
+Rotation does not drain inflight requests; their outcomes and stream results can enter the next epoch.
+
+### Client Retry Linkage
+
+Validate bounded `client_request_id` syntax; bind it to the principal and destination in the intent.
+Defer client retry deduplication and mapping an attempt ID to upstream `Idempotency-Key`.
+Providers differ in key support and retention. A new gateway attempt can repeat an external side effect.
+Claim no exactly-once behavior; add no unsupported provider header.
+The [owner defaults](README.md#open-decisions) disable automatic replay.
+
+### Group-Commit Cost
+
+Intent and outcome cannot share a transaction for their own request.
+Other requests can still share each batch's fsync.
+Allowed buffered attempts require 2 durable boundaries instead of 1.
+SSE adds a later terminal record as well.
+
+`bench/RESULTS.md` is absent on the pinned main baseline.
+[Phase 0 PR #50](https://github.com/Clawdlinux/agentgate/pull/50), head `29ef207`, reports these added-overhead p99 measurements:
+
+| Workload | Existing 1-receipt path |
+| --- | --- |
+| Closed loop, 0 ms upstream, concurrency 1 | 0.34 ms |
+| Closed loop, 0 ms upstream, concurrency 16 | 3.42 ms |
+| Closed loop, 0 ms upstream, concurrency 64 | 6.22 ms |
+| Open loop, 0 ms upstream, 25% load | 7.00 ms |
+
+The PR used Apple M4, `GOMAXPROCS=4`, and 20,000 requests per row.
+It did not meet the under-1-ms p99 gate.
+These are end-to-end overheads, not fsync measurements. Doubling them cannot predict two-phase p99.
+Budget 1 extra serialized commit wait before dispatch, then benchmark it on Linux with dedicated 4-vCPU capacity.
+
+## Consequences
+
+### Positive
+
+Crashes leave inspectable obligations and explicit unknown outcomes.
+
+### Negative
+
+Writes and latency increase. Recovery needs lifecycle indexes.
+
+### Risks and Mitigations
+
+An unknown outcome may hide an executed action. Require operator reconciliation, not automatic replay.
+SSE can fail after response headers. End the stream and persist uncertainty when storage recovers.
+
+## Implementation Notes
+
+1. Freeze lifecycle references and optional fields in [v2](../spec/RECEIPT-v2.md).
+   Acceptance: reject missing, duplicate, or mismatched intent references.
+2. Add intent ordering instrumentation.
+   Acceptance: the mock upstream sees no call before intent commit acknowledgment.
+   Acceptance: runtime cancellation before dispatch yields `not_dispatched` and zero attempts; retry counts include every action dispatch.
+3. Run subprocess `kill -9` tests before intent commit, after commit, during dispatch, and after provider success.
+   Also kill between the upstream call and outcome commit, after outcome commit, and during SSE.
+   Acceptance: recovery is idempotent, records unknown states, and never re-dispatches.
+4. Re-run Phase 0 closed-loop and open-loop workloads.
+   Acceptance: publish intent-wait and outcome-wait percentiles separately; obtain owner approval for the latency budget.
+
+## Related ADRs
+
+- [0001: References](0001-per-replica-chains.md).
+- [0003: Shipping](0003-durable-shipping.md).
+- [0006: Closed failure mode](0006-receipt-failure-modes.md).
+- [0007: Stream drain](0007-kubernetes-lifecycle.md).