diff --git a/docs/design/0000-phase2-sequencing.md b/docs/design/0000-phase2-sequencing.md new file mode 100644 index 0000000..cf31aee --- /dev/null +++ b/docs/design/0000-phase2-sequencing.md @@ -0,0 +1,85 @@ +# ADR-0000: Phase 2 sequencing + +## Status + +Proposed + +## Date + +2026-10-18 + +## Context + +The [v1 contract](../spec/RECEIPT-v1.md) is frozen. +The [ledger](../../internal/receipt/ledger.go) and [signer](../../internal/signer/store.go) already coordinate group commits and rotation leases. +Phase 2 proposes new protocols around those guarantees. +These documents change no production code. + +## Decision Drivers + +- Preserve v1 evidence and offline verification. +- Separate schema agreement from implementation. +- Keep air-gapped deployments independent of cloud services. + +## Considered Options + +| Option | Pros | Cons | +| --- | --- | --- | +| Implement all proposals together | Few temporary interfaces | Hard failure attribution and broad review | +| Sequence contracts, then independent components | Smaller tests and clearer dependencies | Requires agreed artifact fixtures first | + +## Decision + +Propose this dependency order: + +| Step | ADR | Prerequisites | Acceptance gate | +| --- | --- | --- | --- | +| 1 | [0001](0001-per-replica-chains.md) | None | Agree v2 chain identity and unchanged v1 fixtures | +| 2 | [0005](0005-root-certified-keys.md) | 0001 | Agree certificate scopes and epoch rotation fixtures | +| 2a | [0002](0002-two-phase-receipts.md) | 0001 | Verify durable intent ordering and crash recovery with pinned raw keys | +| 3b | [0003](0003-durable-shipping.md) | 0001, 0005 | Recover and verify sealed segments offline | +| 4a | [0004](0004-merkle-checkpoints.md) | 0003, 0005 | Verify registry-backed head vectors; gate later Merkle proofs | +| 3c | [0006](0006-receipt-failure-modes.md) | 0001, 0002 | Verify 503 responses and authenticated control receipts | +| 5 | [0007](0007-kubernetes-lifecycle.md) | 0001, 0002, 0003, 0005 | Pass enrolment, drain, volume fencing, and outage tests | + +0004 is not a prerequisite for Kubernetes ledger availability. +Coverage-publication tests require 0004, independently of gateway readiness. +0002 does not require 0005. Pinned raw keys are sufficient for its v2 lifecycle tests. +0007 needs 0006's spool protocol only if the owner later enables the deferred spool. +That deferred protocol also depends on 0003 for direct segment shipping. +Multi-replica release remains blocked until authorization and vault provisioning are solved. + +## Consequences + +### Positive + +0002 and 0005 can proceed independently after the chain contract settles. +0003 and the fail-closed portion of 0006 can then proceed independently. +0004 publication does not block 0006. +Tessera POSIX feasibility tests, benchmark reruns, and Kubernetes manifest drafts can start earlier. +V1 golden-vector regression work is independent of every new protocol. + +### Negative + +Final integration waits for stable fixtures and key provisioning. +Manifest-signer validity-window hardening remains separate work, not a prerequisite for preserving v1 bytes. + +### Risks and Mitigations + +Interface drift could invalidate parallel work. Freeze shared fixtures before merging dependent implementations. +Additional commits could worsen latency. Re-run Phase 0 workloads before accepting performance claims. + +## Implementation Notes + +1. Review [RECEIPT-v2](../spec/RECEIPT-v2.md) before any producer emits v2. +2. Land compatibility tests before new encoding or epoch storage. +3. Require each ADR's acceptance tests before enabling its feature. +4. Run integrated crash tests and disconnected verification before declaring Phase 2 usable. + +### Owner Decisions + +The [open decisions](README.md#open-decisions) list recommended defaults and remaining approval gates. + +## Related ADRs + +[Index](README.md) links all 7 proposed decisions. diff --git a/docs/design/0001-per-replica-chains.md b/docs/design/0001-per-replica-chains.md new file mode 100644 index 0000000..a1cc304 --- /dev/null +++ b/docs/design/0001-per-replica-chains.md @@ -0,0 +1,141 @@ +# ADR-0001: Per-replica chains and epochs + +## Status + +Proposed + +## Date + +2026-10-18 + +## Context + +The [ledger](../../internal/receipt/ledger.go) reads 1 head per `BEGIN IMMEDIATE` batch. +[Sequence](../../internal/db/migrations/002_receipts.sql) is global within its database. +The [signer](../../internal/signer/store.go) supports only 1 gateway writer per database. + +[V1](../spec/RECEIPT-v1.md) authenticates neither a replica identifier nor an epoch. +Unsigned metadata cannot add authenticated identity. + +## Decision Drivers + +- Avoid cross-replica SQLite locks. +- Distinguish rotation and replacement. +- Preserve v1 bytes and offline verification. + +## Considered Options + +| Option | Pros | Cons | +| --- | --- | --- | +| Shared global chain | Simple total order | Shared writer bottleneck and coordination dependency | +| Per-replica chains without epochs | Local availability | Recovery and key replacement can reuse identities | +| Per-replica epoch chains | Local commits and explicit boundaries | Auditors must inspect several chains | + +## Decision + +Propose a required authenticated v2 field: `chain_id = :`. +The [v2 outline](../spec/RECEIPT-v2.md) records encoding boundaries. +`replica-id` is a provisioned UUID, not a reusable pod ordinal. +Persist a positive, monotonically increasing `epoch` with replica identity. +The signed epoch-open control record is exactly `seq = 1`, with a zero previous hash. +The uniqueness key becomes `(chain_id, seq)`. + +Intact restarts resume their epoch unless a loss or takeover cutoff requires another. +Rotation starts another epoch. +Lost storage requires a new UUID, even for reused ordinals. +There is no total cross-replica execution order. + +### Fencing and Replica Authority + +Propose an external epoch lease with a cached ownership view checked before admission and every writer batch. +Use a Kubernetes Lease by default. Non-Kubernetes installations may use conditional object updates on an epoch-open lease. +Renew in the background. Never call the API server per batch. +Use a monotonic self-fencing deadline, approximately 10 seconds with 5-second renewals and a 15-second lease. +The deadline must be shorter than lease duration minus maximum clock drift. +Measure from the start of the last successful renewal attempt, not its delayed acknowledgment. +Deadline expiry stops admission and signing, including batches still waiting for SQLite. +The authoritative registry allocates monotonically increasing fencing tokens through compare-and-swap. +Lease acquisition requires the current workload identity, not merely a copied receipt certificate. +Include the token in epoch-open, subsequent records, and segment descriptors. +Confirmed lease loss fences immediately; failed renewals never extend the cached deadline. +Loss declarations and lease takeovers for the same UUID require an intermediate-signed `trust_status` cutoff at the last covered head. +Bind chain ID, sequence, and head hash; without coverage, use sequence 0 and the zero hash. +Verifiers reject every old-epoch record beyond that cutoff, including an unshipped local suffix. +Persist the cutoff before activating takeover; continue under a newly certified epoch, never beyond the old cutoff. +Registry handoffs bind that covered head and token boundary after storage fencing. +Already-dispatched upstream calls cannot be recalled by a lease. + +The intermediate issuer refuses to certify the same `(replica UUID, epoch)` twice. +That rule alone cannot stop a clone reusing an existing certificate and private key. +The external lease and storage fencing are therefore required, not optional certificate features. +Kubernetes Lease ownership alone does not physically fence a partitioned volume writer. +Before takeover, establish storage fencing or refuse failover. + +Propose an authoritative replica registry backed by an append-only, signed issued-certificate log. +Store it outside gateway PVCs, so replica loss cannot erase registration. +The intermediate signs registrations, certificate digests, fencing generations, replacements, and loss declarations. +Each entry binds organization, installation, replica UUID, epoch, and deployment slot. +Inventories bind the registry revision and digest; offline bundles include its signed snapshot and required log entries. +A registered chain with neither epoch-close nor inventory coverage is reported `lost`, even when no receipt survives. +This reports missing evidence, not proof that the replica never executed work. +Unclosed chains with partial coverage report the last authenticated head and an unknown suffix. +Missing registry history prevents a complete installation-level coverage claim. +Loss detection is relative to the supplied registry revision. +Without independently trusted freshness, unseen registrations cannot be ruled out. + +Epoch-open references the predecessor's terminal hash and final segment digest when available. +Epoch-close is the final record and fixes the epoch's terminal sequence. +A missing predecessor is explicit. Restoring an old backup requires enrolment with a new UUID. + +Propose dual-version verification with frozen v1 preimages, signatures, and accepted semantics. +V1 receipts are never re-encoded during migration or shipping. +Keep original JSONL bytes and vectors. +Existing v1-only binaries will still reject v2. + +Migration closes the v1 ledger at a durable head. +A migration descriptor binds head, original exports, legacy keys, and new v2 chain. +The organization root and final v1 signing key must co-sign the same descriptor preimage. +Neither intermediate signatures nor embedded keys replace those approvals. +That descriptor does not add fields to historical receipts. +The legacy export retains its original trust requirements. + +## Consequences + +### Positive + +Local group commit remains. +Epochs separate signing histories. +Offline tooling finds missing registered chains, not just surviving files. + +### Negative + +Storage must separate epoch heads. +Leases and registry add local dependencies. + +### Risks and Mitigations + +Replica cloning can create forks. Require external fencing and reject duplicate positions with different hashes. +An unpublished chain can disappear. Compare the issued-certificate log with signed inventory coverage. +Epoch resets could hide missing history. Require explicit predecessor references and report unverifiable gaps. + +## Implementation Notes + +1. Freeze v2 hash fields and chain-ID grammar with cross-language fixtures before implementing storage. +2. Add dual-version dispatch without changing the v1 parser or golden encodings. + Acceptance: all existing v1 vectors retain their results. +3. Introduce durable replica identity, epoch allocation, and separate chain heads. + Acceptance: concurrent replicas can both emit sequence 1 without collision. +4. Test concurrent lease acquisition, stale tokens, renewal loss, and copied certificates. + Acceptance: only the fenced owner admits work; duplicate certificate issuance is refused. + Acceptance: cached checks make no per-batch API calls; monotonic deadline expiry fences despite wall-clock changes. +5. Remove every segment of a registered chain. + Acceptance: offline coverage reports it lost, rather than silently omitting it. +6. Interrupt migration and verify both co-signatures. + Acceptance: either missing signature fails; v1 bytes remain identical; epoch-open occupies sequence 1. + +## Related ADRs + +- [0002: Lifecycle references](0002-two-phase-receipts.md). +- [0003: Segment boundaries](0003-durable-shipping.md). +- [0005: Issuance](0005-root-certified-keys.md). +- [0007: Identity and fencing](0007-kubernetes-lifecycle.md). diff --git a/docs/design/0002-two-phase-receipts.md b/docs/design/0002-two-phase-receipts.md new file mode 100644 index 0000000..012398a --- /dev/null +++ b/docs/design/0002-two-phase-receipts.md @@ -0,0 +1,134 @@ +# ADR-0002: Durable intent and outcome receipts + +## Status + +Proposed + +## Date + +2026-10-18 + +## Context + +The [gateway](../../internal/gateway/gateway.go) executes upstream work, commits its v1 receipt, then responds. +The [ledger](../../internal/receipt/ledger.go) documents the crash gap: an action can succeed before its receipt commits. + +[SSE](../../internal/gateway/stream.go) starts only after a header-time receipt commits. +Digests reach logs, not receipts. The [stream-digest proposal](../spec/proposals/receipt-stream-digest.md) is unimplemented. + +## Decision Drivers + +- Record a durable obligation before dispatch. +- Report uncertainty without replaying external side effects. +- Preserve receipt-before-response, including SSE headers. +- Measure added commit latency. + +## Considered Options + +| Option | Pros | Cons | +| --- | --- | --- | +| Outcome only | Current latency and simple schema | Crash can erase evidence of an action | +| Intent only | Records attempted dispatch | Cannot attest observed outcomes | +| Durable intent, then outcome | Exposes missing outcomes | Adds latency and recovery state | + +## Decision + +Propose v2 lifecycle records without changing v1. +After authentication, validation, and policy evaluation, allocate an attempt ID. +Persist a signed `intent` before any upstream call, including credential refresh. +Bind identity, delegation, params digest, policy, action, and any validated client request ID. +Denied attempts get a terminal decision record without an upstream intent. + +An acknowledged intent permits 1 logical execution under existing safe retries. +Append timeout can still leave a committed record. Resolve ambiguity before dispatch, never by retrying upstream. + +Append a signed `outcome` after observing the upstream result. +It references the intent's `chain_id`, sequence, entry hash, and attempt ID. +Include `upstream_attempt_count`, counting actual action dispatches, including transport retries. +Exclude credential refresh calls. +Use `observation = not_dispatched` only when the running process knows no action dispatch occurred. +That observation requires a known zero count, not an invented HTTP status. +Crash recovery still writes `outcome_unknown`, with unknown attempt count unless durable evidence establishes it. +For buffered responses, commit the outcome before returning status or body. +Use a bounded commit context detached from client cancellation, as today. + +For SSE, commit an `outcome` with `observation = headers_accepted` before forwarding headers. +That outcome does not claim stream completion. +Append a supplemental `stream_result` at termination. +Bind outcome hash, written-byte SHA-256, byte count, and termination reason. +The digest covers downstream writes, not proven client delivery. +A missing terminal record means unknown completion, never success. +Keep terminal artifacts out of forwarded SSE bytes. +This proposed lifecycle replaces the earlier stream proposal's single-action-receipt rule for v2 only. + +Before admitting traffic after restart, scan committed intents without outcomes. +Append recovery outcomes marked `outcome_unknown`. Never call upstream during recovery. +For header outcomes without terminal stream records, append `stream_result` with unknown termination and no fabricated digest. +Fenced recovery and unique references must prevent duplicate terminal records. +Recovery records can reference an earlier epoch. +Rotation does not drain inflight requests; their outcomes and stream results can enter the next epoch. + +### Client Retry Linkage + +Validate bounded `client_request_id` syntax; bind it to the principal and destination in the intent. +Defer client retry deduplication and mapping an attempt ID to upstream `Idempotency-Key`. +Providers differ in key support and retention. A new gateway attempt can repeat an external side effect. +Claim no exactly-once behavior; add no unsupported provider header. +The [owner defaults](README.md#open-decisions) disable automatic replay. + +### Group-Commit Cost + +Intent and outcome cannot share a transaction for their own request. +Other requests can still share each batch's fsync. +Allowed buffered attempts require 2 durable boundaries instead of 1. +SSE adds a later terminal record as well. + +`bench/RESULTS.md` is absent on the pinned main baseline. +[Phase 0 PR #50](https://github.com/Clawdlinux/agentgate/pull/50), head `29ef207`, reports these added-overhead p99 measurements: + +| Workload | Existing 1-receipt path | +| --- | --- | +| Closed loop, 0 ms upstream, concurrency 1 | 0.34 ms | +| Closed loop, 0 ms upstream, concurrency 16 | 3.42 ms | +| Closed loop, 0 ms upstream, concurrency 64 | 6.22 ms | +| Open loop, 0 ms upstream, 25% load | 7.00 ms | + +The PR used Apple M4, `GOMAXPROCS=4`, and 20,000 requests per row. +It did not meet the under-1-ms p99 gate. +These are end-to-end overheads, not fsync measurements. Doubling them cannot predict two-phase p99. +Budget 1 extra serialized commit wait before dispatch, then benchmark it on Linux with dedicated 4-vCPU capacity. + +## Consequences + +### Positive + +Crashes leave inspectable obligations and explicit unknown outcomes. + +### Negative + +Writes and latency increase. Recovery needs lifecycle indexes. + +### Risks and Mitigations + +An unknown outcome may hide an executed action. Require operator reconciliation, not automatic replay. +SSE can fail after response headers. End the stream and persist uncertainty when storage recovers. + +## Implementation Notes + +1. Freeze lifecycle references and optional fields in [v2](../spec/RECEIPT-v2.md). + Acceptance: reject missing, duplicate, or mismatched intent references. +2. Add intent ordering instrumentation. + Acceptance: the mock upstream sees no call before intent commit acknowledgment. + Acceptance: runtime cancellation before dispatch yields `not_dispatched` and zero attempts; retry counts include every action dispatch. +3. Run subprocess `kill -9` tests before intent commit, after commit, during dispatch, and after provider success. + Also kill between the upstream call and outcome commit, after outcome commit, and during SSE. + Acceptance: recovery is idempotent, records unknown states, and never re-dispatches. +4. Re-run Phase 0 closed-loop and open-loop workloads. + Acceptance: publish intent-wait and outcome-wait percentiles separately; obtain owner approval for the latency budget. + +## Related ADRs + +- [0001: References](0001-per-replica-chains.md). +- [0003: Shipping](0003-durable-shipping.md). +- [0006: Closed failure mode](0006-receipt-failure-modes.md). +- [0007: Stream drain](0007-kubernetes-lifecycle.md). diff --git a/docs/design/0003-durable-shipping.md b/docs/design/0003-durable-shipping.md new file mode 100644 index 0000000..c94b74a --- /dev/null +++ b/docs/design/0003-durable-shipping.md @@ -0,0 +1,146 @@ +# ADR-0003: SQLite outbox and signed sealed segments + +## Status + +Proposed + +## Date + +2026-10-18 + +## Context + +The [ledger](../../internal/receipt/ledger.go) commits signed SQLite batches. +[WAL](../../internal/db/sqlite.go) is local storage, not replication. +[Exports](../../internal/receipt/export.go) provide signed snapshots, not shipping. +The [schema](../../internal/db/migrations/002_receipts.sql) prohibits receipt updates and deletes. +Clawdlinux requires air-gapped operation. Postgres is PARKED. + +## Decision Drivers + +- Continue local commits during destination outages. +- Ship verifiable artifacts, not live WAL files. +- Bound growth without deleting pending evidence. + +## Considered Options + +| Option | Pros | Cons | +| --- | --- | --- | +| SQLite plus signed segments | Existing writer; local MinIO or filesystem | Async loss window; sealing implementation | +| Postgres outbox | Transactions and replication | PARKED; database and network operations | +| Kafka-like log | Replay and throughput | Quorum and partition operations | + +Postgres and Kafka can run offline. Their extra operations, not cloud dependence, argue against initial adoption. + +## Decision + +Propose replica-local SQLite WAL plus a durable shipping outbox. +Ship immutable segments to configurable S3, in-cluster MinIO, or filesystem export. +Uploads never block request commits. + +Seal after 4,096 records, 16 MiB, or 10 seconds, whichever comes first. +Also seal on epoch closure and drain. +Each segment covers 1 contiguous, nonempty chain range. +Store original receipt bytes. Never re-encode v1 receipts. +Capture existing SQLite history through the current v1 exporter, then preserve those bytes. +Never ship live SQLite, WAL, or SHM files. + +A versioned descriptor binds chain, fencing token, range, count, anchor, terminal hash, and predecessor digest. +Also bind payload SHA-256 and certificate or legacy keyset digests. +Use separate signing domains and strict range validation. Bundle certificates; independently pin the root. + +### Artifact Signers + +Use [0005's purposes](0005-root-certified-keys.md). +Epoch keys sign descriptors with `segment_descriptor` and control records with `control`. +Destroy an epoch key only after its final segment descriptor and directory are durable. + +The intermediate-certified singleton publisher signs cross-replica inventories and head vectors with `inventory`. +Replica keys cannot assert cross-replica coverage. +The intermediate signs the registry with `replica_registry`. +Inventories bind registry revision/digest, ranges, close records, and final descriptors. + +The offline root and final v1 key co-sign migration descriptors using the same domain-separated preimage. +Bind original exports, final v1 head, legacy key history, and destination v2 identity. +A dedicated intermediate-certified archival publisher signs legacy descriptors with `legacy_segment`. +It references that migration envelope without re-signing or re-encoding v1 receipts. +Without migration, legacy verification still requires independently trusted v1 keys. + +### Durable Materialization and Upload + +Reserve range and outbox ID in a ledger-writer transaction. +Materialize immutable rows from a snapshot. +Write and fsync payload and descriptor, then atomically rename and fsync their directory. +Mark the outbox item sealed only after those files are durable. +Recreate interrupted reservations from retained rows. Uploads hold no SQLite write transaction. + +Use object names containing chain ID, range, and payload digest. +Require conditional creation such as `If-None-Match: *`; object lock may add immutable retention. +Object lock alone must prevent replacement or bind references to immutable versions. +For filesystem export, create staging files with `O_CREAT|O_EXCL`, fsync, then publish through atomic no-replace linking. +Fsync the destination directory before publishing inventory references. +Retry identical names and bytes after uncertain acknowledgments; compare existing objects. +Verify the stored digest independently of multipart ETags. +Different existing bytes fail integrity. +Upload payload before descriptor; publish the inventory reference last. +Only the final reference advertises completeness. +Persist acknowledgments and inventory progress locally. + +### Retention and Offline Consumption + +Retain unuploaded segments and their source rows regardless of age. +Apply retention only after upload, read-back verification, and required publication. +Do not bypass current no-delete triggers. +The storage model remains an [owner decision](README.md#open-decisions): + +- File per epoch: reclaim closed files after durable sealing, verified shipping, and required coverage. +- Single database: require an append-only-preserving compaction or archival migration before reclamation. + +Recommend files per epoch, not yet implemented. +Neither model permits deleting pending intents or unshipped evidence. +Retain trusted anchors, certificates, and checkpoints as long as retained evidence needs them. +Quota exhaustion blocks admission rather than evicting pending evidence. + +Verifiers validate inventory, descriptors, payloads, chains, and certificate bounds. +Sort segments by authenticated range, not object listing order. +Reject gaps and conflicting overlaps; identical retries are harmless. +Compare registered chains against coverage; report omissions. +[0004](0004-merkle-checkpoints.md) starts with head vectors, then optional compact proofs. +Inventory completeness is only relative to its signed capture point. + +## Consequences + +### Positive + +Local admission tolerates upload outages. +Artifacts work through MinIO, S3, or removable media. + +### Negative + +Sealing needs implementation; off-node durability lags local acknowledgment. + +### Risks and Mitigations + +Volume loss loses unshipped evidence. Publish lag and agree acceptable loss. +Outages fill disks. Alert on capacity and stop admission before reserved outcome space disappears. +Storage can roll back inventories. Compare pinned coverage statements; otherwise report freshness unknown. + +## Implementation Notes + +1. Freeze descriptor, inventory, and outbox state-machine fixtures. + Acceptance: original v1 export bytes and v2 signatures verify offline. +2. Inject crashes around reservation, fsync, rename, upload, and acknowledgment. + Acceptance: retries never skip ranges or publish partial segments. +3. Race conflicting conditional puts and filesystem exclusive creates; lose upload replies against in-cluster MinIO. + Acceptance: exactly 1 immutable object wins; matching retries succeed; conflicting bytes fail. +4. Test destination disconnection and quota pressure. + Acceptance: no pending evidence is deleted, and retention never bypasses append-only triggers. +5. Rotate before the final descriptor is written, and attempt cross-replica signing with a replica certificate. + Acceptance: the old key is retained; unauthorized inventory and legacy-segment signers fail. + +## Related ADRs + +- [0001: Epoch boundaries](0001-per-replica-chains.md). +- [0004: Publication](0004-merkle-checkpoints.md). +- [0005: Signing purposes](0005-root-certified-keys.md). +- [0006: Deferred direct spool shipping](0006-receipt-failure-modes.md). diff --git a/docs/design/0004-merkle-checkpoints.md b/docs/design/0004-merkle-checkpoints.md new file mode 100644 index 0000000..783df35 --- /dev/null +++ b/docs/design/0004-merkle-checkpoints.md @@ -0,0 +1,143 @@ +# ADR-0004: Signed head vectors before tiled proofs + +## Status + +Proposed + +## Date + +2026-10-18 + +## Context + +[V1 verification](../../pkg/receiptspec/verifier.go) authenticates chains against keys and optional endpoints. +A valid prefix cannot expose every omission. +[agentgate-verify](../../cmd/agentgate-verify/main.go) has no Merkle proofs. + +## Decision Drivers + +- Expose missing chains. +- Publish offline, outside request commits. +- Prefer standard proof formats. + +## Considered Options + +| Option | Pros | Cons | +| --- | --- | --- | +| Signed `(chain_id, seq, head_hash)` vector plus inventories | Fewer dependencies; missing-chain coverage | Reads segments; no compact inclusion proofs | +| Trillian Tessera | Standard tiles; Go library; POSIX support | New dependency, publisher, and backend qualification | +| Hand-written Merkle tree | Small initial implementation | Team owns cryptographic correctness and recovery | +| Sigstore Rekor | Existing APIs and tooling | Supply-chain schema adaptation and service operations | + +[Tessera](https://github.com/transparency-dev/tessera) succeeds older Trillian approaches. +[Rekor](https://github.com/sigstore/rekor) can run offline; its broader protocol is unnecessary here. + +## Decision + +### Initial Publication + +Recommend signed vectors and inventories first; add Tessera when compact inclusion proofs justify its cost. +Vectors bind organization, installation, publication revision, predecessor digest, and registry revision. +Include every registered chain with its covered `(chain_id, seq, head_hash)`, close status, and inventory digest. +Explicitly list lost or uncovered chains. +A head vector is not a C2SP checkpoint. +Verify selected segments against its endpoints. + +Propose publication every 10 seconds or 4,096 newly covered records, whichever comes first. +Publish descriptors and inventories before their vector. +Retain immutable vector history with conditional latest-pointer updates. +Coverage alone cannot prove freshness. + +### Publisher Ownership and Custody + +Use 1 logical publisher per installation with active/standby failover. +Its `inventory` key is intermediate-certified and cannot be a per-replica key. +Protect keys in local HSM, Vault transit, or approved encrypted storage. +Use a separate origin-scoped `checkpoint` key for the later log. +Keep both private keys outside gateway replicas. + +Validate segments against the independently pinned root and scoped intermediate. +Check registry, fencing token, purpose, chains, digests, and terminal bounds. +Embedded certificates alone are not trusted roots. + +The publisher acquires an external lease and monotonic fencing token. +Persist head map, predecessor digest, and any tree cursor before signing successors. +Publish conditionally against the exact predecessor. +Fence the former writer's storage before recovering its durable state on standby. +Kubernetes Lease expiry alone is insufficient. +Unproven fencing or continuity stops publication. +For POSIX Tessera, never attach its writable data volume to competing publishers. +Reconcile the last signed checkpoint before signing after failover. + +### Later Merkle Profile + +Tessera adds compact inclusion proofs and standard tooling, not missing-chain discovery. +Qualify a pinned POSIX release offline before adoption. +Do not assume a Tessera cloud backend supports MinIO. + +Adopt [C2SP tlog-checkpoint v1.1.0](https://c2sp.org/tlog-checkpoint@v1.1.0) +and [tlog-tiles v1.0.0](https://c2sp.org/tlog-tiles@v1.0.0). +Use RFC 6962 SHA-256: +Leaf hashes are `SHA256(0x00 || leaf_bytes)`. +Internal hashes are `SHA256(0x01 || left_hash || right_hash)`. +Use signed-note Ed25519 checkpoints without extension lines. + +Freeze leaf bytes binding receipt version, scope, sequence, and entry hash. +Use authenticated v2 `chain_id`, or v1's signed legacy-descriptor scope without modifying receipts. +Membership never replaces receipt signatures or chain verification. + +Persist `(chain_scope, seq, entry_hash)` to leaf-index mappings. +Reconcile ambiguous integration before accepting retries; reject conflicting positions. +Integration order reflects arrival, not external execution order. + +Checkpoint every 10 seconds or 4,096 integrated leaves, whichever comes first. +Publish tiles and entry bundles durably before publishing their checkpoint. +Retain immutable checkpoint history plus a mutable latest pointer. +Idle logs need not issue identical checkpoints. +Publish an inventory beside each checkpoint for per-chain coverage. + +Extend `agentgate-verify` for local proofs and tiles. +Check certificate path, checkpoint signature/origin, leaf identity, inclusion, and tree size. +Check consistency against an earlier pinned checkpoint when supplied. +A single checkpoint proves neither freshness nor absence of split views. + +Air-gapped publication starts with signed vector files, certificates, registry snapshots, and inventories. +The later log also exports signed checkpoint files and tiles. +Copy those artifacts and segments through approved removable media. +Verification requires no public services or network fetches. +Witness cosigning is optional and disabled by default. +Local witness quorum is optional. Without it, claim no witness-backed anti-equivocation. + +## Consequences + +### Positive + +Vectors expose omissions without a tree service. +Later proofs reduce selective verification cost. + +### Negative + +Both stages need a fenced singleton publisher and durable state. +Initial selective verification is linear in the supplied segment range. + +### Risks and Mitigations + +A publisher can equivocate. Compare pinned checkpoints and optionally require independent local witness signatures. +Delayed publication can hide recent loss. Report integrated coverage separately from committed and uploaded heads. +Dependencies can change. Pin releases and run offline interoperability fixtures. + +## Implementation Notes + +1. Publish vectors from registry-backed inventories and fail over the active publisher during publication. + Acceptance: no registered chain vanishes and no predecessor has competing accepted successors. +2. Lose the publisher node, block its lease renewal, and simulate failed storage fencing. + Acceptance: an unfenced standby cannot publish; recovery preserves signed history. +3. Gate Tessera adoption on offline POSIX recovery and stable leaf indices after duplicate ingestion. +4. Add C2SP interoperability and CLI fixtures before enabling Merkle mode. + Acceptance: wrong roots, origins, inclusion paths, or consistency paths fail; existing v1 results remain unchanged. + +## Related ADRs + +- [0001: Chains](0001-per-replica-chains.md) defines leaf identity. +- [0003: Segments](0003-durable-shipping.md) supplies validated inputs. +- [0005: Root keys](0005-root-certified-keys.md) certifies the log signing role. diff --git a/docs/design/0005-root-certified-keys.md b/docs/design/0005-root-certified-keys.md new file mode 100644 index 0000000..48a6802 --- /dev/null +++ b/docs/design/0005-root-certified-keys.md @@ -0,0 +1,159 @@ +# ADR-0005: Offline root, scoped issuer, and epoch keys + +## Status + +Proposed + +## Date + +2026-10-18 + +## Context + +The [signer store](../../internal/signer/store.go) caches Ed25519 keys. +`WithActiveKey` leases exclude rotation through batch commit. +`Rotate` switches global sequence windows and clears old keys, without epochs or certificates. + +[V1](../spec/RECEIPT-v1.md) trusts keys with inclusive sequence windows. +[Manifest verification](../../pkg/receiptspec/manifest.go) currently omits signer-window checks. +That hardening is in progress elsewhere, not shipped here. + +## Decision Drivers + +- Pin 1 offline root. +- Constrain installation, purpose, and epoch. +- Preserve batch signing leases. + +## Considered Options + +### Issuance and Custody + +| Option | Pros | Cons | +| --- | --- | --- | +| Direct offline-root issuance | Minimal delegation | Daily ceremonies | +| Offline root with local issuer | Automated epochs; offline root | Intermediate compromise risk | +| Online KMS/HSM/Vault root | Automated custody | Availability and algorithm constraints | + +Cloud KMS is optional. HSM or Vault transit can operate locally. + +### Replica Certificate + +| Option | Pros | Cons | +| --- | --- | --- | +| Length-prefixed statement | V1 conventions; explicit scopes | New certificate fixtures | +| X.509 | Existing PKI tooling | Custom scopes and path validation | +| Canonical JSON | Readable fields | Another canonicalization profile | + +## Decision + +Propose an offline root delegating to a longer-lived, in-cluster intermediate issuer. +Bind organization, installation, allowed purposes, and a prohibition on further intermediate delegation. +Recommend annual intermediate renewal and 24-hour epoch rotation, without public connectivity. +Prefer local HSM or Vault transit; encrypted issuer storage needs owner approval. +Record issuance in [0001's registry](0001-per-replica-chains.md) and refuse duplicate `(UUID, epoch)` certification. +Reserve that pair and durably record its signed certificate digest before releasing the certificate. +Crashes may leave unused registrations, never usable unregistered certificates. + +Choose v1-style length-prefixed statements, not JSON or X.509. +Use NUL-terminated domains, `U32LE` lengths, `U64LE` integers, raw keys, and absence markers. +Ed25519 signs SHA-256 of that preimage. +Bind issuer and subject fingerprints, organization, installation, purposes, and scope. +Freeze ordering and fixtures. Reject ambiguous key IDs and cross-purpose use. + +### Signing Roles and Purposes + +| Purpose | Authorized signer and scope | +| --- | --- | +| `certificate_issuance` | Intermediate; its installation and permitted child purposes | +| `replica_registry` | Intermediate; issuance, lease generations, replacements, losses | +| `receipt` | Epoch key; intents, outcomes, decisions, stream results | +| `control` | Epoch key; boundaries, mode changes, future sink-transition markers | +| `segment_descriptor` | Epoch key; descriptors for that same epoch | +| `legacy_segment` | Certified archival publisher; migrated v1 exports only | +| `inventory` | Certified singleton publisher; cross-replica inventories and head vectors | +| `checkpoint` | Separate certified log key; installation and origin | +| `trust_status` | Intermediate; signed status of its delegated keys and sequence cutoffs | +| `migration` | Offline root and final v1 key co-sign; neither signature may be delegated | + +### Validity and Rotation + +Epoch and sequence bounds govern validity. +Certify exactly 1 UUID and epoch, starting at sequence 1. +Its signed epoch-close fixes the inclusive terminal sequence. +Verifiers reject records beyond that bound. +Terminal coverage requires a signed inventory binding epoch-close and the final descriptor. +When Merkle mode is enabled, also prove inclusion of epoch-close against its checkpoint. +Unclosed epochs remain provisional prefixes. +Time windows guide renewal only; untrusted timestamps cannot establish cryptographic expiry. +Elapsed time alone cannot invalidate historical evidence. + +Propose `agentgate keys rotate` at a batch boundary. +Keep admission open; bounded queues may briefly wait. +Obtain the next certificate before closing the current epoch. +If the intermediate is unavailable, remain in the current epoch and alert instead of beginning rotation. +Continued signing still requires valid lease ownership and no applicable trust-status cutoff. +After the current batch, append epoch-close and durably seal the final segment descriptor. +Retain the old key until descriptor and directory fsync. +Durably activate the next certified epoch and write epoch-open at sequence 1 before queued work. +Inflight upstream calls continue. +Later outcomes reference earlier intents across epochs. +Epoch-close closes signing extent, not external-work completion. +Preserve lease-before-SQLite lock ordering; recover activation from durable phase metadata. +Current `Rotate(atSeq)` needs epoch-aware changes; calling `Rotate(1)` is not valid. + +### Revocation and Offline Verification + +Revocation policy remains an [owner decision](README.md#open-decisions). +Recommend intermediate-signed status every 6 hours; `next_update` is 24 hours after issuance. +Bind revision, predecessor digest, certificate IDs, and revoked epoch or sequence cutoffs. +Every registry loss declaration or same-UUID lease takeover must issue a cutoff at the last covered head. +Bind the affected epoch, terminal sequence, and head hash; verifiers reject records beyond that bound. +Without coverage, sequence 0 rejects the entire affected epoch. +Publish the cutoff before takeover activation; any continuation requires a newly certified epoch. +Offline verification reports bundled status as-of time. +Freshness needs trusted verifier time or a separately pinned status revision. +Otherwise report freshness unknown, not unseen revocation checks. +Stale or missing status prevents current-trust success; historical checks report as-of status. +Revoke compromised intermediates through offline-root-signed replacement status. + +Bundle certificates, registry history, status, descriptors, and applicable proofs or checkpoints. +Independently pin the root public key. +Raw-key v2 lifecycle tests do not require this hierarchy. +Legacy v1 retains its original trust rules outside a root-and-final-v1-key co-signed migration envelope. + +## Consequences + +### Positive + +Root pinning supports automated local issuance. +Sequence bounds remain meaningful without trusted receipt clocks. + +### Negative + +The intermediate and registry require protected local state. +Revocation freshness remains a policy requirement. + +### Risks and Mitigations + +A compromised issuer can certify attacker keys. Restrict scope and audit issuance. +Cloned certificates remain usable. Require external fencing. +Coordinate manifest hardening without changing frozen v1 encoding. + +## Implementation Notes + +1. Freeze binary certificate fixtures and an offline root-pin procedure. + Acceptance: wrong installation, purpose, epoch, or issuer delegation fails verification. +2. Rotate under continuous admission and inflight SSE, then crash at each boundary. + Acceptance: no admission drain, exactly 1 epoch-open, and the old key survives until final descriptor durability. +3. Append records beyond epoch-close and remove its inventory coverage. + Acceptance: out-of-bounds records fail; missing terminal coverage cannot pass as complete. +4. Test stale, missing, and revoked trust-status bundles without networking. + Acceptance: report exact revocation scope and unknown freshness without fabricating clock-based trust. + Acceptance: loss and takeover reject old-epoch suffixes beyond covered heads; issuer outages preserve the current epoch. + +## Related ADRs + +- [0001: Identity and registry](0001-per-replica-chains.md). +- [0002: Cross-epoch outcomes](0002-two-phase-receipts.md). +- [0003: Descriptors](0003-durable-shipping.md). +- [0004: Publisher keys](0004-merkle-checkpoints.md). diff --git a/docs/design/0006-receipt-failure-modes.md b/docs/design/0006-receipt-failure-modes.md new file mode 100644 index 0000000..7e93cdc --- /dev/null +++ b/docs/design/0006-receipt-failure-modes.md @@ -0,0 +1,124 @@ +# ADR-0006: Explicit receipt failure modes + +## Status + +Proposed + +## Date + +2026-10-18 + +## Context + +The [gateway](../../internal/gateway/gateway.go) returns 500 on receipt failure, withholding the outcome even after upstream execution. +[Ledger cancellation](../../internal/receipt/ledger.go) can leave committed receipts. +There is no fallback spool or failure-mode setting. + +## Decision Drivers + +- Keep durable evidence as the default admission requirement. +- Require a named failure class before adding fallback storage. +- Never interpret fail-open as bypassing authentication or policy. +- Make configuration changes auditable even when the primary sink fails. + +## Considered Options + +| Option | Pros | Cons | +| --- | --- | --- | +| `fail_closed` only | Smallest failure surface | Local database faults stop all actions | +| Unbounded or memory-only fail-open | Highest apparent availability | Evidence loss and uncontrolled disk growth | +| Default closed, defer durable spool | Clear initial guarantee; avoids unsupported availability claims | No fallback until its failure model is approved | + +## Decision + +Propose shipping only `receipt_failure_mode: fail_closed` in the initial implementation. +These documents do not implement that behavior. +An unavailable durable intent sink prevents upstream dispatch and returns HTTP 503. +Failure after dispatch also returns 503 before response headers, without claiming the external action failed. +The unresolved committed intent is recovered as `outcome_unknown`. +This proposes an API behavior change from the current 500 response. + +Defer `fail_open_spool` until the owner names a covered failure class and demonstrates recovery under that fault. +Reject that mode as unsupported for now. +A same-volume spool does not cover volume loss or full-disk failure. +It also cannot resolve an uncertain SQLite commit or replace a missing authorized signer. + +### Authenticated Configuration + +Log initial configuration and every later accepted mode change as signed v2 control receipts. +Bind old mode, new mode, actor, reason, and configuration digest. +The epoch key signs them with `control` authority. +Commit the control receipt before activating the configuration. +If it cannot commit, reject the change and retain the last committed configuration. +503 handling does not bypass authentication, action policy, or egress enforcement. + +For human changes, require the existing [authenticated admin session](../../internal/admin/session.go). +Resolve `AdminID` and `OrgID` server-side and require installation configuration authority. +Apply [mutating-request CSRF checks](../../internal/admin/handler.go). +Never trust a request-supplied actor name. +The legacy shared `X-Admin-Secret` alone lacks an individual actor and is insufficient for this endpoint. +Bootstrap verifies the issuer-signed enrolment statement and binds its deployment identity to the configuration digest. +Unattended control authentication beyond that bootstrap remains deferred, not an anonymous local bypass. + +## Consequences + +### Positive + +Default operation keeps a clear durability boundary. +No fallback availability is claimed without a demonstrated failure model. + +### Negative + +Primary-sink outages stop admission. +503 does not mean an already-dispatched external action was rolled back. + +### Risks and Mitigations + +Unlogged configuration can hide evidence loss. Require authenticated actors and committed control receipts before activation. +Started SSE cannot change HTTP status. Close the stream on terminal persistence failure and recover unknown completion later. + +## Implementation Notes + +1. Add 503 handling to the two-phase state machine. + Acceptance: no intent acknowledgment means no upstream dispatch; post-dispatch failures never imply rollback. +2. Add authenticated configuration control. + Acceptance: forged actors, wrong organizations, missing authority, and missing CSRF checks cannot change mode. +3. Fail the control append and request the deferred spool mode. + Acceptance: neither change activates; the last committed mode survives restart. +4. Any later spool implementation must pass the appendix's crash and evidence-loss tests. + +## Deferred Appendix: Bounded Durable Spool + +This is a design sketch, not an enabled Phase 2 mode. +Approval requires a named primary-sink fault that leaves independent spool writes and signing safe. +Keep 1 writer and 1 authoritative chain. +Fsync intent before dispatch and outcome before response; preserve SSE's header and terminal boundaries. +Resolve uncertain commits before handoff. Never replay requests, re-sign receipts, or allocate replacement sequences. + +Persist a signed sink-transition marker outside the spool before entering it. +Keep that marker in a separate durable control file, not solely SQLite or the spool directory. +Bind previous SQLite head, sink generation, spool identity, and fencing token. +Update its recorded spool head durably before acknowledging each spool append. +After a crash, scan durable spool records to reconcile any append newer than the marker. +An absent or corrupt spool while its marker remains is evidence loss, not permission to resume SQLite. +Refuse SQLite admission past a recorded spool head until exact signed records have been replayed and reconciled. +Only then fsync a return-to-SQLite marker and commit the corresponding control receipt. + +Ship signed spool segments directly, without waiting for SQLite replay. +Their sealing metadata and upload progress must remain durable outside the failed SQLite sink. +Preserve the marker and pending segments if replay cannot finish. +Identical existing records are idempotent; conflicting positions halt recovery. +Delete nothing until replay, verified shipping, and retention requirements permit it. + +Bound bytes, count, and age using owner-approved quotas. +Reserve intent, outcome, stream-terminal, and control capacity before admission. +Overflow rejects new work with 503 and never evicts evidence. +Tests must remove spool files while retaining the marker and interrupt every transition, replay, and upload boundary. +Acceptance: no SQLite restart can conceal a recorded spool suffix, and direct shipping works while SQLite remains unavailable. + +## Related ADRs + +- [0001: Chain positions](0001-per-replica-chains.md). +- [0002: Durability boundaries](0002-two-phase-receipts.md). +- [0003: Shipping and deletion](0003-durable-shipping.md). +- [0005: Signing authority](0005-root-certified-keys.md). diff --git a/docs/design/0007-kubernetes-lifecycle.md b/docs/design/0007-kubernetes-lifecycle.md new file mode 100644 index 0000000..3b09b0d --- /dev/null +++ b/docs/design/0007-kubernetes-lifecycle.md @@ -0,0 +1,152 @@ +# ADR-0007: Kubernetes identity, drain, and WAL ownership + +## Status + +Proposed + +## Date + +2026-10-18 + +## Context + +This baseline has [Compose](../../docker-compose.yaml) and [configuration YAML](../../configs/agentgate.yaml), not Kubernetes workload manifests. +The [health endpoint](../../internal/gateway/gateway.go) reports service count, not ledger readiness. +[HTTP shutdown](../../cmd/agentgw/main.go) currently allows 10 seconds. +Then [ledger closure](../../internal/receipt/ledger.go) finishes submitted transactions but rejects pending requests, without sealing. +Signer and database closure follow. + +SQLite also stores authorization, vault, and organization state. Receipt isolation does not replicate it. + +## Decision Drivers + +- Preserve identity and exclusive storage. +- Drain safely without restart loops. +- Keep governance runtime-independent. + +## Considered Options + +| Option | Pros | Cons | +| --- | --- | --- | +| StatefulSet with per-pod PVC | Stable attachment and identity | Storage fencing and zone recovery | +| Deployment with PVC per replica | Familiar controller | Additional claim-ownership machinery | +| Shared or ephemeral storage | Simple scheduling | Writer conflicts or evidence loss | + +## Decision + +Propose a StatefulSet with 1 persistent data volume per replica. +Use `volumeClaimTemplates` and `ReadWriteOncePod` where supported. +Otherwise require storage fencing beyond `ReadWriteOnce`. Test filesystem locking and fsync. +Never share a writable SQLite database between replicas. + +Persist the provisioned replica UUID and epoch alongside SQLite. +Ordinals locate PVCs; they are not cryptographic identities. +Empty PVCs stay unready until explicit intermediate enrolment. +Never silently initialize replacement identity. +The signed registry binds ordinal, new UUID, and replacement of the prior UUID. +Report the old chain lost from its last shipped or checkpointed head, with unknown suffix. +Without coverage, report it lost with no known head. +Intact restarts require recovery and lease checks before resuming. +Retain PVCs on scale-down. Audit explicit deletion. + +Check [0001's cached lease view and token](0001-per-replica-chains.md) before admission and each batch, without per-batch API calls. +Background renewal sets a monotonic self-fencing deadline, approximately 10 seconds with the 5-second/15-second defaults. +Keep that deadline shorter than lease duration minus maximum clock drift; self-fence when it expires. +Kubernetes Lease expiry alone does not fence a partitioned node. +Force-detach requires proven old-node storage fencing, then lease takeover with a newer token. +Loss declarations and same-UUID takeovers require a `trust_status` cutoff at the last covered head before activation. +Verifiers reject the affected epoch's suffix beyond that head; a takeover must use a newly certified epoch. +Unproven fencing blocks takeover. + +The only proposed gateway-owned writable state is its WAL data volume. +It includes SQLite, WAL, SHM, encrypted keys, outbox, sealed files, and any future spool. +Existing authorization and vault rows remain on that volume. +Configuration and mounted secrets stay read-only. +Object storage and optional Tessera publication require separate durable infrastructure. +Multi-replica AgentGate cannot ship until per-replica auth/vault state is solved. +Test provisioning, rotation, and revocation propagation before enabling replicas beyond 1. +Never copy live WAL files to provision replicas. + +Propose separate `/livez` and `/readyz` endpoints plus a startup probe. +Liveness checks process progress, not upstream or object storage. +Readiness checks recovery, exclusive volume ownership, signer authority, and capacity. +The initial deployment is fail-closed. The deferred spool is neither required nor enabled. +Only later spool-enabled deployments require [0006's spool protocol](0006-receipt-failure-modes.md). +Upload outages preserve readiness until local quotas block admission. +Probes must not append synthetic receipts. + +Use authenticated local drain control for `preStop` and SIGTERM. +Fail readiness and stop admission. +Allow endpoint propagation, then drain HTTP work. +Bound long-lived SSE streams and write their terminal or unknown records. +Wait for append acknowledgments before `Ledger.Close`. +Seal the final segment and persist its outbox state. +Never wait indefinitely for upload. +Close ledger, clear signer keys after descriptor durability, then close SQLite. +Recover interrupted drain from that volume. + +Propose `terminationGracePeriodSeconds: 60`, including hooks. +Allocate 5 seconds for propagation, 40 for requests/streams, 10 for ledger/sealing, and 5 for closure. +Change the current shutdown timeout accordingly. +The 40-second stream budget cuts longer SSE streams. +The expected `stream_result` has `termination = unknown` and reason `drain_deadline`. +Any digest covers only the observed prefix. +Document these rollout cuts; unlimited stream continuity is not promised. + +For 2 or more replicas, propose a PDB with `maxUnavailable: 1`. +Spread across hostnames; prefer zones when available. +Use rolling updates with at most 1 unavailable replica. +Document single-replica maintenance downtime. +PDBs neither prevent node failures nor replicate unshipped evidence. + +Clawdlinux's operator applies sandbox labels and egress policy independently of runtime. +Select runtimes through its `pkg/runtime.Registry`, not gateway branches. +Keep its gVisor label injector and runtime-class integration. +Adapters must not reimplement sandbox or egress policy. +Air-gapped profiles require no public-cloud endpoint. + +## Consequences + +### Positive + +Replica identity follows durable ownership rather than pod scheduling. +Drain preserves local evidence without requiring remote availability. + +### Negative + +Stateful storage complicates rescheduling and disaster recovery. +Unsolved replica authorization state blocks scale-out release. + +### Risks and Mitigations + +Split-brain attachments can fork evidence. Require lease and storage fencing. +PVC loss loses unshipped evidence. Preserve registry history and report loss. +Slow streams exhaust shutdown time. Cut them and record unknown termination. +Spread constraints can block scheduling. Document installation-specific degraded availability. + +## Implementation Notes + +1. Add recovery-aware probes and idempotent drain. + Acceptance: unhealthy storage removes readiness without restart loops. +2. Test a 2-replica rolling update with continuous traffic and SSE. + Acceptance: draining pods reject new attempts; committed receipts survive rescheduling. +3. Partition a node with active upstream work, expire its lease, then request CSI force-detach. + Attempt takeover before and after proving storage fencing. + Acceptance: unfenced takeover fails; replacement tokens increase; the old writer cannot admit or publish. + Acceptance: renewal failure self-fences on monotonic deadline; takeover status rejects records beyond the last covered head. +4. Replace its PVC with an empty one. + Acceptance: enrolment gates readiness; registry records replacement and old-chain loss. +5. Drain an SSE stream longer than 40 seconds. + Acceptance: stream cuts produce unknown termination, never completion. +6. Disconnect external networking and the optional upload destination. + Acceptance: local governance and offline verification work; capacity exhaustion fails closed. +7. Validate equivalent sandbox labels and egress policy across supported operator runtimes. + Acceptance: changing runtime selection does not change receipt or sealing guarantees. + +## Related ADRs + +- [0001: Fencing and volume loss](0001-per-replica-chains.md). +- [0002: Drain recovery](0002-two-phase-receipts.md). +- [0003: Sealing](0003-durable-shipping.md). +- [0005: Enrolment and rotation](0005-root-certified-keys.md). +- [0006: Dependency only for enabled spool](0006-receipt-failure-modes.md). diff --git a/docs/design/README.md b/docs/design/README.md new file mode 100644 index 0000000..a1fb86d --- /dev/null +++ b/docs/design/README.md @@ -0,0 +1,77 @@ +# AgentGate Phase 2 decisions + +Status: Proposed +Date: 2026-10-18 + +AgentGate is part of Clawdlinux, an air-gapped agent governance and attestation plane. +These documents propose Phase 2 designs. They do not describe shipped features. +Air-gapped operation and offline verification are required. No proposal requires a public cloud. + +The code baseline is `origin/main` at `d740dc6428865d0ca6e4998d769159f461a2fdfb`. +The [v1 wire contract](../spec/RECEIPT-v1.md) remains frozen. +The [v2 outline](../spec/RECEIPT-v2.md) is a proposal, not an interoperable specification. + +## Index + +| ADR | Proposed decision | Status | Date | +| --- | --- | --- | --- | +| [0000](0000-phase2-sequencing.md) | Dependency order and independent work | Proposed | 2026-10-18 | +| [0001](0001-per-replica-chains.md) | Separate replica epochs with authenticated v2 chain IDs | Proposed | 2026-10-18 | +| [0002](0002-two-phase-receipts.md) | Commit intent before dispatch and outcome before response | Proposed | 2026-10-18 | +| [0003](0003-durable-shipping.md) | Ship signed segments from replica-local SQLite | Proposed | 2026-10-18 | +| [0004](0004-merkle-checkpoints.md) | Publish signed head vectors first; defer Tessera proofs | Proposed | 2026-10-18 | +| [0005](0005-root-certified-keys.md) | Delegate an offline root to a scoped local epoch-key issuer | Proposed | 2026-10-18 | +| [0006](0006-receipt-failure-modes.md) | Implement fail-closed first; defer spooling pending a named failure class | Proposed | 2026-10-18 | +| [0007](0007-kubernetes-lifecycle.md) | Use StatefulSet identities and isolated WAL volumes | Proposed | 2026-10-18 | + +## Current implementation + +- [Receipt protocol](../../pkg/receiptspec/receipt.go) and [encoding](../../pkg/receiptspec/encoding.go) support v1 only. +- [Ledger](../../internal/receipt/ledger.go) uses a single writer goroutine per ledger. +- Production creates 1 ledger per process. Each batch uses `BEGIN IMMEDIATE` and replies after `COMMIT`. +- [SQLite configuration](../../internal/db/sqlite.go) requests WAL mode. +- [Signer store](../../internal/signer/store.go) caches private keys and excludes rotation during batch leases. +- [Gateway](../../internal/gateway/gateway.go) commits after upstream execution, before returning its response. +- [SSE forwarding](../../internal/gateway/stream.go) starts after the header-time receipt commits. +- [Offline verifier](../../cmd/agentgate-verify/main.go) reads SQLite or JSONL. It does not verify Merkle proofs. +- [Compose](../../docker-compose.yaml) mounts local data. This baseline has no Helm chart or Kubernetes workload manifests. + +The [Phase 0 benchmark PR](https://github.com/Clawdlinux/agentgate/pull/50) contains results absent from this baseline. +Its reported latency gate is not met. +Phase 2 must not claim that extra durable writes meet that gate. + +## Open decisions + +The owner must approve these choices before dependent features are enabled. +Defaults below are recommendations, not shipped configuration. +All certificate and epoch changes remain subject to the frozen v1 compatibility boundary. + +| Decision | Recommended default | Related ADRs | +| --- | --- | --- | +| V2 protocol approval | Assign a protocol owner; approve binary preimages and fixtures before emitting v2. | [0001](0001-per-replica-chains.md), [v2](../spec/RECEIPT-v2.md) | +| Fencing backend and timing | Kubernetes Lease with registry-allocated tokens; renew every 5 seconds within a 15-second lease. Require storage fencing before takeover. | [0001](0001-per-replica-chains.md), [0007](0007-kubernetes-lifecycle.md) | +| Registry storage backend | Recommend a CRD backed by cluster etcd for air-gapped Kubernetes. Conditional object storage is an alternative; filesystem export supports offline inspection, not live lease authority. | [0001](0001-per-replica-chains.md) | +| Issuer custody and renewal | Offline root; local HSM or Vault transit for the intermediate. Renew annually; target epoch rotation every 24 hours. | [0005](0005-root-certified-keys.md) | +| Revocation scope and freshness | Intermediate-signed status every 6 hours, valid for freshness checks for 24 hours. Use explicit epoch or sequence cutoffs. | [0005](0005-root-certified-keys.md) | +| SQLite storage model | File per epoch, enabling whole-file archival. A single database instead needs an approved append-only-preserving reclamation migration. | [0003](0003-durable-shipping.md) | +| Evidence retention | Propose 30 days locally and 365 days in verified archive. Never delete unshipped evidence or required trust material. | [0003](0003-durable-shipping.md) | +| Publication profile | Signed head vectors and inventories first. Enable Tessera only after compact-proof demand and offline backend qualification. | [0004](0004-merkle-checkpoints.md) | +| Publisher custody and failover | Separate protected publisher keys; 1 active writer with storage-fenced standby. Refuse unproven takeover. | [0004](0004-merkle-checkpoints.md) | +| Accepted unshipped-loss risk | Target 10-second healthy shipping. Require explicit acceptance of asynchronous suffix loss; do not claim zero-loss durability. | [0003](0003-durable-shipping.md) | +| Replica authorization and vault state | Keep replicas at 1 until local provisioning, credential rotation, and revocation propagation pass cross-replica tests. | [0007](0007-kubernetes-lifecycle.md) | +| Client retry semantics | Bind client request IDs now. Defer retry deduplication and provider idempotency-key mapping; never automatically replay unknown outcomes. | [0002](0002-two-phase-receipts.md) | +| Spool failure class | Keep `fail_closed` and reject `fail_open_spool`. Reconsider only after a named fault and independent durability test exist. | [0006](0006-receipt-failure-modes.md) | +| Added-latency budget | The owner must re-set the gate. Phase 0 single-phase p99 already exceeds 1 ms at c16: 3.42 ms ([PR #50](https://github.com/Clawdlinux/agentgate/pull/50)). Recommend a measured, concurrency-qualified budget after the tail-latency investigation lands. | [0002](0002-two-phase-receipts.md) | +| Long-stream rollout policy | Keep the 40-second drain budget. Document stream cuts and expected unknown termination instead of promising uninterrupted SSE. | [0007](0007-kubernetes-lifecycle.md) | +| Witness policy | Disabled by default. Require explicit local quorum policy before making witness-backed anti-equivocation claims. | [0004](0004-merkle-checkpoints.md) | + +Issuer software and certificate-format approval do not solve replica-state distribution. +Multi-replica AgentGate remains blocked until that separate state problem is solved. +Untrusted clocks cannot prove status freshness; report unknown unless verifier time or a status revision is independently trusted. + +## Review boundaries + +Every decision remains proposed until owner review. +Implementation notes describe future acceptance tests, not completed validation. +Each ADR records tradeoffs and links its dependencies. +The [sequencing document](0000-phase2-sequencing.md) separates prerequisites from deferred features. diff --git a/docs/spec/RECEIPT-v2.md b/docs/spec/RECEIPT-v2.md new file mode 100644 index 0000000..504ebad --- /dev/null +++ b/docs/spec/RECEIPT-v2.md @@ -0,0 +1,200 @@ +# AgentGate receipt v2: proposed outline + +Status: Proposed. Not implemented. Not a frozen wire contract. +Date: 2026-10-18 + +This outline collects Phase 2 protocol decisions for review. +It is not sufficient for interoperable producers. +No production code in the baseline accepts v2. +The [v1 contract](RECEIPT-v1.md) remains frozen and independently verifiable. + +## 1. Version Selection and Compatibility + +Propose JSONL `format_version = 2` for v2 receipt records. +Use a new hash domain, proposed as `agentgate.receipt.hash.v2` followed by NUL. +Version dispatch must happen before selecting field validation or a hash preimage. +Never reinterpret v1 bytes under v2 rules. +Unknown versions must fail explicitly. + +A future verifier accepts both versions using separate validation and encoding paths. +Existing v1-only binaries still reject v2. +Preserve v1 preimages, public test vectors, signatures, and accepted verification semantics. +Do not update v1 encoding locks to accommodate v2. +Never re-encode historical v1 receipts during migration or export. + +Retain original v1 JSONL exports and their independently trusted key history. +A migration descriptor binds original export digests, terminal head, legacy key history, and destination v2 chain. +The offline root and final v1 key co-sign the same preimage. +It does not rewrite old receipts or manufacture authenticated v1 chain IDs. +V2 and legacy manifests require separate explicit version dispatch. + +## 2. Chain Identity + +Propose required authenticated `chain_id = :`. +`replica-id` is a lowercase provisioned UUID. +`epoch` is positive decimal without leading zeros. +Persist both; never derive cryptographic identity solely from a reusable Kubernetes ordinal. +The epoch-open control record occupies sequence 1 with a zero previous hash. +Within that chain, sequence and previous-hash checks remain mandatory. +No global ordering between replica chains is implied. + +Proposed epoch-open records reference the preceding terminal hash and segment digest when available. +Key rotation creates a new epoch. +An intact restart resumes its current epoch unless a loss or takeover cutoff requires another. +Lost or restored storage requires fencing and a new replica UUID. +An external epoch lease and monotonic fencing token are checked before admission and each writer batch. +Use cached ownership and a monotonic renewal deadline, not a per-batch API-server request. +Self-fence after approximately 10 seconds with 5-second renewals and a 15-second lease. +The deadline must be shorter than lease duration minus maximum clock drift. +Bind that token into signed records and descriptors. +The issuer refuses duplicate `(UUID, epoch)` certification, but a copied certificate still requires lease-based fencing. +Offline coverage compares inventories with the intermediate-signed replica registry and issued-certificate log. +Registered chains with neither epoch-close nor inventory coverage are reported lost, even when no receipt survives. +Empty replacement PVCs require explicit issuer enrolment and a registry entry replacing the earlier UUID for that ordinal. + +## 3. Proposed Record Kinds + +All records authenticate their kind, chain identity, fencing token, sequence, previous hash, timestamp, and signing key. +Required action fields and optional result fields must be specified per kind. +Do not copy v1's required HTTP status into an intent with a fabricated value. + +| Kind | Proposed authenticated content | +| --- | --- | +| `intent` | Attempt ID, optional client request ID, principal, agent, delegation, destination action, params digest, policy | +| `outcome` | Full intent reference, observation, upstream attempt count, known status, latency, bounded error code | +| `stream_result` | Header-outcome reference, available byte digest and count, termination classification | +| `decision` | Authenticated denied or rate-limited attempt without upstream dispatch | +| `control` | Typed mode changes or epoch boundaries with actor, reason, and bound configuration | + +An intent reference includes chain ID, sequence, entry hash, and attempt ID. +Every attempt ID is gateway-generated and unique within its replica identity. +Specify its exact size and encoding before implementation. +Unknown outcomes use `outcome_unknown`, without invented provider status or successful completion. +Runtime-certain `not_dispatched` requires a known zero action-dispatch count. +Crash recovery cannot infer that state from an orphan intent. +Count action transport retries in `upstream_attempt_count`; unknown counts remain explicitly absent. +Propose a distinct observation field rather than extending v1's frozen policy enum. +Bind any validated client request ID into the intent. +Client retry deduplication and upstream `Idempotency-Key` mapping remain deferred. + +SSE outcomes attest header acceptance before headers are forwarded. +Supplemental terminal records bind SHA-256 of bytes accepted by downstream writes. +They do not prove client delivery. +Recovery emits unknown outcomes or unknown stream termination without replaying upstream calls. +References may cross epoch boundaries. +Duplicate or contradictory terminal records must be rejected. +Draining streams after 40 seconds may produce unknown termination with reason `drain_deadline`. +That expected interruption is not a successful stream outcome. + +The [earlier stream proposal](proposals/receipt-stream-digest.md) remains unimplemented. +This outline proposes replacing its single-action-receipt rule for v2 lifecycle records. + +## 4. Hashing and Signature Work Remaining + +Propose SHA-256 entry hashes and Ed25519 receipt signatures over entry hashes. +Authenticate version and kind under the new domain. +Include chain identity and lifecycle references in the signed preimage. +Keep gateway params canonicalization unchanged unless separately versioned and reviewed. +Do not treat the narrower decision-record canonicalizer as interchangeable with gateway params. + +Before acceptance, specify exact field order, integer widths, endianness, string prefixes, and absence markers. +Also freeze string bounds, enum values, timestamp rules, attempt-ID grammar, and cross-field validation. +Assign separate domains to certificates, segment descriptors, inventories, and migration descriptors. +Define strict duplicate-field and unknown-field handling for new signed containers. +No placeholder encoding in this outline authorizes production emission. + +## 5. Keys and Export Artifacts + +Propose an offline organization root delegating to a longer-lived, in-cluster intermediate issuer. +The root delegation binds organization, installation, allowed purposes, and no further intermediate delegation. +The issuer certifies epoch keys and records each issuance in the signed replica registry. +Use v1-style length-prefixed binary certificate preimages, not canonical JSON. +Freeze domain strings, field order, `U32LE` lengths, `U64LE` integers, raw keys, and absence markers. +Ed25519 signs the SHA-256 certificate preimage digest. + +| Purpose | Authorized scope and signer | +| --- | --- | +| `certificate_issuance` | Offline-root-delegated intermediate; permitted installation and child purposes only | +| `replica_registry` | Intermediate; registrations, issued certificates, fencing generations, replacements, and losses | +| `receipt` | Epoch key; action lifecycle records in exactly 1 UUID and epoch | +| `control` | Epoch key; epoch boundaries, authenticated mode changes, and future sink-transition markers | +| `segment_descriptor` | Epoch key; its epoch's signed segment ranges | +| `legacy_segment` | Intermediate-certified archival publisher; migrated v1 export scope | +| `inventory` | Intermediate-certified singleton publisher; cross-replica inventories and head vectors | +| `checkpoint` | Separately certified log key; installation and checkpoint origin | +| `trust_status` | Intermediate; delegated-key status and explicit sequence or epoch revocation cutoffs | +| `migration` | Offline root plus final v1 signing key; mandatory co-signatures | + +Epoch and sequence bounds govern validity, not receipt clock values. +The certificate binds exactly 1 epoch starting at sequence 1. +Signed epoch-close fixes its inclusive terminal sequence. +Verifiers reject later records and require signed inventory coverage of close and final descriptor for terminal claims. +When Merkle mode is enabled, also prove epoch-close inclusion against its checkpoint. +Without that coverage, report an unclosed prefix or missing terminal evidence, not a complete chain. +Time windows are operational renewal hygiene only. +Recommend annual intermediate renewal and 24-hour epoch rotation. +Do not reject historical evidence merely because its time window elapsed. + +Rotate at a writer batch boundary without draining admission or inflight calls. +Obtain the next certificate first; intermediate outages leave the replica in its current epoch and trigger alerts. +Existing lease and trust-status bounds still govern continued signing. +Durably seal the old epoch's final descriptor before destroying its key. +The new epoch starts with epoch-open; later outcomes may reference prior-epoch intents. +Registry and activation recovery must prevent duplicate issuance or competing active epochs. + +Recommend intermediate-signed trust-status statements every 6 hours, with a 24-hour freshness limit. +Bind revision, predecessor digest, affected certificates, and explicit revocation cutoffs. +Registry loss declarations and same-UUID lease takeovers require a cutoff at the last covered epoch head. +Verifiers reject every old-epoch record beyond its sequence and hash boundary; uncovered epochs use a sequence-0 cutoff. +Persist that status before takeover activation and use a newly certified epoch for continuation. +Freshness requires trusted verifier time or a separately pinned revision; otherwise report freshness unknown. +Stale or missing status cannot produce a current-trust success claim. +Root-signed status handles intermediate replacement. +Preserve v1 trust behavior outside the co-signed migration envelope. + +Proposed sealed segments contain original receipt bytes and signed range descriptors. +Descriptors bind payload digest, chain scope, anchor, terminal hash, predecessor segment, and certificate digests. +Specify export-container versions independently of receipt versions. +Offline bundles include required certificates and signed trust-status information. +They also include registry history and the signed inventory or head vector. +Their trust anchor is an independently pinned organization root, never an embedded self-asserted root. +Descriptors and inventories require conditional immutable creation; filesystem export requires exclusive creation and fsync. +The initial failure mode is `fail_closed`, returning 503 when required durable recording fails. +Spooling remains deferred until its covered failure class and external sink-transition marker protocol are approved. + +## 6. Transparency Proof Profile + +Recommend signed head vectors and inventories first. They do not provide compact Merkle inclusion proofs. +The fenced singleton publisher validates segment certificate paths against the independently pinned root. +Its durable predecessor state and storage fencing are required during failover. +For the later Tessera profile, propose C2SP [tlog-checkpoint](https://c2sp.org/tlog-checkpoint@v1.1.0) +and [tlog-tiles](https://c2sp.org/tlog-tiles@v1.0.0) with SHA-256. +Freeze leaf bytes binding receipt version, authenticated chain scope, sequence, and entry hash. +Legacy scope comes from its signed descriptor, not modified v1 receipts. +Specify inclusion bundles and consistency proofs with independent fixtures. +Use standard signed-note checkpoint signatures. +Export checkpoint files, tiles, inventories, certificates, and receipts for disconnected verification. +Witness quorum remains optional. + +## 7. Acceptance Gates and Open Questions + +1. Approve a complete byte-level specification and deterministic vectors for each record and artifact kind. +2. Preserve all v1 vector outcomes and byte locks under dual-version verification. +3. Reject mismatched chain identity, duplicate terminal references, conflicting segments, and unauthorized certificate purposes. +4. Run crash recovery across intent, outcome, fencing, sealing, and batch-boundary epoch rotation. +5. Verify vectors offline; gate later Merkle mode on offline inclusion and consistency fixtures. +6. Document registry coverage, revocation freshness, storage-model retention, and certificate renewal targets. + +The [open decisions](../design/README.md#open-decisions) record recommended defaults. +Proof freshness requires a separately trusted checkpoint or time policy. +No artifact alone proves that every external side effect was recorded. + +## Related ADRs + +- [0001: Per-replica chains](../design/0001-per-replica-chains.md) +- [0002: Intent and outcome](../design/0002-two-phase-receipts.md) +- [0003: Sealed segments](../design/0003-durable-shipping.md) +- [0004: Merkle checkpoints](../design/0004-merkle-checkpoints.md) +- [0005: Scoped issuer and epoch keys](../design/0005-root-certified-keys.md) +- [0006: Failure modes](../design/0006-receipt-failure-modes.md) +- [0007: Kubernetes lifecycle](../design/0007-kubernetes-lifecycle.md)