From 90695d31df6e3bdf07308eb36abedc3e6e039aec Mon Sep 17 00:00:00 2001 From: Shreyansh Sancheti <43677304+shreyanshjain7174@users.noreply.github.com> Date: Sun, 18 Oct 2026 06:17:34 +0530 Subject: [PATCH 1/3] docs(design): propose phase 2 receipt ADRs Signed-off-by: Shreyansh Sancheti <43677304+shreyanshjain7174@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/design/0000-phase2-sequencing.md | 86 ++++++++++++++ docs/design/0001-per-replica-chains.md | 103 +++++++++++++++++ docs/design/0002-two-phase-receipts.md | 127 ++++++++++++++++++++ docs/design/0003-durable-shipping.md | 122 ++++++++++++++++++++ docs/design/0004-merkle-checkpoints.md | 118 +++++++++++++++++++ docs/design/0005-root-certified-keys.md | 123 ++++++++++++++++++++ docs/design/0006-receipt-failure-modes.md | 114 ++++++++++++++++++ docs/design/0007-kubernetes-lifecycle.md | 132 +++++++++++++++++++++ docs/design/README.md | 48 ++++++++ docs/spec/RECEIPT-v2.md | 134 ++++++++++++++++++++++ 10 files changed, 1107 insertions(+) create mode 100644 docs/design/0000-phase2-sequencing.md create mode 100644 docs/design/0001-per-replica-chains.md create mode 100644 docs/design/0002-two-phase-receipts.md create mode 100644 docs/design/0003-durable-shipping.md create mode 100644 docs/design/0004-merkle-checkpoints.md create mode 100644 docs/design/0005-root-certified-keys.md create mode 100644 docs/design/0006-receipt-failure-modes.md create mode 100644 docs/design/0007-kubernetes-lifecycle.md create mode 100644 docs/design/README.md create mode 100644 docs/spec/RECEIPT-v2.md diff --git a/docs/design/0000-phase2-sequencing.md b/docs/design/0000-phase2-sequencing.md new file mode 100644 index 0000000..616d8cd --- /dev/null +++ b/docs/design/0000-phase2-sequencing.md @@ -0,0 +1,86 @@ +# ADR-0000: Phase 2 sequencing + +## Status + +Proposed + +## Date + +2026-10-18 + +## Context + +The [v1 contract](../spec/RECEIPT-v1.md) is frozen. +The [ledger](../../internal/receipt/ledger.go) and [signer](../../internal/signer/store.go) already coordinate group commits and rotation leases. +Phase 2 proposes new protocols around those guarantees. +These documents change no production code. + +## Decision Drivers + +- Preserve v1 evidence and offline verification. +- Separate schema agreement from implementation. +- Keep air-gapped deployments independent of cloud services. + +## Considered Options + +| Option | Pros | Cons | +| --- | --- | --- | +| Implement all proposals together | Few temporary interfaces | Hard failure attribution and broad review | +| Sequence contracts, then independent components | Smaller tests and clearer dependencies | Requires agreed artifact fixtures first | + +## Decision + +Propose this dependency order: + +| Step | ADR | Prerequisites | Acceptance gate | +| --- | --- | --- | --- | +| 1 | [0001](0001-per-replica-chains.md) | None | Agree v2 chain identity and unchanged v1 fixtures | +| 2 | [0005](0005-root-certified-keys.md) | 0001 | Agree certificate scopes and epoch rotation fixtures | +| 3a | [0002](0002-two-phase-receipts.md) | 0001, 0005 | Verify durable intent ordering and crash recovery | +| 3b | [0003](0003-durable-shipping.md) | 0001, 0005 | Recover and verify sealed segments offline | +| 4a | [0004](0004-merkle-checkpoints.md) | 0003, 0005 | Verify segment receipts against offline checkpoints | +| 4b | [0006](0006-receipt-failure-modes.md) | 0002, 0003 | Prove bounded spool admission and exact replay | +| 5 | [0007](0007-kubernetes-lifecycle.md) | 0001, 0002, 0003, 0005, 0006 | Pass restart, drain, volume fencing, and outage tests | + +0004 is not a prerequisite for Kubernetes ledger availability. +Kubernetes proof-publication tests require 0004 when that optional publisher is enabled. +0007 follows 0005 transitively through its other prerequisites. + +## Consequences + +### Positive + +0002 and 0003 can be implemented independently after shared contracts settle. +0004 and 0006 can then proceed independently. +Tessera POSIX feasibility tests, benchmark reruns, and Kubernetes manifest drafts can start earlier. +V1 golden-vector regression work is independent of every new protocol. + +### Negative + +Final integration waits for stable fixtures and key provisioning. +Manifest-signer validity-window hardening remains separate work, not a prerequisite for preserving v1 bytes. + +### Risks and Mitigations + +Interface drift could invalidate parallel work. Freeze shared fixtures before merging dependent implementations. +Additional commits could worsen latency. Re-run Phase 0 workloads before accepting performance claims. + +## Implementation Notes + +1. Review [RECEIPT-v2](../spec/RECEIPT-v2.md) before any producer emits v2. +2. Land compatibility tests before new encoding or epoch storage. +3. Require each ADR's acceptance tests before enabling its feature. +4. Run integrated crash tests and disconnected verification before declaring Phase 2 usable. + +### Owner Questions + +- Who approves v2 fields and the v1-to-v2 migration envelope? +- What latency budget applies after the Phase 0 gate failed? +- What certificate lifetime and offline renewal interval fit each installation? +- What retention period, spool quota, and accepted unshipped-loss window apply? +- How will replicas provision consistent authorization and vault data currently stored in SQLite? +- Who operates the optional log publisher, and does any installation require witness quorum? + +## Related ADRs + +[Index](README.md) links all 7 proposed decisions. diff --git a/docs/design/0001-per-replica-chains.md b/docs/design/0001-per-replica-chains.md new file mode 100644 index 0000000..4ef5c8c --- /dev/null +++ b/docs/design/0001-per-replica-chains.md @@ -0,0 +1,103 @@ +# ADR-0001: Per-replica chains and epochs + +## Status + +Proposed + +## Date + +2026-10-18 + +## Context + +The [ledger](../../internal/receipt/ledger.go) reads a single committed head inside each `BEGIN IMMEDIATE` batch. +Its primary key is currently [global `seq`](../../internal/db/migrations/002_receipts.sql) within 1 database. +The [signer store](../../internal/signer/store.go) explicitly supports 1 gateway writer per database. +Sharing a live database between replicas would violate that signing model. + +[V1](../spec/RECEIPT-v1.md) authenticates neither a replica identifier nor an epoch. +Adding unsigned JSON metadata cannot turn either into authenticated identity. + +## Decision Drivers + +- Scale replicas without cross-replica SQLite locking. +- Make sequence reuse distinguishable after rotation or replacement. +- Preserve all v1 bytes and verification semantics. +- Support disconnected verification without a global sequencer. + +## Considered Options + +| Option | Pros | Cons | +| --- | --- | --- | +| Shared global chain | Simple total order | Shared writer bottleneck and coordination dependency | +| Per-replica chains without epochs | Local availability | Recovery and key replacement can reuse identities | +| Per-replica epoch chains | Local commits and explicit boundaries | Auditors must inspect several chains | + +## Decision + +Propose a required authenticated v2 field: `chain_id = :`. +The [v2 outline](../spec/RECEIPT-v2.md) proposes its encoding boundaries. +`replica-id` is a provisioned UUID, not a reusable pod ordinal. +`epoch` is a positive, monotonically increasing integer persisted with the replica identity. +Each epoch starts at `seq = 1` with a zero previous hash. +The uniqueness key becomes `(chain_id, seq)`. + +A healthy restart with intact storage resumes its existing epoch. +Key rotation creates a new epoch. +Lost storage requires a new replica UUID, even when Kubernetes reuses an ordinal. +An exclusive volume fence must prevent concurrent writers and restored-volume clones from reusing a live identity. +No proposal establishes a total execution order between replicas. + +Propose signed epoch-open and epoch-close control records. +The opening record binds the preceding epoch's terminal hash and closing segment digest when available. +A missing predecessor is explicit, not presented as continuity. +Restoring an old backup requires fencing and a new replica UUID. + +Propose a version-aware verifier that accepts v1 and v2. +Its v1 path retains the frozen preimages, signatures, and accepted semantics. +V1 receipts are never re-encoded during migration or shipping. +Keep their original JSONL bytes and existing verification vectors. +Existing v1-only binaries will still reject v2. + +Migration closes the v1 ledger at a durable head. +A separately signed migration descriptor binds that head, its original export digest, and the new v2 chain. +That descriptor does not add fields to historical receipts. +The legacy export retains its original trust requirements. + +## Consequences + +### Positive + +Each process retains a local group-commit writer. +Epoch identity separates key lifetimes and recovery histories. +Offline tooling can report missing epochs without querying a coordinator. + +### Negative + +Storage migrations must separate epoch metadata from the old global sequence model. +Chain aggregation and inventory become explicit operational work. + +### Risks and Mitigations + +Replica cloning can create forks. Require exclusive ownership and reject duplicate `(chain_id, seq)` values with different hashes. +An unpublished suffix can disappear. Compare signed inventories and checkpoints, not chain links alone. +Epoch resets could hide missing history. Require explicit predecessor references and report unverifiable gaps. + +## Implementation Notes + +1. Freeze v2 hash fields and chain-ID grammar with cross-language fixtures before implementing storage. +2. Add dual-version dispatch without changing the v1 parser or golden encodings. + Acceptance: all existing v1 vectors retain their results. +3. Introduce durable replica identity, epoch allocation, and separate chain heads. + Acceptance: concurrent replicas can both emit sequence 1 without collision. +4. Fence migration and rotation around writer admission. + Acceptance: restart resumes the same head; rotation starts exactly 1 new epoch. +5. Exercise lost volumes, cloned backups, and interrupted migration. + Acceptance: no chain identity is silently reused, and original v1 exports remain byte-identical. + +## Related ADRs + +- [0002: Lifecycle records](0002-two-phase-receipts.md) references intents by chain and hash. +- [0003: Segments](0003-durable-shipping.md) preserves epoch boundaries. +- [0005: Keys](0005-root-certified-keys.md) certifies epoch-scoped signing keys. +- [0007: Kubernetes](0007-kubernetes-lifecycle.md) provides persistent identity and fencing. diff --git a/docs/design/0002-two-phase-receipts.md b/docs/design/0002-two-phase-receipts.md new file mode 100644 index 0000000..074a6d5 --- /dev/null +++ b/docs/design/0002-two-phase-receipts.md @@ -0,0 +1,127 @@ +# ADR-0002: Durable intent and outcome receipts + +## Status + +Proposed + +## Date + +2026-10-18 + +## Context + +The [gateway](../../internal/gateway/gateway.go) executes upstream work before appending its v1 receipt. +It returns the response only after that append commits. +The [ledger](../../internal/receipt/ledger.go) documents the resulting external-side-effect crash gap. +An upstream action can succeed without a later committed receipt. + +[SSE](../../internal/gateway/stream.go) starts only after a header-time receipt commits. +Final stream digests currently reach logs, not signed receipts. +The [stream-digest proposal](../spec/proposals/receipt-stream-digest.md) is unimplemented. + +## Decision Drivers + +- Record a durable obligation before dispatch. +- Report uncertainty without replaying external side effects. +- Preserve the receipt-before-response boundary, including SSE headers. +- Measure the extra durable write rather than claiming free batching. + +## Considered Options + +| Option | Pros | Cons | +| --- | --- | --- | +| Outcome only | Current latency and simple schema | Crash can erase evidence of an action | +| Intent only | Records attempted dispatch | Cannot attest observed outcomes | +| Durable intent, then outcome | Exposes missing outcomes | Adds latency and recovery state | + +## Decision + +Propose v2 lifecycle records, not new meanings for v1 fields. +After authentication, shape validation, and policy evaluation, allocate a gateway-generated attempt ID. +Persist a signed `intent` before any upstream call, including credential refresh. +The intent binds identity, delegation, params digest, policy, and destination action. +Denied attempts get a terminal decision record without an upstream intent. + +An acknowledged intent permits 1 logical execution under the existing safe retry policy. +An append timeout can still leave a committed record. +Resolve that attempt's durable state before dispatch; ambiguity must not trigger an upstream retry. +Provider retries and idempotency remain separate policies. Receipts do not guarantee exactly-once external execution. + +Append a signed `outcome` after observing the upstream result. +It references the intent's `chain_id`, sequence, entry hash, and attempt ID. +For buffered responses, commit the outcome before returning status or body. +Persist outcomes with a bounded context detached from client cancellation, matching the current gateway boundary. + +For SSE, commit an `outcome` with `observation = headers_accepted` before forwarding headers. +That outcome does not claim stream completion. +Append a supplemental `stream_result` at termination. +It binds the outcome hash, written-byte SHA-256, byte count, and termination reason. +The digest covers bytes accepted by downstream writes, not proven client delivery. +A missing terminal record means unknown completion, never success. +Keep terminal artifacts out of forwarded SSE bytes. +This proposed lifecycle replaces the earlier stream proposal's single-action-receipt rule for v2 only. + +Before admitting traffic after restart, scan committed intents without outcomes. +Append recovery outcomes marked `outcome_unknown`. Never call upstream during recovery. +For header outcomes without terminal stream records, append `stream_result` with unknown termination and no fabricated digest. +Fenced recovery and unique references must prevent duplicate terminal records. +Recovery records can reference an earlier epoch. + +### Group-Commit Cost + +Intent and outcome cannot share a transaction for their own request. +Other requests can still share each batch's fsync. +Allowed buffered attempts require 2 durable boundaries instead of 1. +SSE adds a later terminal record as well. + +`bench/RESULTS.md` is absent on the pinned main baseline. +[Phase 0 PR #50](https://github.com/Clawdlinux/agentgate/pull/50), head `29ef207`, reports these added-overhead p99 measurements: + +| Workload | Existing 1-receipt path | +| --- | --- | +| Closed loop, 0 ms upstream, concurrency 1 | 0.34 ms | +| Closed loop, 0 ms upstream, concurrency 16 | 3.42 ms | +| Closed loop, 0 ms upstream, concurrency 64 | 6.22 ms | +| Open loop, 0 ms upstream, 25% load | 7.00 ms | + +The PR used Apple M4, `GOMAXPROCS=4`, and 20,000 requests per row. +It did not meet the under-1-ms p99 gate. +Those are end-to-end overheads, not isolated fsync measurements. +Doubling them would not predict two-phase p99. +Budget 1 extra serialized commit wait before dispatch, then benchmark it on Linux with dedicated 4-vCPU capacity. + +## Consequences + +### Positive + +Crashes leave inspectable obligations. +Offline verification can distinguish observed results from unknown outcomes. + +### Negative + +Write volume and request latency increase. +Recovery requires lifecycle indexes beyond a linear hash chain. + +### Risks and Mitigations + +An unknown outcome may hide an executed action. Require operator reconciliation, not automatic replay. +SSE can fail after response headers. End the stream and persist uncertainty when storage recovers. + +## Implementation Notes + +1. Freeze lifecycle references and optional fields in [v2](../spec/RECEIPT-v2.md). + Acceptance: reject missing, duplicate, or mismatched intent references. +2. Add intent ordering instrumentation. + Acceptance: the mock upstream sees no call before intent commit acknowledgment. +3. Run subprocess `kill -9` tests before intent commit, after commit, during dispatch, and after provider success. + Also kill between the upstream call and outcome commit, after outcome commit, and during SSE. + Acceptance: recovery is idempotent, records unknown states, and never re-dispatches. +4. Re-run Phase 0 closed-loop and open-loop workloads. + Acceptance: publish intent-wait and outcome-wait percentiles separately; obtain owner approval for the latency budget. + +## Related ADRs + +- [0001: Chains](0001-per-replica-chains.md) supplies stable references. +- [0003: Shipping](0003-durable-shipping.md) retains recovery obligations. +- [0006: Failure modes](0006-receipt-failure-modes.md) defines alternate durable recording. +- [0007: Drain](0007-kubernetes-lifecycle.md) bounds unfinished streams. diff --git a/docs/design/0003-durable-shipping.md b/docs/design/0003-durable-shipping.md new file mode 100644 index 0000000..c26e56f --- /dev/null +++ b/docs/design/0003-durable-shipping.md @@ -0,0 +1,122 @@ +# ADR-0003: SQLite outbox and signed sealed segments + +## Status + +Proposed + +## Date + +2026-10-18 + +## Context + +The [ledger](../../internal/receipt/ledger.go) already serializes signed batches into SQLite. +[WAL configuration](../../internal/db/sqlite.go) provides local storage, not off-node replication. +Current [exports](../../internal/receipt/export.go) provide signed snapshot ranges, not durable shipping. +The [schema](../../internal/db/migrations/002_receipts.sql) rejects receipt updates and deletes. + +AgentGate belongs to Clawdlinux's air-gapped governance and attestation plane. +Postgres is PARKED for Phase 2. +No storage design may require public-cloud connectivity. + +## Decision Drivers + +- Preserve local commits during destination outages. +- Ship independently verifiable artifacts without exporting raw live WAL files. +- Bound disk growth without silently deleting unshipped evidence. +- Support disconnected installations. + +## Considered Options + +| Option | Pros | Cons | +| --- | --- | --- | +| Replica SQLite plus signed segments | Fits current writer; S3 or in-cluster MinIO; disconnected export | Asynchronous loss window; custom sealing and replay | +| Postgres transactional outbox | Shared SQL transactions and mature replication | PARKED; requires database operations and network availability | +| Kafka-like durable log | Replay and partitioned throughput | Additional quorum, retention, and partition ownership operations | + +Postgres and Kafka-like systems can run offline. Cloud dependence is not the reason to reject them. +They add dependencies before the artifact protocol settles. + +## Decision + +Propose replica-local SQLite WAL plus a durable shipping outbox. +Ship signed immutable segments to configurable S3-compatible storage. +Support in-cluster MinIO and filesystem export for fully disconnected installations. +Postgres remains PARKED. +Uploads stay outside the request commit path. + +Seal at the proposed initial limits of 4,096 records, 16 MiB, or 10 seconds, whichever comes first. +Also seal on epoch closure and graceful drain. +Each segment covers a contiguous, nonempty range from exactly 1 chain. +Store original receipt bytes. Never re-encode v1 receipts. +For existing SQLite history, capture its initial v1 export with the current exporter and preserve that artifact unchanged. +Do not ship a live SQLite database, WAL, or SHM file as a segment. + +A versioned signed descriptor binds chain identity, range, count, anchor, terminal hash, and previous segment digest. +Also bind payload SHA-256 and certificate or legacy keyset digests. +Use a separate signing domain and strict range validation, not an undocumented extension of v1 manifests. +Bundle certificates, but independently pin the trusted root. + +The ledger writer reserves a segment range and outbox ID transactionally. +A read snapshot supplies immutable rows for deterministic materialization. +Write and fsync payload and descriptor, then atomically rename and fsync their directory. +Mark the outbox item sealed only after those files are durable. +An interrupted reservation is recreated from retained rows after restart. +Uploads never hold the SQLite write transaction. + +Use object names containing chain ID, range, and payload digest. +Retry the same name and bytes after uncertain acknowledgments. +Verify the stored digest independently of multipart ETags. +A pre-existing different object fails integrity checks. +Upload payload before descriptor; publish the inventory reference last. +Only the final reference advertises completeness. +Persist acknowledgments and inventory progress locally. + +Retain unuploaded segments and their source rows regardless of age. +Apply an owner-set retention period only after upload, read-back verification, and any required proof publication. +Do not bypass current no-delete triggers. +Reclaim whole closed epoch databases and segment files through a future archival workflow. +Retain trusted anchors, certificates, and checkpoints as long as retained evidence needs them. +Quota exhaustion blocks admission rather than evicting pending evidence. + +Verifiers read a signed inventory, then validate descriptors, payload digests, chain links, and epoch certificates. +Sort segments by authenticated range, not object listing order. +Reject gaps and conflicting overlaps; identical retries are harmless. +Optional proofs bind these receipts to [0004 checkpoints](0004-merkle-checkpoints.md). +Inventory completeness is only relative to its signed capture point. + +## Consequences + +### Positive + +Local admission does not depend on storage network availability. +The same artifacts work through MinIO, S3, or removable media. + +### Negative + +Sealing and archival recovery require new implementation. +Off-node durability lags local acknowledgment. + +### Risks and Mitigations + +A lost volume can lose an unshipped suffix. Publish lag and agree an accepted loss window. +Destination outages can fill disks. Alert on age and capacity, then stop admission before reserved outcome space disappears. +Malicious storage can roll back inventory. Compare separately pinned checkpoints and report freshness as unknown when unpinned. + +## Implementation Notes + +1. Freeze descriptor, inventory, and outbox state-machine fixtures. + Acceptance: original v1 export bytes and v2 signatures verify offline. +2. Inject crashes around reservation, fsync, rename, upload, and acknowledgment. + Acceptance: retries never skip ranges or publish partial segments. +3. Test conflicting object names and lost upload replies against in-cluster MinIO. + Acceptance: exact retries succeed; conflicting bytes fail verification. +4. Test destination disconnection and quota pressure. + Acceptance: no pending evidence is deleted, and retention never bypasses append-only triggers. + +## Related ADRs + +- [0001: Epochs](0001-per-replica-chains.md) defines segment boundaries. +- [0004: Merkle publication](0004-merkle-checkpoints.md) consumes sealed segments. +- [0005: Certificates](0005-root-certified-keys.md) authorizes descriptor signers. +- [0006: Spool](0006-receipt-failure-modes.md) feeds the same shipping path. diff --git a/docs/design/0004-merkle-checkpoints.md b/docs/design/0004-merkle-checkpoints.md new file mode 100644 index 0000000..8d61328 --- /dev/null +++ b/docs/design/0004-merkle-checkpoints.md @@ -0,0 +1,118 @@ +# ADR-0004: Tiled Merkle log and offline checkpoints + +## Status + +Proposed + +## Date + +2026-10-18 + +## Context + +[V1 verification](../../pkg/receiptspec/verifier.go) authenticates a linear chain against supplied keys and an optional endpoint. +A valid prefix alone cannot detect a removed suffix or a conflicting view. +[agentgate-verify](../../cmd/agentgate-verify/main.go) currently has no Merkle inclusion or consistency checks. +Signed segments propose durable inputs, not an existing transparency log. + +## Decision Drivers + +- Provide compact inclusion evidence and append-only checkpoint comparisons. +- Use published formats instead of inventing tree cryptography. +- Publish inside an air gap without public Sigstore services. +- Keep request commits independent of proof publication. + +## Considered Options + +| Option | Pros | Cons | +| --- | --- | --- | +| Trillian Tessera | Go library; tiled log formats; upstream POSIX support | New dependency and publisher integration; backend qualification needed | +| Hand-written Merkle tree | Small initial code and direct SQLite integration | Team owns proof correctness, recovery, and long-term format maintenance | +| Sigstore Rekor | Existing transparency APIs and verification ecosystem | Supply-chain schema adaptation and extra service operations | + +[Tessera](https://github.com/transparency-dev/tessera) is Trillian's tiled-log successor, not the older Trillian service stack. +[Rekor](https://github.com/sigstore/rekor) can be self-hosted. Public-network dependence is not an inherent requirement. +Its broader application protocol is unnecessary for the proposed receipt log. + +## Decision + +Propose Tessera behind a narrow asynchronous publisher interface. +Qualify a pinned release with its POSIX backend inside a disconnected cluster before adoption. +Do not assume S3 compatibility means every Tessera backend works with MinIO. +The segment shipper and tree backend are separate concerns. +If offline backend qualification fails, revisit this ADR before implementation. + +Adopt [C2SP tlog-checkpoint v1.1.0](https://c2sp.org/tlog-checkpoint@v1.1.0) +and [tlog-tiles v1.0.0](https://c2sp.org/tlog-tiles@v1.0.0). +Use SHA-256 with RFC 6962 domain separation. +Leaf hashes are `SHA256(0x00 || leaf_bytes)`. +Internal hashes are `SHA256(0x01 || left_hash || right_hash)`. +Use standard signed-note checkpoint encoding with an Ed25519 log signature. +Do not invent JSON checkpoint signatures or optional extension lines. + +Propose a small versioned leaf encoding binding receipt version, chain scope, sequence, and entry hash. +The v2 scope is its authenticated `chain_id`. +For v1, scope comes from the signed legacy segment descriptor, without modifying the receipt. +Specify leaf bytes and proof fixtures before implementation. +Tree membership does not replace receipt signatures or chain verification. + +The publisher validates sealed segments before integration. +A durable mapping from `(chain_scope, seq, entry_hash)` to leaf index prevents upload retries from adding duplicate leaves. +Reject conflicting hashes for an already published chain position. +Cross-replica integration order records publisher arrival, not external execution order. +Crash recovery must reconcile ambiguous integration before accepting new work. + +Propose a checkpoint every 10 seconds or 4,096 newly integrated leaves, whichever comes first. +Publish tiles and entry bundles durably before publishing their checkpoint. +Retain immutable checkpoint history plus a mutable latest pointer. +Idle logs need not issue identical checkpoints. +Checkpoint size authenticates coverage, not wall-clock freshness. +Publish a signed segment inventory alongside each checkpoint to expose per-chain coverage. + +Propose `agentgate-verify` support for local proof bundles and tile directories. +Verify the root-certified log key, checkpoint origin and signature, leaf identity, inclusion path, and tree size. +Verify receipt signatures and segment boundaries separately. +Check consistency against an earlier trusted checkpoint when supplied. +A single checkpoint does not prove global freshness or absence of split views. + +Air-gapped publication writes signed checkpoint files, tiles, certificates, and inventories to in-cluster storage. +Offline export copies those files and receipt segments through approved removable media. +No public Rekor, cloud KMS, or network fetch is required by verification. +Witness cosigning is optional and disabled by default. +An installation may configure local witnesses and a quorum policy. +Without that policy, make no witness-backed anti-equivocation claim. + +## Consequences + +### Positive + +Proof verification can remain local and independent of the gateway process. +Published formats reduce custom cryptographic surface. + +### Negative + +Proof publication introduces asynchronous lag and another durable service role. +Tiles and historical checkpoints consume storage beyond the gateway volume. + +### Risks and Mitigations + +A publisher can equivocate. Compare pinned checkpoints and optionally require independent local witness signatures. +Delayed publication can hide recent loss. Report integrated coverage separately from committed and uploaded heads. +Dependency behavior can change. Pin releases and run disconnected interoperability fixtures before upgrades. + +## Implementation Notes + +1. Qualify Tessera POSIX recovery without external connectivity. + Acceptance: restart preserves tree size, roots, and leaf indices. +2. Freeze leaf and checkpoint-key certificate formats. + Acceptance: independent C2SP tooling verifies checkpoints and reconstructs tile proofs. +3. Test duplicate segment ingestion and crashes during integration. + Acceptance: each unique chain position has 1 stable leaf index. +4. Add CLI fixtures for wrong roots, wrong origins, missing tiles, and invalid inclusion or consistency paths. + Acceptance: offline checks reject each mutation while existing v1 verification remains unchanged. + +## Related ADRs + +- [0001: Chains](0001-per-replica-chains.md) defines leaf identity. +- [0003: Segments](0003-durable-shipping.md) supplies validated inputs. +- [0005: Root keys](0005-root-certified-keys.md) certifies the log signing role. diff --git a/docs/design/0005-root-certified-keys.md b/docs/design/0005-root-certified-keys.md new file mode 100644 index 0000000..b065602 --- /dev/null +++ b/docs/design/0005-root-certified-keys.md @@ -0,0 +1,123 @@ +# ADR-0005: Root-certified replica signing keys + +## Status + +Proposed + +## Date + +2026-10-18 + +## Context + +The [signer store](../../internal/signer/store.go) caches decrypted Ed25519 keys in memory. +`WithActiveKey` holds a lease through each ledger transaction, excluding `Rotate` and `Close`. +`Rotate` switches keys at the next global sequence and clears old private bytes. +It creates neither epochs nor root certificates. + +[V1 key history](../spec/RECEIPT-v1.md) uses independently trusted public keys with inclusive sequence windows. +Embedded export keys alone prove consistency, not organization identity. +Current [manifest verification](../../pkg/receiptspec/manifest.go) does not enforce its signer's validity window. +Manifest-signer validity-window hardening is in progress on another branch, not shipped in this baseline. + +## Decision Drivers + +- Let an offline verifier trust 1 independently provisioned organization root. +- Limit a compromised replica key's scope and lifetime. +- Keep roots outside request handling. +- Preserve cached signing and batch rotation leases. + +## Considered Options + +### Root Custody + +| Option | Pros | Cons | +| --- | --- | --- | +| KMS or HSM | Managed custody or hardware protection | Product-specific algorithms, cost, and availability | +| Local Vault transit | Local automation and access policy | Additional service and recovery operations | +| Offline root ceremony | No online root dependency | Manual issuance and renewal planning | + +Cloud KMS is optional. Local HSM, Vault transit, or an offline root must support air-gapped issuance. + +### Replica Certificate + +| Option | Pros | Cons | +| --- | --- | --- | +| Minimal signed statement | Small offline parser and explicit chain scopes | Team must specify encoding and constraints | +| X.509 | Existing tooling and enterprise PKI integration | Custom chain-scope extensions and path-validation complexity | +| Raw trusted replica keys | Matches v1 | Every verifier needs independently distributed key history | + +## Decision + +Propose an organization root held in an approved KMS, HSM, local Vault transit, or offline ceremony. +The initial proposed certificate profile uses Ed25519. +Reject incompatible backends instead of exporting root keys. +No request or offline verification requires reaching the root service. + +Choose a minimal, versioned signed certificate statement over X.509. +Propose canonical JSON, restricted integers, and a dedicated signature domain. +Gateway params canonicalization does not define certificate semantics. +The root signs the statement digest. +Bind organization ID, issuer fingerprint, subject public key, key ID, allowed purpose, and validity timestamps. +Replica certificates also bind exactly 1 replica UUID and epoch. +Log certificates instead bind a checkpoint origin and log-signing purpose. +Freeze bytes, timestamp units, and domains before implementation. +Reject duplicate or ambiguous key IDs. + +Propose a default replica key lifetime of 24 hours, configurable for offline renewal ceremonies. +Offline installations must provision renewal material or approve longer intervals. +Expiry stops new signing. Never silently extend trust or fall back to cloud services. +Keep private replica keys encrypted at rest and cached under the current lease model. +Retain certificates after private-key destruction. + +Propose `agentgate keys rotate` as a new command, not an existing CLI feature. +Stop admission, resolve inflight attempts, then close and seal the authorized epoch. +Atomically persist the next certified key and epoch before resuming. +Preserve lease-before-SQLite lock ordering. +Current `Rotate(atSeq)` uses global windows; v2 needs an epoch-aware transaction, not `Rotate(1)`. +Interrupted rotation must resume at exactly 1 committed epoch boundary. + +Offline verification needs only receipts, proofs, checkpoint, and an independently pinned root public key. +Bundles include certificates, descriptors, and signed trust-status statements. +Verify purpose, chain scope, key bytes, and applicable sequence or timestamp windows. +Do not reject historical receipts merely because their certificate is expired today. +Receipt timestamps alone do not prove signing time. +Without independently trusted time, report backdating limitations. +Offline revocation reflects bundled signed status, not unseen later revocations. +Legacy v1 exports still require their original trusted key history unless a reviewed root-signed migration descriptor binds it. + +## Consequences + +### Positive + +Root pinning replaces manual trust distribution for each new replica key. +Purpose and epoch constraints reduce cross-role key misuse. + +### Negative + +Certificate issuance and root recovery require operational procedures. +Offline installations trade short lifetimes against renewal workload. + +### Risks and Mitigations + +A compromised root can certify attacker keys. Require protected custody and an audited root replacement procedure. +Compromised replicas can backdate receipts. Document clock limits and compare independently retained checkpoints. +Manifest hardening may change verifier rejection cases. Coordinate that branch without changing frozen v1 encoding. + +## Implementation Notes + +1. Freeze certificate fixtures and an offline root-pin procedure. + Acceptance: tampered scope, purpose, validity, or public key fails verification. +2. Integrate epoch-aware activation with existing cached-key leases. + Acceptance: rotation during a batch waits; no batch uses a cleared or wrong-epoch key. +3. Crash-test each rotation boundary. + Acceptance: restart selects exactly 1 active epoch and never emits uncertified receipts. +4. Test root outages, expiration, offline renewal, and archived evidence. + Acceptance: new signing stops on expiry; historical verification works without any network. + +## Related ADRs + +- [0001: Epoch identity](0001-per-replica-chains.md) scopes replica certificates. +- [0002: Recovery](0002-two-phase-receipts.md) resolves inflight attempts before rotation. +- [0003: Segments](0003-durable-shipping.md) carries public certificate history. +- [0004: Checkpoints](0004-merkle-checkpoints.md) uses a separately certified log purpose. diff --git a/docs/design/0006-receipt-failure-modes.md b/docs/design/0006-receipt-failure-modes.md new file mode 100644 index 0000000..4d3ea9a --- /dev/null +++ b/docs/design/0006-receipt-failure-modes.md @@ -0,0 +1,114 @@ +# ADR-0006: Explicit receipt failure modes + +## Status + +Proposed + +## Date + +2026-10-18 + +## Context + +The [gateway](../../internal/gateway/gateway.go) currently returns HTTP 500 when receipt append fails. +It never returns the would-be outcome without an acknowledged receipt commit. +An upstream side effect may already exist. +The [ledger](../../internal/receipt/ledger.go) notes that cancellation after submission can still leave a committed receipt. +There is no durable fallback spool or failure-mode setting today. + +## Decision Drivers + +- Keep durable evidence as the default admission requirement. +- Offer bounded availability during specific SQLite failures. +- Never interpret fail-open as bypassing authentication or policy. +- Make configuration changes auditable even when the primary sink fails. + +## Considered Options + +| Option | Pros | Cons | +| --- | --- | --- | +| `fail_closed` only | Smallest failure surface | Local database faults stop all actions | +| Unbounded or memory-only fail-open | Highest apparent availability | Evidence loss and uncontrolled disk growth | +| Default closed, optional durable spool | Bounded fallback with explicit risk | More recovery states and capacity accounting | + +## Decision + +Propose `receipt_failure_mode: fail_closed` as the default. +An unavailable durable intent sink prevents upstream dispatch and returns HTTP 503. +Failure after dispatch also returns 503 before response headers, without claiming the external action failed. +The unresolved committed intent is recovered as `outcome_unknown`. +This proposes an API behavior change from the current 500 response. + +Propose opt-in `fail_open_spool`, meaning open relative to the primary SQLite sink only. +It still requires durable signed evidence before dispatch and response. +Use a bounded append-only spool on the replica's persistent data volume. +Fsync its intent before upstream dispatch and its outcome before returning a buffered response. +For SSE, fsync the header outcome before headers and persist the terminal artifact afterward. +The spool is not a raw request queue. Never store credentials or replay upstream requests. + +Keep 1 writer and 1 authoritative chain across SQLite and the spool. +Fallback is allowed only with an unambiguous committed head and an available certified signer. +An uncertain SQLite commit cannot be converted into a second signed entry. +Fence admission until durable reconciliation resolves that attempt. +Disk failure, signer failure, expired certificates, or unknown chain state still fail closed. +Sharing a volume means the spool does not protect against volume loss or full-disk failure. + +Propose explicit limits for bytes, record count, and oldest unshipped age. +Initial suggested limits are 256 MiB, 100,000 records, and 24 hours. +Owner approval and workload sizing are required before enabling them. +Reserve maximum bounded intent, outcome, and stream-terminal space before admitting each attempt. +Reserve control-record capacity separately. +Never evict unshipped records. +When any limit blocks reservation, reject new attempts with 503 and `Retry-After`. +Inflight work consumes its reserved space, including recovery outcomes. + +Log every configured mode change as a signed v2 control receipt. +Bind old mode, new mode, actor, reason, and configuration digest. +Also record effective sink transitions between SQLite and spool. +Commit that receipt to the current authoritative sink before activating the change. +If neither sink can durably record it, reject the change and retain the last committed configuration. +Bootstrap records the initial configuration before admission. +Do not silently downgrade modes on restart. + +On recovery, reconcile spool and SQLite positions before accepting traffic. +Replay exact signed records into SQLite with their original identities and bytes. +Identical existing entries are idempotent; different hashes at the same position halt recovery. +Do not allocate replacement sequences, re-sign, or re-dispatch upstream. +Ship spooled evidence through the same sealed-segment protocol. +Delete spool files only after replay, shipping verification, and retention requirements are satisfied. + +## Consequences + +### Positive + +Default operation keeps a clear durability boundary. +Opt-in fallback survives some primary-sink faults without inventing successful attestations. + +### Negative + +Capacity reservation and sink handoff add complexity. +503 does not mean an already-dispatched external action was rolled back. + +### Risks and Mitigations + +Ambiguous handoff can fork a chain. Require fencing and durable reconciliation before selecting another sink. +Spool exhaustion can strand outcomes. Reserve terminal capacity before dispatch and test worst-case record sizes. +Started SSE cannot change HTTP status. Close the stream on terminal persistence failure and recover unknown completion later. + +## Implementation Notes + +1. Implement the two-phase state machine before exposing fallback configuration. + Acceptance: neither mode dispatches without a durable intent. +2. Inject unambiguous database failures and uncertain commit acknowledgments separately. + Acceptance: only the former can enter the spool; the latter blocks until reconciled. +3. Fill each quota and exhaust reserved control capacity. + Acceptance: no evidence is evicted, and unrecordable mode changes are refused. +4. Restart during sink handoff, spool fsync, replay, and segment shipping. + Acceptance: exact replay produces no duplicates, forks, or upstream calls. + +## Related ADRs + +- [0001: Chains](0001-per-replica-chains.md) defines unique positions. +- [0002: Two-phase receipts](0002-two-phase-receipts.md) defines durability boundaries. +- [0003: Shipping](0003-durable-shipping.md) governs deletion and acknowledgment. +- [0005: Keys](0005-root-certified-keys.md) defines signing availability. diff --git a/docs/design/0007-kubernetes-lifecycle.md b/docs/design/0007-kubernetes-lifecycle.md new file mode 100644 index 0000000..738878f --- /dev/null +++ b/docs/design/0007-kubernetes-lifecycle.md @@ -0,0 +1,132 @@ +# ADR-0007: Kubernetes identity, drain, and WAL ownership + +## Status + +Proposed + +## Date + +2026-10-18 + +## Context + +This baseline includes [Compose](../../docker-compose.yaml) and [configuration YAML](../../configs/agentgate.yaml), not Helm or Kubernetes workload manifests. +The [health endpoint](../../internal/gateway/gateway.go) reports service count, not ledger readiness. +[HTTP shutdown](../../cmd/agentgw/main.go) currently allows 10 seconds. +Deferred closure then stops the ledger, clears signer keys, and closes SQLite. +[Ledger closure](../../internal/receipt/ledger.go) finishes an already submitted transaction but rejects pending requests. +It does not seal segments. + +The same SQLite database also stores authorization, vault, and organization state. +Receipt isolation does not replicate that state. + +## Decision Drivers + +- Give every replica exclusive, restart-stable storage. +- Stop admission before draining work. +- Distinguish storage pressure from process failure. +- Preserve runtime-independent governance. + +## Considered Options + +| Option | Pros | Cons | +| --- | --- | --- | +| StatefulSet with per-pod PVC | Stable attachment and explicit replica lifecycle | Volume fencing and zone recovery still need operations | +| Deployment with PVC per replica | Familiar rollout controller | Dynamic claim ownership needs extra identity machinery | +| Deployment with shared writable PVC | Simple manifest | Violates isolated writer and signer-cache assumptions | +| Ephemeral Deployment storage | Easy replacement | Loses committed, unshipped evidence | + +## Decision + +Propose a StatefulSet with 1 persistent data volume per replica. +Use `volumeClaimTemplates` and `ReadWriteOncePod` where supported. +Otherwise require storage-level single-writer fencing beyond `ReadWriteOnce`. +Test filesystem locking and fsync behavior. +Never share a writable SQLite database between replicas. + +Persist the provisioned replica UUID and epoch alongside SQLite. +The StatefulSet ordinal locates its PVC but is not the cryptographic replica ID. +Replacement PVCs receive new UUIDs; intact restarts resume their epoch. +Retain PVCs on scale-down. Audit explicit deletion. + +The only proposed gateway-owned writable state is its WAL data volume. +That includes the main SQLite file, WAL, SHM, encrypted keys, outbox, sealed files, and optional spool. +Existing authorization and vault rows remain on that volume. +Configuration and mounted secrets stay read-only. +Object storage and optional Tessera publication require separate durable infrastructure. +Consistent authorization and vault provisioning across replicas remains a rollout prerequisite. +Never copy live WAL files to provision replicas. + +Propose separate `/livez` and `/readyz` endpoints plus a startup probe. +Liveness checks process progress, not upstream reachability or object-store availability. +Readiness requires completed recovery, exclusive volume ownership, an authorized signer, and admission capacity. +Spool mode requires healthy reserved capacity. +Upload outages preserve readiness until local quotas block admission. +Probes must not append synthetic receipts. + +Propose an authenticated local drain control used by both `preStop` and SIGTERM. +First fail readiness and stop accepting new actions. +Allow endpoint propagation, then drain HTTP work. +Bound long-lived SSE streams and write their terminal or unknown records. +Wait for outstanding ledger append acknowledgments before calling `Ledger.Close`. +Seal the final segment and persist its outbox state. +Never wait indefinitely for upload. +Finally close the ledger, clear signer keys, and close SQLite. +Recover interrupted drain from that volume. + +Propose `terminationGracePeriodSeconds: 60` as a starting budget. +Budget 5 seconds for endpoint propagation, 40 for requests and streams, and 10 for ledger completion and sealing. +Reserve 5 seconds for closure. +This requires changing the current shutdown timeout. +Hook time counts within the grace period. + +For installations with at least 2 replicas, propose a PDB with `maxUnavailable: 1`. +Spread across hostnames; prefer zones when available. +Use rolling updates with at most 1 unavailable replica. +Document single-replica maintenance downtime. +PDBs neither prevent node failures nor replicate unshipped evidence. + +Clawdlinux's operator applies sandbox labels and egress policy independently of the agent runtime. +Runtime selection belongs to the operator's `pkg/runtime.Registry`, not gateway branches. +Keep the gVisor label injector and configure runtime classes through that integration. +No runtime adapter should reimplement sandbox or egress policy. +Air-gapped profiles require no public-cloud endpoint. + +## Consequences + +### Positive + +Replica identity follows durable ownership rather than pod scheduling. +Drain preserves local evidence without requiring remote availability. + +### Negative + +Stateful storage complicates rescheduling and disaster recovery. +Replica authorization provisioning remains separate work. + +### Risks and Mitigations + +A split-brain volume attachment can fork evidence. Require fencing and test forced node recovery. +Slow streams can exhaust shutdown time. Enforce drain deadlines and recover missing terminal records. +Topology spread can leave pods unschedulable. Use installation-specific constraints and document degraded availability. + +## Implementation Notes + +1. Add recovery-aware probes and idempotent drain control before publishing manifests. + Acceptance: unhealthy storage removes readiness without triggering a restart loop. +2. Test a 2-replica rolling update with continuous traffic and SSE. + Acceptance: no new attempts enter a draining pod; committed receipts survive rescheduling. +3. Kill a pod during drain and force a node replacement. + Acceptance: outbox recovery resumes, and no two writers acquire the same replica identity. +4. Disconnect external networking and the optional upload destination. + Acceptance: local governance and offline verification work; capacity exhaustion fails closed. +5. Validate equivalent sandbox labels and egress policy across supported operator runtimes. + Acceptance: changing runtime selection does not change receipt or sealing guarantees. + +## Related ADRs + +- [0001: Identity](0001-per-replica-chains.md) governs volume loss and epochs. +- [0002: Lifecycle](0002-two-phase-receipts.md) supplies drain recovery records. +- [0003: Segments](0003-durable-shipping.md) defines sealing and retention. +- [0005: Rotation](0005-root-certified-keys.md) defines key expiry and drain interactions. +- [0006: Failure modes](0006-receipt-failure-modes.md) determines readiness capacity. diff --git a/docs/design/README.md b/docs/design/README.md new file mode 100644 index 0000000..e1eb73c --- /dev/null +++ b/docs/design/README.md @@ -0,0 +1,48 @@ +# AgentGate Phase 2 decisions + +Status: Proposed +Date: 2026-10-18 + +AgentGate is part of Clawdlinux, an air-gapped agent governance and attestation plane. +These documents propose Phase 2 designs. They do not describe shipped features. +Air-gapped operation and offline verification are required. No proposal requires a public cloud. + +The code baseline is `origin/main` at `d740dc6428865d0ca6e4998d769159f461a2fdfb`. +The [v1 wire contract](../spec/RECEIPT-v1.md) remains frozen. +The [v2 outline](../spec/RECEIPT-v2.md) is a proposal, not an interoperable specification. + +## Index + +| ADR | Proposed decision | Status | Date | +| --- | --- | --- | --- | +| [0000](0000-phase2-sequencing.md) | Dependency order and independent work | Proposed | 2026-10-18 | +| [0001](0001-per-replica-chains.md) | Separate replica epochs with authenticated v2 chain IDs | Proposed | 2026-10-18 | +| [0002](0002-two-phase-receipts.md) | Commit intent before dispatch and outcome before response | Proposed | 2026-10-18 | +| [0003](0003-durable-shipping.md) | Ship signed segments from replica-local SQLite | Proposed | 2026-10-18 | +| [0004](0004-merkle-checkpoints.md) | Evaluate Tessera with C2SP tiles and signed checkpoints | Proposed | 2026-10-18 | +| [0005](0005-root-certified-keys.md) | Certify short-lived replica keys with an organization root | Proposed | 2026-10-18 | +| [0006](0006-receipt-failure-modes.md) | Default closed; permit bounded durable spooling by opt-in | Proposed | 2026-10-18 | +| [0007](0007-kubernetes-lifecycle.md) | Use StatefulSet identities and isolated WAL volumes | Proposed | 2026-10-18 | + +## Current implementation + +- [Receipt protocol](../../pkg/receiptspec/receipt.go) and [encoding](../../pkg/receiptspec/encoding.go) support v1 only. +- [Ledger](../../internal/receipt/ledger.go) uses a single writer goroutine per ledger. +- Production creates 1 ledger per process. Each batch uses `BEGIN IMMEDIATE` and replies after `COMMIT`. +- [SQLite configuration](../../internal/db/sqlite.go) requests WAL mode. +- [Signer store](../../internal/signer/store.go) caches private keys and excludes rotation during batch leases. +- [Gateway](../../internal/gateway/gateway.go) commits after upstream execution, before returning its response. +- [SSE forwarding](../../internal/gateway/stream.go) starts after the header-time receipt commits. +- [Offline verifier](../../cmd/agentgate-verify/main.go) reads SQLite or JSONL. It does not verify Merkle proofs. +- [Compose](../../docker-compose.yaml) mounts local data. This baseline has no Helm chart or Kubernetes workload manifests. + +The [Phase 0 benchmark PR](https://github.com/Clawdlinux/agentgate/pull/50) contains results absent from this baseline. +Its reported latency gate is not met. +Phase 2 must not claim that extra durable writes meet that gate. + +## Review boundaries + +Every decision remains proposed until owner review. +Implementation notes describe future acceptance tests, not completed validation. +Each ADR records tradeoffs and links its dependencies. +The sequencing document lists remaining owner decisions. diff --git a/docs/spec/RECEIPT-v2.md b/docs/spec/RECEIPT-v2.md new file mode 100644 index 0000000..22c1cf3 --- /dev/null +++ b/docs/spec/RECEIPT-v2.md @@ -0,0 +1,134 @@ +# AgentGate receipt v2: proposed outline + +Status: Proposed. Not implemented. Not a frozen wire contract. +Date: 2026-10-18 + +This outline collects Phase 2 protocol decisions for review. +It is not sufficient for interoperable producers. +No production code in the baseline accepts v2. +The [v1 contract](RECEIPT-v1.md) remains frozen and independently verifiable. + +## 1. Version Selection and Compatibility + +Propose JSONL `format_version = 2` for v2 receipt records. +Use a new hash domain, proposed as `agentgate.receipt.hash.v2` followed by NUL. +Version dispatch must happen before selecting field validation or a hash preimage. +Never reinterpret v1 bytes under v2 rules. +Unknown versions must fail explicitly. + +A future verifier accepts both versions using separate validation and encoding paths. +Existing v1-only binaries still reject v2. +Preserve v1 preimages, public test vectors, signatures, and accepted verification semantics. +Do not update v1 encoding locks to accommodate v2. +Never re-encode historical v1 receipts during migration or export. + +Retain original v1 JSONL exports and their independently trusted key history. +A proposed root-signed migration descriptor may bind their exact payload digest and terminal head. +It does not rewrite old receipts or manufacture authenticated v1 chain IDs. +V2 and legacy manifests require separate explicit version dispatch. + +## 2. Chain Identity + +Propose required authenticated `chain_id = :`. +`replica-id` is a lowercase provisioned UUID. +`epoch` is positive decimal without leading zeros. +Persist both; never derive cryptographic identity solely from a reusable Kubernetes ordinal. +Each chain starts at sequence 1 with a zero previous hash. +Within that chain, sequence and previous-hash checks remain mandatory. +No global ordering between replica chains is implied. + +Proposed epoch-open records reference the preceding terminal hash and segment digest when available. +Key rotation creates a new epoch. +An intact restart resumes its current epoch. +Lost or restored storage requires fencing and a new replica UUID. + +## 3. Proposed Record Kinds + +All records authenticate their kind, chain identity, sequence, previous hash, timestamp, and signing key. +Required action fields and optional result fields must be specified per kind. +Do not copy v1's required HTTP status into an intent with a fabricated value. + +| Kind | Proposed authenticated content | +| --- | --- | +| `intent` | Attempt ID, principal, agent, delegation, destination action, params digest, policy | +| `outcome` | Full intent reference, observation class, known status, latency, bounded error code | +| `stream_result` | Header-outcome reference, available byte digest and count, termination classification | +| `decision` | Authenticated denied or rate-limited attempt without upstream dispatch | +| `control` | Typed mode changes or epoch boundaries with actor, reason, and bound configuration | + +An intent reference includes chain ID, sequence, entry hash, and attempt ID. +Every attempt ID is gateway-generated and unique within its replica identity. +Specify its exact size and encoding before implementation. +Unknown outcomes use `outcome_unknown`, without invented provider status or successful completion. +Propose a distinct observation field rather than extending v1's frozen policy enum. + +SSE outcomes attest header acceptance before headers are forwarded. +Supplemental terminal records bind SHA-256 of bytes accepted by downstream writes. +They do not prove client delivery. +Recovery emits unknown outcomes or unknown stream termination without replaying upstream calls. +References may cross epoch boundaries. +Duplicate or contradictory terminal records must be rejected. + +The [earlier stream proposal](proposals/receipt-stream-digest.md) remains unimplemented. +This outline proposes replacing its single-action-receipt rule for v2 lifecycle records. + +## 4. Hashing and Signature Work Remaining + +Propose SHA-256 entry hashes and Ed25519 receipt signatures over entry hashes. +Authenticate version and kind under the new domain. +Include chain identity and lifecycle references in the signed preimage. +Keep gateway params canonicalization unchanged unless separately versioned and reviewed. +Do not treat the narrower decision-record canonicalizer as interchangeable with gateway params. + +Before acceptance, specify exact field order, integer widths, endianness, string prefixes, and absence markers. +Also freeze string bounds, enum values, timestamp rules, attempt-ID grammar, and cross-field validation. +Assign separate domains to certificates, segment descriptors, inventories, and migration descriptors. +Define strict duplicate-field and unknown-field handling for new signed containers. +No placeholder encoding in this outline authorizes production emission. + +## 5. Keys and Export Artifacts + +Propose root-certified replica keys scoped to exactly 1 replica UUID and epoch. +Certificates bind purpose, public key, and validity under a versioned minimal signed statement. +A separately scoped log certificate authorizes checkpoint signatures for a specific origin. +Preserve v1 legacy trust behavior outside an explicit migration envelope. + +Proposed sealed segments contain original receipt bytes and signed range descriptors. +Descriptors bind payload digest, chain scope, anchor, terminal hash, predecessor segment, and certificate digests. +Specify export-container versions independently of receipt versions. +Offline bundles include required certificates and signed trust-status information. +Their trust anchor is an independently pinned organization root, never an embedded self-asserted root. + +## 6. Transparency Proof Profile + +Propose C2SP [tlog-checkpoint](https://c2sp.org/tlog-checkpoint@v1.1.0) +and [tlog-tiles](https://c2sp.org/tlog-tiles@v1.0.0) with SHA-256. +Freeze leaf bytes binding receipt version, authenticated chain scope, sequence, and entry hash. +Legacy scope comes from its signed descriptor, not modified v1 receipts. +Specify inclusion bundles and consistency proofs with independent fixtures. +Use standard signed-note checkpoint signatures. +Export checkpoint files, tiles, inventories, certificates, and receipts for disconnected verification. +Witness quorum remains optional. + +## 7. Acceptance Gates and Open Questions + +1. Approve a complete byte-level specification and deterministic vectors for each record and artifact kind. +2. Preserve all v1 vector outcomes and byte locks under dual-version verification. +3. Reject mismatched chain identity, duplicate terminal references, conflicting segments, and unauthorized certificate purposes. +4. Run crash recovery across intent, outcome, spool, sealing, and epoch rotation boundaries. +5. Verify inclusion and consistency proofs with no network or private keys. +6. Document clock limitations, revocation freshness, retention, and owner-approved signing lifetimes. + +Open questions include certificate renewal policy, accepted publication lag, and archival retention. +Proof freshness requires a separately trusted checkpoint or time policy. +No artifact alone proves that every external side effect was recorded. + +## Related ADRs + +- [0001: Per-replica chains](../design/0001-per-replica-chains.md) +- [0002: Intent and outcome](../design/0002-two-phase-receipts.md) +- [0003: Sealed segments](../design/0003-durable-shipping.md) +- [0004: Merkle checkpoints](../design/0004-merkle-checkpoints.md) +- [0005: Root-certified keys](../design/0005-root-certified-keys.md) +- [0006: Failure modes](../design/0006-receipt-failure-modes.md) +- [0007: Kubernetes lifecycle](../design/0007-kubernetes-lifecycle.md) From b47ed84895033678a8e01490a4a6b892e70c7228 Mon Sep 17 00:00:00 2001 From: Shreyansh Sancheti <43677304+shreyanshjain7174@users.noreply.github.com> Date: Sun, 18 Oct 2026 06:38:46 +0530 Subject: [PATCH 2/3] docs(design): revise phase 2 trust boundaries Bind proposed chains to fenced ownership and signed registry history. Defer spooling until a covered failure class is approved. Signed-off-by: Shreyansh Sancheti <43677304+shreyanshjain7174@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/design/0000-phase2-sequencing.md | 29 ++-- docs/design/0001-per-replica-chains.md | 112 +++++++++----- docs/design/0002-two-phase-receipts.md | 63 ++++---- docs/design/0003-durable-shipping.md | 130 +++++++++------- docs/design/0004-merkle-checkpoints.md | 153 ++++++++++-------- docs/design/0005-root-certified-keys.md | 180 +++++++++++++--------- docs/design/0006-receipt-failure-modes.md | 130 ++++++++-------- docs/design/0007-kubernetes-lifecycle.md | 121 ++++++++------- docs/design/README.md | 36 ++++- docs/spec/RECEIPT-v2.md | 88 +++++++++-- 10 files changed, 632 insertions(+), 410 deletions(-) diff --git a/docs/design/0000-phase2-sequencing.md b/docs/design/0000-phase2-sequencing.md index 616d8cd..cf31aee 100644 --- a/docs/design/0000-phase2-sequencing.md +++ b/docs/design/0000-phase2-sequencing.md @@ -36,22 +36,26 @@ Propose this dependency order: | --- | --- | --- | --- | | 1 | [0001](0001-per-replica-chains.md) | None | Agree v2 chain identity and unchanged v1 fixtures | | 2 | [0005](0005-root-certified-keys.md) | 0001 | Agree certificate scopes and epoch rotation fixtures | -| 3a | [0002](0002-two-phase-receipts.md) | 0001, 0005 | Verify durable intent ordering and crash recovery | +| 2a | [0002](0002-two-phase-receipts.md) | 0001 | Verify durable intent ordering and crash recovery with pinned raw keys | | 3b | [0003](0003-durable-shipping.md) | 0001, 0005 | Recover and verify sealed segments offline | -| 4a | [0004](0004-merkle-checkpoints.md) | 0003, 0005 | Verify segment receipts against offline checkpoints | -| 4b | [0006](0006-receipt-failure-modes.md) | 0002, 0003 | Prove bounded spool admission and exact replay | -| 5 | [0007](0007-kubernetes-lifecycle.md) | 0001, 0002, 0003, 0005, 0006 | Pass restart, drain, volume fencing, and outage tests | +| 4a | [0004](0004-merkle-checkpoints.md) | 0003, 0005 | Verify registry-backed head vectors; gate later Merkle proofs | +| 3c | [0006](0006-receipt-failure-modes.md) | 0001, 0002 | Verify 503 responses and authenticated control receipts | +| 5 | [0007](0007-kubernetes-lifecycle.md) | 0001, 0002, 0003, 0005 | Pass enrolment, drain, volume fencing, and outage tests | 0004 is not a prerequisite for Kubernetes ledger availability. -Kubernetes proof-publication tests require 0004 when that optional publisher is enabled. -0007 follows 0005 transitively through its other prerequisites. +Coverage-publication tests require 0004, independently of gateway readiness. +0002 does not require 0005. Pinned raw keys are sufficient for its v2 lifecycle tests. +0007 needs 0006's spool protocol only if the owner later enables the deferred spool. +That deferred protocol also depends on 0003 for direct segment shipping. +Multi-replica release remains blocked until authorization and vault provisioning are solved. ## Consequences ### Positive -0002 and 0003 can be implemented independently after shared contracts settle. -0004 and 0006 can then proceed independently. +0002 and 0005 can proceed independently after the chain contract settles. +0003 and the fail-closed portion of 0006 can then proceed independently. +0004 publication does not block 0006. Tessera POSIX feasibility tests, benchmark reruns, and Kubernetes manifest drafts can start earlier. V1 golden-vector regression work is independent of every new protocol. @@ -72,14 +76,9 @@ Additional commits could worsen latency. Re-run Phase 0 workloads before accepti 3. Require each ADR's acceptance tests before enabling its feature. 4. Run integrated crash tests and disconnected verification before declaring Phase 2 usable. -### Owner Questions +### Owner Decisions -- Who approves v2 fields and the v1-to-v2 migration envelope? -- What latency budget applies after the Phase 0 gate failed? -- What certificate lifetime and offline renewal interval fit each installation? -- What retention period, spool quota, and accepted unshipped-loss window apply? -- How will replicas provision consistent authorization and vault data currently stored in SQLite? -- Who operates the optional log publisher, and does any installation require witness quorum? +The [open decisions](README.md#open-decisions) list recommended defaults and remaining approval gates. ## Related ADRs diff --git a/docs/design/0001-per-replica-chains.md b/docs/design/0001-per-replica-chains.md index 4ef5c8c..0e15d82 100644 --- a/docs/design/0001-per-replica-chains.md +++ b/docs/design/0001-per-replica-chains.md @@ -10,20 +10,18 @@ Proposed ## Context -The [ledger](../../internal/receipt/ledger.go) reads a single committed head inside each `BEGIN IMMEDIATE` batch. -Its primary key is currently [global `seq`](../../internal/db/migrations/002_receipts.sql) within 1 database. -The [signer store](../../internal/signer/store.go) explicitly supports 1 gateway writer per database. -Sharing a live database between replicas would violate that signing model. +The [ledger](../../internal/receipt/ledger.go) reads 1 head per `BEGIN IMMEDIATE` batch. +[Sequence](../../internal/db/migrations/002_receipts.sql) is global within its database. +The [signer](../../internal/signer/store.go) supports only 1 gateway writer per database. [V1](../spec/RECEIPT-v1.md) authenticates neither a replica identifier nor an epoch. -Adding unsigned JSON metadata cannot turn either into authenticated identity. +Unsigned metadata cannot add authenticated identity. ## Decision Drivers -- Scale replicas without cross-replica SQLite locking. -- Make sequence reuse distinguishable after rotation or replacement. -- Preserve all v1 bytes and verification semantics. -- Support disconnected verification without a global sequencer. +- Avoid cross-replica SQLite locks. +- Distinguish rotation and replacement. +- Preserve v1 bytes and offline verification. ## Considered Options @@ -36,31 +34,59 @@ Adding unsigned JSON metadata cannot turn either into authenticated identity. ## Decision Propose a required authenticated v2 field: `chain_id = :`. -The [v2 outline](../spec/RECEIPT-v2.md) proposes its encoding boundaries. +The [v2 outline](../spec/RECEIPT-v2.md) records encoding boundaries. `replica-id` is a provisioned UUID, not a reusable pod ordinal. -`epoch` is a positive, monotonically increasing integer persisted with the replica identity. -Each epoch starts at `seq = 1` with a zero previous hash. +Persist a positive, monotonically increasing `epoch` with replica identity. +The signed epoch-open control record is exactly `seq = 1`, with a zero previous hash. The uniqueness key becomes `(chain_id, seq)`. -A healthy restart with intact storage resumes its existing epoch. -Key rotation creates a new epoch. -Lost storage requires a new replica UUID, even when Kubernetes reuses an ordinal. -An exclusive volume fence must prevent concurrent writers and restored-volume clones from reusing a live identity. -No proposal establishes a total execution order between replicas. - -Propose signed epoch-open and epoch-close control records. -The opening record binds the preceding epoch's terminal hash and closing segment digest when available. -A missing predecessor is explicit, not presented as continuity. -Restoring an old backup requires fencing and a new replica UUID. - -Propose a version-aware verifier that accepts v1 and v2. -Its v1 path retains the frozen preimages, signatures, and accepted semantics. +Intact restarts resume their epoch. Rotation starts another. +Lost storage requires a new UUID, even for reused ordinals. +There is no total cross-replica execution order. + +### Fencing and Replica Authority + +Propose an external epoch lease, checked before admission and every writer batch. +Use a Kubernetes Lease by default. Non-Kubernetes installations may use conditional object updates on an epoch-open lease. +The authoritative registry allocates monotonically increasing fencing tokens through compare-and-swap. +Lease acquisition requires the current workload identity, not merely a copied receipt certificate. +Include the token in epoch-open, subsequent records, and segment descriptors. +Lease loss or an uncertain renewal stops admission and further signing. +Registry handoffs bind the recovered head and token cutover boundary after storage fencing. +Publishers reject stale writes beyond that boundary, not legitimate historical receipts from the earlier token. +Already-dispatched upstream calls cannot be recalled by a lease. + +The intermediate issuer refuses to certify the same `(replica UUID, epoch)` twice. +That rule alone cannot stop a clone reusing an existing certificate and private key. +The external lease and storage fencing are therefore required, not optional certificate features. +Kubernetes Lease ownership alone does not physically fence a partitioned volume writer. +Before takeover, establish storage fencing or refuse failover. + +Propose an authoritative replica registry backed by an append-only, signed issued-certificate log. +Store it outside gateway PVCs, so replica loss cannot erase registration. +The intermediate signs registrations, certificate digests, fencing generations, replacements, and loss declarations. +Each entry binds organization, installation, replica UUID, epoch, and deployment slot. +Inventories bind the registry revision and digest; offline bundles include its signed snapshot and required log entries. +A registered chain with neither epoch-close nor inventory coverage is reported `lost`, even when no receipt survives. +This reports missing evidence, not proof that the replica never executed work. +Unclosed chains with partial coverage report the last authenticated head and an unknown suffix. +Missing registry history prevents a complete installation-level coverage claim. +Loss detection is relative to the supplied registry revision. +Without independently trusted freshness, unseen registrations cannot be ruled out. + +Epoch-open references the predecessor's terminal hash and final segment digest when available. +Epoch-close is the final record and fixes the epoch's terminal sequence. +A missing predecessor is explicit. Restoring an old backup requires enrolment with a new UUID. + +Propose dual-version verification with frozen v1 preimages, signatures, and accepted semantics. V1 receipts are never re-encoded during migration or shipping. -Keep their original JSONL bytes and existing verification vectors. +Keep original JSONL bytes and vectors. Existing v1-only binaries will still reject v2. Migration closes the v1 ledger at a durable head. -A separately signed migration descriptor binds that head, its original export digest, and the new v2 chain. +A migration descriptor binds head, original exports, legacy keys, and new v2 chain. +The organization root and final v1 signing key must co-sign the same descriptor preimage. +Neither intermediate signatures nor embedded keys replace those approvals. That descriptor does not add fields to historical receipts. The legacy export retains its original trust requirements. @@ -68,19 +94,19 @@ The legacy export retains its original trust requirements. ### Positive -Each process retains a local group-commit writer. -Epoch identity separates key lifetimes and recovery histories. -Offline tooling can report missing epochs without querying a coordinator. +Local group commit remains. +Epochs separate signing histories. +Offline tooling finds missing registered chains, not just surviving files. ### Negative -Storage migrations must separate epoch metadata from the old global sequence model. -Chain aggregation and inventory become explicit operational work. +Storage must separate epoch heads. +Leases and registry add local dependencies. ### Risks and Mitigations -Replica cloning can create forks. Require exclusive ownership and reject duplicate `(chain_id, seq)` values with different hashes. -An unpublished suffix can disappear. Compare signed inventories and checkpoints, not chain links alone. +Replica cloning can create forks. Require external fencing and reject duplicate positions with different hashes. +An unpublished chain can disappear. Compare the issued-certificate log with signed inventory coverage. Epoch resets could hide missing history. Require explicit predecessor references and report unverifiable gaps. ## Implementation Notes @@ -90,14 +116,16 @@ Epoch resets could hide missing history. Require explicit predecessor references Acceptance: all existing v1 vectors retain their results. 3. Introduce durable replica identity, epoch allocation, and separate chain heads. Acceptance: concurrent replicas can both emit sequence 1 without collision. -4. Fence migration and rotation around writer admission. - Acceptance: restart resumes the same head; rotation starts exactly 1 new epoch. -5. Exercise lost volumes, cloned backups, and interrupted migration. - Acceptance: no chain identity is silently reused, and original v1 exports remain byte-identical. +4. Test concurrent lease acquisition, stale tokens, renewal loss, and copied certificates. + Acceptance: only the fenced owner admits work; duplicate certificate issuance is refused. +5. Remove every segment of a registered chain. + Acceptance: offline coverage reports it lost, rather than silently omitting it. +6. Interrupt migration and verify both co-signatures. + Acceptance: either missing signature fails; v1 bytes remain identical; epoch-open occupies sequence 1. ## Related ADRs -- [0002: Lifecycle records](0002-two-phase-receipts.md) references intents by chain and hash. -- [0003: Segments](0003-durable-shipping.md) preserves epoch boundaries. -- [0005: Keys](0005-root-certified-keys.md) certifies epoch-scoped signing keys. -- [0007: Kubernetes](0007-kubernetes-lifecycle.md) provides persistent identity and fencing. +- [0002: Lifecycle references](0002-two-phase-receipts.md). +- [0003: Segment boundaries](0003-durable-shipping.md). +- [0005: Issuance](0005-root-certified-keys.md). +- [0007: Identity and fencing](0007-kubernetes-lifecycle.md). diff --git a/docs/design/0002-two-phase-receipts.md b/docs/design/0002-two-phase-receipts.md index 074a6d5..012398a 100644 --- a/docs/design/0002-two-phase-receipts.md +++ b/docs/design/0002-two-phase-receipts.md @@ -10,21 +10,18 @@ Proposed ## Context -The [gateway](../../internal/gateway/gateway.go) executes upstream work before appending its v1 receipt. -It returns the response only after that append commits. -The [ledger](../../internal/receipt/ledger.go) documents the resulting external-side-effect crash gap. -An upstream action can succeed without a later committed receipt. +The [gateway](../../internal/gateway/gateway.go) executes upstream work, commits its v1 receipt, then responds. +The [ledger](../../internal/receipt/ledger.go) documents the crash gap: an action can succeed before its receipt commits. [SSE](../../internal/gateway/stream.go) starts only after a header-time receipt commits. -Final stream digests currently reach logs, not signed receipts. -The [stream-digest proposal](../spec/proposals/receipt-stream-digest.md) is unimplemented. +Digests reach logs, not receipts. The [stream-digest proposal](../spec/proposals/receipt-stream-digest.md) is unimplemented. ## Decision Drivers - Record a durable obligation before dispatch. - Report uncertainty without replaying external side effects. -- Preserve the receipt-before-response boundary, including SSE headers. -- Measure the extra durable write rather than claiming free batching. +- Preserve receipt-before-response, including SSE headers. +- Measure added commit latency. ## Considered Options @@ -36,27 +33,30 @@ The [stream-digest proposal](../spec/proposals/receipt-stream-digest.md) is unim ## Decision -Propose v2 lifecycle records, not new meanings for v1 fields. -After authentication, shape validation, and policy evaluation, allocate a gateway-generated attempt ID. +Propose v2 lifecycle records without changing v1. +After authentication, validation, and policy evaluation, allocate an attempt ID. Persist a signed `intent` before any upstream call, including credential refresh. -The intent binds identity, delegation, params digest, policy, and destination action. +Bind identity, delegation, params digest, policy, action, and any validated client request ID. Denied attempts get a terminal decision record without an upstream intent. -An acknowledged intent permits 1 logical execution under the existing safe retry policy. -An append timeout can still leave a committed record. -Resolve that attempt's durable state before dispatch; ambiguity must not trigger an upstream retry. -Provider retries and idempotency remain separate policies. Receipts do not guarantee exactly-once external execution. +An acknowledged intent permits 1 logical execution under existing safe retries. +Append timeout can still leave a committed record. Resolve ambiguity before dispatch, never by retrying upstream. Append a signed `outcome` after observing the upstream result. It references the intent's `chain_id`, sequence, entry hash, and attempt ID. +Include `upstream_attempt_count`, counting actual action dispatches, including transport retries. +Exclude credential refresh calls. +Use `observation = not_dispatched` only when the running process knows no action dispatch occurred. +That observation requires a known zero count, not an invented HTTP status. +Crash recovery still writes `outcome_unknown`, with unknown attempt count unless durable evidence establishes it. For buffered responses, commit the outcome before returning status or body. -Persist outcomes with a bounded context detached from client cancellation, matching the current gateway boundary. +Use a bounded commit context detached from client cancellation, as today. For SSE, commit an `outcome` with `observation = headers_accepted` before forwarding headers. That outcome does not claim stream completion. Append a supplemental `stream_result` at termination. -It binds the outcome hash, written-byte SHA-256, byte count, and termination reason. -The digest covers bytes accepted by downstream writes, not proven client delivery. +Bind outcome hash, written-byte SHA-256, byte count, and termination reason. +The digest covers downstream writes, not proven client delivery. A missing terminal record means unknown completion, never success. Keep terminal artifacts out of forwarded SSE bytes. This proposed lifecycle replaces the earlier stream proposal's single-action-receipt rule for v2 only. @@ -66,6 +66,15 @@ Append recovery outcomes marked `outcome_unknown`. Never call upstream during re For header outcomes without terminal stream records, append `stream_result` with unknown termination and no fabricated digest. Fenced recovery and unique references must prevent duplicate terminal records. Recovery records can reference an earlier epoch. +Rotation does not drain inflight requests; their outcomes and stream results can enter the next epoch. + +### Client Retry Linkage + +Validate bounded `client_request_id` syntax; bind it to the principal and destination in the intent. +Defer client retry deduplication and mapping an attempt ID to upstream `Idempotency-Key`. +Providers differ in key support and retention. A new gateway attempt can repeat an external side effect. +Claim no exactly-once behavior; add no unsupported provider header. +The [owner defaults](README.md#open-decisions) disable automatic replay. ### Group-Commit Cost @@ -86,21 +95,18 @@ SSE adds a later terminal record as well. The PR used Apple M4, `GOMAXPROCS=4`, and 20,000 requests per row. It did not meet the under-1-ms p99 gate. -Those are end-to-end overheads, not isolated fsync measurements. -Doubling them would not predict two-phase p99. +These are end-to-end overheads, not fsync measurements. Doubling them cannot predict two-phase p99. Budget 1 extra serialized commit wait before dispatch, then benchmark it on Linux with dedicated 4-vCPU capacity. ## Consequences ### Positive -Crashes leave inspectable obligations. -Offline verification can distinguish observed results from unknown outcomes. +Crashes leave inspectable obligations and explicit unknown outcomes. ### Negative -Write volume and request latency increase. -Recovery requires lifecycle indexes beyond a linear hash chain. +Writes and latency increase. Recovery needs lifecycle indexes. ### Risks and Mitigations @@ -113,6 +119,7 @@ SSE can fail after response headers. End the stream and persist uncertainty when Acceptance: reject missing, duplicate, or mismatched intent references. 2. Add intent ordering instrumentation. Acceptance: the mock upstream sees no call before intent commit acknowledgment. + Acceptance: runtime cancellation before dispatch yields `not_dispatched` and zero attempts; retry counts include every action dispatch. 3. Run subprocess `kill -9` tests before intent commit, after commit, during dispatch, and after provider success. Also kill between the upstream call and outcome commit, after outcome commit, and during SSE. Acceptance: recovery is idempotent, records unknown states, and never re-dispatches. @@ -121,7 +128,7 @@ SSE can fail after response headers. End the stream and persist uncertainty when ## Related ADRs -- [0001: Chains](0001-per-replica-chains.md) supplies stable references. -- [0003: Shipping](0003-durable-shipping.md) retains recovery obligations. -- [0006: Failure modes](0006-receipt-failure-modes.md) defines alternate durable recording. -- [0007: Drain](0007-kubernetes-lifecycle.md) bounds unfinished streams. +- [0001: References](0001-per-replica-chains.md). +- [0003: Shipping](0003-durable-shipping.md). +- [0006: Closed failure mode](0006-receipt-failure-modes.md). +- [0007: Stream drain](0007-kubernetes-lifecycle.md). diff --git a/docs/design/0003-durable-shipping.md b/docs/design/0003-durable-shipping.md index c26e56f..c94b74a 100644 --- a/docs/design/0003-durable-shipping.md +++ b/docs/design/0003-durable-shipping.md @@ -10,98 +10,120 @@ Proposed ## Context -The [ledger](../../internal/receipt/ledger.go) already serializes signed batches into SQLite. -[WAL configuration](../../internal/db/sqlite.go) provides local storage, not off-node replication. -Current [exports](../../internal/receipt/export.go) provide signed snapshot ranges, not durable shipping. -The [schema](../../internal/db/migrations/002_receipts.sql) rejects receipt updates and deletes. - -AgentGate belongs to Clawdlinux's air-gapped governance and attestation plane. -Postgres is PARKED for Phase 2. -No storage design may require public-cloud connectivity. +The [ledger](../../internal/receipt/ledger.go) commits signed SQLite batches. +[WAL](../../internal/db/sqlite.go) is local storage, not replication. +[Exports](../../internal/receipt/export.go) provide signed snapshots, not shipping. +The [schema](../../internal/db/migrations/002_receipts.sql) prohibits receipt updates and deletes. +Clawdlinux requires air-gapped operation. Postgres is PARKED. ## Decision Drivers -- Preserve local commits during destination outages. -- Ship independently verifiable artifacts without exporting raw live WAL files. -- Bound disk growth without silently deleting unshipped evidence. -- Support disconnected installations. +- Continue local commits during destination outages. +- Ship verifiable artifacts, not live WAL files. +- Bound growth without deleting pending evidence. ## Considered Options | Option | Pros | Cons | | --- | --- | --- | -| Replica SQLite plus signed segments | Fits current writer; S3 or in-cluster MinIO; disconnected export | Asynchronous loss window; custom sealing and replay | -| Postgres transactional outbox | Shared SQL transactions and mature replication | PARKED; requires database operations and network availability | -| Kafka-like durable log | Replay and partitioned throughput | Additional quorum, retention, and partition ownership operations | +| SQLite plus signed segments | Existing writer; local MinIO or filesystem | Async loss window; sealing implementation | +| Postgres outbox | Transactions and replication | PARKED; database and network operations | +| Kafka-like log | Replay and throughput | Quorum and partition operations | -Postgres and Kafka-like systems can run offline. Cloud dependence is not the reason to reject them. -They add dependencies before the artifact protocol settles. +Postgres and Kafka can run offline. Their extra operations, not cloud dependence, argue against initial adoption. ## Decision Propose replica-local SQLite WAL plus a durable shipping outbox. -Ship signed immutable segments to configurable S3-compatible storage. -Support in-cluster MinIO and filesystem export for fully disconnected installations. -Postgres remains PARKED. -Uploads stay outside the request commit path. - -Seal at the proposed initial limits of 4,096 records, 16 MiB, or 10 seconds, whichever comes first. -Also seal on epoch closure and graceful drain. -Each segment covers a contiguous, nonempty range from exactly 1 chain. +Ship immutable segments to configurable S3, in-cluster MinIO, or filesystem export. +Uploads never block request commits. + +Seal after 4,096 records, 16 MiB, or 10 seconds, whichever comes first. +Also seal on epoch closure and drain. +Each segment covers 1 contiguous, nonempty chain range. Store original receipt bytes. Never re-encode v1 receipts. -For existing SQLite history, capture its initial v1 export with the current exporter and preserve that artifact unchanged. -Do not ship a live SQLite database, WAL, or SHM file as a segment. +Capture existing SQLite history through the current v1 exporter, then preserve those bytes. +Never ship live SQLite, WAL, or SHM files. -A versioned signed descriptor binds chain identity, range, count, anchor, terminal hash, and previous segment digest. +A versioned descriptor binds chain, fencing token, range, count, anchor, terminal hash, and predecessor digest. Also bind payload SHA-256 and certificate or legacy keyset digests. -Use a separate signing domain and strict range validation, not an undocumented extension of v1 manifests. -Bundle certificates, but independently pin the trusted root. +Use separate signing domains and strict range validation. Bundle certificates; independently pin the root. + +### Artifact Signers + +Use [0005's purposes](0005-root-certified-keys.md). +Epoch keys sign descriptors with `segment_descriptor` and control records with `control`. +Destroy an epoch key only after its final segment descriptor and directory are durable. + +The intermediate-certified singleton publisher signs cross-replica inventories and head vectors with `inventory`. +Replica keys cannot assert cross-replica coverage. +The intermediate signs the registry with `replica_registry`. +Inventories bind registry revision/digest, ranges, close records, and final descriptors. -The ledger writer reserves a segment range and outbox ID transactionally. -A read snapshot supplies immutable rows for deterministic materialization. +The offline root and final v1 key co-sign migration descriptors using the same domain-separated preimage. +Bind original exports, final v1 head, legacy key history, and destination v2 identity. +A dedicated intermediate-certified archival publisher signs legacy descriptors with `legacy_segment`. +It references that migration envelope without re-signing or re-encoding v1 receipts. +Without migration, legacy verification still requires independently trusted v1 keys. + +### Durable Materialization and Upload + +Reserve range and outbox ID in a ledger-writer transaction. +Materialize immutable rows from a snapshot. Write and fsync payload and descriptor, then atomically rename and fsync their directory. Mark the outbox item sealed only after those files are durable. -An interrupted reservation is recreated from retained rows after restart. -Uploads never hold the SQLite write transaction. +Recreate interrupted reservations from retained rows. Uploads hold no SQLite write transaction. Use object names containing chain ID, range, and payload digest. -Retry the same name and bytes after uncertain acknowledgments. +Require conditional creation such as `If-None-Match: *`; object lock may add immutable retention. +Object lock alone must prevent replacement or bind references to immutable versions. +For filesystem export, create staging files with `O_CREAT|O_EXCL`, fsync, then publish through atomic no-replace linking. +Fsync the destination directory before publishing inventory references. +Retry identical names and bytes after uncertain acknowledgments; compare existing objects. Verify the stored digest independently of multipart ETags. -A pre-existing different object fails integrity checks. +Different existing bytes fail integrity. Upload payload before descriptor; publish the inventory reference last. Only the final reference advertises completeness. Persist acknowledgments and inventory progress locally. +### Retention and Offline Consumption + Retain unuploaded segments and their source rows regardless of age. -Apply an owner-set retention period only after upload, read-back verification, and any required proof publication. +Apply retention only after upload, read-back verification, and required publication. Do not bypass current no-delete triggers. -Reclaim whole closed epoch databases and segment files through a future archival workflow. +The storage model remains an [owner decision](README.md#open-decisions): + +- File per epoch: reclaim closed files after durable sealing, verified shipping, and required coverage. +- Single database: require an append-only-preserving compaction or archival migration before reclamation. + +Recommend files per epoch, not yet implemented. +Neither model permits deleting pending intents or unshipped evidence. Retain trusted anchors, certificates, and checkpoints as long as retained evidence needs them. Quota exhaustion blocks admission rather than evicting pending evidence. -Verifiers read a signed inventory, then validate descriptors, payload digests, chain links, and epoch certificates. +Verifiers validate inventory, descriptors, payloads, chains, and certificate bounds. Sort segments by authenticated range, not object listing order. Reject gaps and conflicting overlaps; identical retries are harmless. -Optional proofs bind these receipts to [0004 checkpoints](0004-merkle-checkpoints.md). +Compare registered chains against coverage; report omissions. +[0004](0004-merkle-checkpoints.md) starts with head vectors, then optional compact proofs. Inventory completeness is only relative to its signed capture point. ## Consequences ### Positive -Local admission does not depend on storage network availability. -The same artifacts work through MinIO, S3, or removable media. +Local admission tolerates upload outages. +Artifacts work through MinIO, S3, or removable media. ### Negative -Sealing and archival recovery require new implementation. -Off-node durability lags local acknowledgment. +Sealing needs implementation; off-node durability lags local acknowledgment. ### Risks and Mitigations -A lost volume can lose an unshipped suffix. Publish lag and agree an accepted loss window. -Destination outages can fill disks. Alert on age and capacity, then stop admission before reserved outcome space disappears. -Malicious storage can roll back inventory. Compare separately pinned checkpoints and report freshness as unknown when unpinned. +Volume loss loses unshipped evidence. Publish lag and agree acceptable loss. +Outages fill disks. Alert on capacity and stop admission before reserved outcome space disappears. +Storage can roll back inventories. Compare pinned coverage statements; otherwise report freshness unknown. ## Implementation Notes @@ -109,14 +131,16 @@ Malicious storage can roll back inventory. Compare separately pinned checkpoints Acceptance: original v1 export bytes and v2 signatures verify offline. 2. Inject crashes around reservation, fsync, rename, upload, and acknowledgment. Acceptance: retries never skip ranges or publish partial segments. -3. Test conflicting object names and lost upload replies against in-cluster MinIO. - Acceptance: exact retries succeed; conflicting bytes fail verification. +3. Race conflicting conditional puts and filesystem exclusive creates; lose upload replies against in-cluster MinIO. + Acceptance: exactly 1 immutable object wins; matching retries succeed; conflicting bytes fail. 4. Test destination disconnection and quota pressure. Acceptance: no pending evidence is deleted, and retention never bypasses append-only triggers. +5. Rotate before the final descriptor is written, and attempt cross-replica signing with a replica certificate. + Acceptance: the old key is retained; unauthorized inventory and legacy-segment signers fail. ## Related ADRs -- [0001: Epochs](0001-per-replica-chains.md) defines segment boundaries. -- [0004: Merkle publication](0004-merkle-checkpoints.md) consumes sealed segments. -- [0005: Certificates](0005-root-certified-keys.md) authorizes descriptor signers. -- [0006: Spool](0006-receipt-failure-modes.md) feeds the same shipping path. +- [0001: Epoch boundaries](0001-per-replica-chains.md). +- [0004: Publication](0004-merkle-checkpoints.md). +- [0005: Signing purposes](0005-root-certified-keys.md). +- [0006: Deferred direct spool shipping](0006-receipt-failure-modes.md). diff --git a/docs/design/0004-merkle-checkpoints.md b/docs/design/0004-merkle-checkpoints.md index 8d61328..783df35 100644 --- a/docs/design/0004-merkle-checkpoints.md +++ b/docs/design/0004-merkle-checkpoints.md @@ -1,4 +1,4 @@ -# ADR-0004: Tiled Merkle log and offline checkpoints +# ADR-0004: Signed head vectors before tiled proofs ## Status @@ -10,106 +10,131 @@ Proposed ## Context -[V1 verification](../../pkg/receiptspec/verifier.go) authenticates a linear chain against supplied keys and an optional endpoint. -A valid prefix alone cannot detect a removed suffix or a conflicting view. -[agentgate-verify](../../cmd/agentgate-verify/main.go) currently has no Merkle inclusion or consistency checks. -Signed segments propose durable inputs, not an existing transparency log. +[V1 verification](../../pkg/receiptspec/verifier.go) authenticates chains against keys and optional endpoints. +A valid prefix cannot expose every omission. +[agentgate-verify](../../cmd/agentgate-verify/main.go) has no Merkle proofs. ## Decision Drivers -- Provide compact inclusion evidence and append-only checkpoint comparisons. -- Use published formats instead of inventing tree cryptography. -- Publish inside an air gap without public Sigstore services. -- Keep request commits independent of proof publication. +- Expose missing chains. +- Publish offline, outside request commits. +- Prefer standard proof formats. ## Considered Options | Option | Pros | Cons | | --- | --- | --- | -| Trillian Tessera | Go library; tiled log formats; upstream POSIX support | New dependency and publisher integration; backend qualification needed | -| Hand-written Merkle tree | Small initial code and direct SQLite integration | Team owns proof correctness, recovery, and long-term format maintenance | -| Sigstore Rekor | Existing transparency APIs and verification ecosystem | Supply-chain schema adaptation and extra service operations | +| Signed `(chain_id, seq, head_hash)` vector plus inventories | Fewer dependencies; missing-chain coverage | Reads segments; no compact inclusion proofs | +| Trillian Tessera | Standard tiles; Go library; POSIX support | New dependency, publisher, and backend qualification | +| Hand-written Merkle tree | Small initial implementation | Team owns cryptographic correctness and recovery | +| Sigstore Rekor | Existing APIs and tooling | Supply-chain schema adaptation and service operations | -[Tessera](https://github.com/transparency-dev/tessera) is Trillian's tiled-log successor, not the older Trillian service stack. -[Rekor](https://github.com/sigstore/rekor) can be self-hosted. Public-network dependence is not an inherent requirement. -Its broader application protocol is unnecessary for the proposed receipt log. +[Tessera](https://github.com/transparency-dev/tessera) succeeds older Trillian approaches. +[Rekor](https://github.com/sigstore/rekor) can run offline; its broader protocol is unnecessary here. ## Decision -Propose Tessera behind a narrow asynchronous publisher interface. -Qualify a pinned release with its POSIX backend inside a disconnected cluster before adoption. -Do not assume S3 compatibility means every Tessera backend works with MinIO. -The segment shipper and tree backend are separate concerns. -If offline backend qualification fails, revisit this ADR before implementation. +### Initial Publication + +Recommend signed vectors and inventories first; add Tessera when compact inclusion proofs justify its cost. +Vectors bind organization, installation, publication revision, predecessor digest, and registry revision. +Include every registered chain with its covered `(chain_id, seq, head_hash)`, close status, and inventory digest. +Explicitly list lost or uncovered chains. +A head vector is not a C2SP checkpoint. +Verify selected segments against its endpoints. + +Propose publication every 10 seconds or 4,096 newly covered records, whichever comes first. +Publish descriptors and inventories before their vector. +Retain immutable vector history with conditional latest-pointer updates. +Coverage alone cannot prove freshness. + +### Publisher Ownership and Custody + +Use 1 logical publisher per installation with active/standby failover. +Its `inventory` key is intermediate-certified and cannot be a per-replica key. +Protect keys in local HSM, Vault transit, or approved encrypted storage. +Use a separate origin-scoped `checkpoint` key for the later log. +Keep both private keys outside gateway replicas. + +Validate segments against the independently pinned root and scoped intermediate. +Check registry, fencing token, purpose, chains, digests, and terminal bounds. +Embedded certificates alone are not trusted roots. + +The publisher acquires an external lease and monotonic fencing token. +Persist head map, predecessor digest, and any tree cursor before signing successors. +Publish conditionally against the exact predecessor. +Fence the former writer's storage before recovering its durable state on standby. +Kubernetes Lease expiry alone is insufficient. +Unproven fencing or continuity stops publication. +For POSIX Tessera, never attach its writable data volume to competing publishers. +Reconcile the last signed checkpoint before signing after failover. + +### Later Merkle Profile + +Tessera adds compact inclusion proofs and standard tooling, not missing-chain discovery. +Qualify a pinned POSIX release offline before adoption. +Do not assume a Tessera cloud backend supports MinIO. Adopt [C2SP tlog-checkpoint v1.1.0](https://c2sp.org/tlog-checkpoint@v1.1.0) and [tlog-tiles v1.0.0](https://c2sp.org/tlog-tiles@v1.0.0). -Use SHA-256 with RFC 6962 domain separation. +Use RFC 6962 SHA-256: Leaf hashes are `SHA256(0x00 || leaf_bytes)`. Internal hashes are `SHA256(0x01 || left_hash || right_hash)`. -Use standard signed-note checkpoint encoding with an Ed25519 log signature. -Do not invent JSON checkpoint signatures or optional extension lines. - -Propose a small versioned leaf encoding binding receipt version, chain scope, sequence, and entry hash. -The v2 scope is its authenticated `chain_id`. -For v1, scope comes from the signed legacy segment descriptor, without modifying the receipt. -Specify leaf bytes and proof fixtures before implementation. -Tree membership does not replace receipt signatures or chain verification. - -The publisher validates sealed segments before integration. -A durable mapping from `(chain_scope, seq, entry_hash)` to leaf index prevents upload retries from adding duplicate leaves. -Reject conflicting hashes for an already published chain position. -Cross-replica integration order records publisher arrival, not external execution order. -Crash recovery must reconcile ambiguous integration before accepting new work. - -Propose a checkpoint every 10 seconds or 4,096 newly integrated leaves, whichever comes first. +Use signed-note Ed25519 checkpoints without extension lines. + +Freeze leaf bytes binding receipt version, scope, sequence, and entry hash. +Use authenticated v2 `chain_id`, or v1's signed legacy-descriptor scope without modifying receipts. +Membership never replaces receipt signatures or chain verification. + +Persist `(chain_scope, seq, entry_hash)` to leaf-index mappings. +Reconcile ambiguous integration before accepting retries; reject conflicting positions. +Integration order reflects arrival, not external execution order. + +Checkpoint every 10 seconds or 4,096 integrated leaves, whichever comes first. Publish tiles and entry bundles durably before publishing their checkpoint. Retain immutable checkpoint history plus a mutable latest pointer. Idle logs need not issue identical checkpoints. -Checkpoint size authenticates coverage, not wall-clock freshness. -Publish a signed segment inventory alongside each checkpoint to expose per-chain coverage. - -Propose `agentgate-verify` support for local proof bundles and tile directories. -Verify the root-certified log key, checkpoint origin and signature, leaf identity, inclusion path, and tree size. -Verify receipt signatures and segment boundaries separately. -Check consistency against an earlier trusted checkpoint when supplied. -A single checkpoint does not prove global freshness or absence of split views. - -Air-gapped publication writes signed checkpoint files, tiles, certificates, and inventories to in-cluster storage. -Offline export copies those files and receipt segments through approved removable media. -No public Rekor, cloud KMS, or network fetch is required by verification. +Publish an inventory beside each checkpoint for per-chain coverage. + +Extend `agentgate-verify` for local proofs and tiles. +Check certificate path, checkpoint signature/origin, leaf identity, inclusion, and tree size. +Check consistency against an earlier pinned checkpoint when supplied. +A single checkpoint proves neither freshness nor absence of split views. + +Air-gapped publication starts with signed vector files, certificates, registry snapshots, and inventories. +The later log also exports signed checkpoint files and tiles. +Copy those artifacts and segments through approved removable media. +Verification requires no public services or network fetches. Witness cosigning is optional and disabled by default. -An installation may configure local witnesses and a quorum policy. -Without that policy, make no witness-backed anti-equivocation claim. +Local witness quorum is optional. Without it, claim no witness-backed anti-equivocation. ## Consequences ### Positive -Proof verification can remain local and independent of the gateway process. -Published formats reduce custom cryptographic surface. +Vectors expose omissions without a tree service. +Later proofs reduce selective verification cost. ### Negative -Proof publication introduces asynchronous lag and another durable service role. -Tiles and historical checkpoints consume storage beyond the gateway volume. +Both stages need a fenced singleton publisher and durable state. +Initial selective verification is linear in the supplied segment range. ### Risks and Mitigations A publisher can equivocate. Compare pinned checkpoints and optionally require independent local witness signatures. Delayed publication can hide recent loss. Report integrated coverage separately from committed and uploaded heads. -Dependency behavior can change. Pin releases and run disconnected interoperability fixtures before upgrades. +Dependencies can change. Pin releases and run offline interoperability fixtures. ## Implementation Notes -1. Qualify Tessera POSIX recovery without external connectivity. - Acceptance: restart preserves tree size, roots, and leaf indices. -2. Freeze leaf and checkpoint-key certificate formats. - Acceptance: independent C2SP tooling verifies checkpoints and reconstructs tile proofs. -3. Test duplicate segment ingestion and crashes during integration. - Acceptance: each unique chain position has 1 stable leaf index. -4. Add CLI fixtures for wrong roots, wrong origins, missing tiles, and invalid inclusion or consistency paths. - Acceptance: offline checks reject each mutation while existing v1 verification remains unchanged. +1. Publish vectors from registry-backed inventories and fail over the active publisher during publication. + Acceptance: no registered chain vanishes and no predecessor has competing accepted successors. +2. Lose the publisher node, block its lease renewal, and simulate failed storage fencing. + Acceptance: an unfenced standby cannot publish; recovery preserves signed history. +3. Gate Tessera adoption on offline POSIX recovery and stable leaf indices after duplicate ingestion. +4. Add C2SP interoperability and CLI fixtures before enabling Merkle mode. + Acceptance: wrong roots, origins, inclusion paths, or consistency paths fail; existing v1 results remain unchanged. ## Related ADRs diff --git a/docs/design/0005-root-certified-keys.md b/docs/design/0005-root-certified-keys.md index b065602..6698770 100644 --- a/docs/design/0005-root-certified-keys.md +++ b/docs/design/0005-root-certified-keys.md @@ -1,4 +1,4 @@ -# ADR-0005: Root-certified replica signing keys +# ADR-0005: Offline root, scoped issuer, and epoch keys ## Status @@ -10,114 +10,142 @@ Proposed ## Context -The [signer store](../../internal/signer/store.go) caches decrypted Ed25519 keys in memory. -`WithActiveKey` holds a lease through each ledger transaction, excluding `Rotate` and `Close`. -`Rotate` switches keys at the next global sequence and clears old private bytes. -It creates neither epochs nor root certificates. +The [signer store](../../internal/signer/store.go) caches Ed25519 keys. +`WithActiveKey` leases exclude rotation through batch commit. +`Rotate` switches global sequence windows and clears old keys, without epochs or certificates. -[V1 key history](../spec/RECEIPT-v1.md) uses independently trusted public keys with inclusive sequence windows. -Embedded export keys alone prove consistency, not organization identity. -Current [manifest verification](../../pkg/receiptspec/manifest.go) does not enforce its signer's validity window. -Manifest-signer validity-window hardening is in progress on another branch, not shipped in this baseline. +[V1](../spec/RECEIPT-v1.md) trusts keys with inclusive sequence windows. +[Manifest verification](../../pkg/receiptspec/manifest.go) currently omits signer-window checks. +That hardening is in progress elsewhere, not shipped here. ## Decision Drivers -- Let an offline verifier trust 1 independently provisioned organization root. -- Limit a compromised replica key's scope and lifetime. -- Keep roots outside request handling. -- Preserve cached signing and batch rotation leases. +- Pin 1 offline root. +- Constrain installation, purpose, and epoch. +- Preserve batch signing leases. ## Considered Options -### Root Custody +### Issuance and Custody | Option | Pros | Cons | | --- | --- | --- | -| KMS or HSM | Managed custody or hardware protection | Product-specific algorithms, cost, and availability | -| Local Vault transit | Local automation and access policy | Additional service and recovery operations | -| Offline root ceremony | No online root dependency | Manual issuance and renewal planning | +| Direct offline-root issuance | Minimal delegation | Daily ceremonies | +| Offline root with local issuer | Automated epochs; offline root | Intermediate compromise risk | +| Online KMS/HSM/Vault root | Automated custody | Availability and algorithm constraints | -Cloud KMS is optional. Local HSM, Vault transit, or an offline root must support air-gapped issuance. +Cloud KMS is optional. HSM or Vault transit can operate locally. ### Replica Certificate | Option | Pros | Cons | | --- | --- | --- | -| Minimal signed statement | Small offline parser and explicit chain scopes | Team must specify encoding and constraints | -| X.509 | Existing tooling and enterprise PKI integration | Custom chain-scope extensions and path-validation complexity | -| Raw trusted replica keys | Matches v1 | Every verifier needs independently distributed key history | +| Length-prefixed statement | V1 conventions; explicit scopes | New certificate fixtures | +| X.509 | Existing PKI tooling | Custom scopes and path validation | +| Canonical JSON | Readable fields | Another canonicalization profile | ## Decision -Propose an organization root held in an approved KMS, HSM, local Vault transit, or offline ceremony. -The initial proposed certificate profile uses Ed25519. -Reject incompatible backends instead of exporting root keys. -No request or offline verification requires reaching the root service. - -Choose a minimal, versioned signed certificate statement over X.509. -Propose canonical JSON, restricted integers, and a dedicated signature domain. -Gateway params canonicalization does not define certificate semantics. -The root signs the statement digest. -Bind organization ID, issuer fingerprint, subject public key, key ID, allowed purpose, and validity timestamps. -Replica certificates also bind exactly 1 replica UUID and epoch. -Log certificates instead bind a checkpoint origin and log-signing purpose. -Freeze bytes, timestamp units, and domains before implementation. -Reject duplicate or ambiguous key IDs. - -Propose a default replica key lifetime of 24 hours, configurable for offline renewal ceremonies. -Offline installations must provision renewal material or approve longer intervals. -Expiry stops new signing. Never silently extend trust or fall back to cloud services. -Keep private replica keys encrypted at rest and cached under the current lease model. -Retain certificates after private-key destruction. - -Propose `agentgate keys rotate` as a new command, not an existing CLI feature. -Stop admission, resolve inflight attempts, then close and seal the authorized epoch. -Atomically persist the next certified key and epoch before resuming. -Preserve lease-before-SQLite lock ordering. -Current `Rotate(atSeq)` uses global windows; v2 needs an epoch-aware transaction, not `Rotate(1)`. -Interrupted rotation must resume at exactly 1 committed epoch boundary. - -Offline verification needs only receipts, proofs, checkpoint, and an independently pinned root public key. -Bundles include certificates, descriptors, and signed trust-status statements. -Verify purpose, chain scope, key bytes, and applicable sequence or timestamp windows. -Do not reject historical receipts merely because their certificate is expired today. -Receipt timestamps alone do not prove signing time. -Without independently trusted time, report backdating limitations. -Offline revocation reflects bundled signed status, not unseen later revocations. -Legacy v1 exports still require their original trusted key history unless a reviewed root-signed migration descriptor binds it. +Propose an offline root delegating to a longer-lived, in-cluster intermediate issuer. +Bind organization, installation, allowed purposes, and a prohibition on further intermediate delegation. +Recommend annual intermediate renewal and 24-hour epoch rotation, without public connectivity. +Prefer local HSM or Vault transit; encrypted issuer storage needs owner approval. +Record issuance in [0001's registry](0001-per-replica-chains.md) and refuse duplicate `(UUID, epoch)` certification. +Reserve that pair and durably record its signed certificate digest before releasing the certificate. +Crashes may leave unused registrations, never usable unregistered certificates. + +Choose v1-style length-prefixed statements, not JSON or X.509. +Use NUL-terminated domains, `U32LE` lengths, `U64LE` integers, raw keys, and absence markers. +Ed25519 signs SHA-256 of that preimage. +Bind issuer and subject fingerprints, organization, installation, purposes, and scope. +Freeze ordering and fixtures. Reject ambiguous key IDs and cross-purpose use. + +### Signing Roles and Purposes + +| Purpose | Authorized signer and scope | +| --- | --- | +| `certificate_issuance` | Intermediate; its installation and permitted child purposes | +| `replica_registry` | Intermediate; issuance, lease generations, replacements, losses | +| `receipt` | Epoch key; intents, outcomes, decisions, stream results | +| `control` | Epoch key; boundaries, mode changes, future sink-transition markers | +| `segment_descriptor` | Epoch key; descriptors for that same epoch | +| `legacy_segment` | Certified archival publisher; migrated v1 exports only | +| `inventory` | Certified singleton publisher; cross-replica inventories and head vectors | +| `checkpoint` | Separate certified log key; installation and origin | +| `trust_status` | Intermediate; signed status of its delegated keys and sequence cutoffs | +| `migration` | Offline root and final v1 key co-sign; neither signature may be delegated | + +### Validity and Rotation + +Epoch and sequence bounds govern validity. +Certify exactly 1 UUID and epoch, starting at sequence 1. +Its signed epoch-close fixes the inclusive terminal sequence. +Verifiers reject records beyond that bound. +Terminal coverage requires a signed inventory binding epoch-close and the final descriptor. +When Merkle mode is enabled, also prove inclusion of epoch-close against its checkpoint. +Unclosed epochs remain provisional prefixes. +Time windows guide renewal only; untrusted timestamps cannot establish cryptographic expiry. +Elapsed time alone cannot invalidate historical evidence. + +Propose `agentgate keys rotate` at a batch boundary. +Keep admission open; bounded queues may briefly wait. +After the current batch, append epoch-close and durably seal the final segment descriptor. +Retain the old key until descriptor and directory fsync. +Durably activate the next certified epoch and write epoch-open at sequence 1 before queued work. +Inflight upstream calls continue. +Later outcomes reference earlier intents across epochs. +Epoch-close closes signing extent, not external-work completion. +Preserve lease-before-SQLite lock ordering; recover activation from durable phase metadata. +Current `Rotate(atSeq)` needs epoch-aware changes; calling `Rotate(1)` is not valid. + +### Revocation and Offline Verification + +Revocation policy remains an [owner decision](README.md#open-decisions). +Recommend intermediate-signed status every 6 hours; `next_update` is 24 hours after issuance. +Bind revision, predecessor digest, certificate IDs, and revoked epoch or sequence cutoffs. +Offline verification reports bundled status as-of time. +Freshness needs trusted verifier time or a separately pinned status revision. +Otherwise report freshness unknown, not unseen revocation checks. +Stale or missing status prevents current-trust success; historical checks report as-of status. +Revoke compromised intermediates through offline-root-signed replacement status. + +Bundle certificates, registry history, status, descriptors, and applicable proofs or checkpoints. +Independently pin the root public key. +Raw-key v2 lifecycle tests do not require this hierarchy. +Legacy v1 retains its original trust rules outside a root-and-final-v1-key co-signed migration envelope. ## Consequences ### Positive -Root pinning replaces manual trust distribution for each new replica key. -Purpose and epoch constraints reduce cross-role key misuse. +Root pinning supports automated local issuance. +Sequence bounds remain meaningful without trusted receipt clocks. ### Negative -Certificate issuance and root recovery require operational procedures. -Offline installations trade short lifetimes against renewal workload. +The intermediate and registry require protected local state. +Revocation freshness remains a policy requirement. ### Risks and Mitigations -A compromised root can certify attacker keys. Require protected custody and an audited root replacement procedure. -Compromised replicas can backdate receipts. Document clock limits and compare independently retained checkpoints. -Manifest hardening may change verifier rejection cases. Coordinate that branch without changing frozen v1 encoding. +A compromised issuer can certify attacker keys. Restrict scope and audit issuance. +Cloned certificates remain usable. Require external fencing. +Coordinate manifest hardening without changing frozen v1 encoding. ## Implementation Notes -1. Freeze certificate fixtures and an offline root-pin procedure. - Acceptance: tampered scope, purpose, validity, or public key fails verification. -2. Integrate epoch-aware activation with existing cached-key leases. - Acceptance: rotation during a batch waits; no batch uses a cleared or wrong-epoch key. -3. Crash-test each rotation boundary. - Acceptance: restart selects exactly 1 active epoch and never emits uncertified receipts. -4. Test root outages, expiration, offline renewal, and archived evidence. - Acceptance: new signing stops on expiry; historical verification works without any network. +1. Freeze binary certificate fixtures and an offline root-pin procedure. + Acceptance: wrong installation, purpose, epoch, or issuer delegation fails verification. +2. Rotate under continuous admission and inflight SSE, then crash at each boundary. + Acceptance: no admission drain, exactly 1 epoch-open, and the old key survives until final descriptor durability. +3. Append records beyond epoch-close and remove its inventory coverage. + Acceptance: out-of-bounds records fail; missing terminal coverage cannot pass as complete. +4. Test stale, missing, and revoked trust-status bundles without networking. + Acceptance: report exact revocation scope and unknown freshness without fabricating clock-based trust. ## Related ADRs -- [0001: Epoch identity](0001-per-replica-chains.md) scopes replica certificates. -- [0002: Recovery](0002-two-phase-receipts.md) resolves inflight attempts before rotation. -- [0003: Segments](0003-durable-shipping.md) carries public certificate history. -- [0004: Checkpoints](0004-merkle-checkpoints.md) uses a separately certified log purpose. +- [0001: Identity and registry](0001-per-replica-chains.md). +- [0002: Cross-epoch outcomes](0002-two-phase-receipts.md). +- [0003: Descriptors](0003-durable-shipping.md). +- [0004: Publisher keys](0004-merkle-checkpoints.md). diff --git a/docs/design/0006-receipt-failure-modes.md b/docs/design/0006-receipt-failure-modes.md index 4d3ea9a..7e93cdc 100644 --- a/docs/design/0006-receipt-failure-modes.md +++ b/docs/design/0006-receipt-failure-modes.md @@ -10,16 +10,14 @@ Proposed ## Context -The [gateway](../../internal/gateway/gateway.go) currently returns HTTP 500 when receipt append fails. -It never returns the would-be outcome without an acknowledged receipt commit. -An upstream side effect may already exist. -The [ledger](../../internal/receipt/ledger.go) notes that cancellation after submission can still leave a committed receipt. -There is no durable fallback spool or failure-mode setting today. +The [gateway](../../internal/gateway/gateway.go) returns 500 on receipt failure, withholding the outcome even after upstream execution. +[Ledger cancellation](../../internal/receipt/ledger.go) can leave committed receipts. +There is no fallback spool or failure-mode setting. ## Decision Drivers - Keep durable evidence as the default admission requirement. -- Offer bounded availability during specific SQLite failures. +- Require a named failure class before adding fallback storage. - Never interpret fail-open as bypassing authentication or policy. - Make configuration changes auditable even when the primary sink fails. @@ -29,86 +27,98 @@ There is no durable fallback spool or failure-mode setting today. | --- | --- | --- | | `fail_closed` only | Smallest failure surface | Local database faults stop all actions | | Unbounded or memory-only fail-open | Highest apparent availability | Evidence loss and uncontrolled disk growth | -| Default closed, optional durable spool | Bounded fallback with explicit risk | More recovery states and capacity accounting | +| Default closed, defer durable spool | Clear initial guarantee; avoids unsupported availability claims | No fallback until its failure model is approved | ## Decision -Propose `receipt_failure_mode: fail_closed` as the default. +Propose shipping only `receipt_failure_mode: fail_closed` in the initial implementation. +These documents do not implement that behavior. An unavailable durable intent sink prevents upstream dispatch and returns HTTP 503. Failure after dispatch also returns 503 before response headers, without claiming the external action failed. The unresolved committed intent is recovered as `outcome_unknown`. This proposes an API behavior change from the current 500 response. -Propose opt-in `fail_open_spool`, meaning open relative to the primary SQLite sink only. -It still requires durable signed evidence before dispatch and response. -Use a bounded append-only spool on the replica's persistent data volume. -Fsync its intent before upstream dispatch and its outcome before returning a buffered response. -For SSE, fsync the header outcome before headers and persist the terminal artifact afterward. -The spool is not a raw request queue. Never store credentials or replay upstream requests. - -Keep 1 writer and 1 authoritative chain across SQLite and the spool. -Fallback is allowed only with an unambiguous committed head and an available certified signer. -An uncertain SQLite commit cannot be converted into a second signed entry. -Fence admission until durable reconciliation resolves that attempt. -Disk failure, signer failure, expired certificates, or unknown chain state still fail closed. -Sharing a volume means the spool does not protect against volume loss or full-disk failure. - -Propose explicit limits for bytes, record count, and oldest unshipped age. -Initial suggested limits are 256 MiB, 100,000 records, and 24 hours. -Owner approval and workload sizing are required before enabling them. -Reserve maximum bounded intent, outcome, and stream-terminal space before admitting each attempt. -Reserve control-record capacity separately. -Never evict unshipped records. -When any limit blocks reservation, reject new attempts with 503 and `Retry-After`. -Inflight work consumes its reserved space, including recovery outcomes. - -Log every configured mode change as a signed v2 control receipt. +Defer `fail_open_spool` until the owner names a covered failure class and demonstrates recovery under that fault. +Reject that mode as unsupported for now. +A same-volume spool does not cover volume loss or full-disk failure. +It also cannot resolve an uncertain SQLite commit or replace a missing authorized signer. + +### Authenticated Configuration + +Log initial configuration and every later accepted mode change as signed v2 control receipts. Bind old mode, new mode, actor, reason, and configuration digest. -Also record effective sink transitions between SQLite and spool. -Commit that receipt to the current authoritative sink before activating the change. -If neither sink can durably record it, reject the change and retain the last committed configuration. -Bootstrap records the initial configuration before admission. -Do not silently downgrade modes on restart. - -On recovery, reconcile spool and SQLite positions before accepting traffic. -Replay exact signed records into SQLite with their original identities and bytes. -Identical existing entries are idempotent; different hashes at the same position halt recovery. -Do not allocate replacement sequences, re-sign, or re-dispatch upstream. -Ship spooled evidence through the same sealed-segment protocol. -Delete spool files only after replay, shipping verification, and retention requirements are satisfied. +The epoch key signs them with `control` authority. +Commit the control receipt before activating the configuration. +If it cannot commit, reject the change and retain the last committed configuration. +503 handling does not bypass authentication, action policy, or egress enforcement. + +For human changes, require the existing [authenticated admin session](../../internal/admin/session.go). +Resolve `AdminID` and `OrgID` server-side and require installation configuration authority. +Apply [mutating-request CSRF checks](../../internal/admin/handler.go). +Never trust a request-supplied actor name. +The legacy shared `X-Admin-Secret` alone lacks an individual actor and is insufficient for this endpoint. +Bootstrap verifies the issuer-signed enrolment statement and binds its deployment identity to the configuration digest. +Unattended control authentication beyond that bootstrap remains deferred, not an anonymous local bypass. ## Consequences ### Positive Default operation keeps a clear durability boundary. -Opt-in fallback survives some primary-sink faults without inventing successful attestations. +No fallback availability is claimed without a demonstrated failure model. ### Negative -Capacity reservation and sink handoff add complexity. +Primary-sink outages stop admission. 503 does not mean an already-dispatched external action was rolled back. ### Risks and Mitigations -Ambiguous handoff can fork a chain. Require fencing and durable reconciliation before selecting another sink. -Spool exhaustion can strand outcomes. Reserve terminal capacity before dispatch and test worst-case record sizes. +Unlogged configuration can hide evidence loss. Require authenticated actors and committed control receipts before activation. Started SSE cannot change HTTP status. Close the stream on terminal persistence failure and recover unknown completion later. ## Implementation Notes -1. Implement the two-phase state machine before exposing fallback configuration. - Acceptance: neither mode dispatches without a durable intent. -2. Inject unambiguous database failures and uncertain commit acknowledgments separately. - Acceptance: only the former can enter the spool; the latter blocks until reconciled. -3. Fill each quota and exhaust reserved control capacity. - Acceptance: no evidence is evicted, and unrecordable mode changes are refused. -4. Restart during sink handoff, spool fsync, replay, and segment shipping. - Acceptance: exact replay produces no duplicates, forks, or upstream calls. +1. Add 503 handling to the two-phase state machine. + Acceptance: no intent acknowledgment means no upstream dispatch; post-dispatch failures never imply rollback. +2. Add authenticated configuration control. + Acceptance: forged actors, wrong organizations, missing authority, and missing CSRF checks cannot change mode. +3. Fail the control append and request the deferred spool mode. + Acceptance: neither change activates; the last committed mode survives restart. +4. Any later spool implementation must pass the appendix's crash and evidence-loss tests. + +## Deferred Appendix: Bounded Durable Spool + +This is a design sketch, not an enabled Phase 2 mode. +Approval requires a named primary-sink fault that leaves independent spool writes and signing safe. +Keep 1 writer and 1 authoritative chain. +Fsync intent before dispatch and outcome before response; preserve SSE's header and terminal boundaries. +Resolve uncertain commits before handoff. Never replay requests, re-sign receipts, or allocate replacement sequences. + +Persist a signed sink-transition marker outside the spool before entering it. +Keep that marker in a separate durable control file, not solely SQLite or the spool directory. +Bind previous SQLite head, sink generation, spool identity, and fencing token. +Update its recorded spool head durably before acknowledging each spool append. +After a crash, scan durable spool records to reconcile any append newer than the marker. +An absent or corrupt spool while its marker remains is evidence loss, not permission to resume SQLite. +Refuse SQLite admission past a recorded spool head until exact signed records have been replayed and reconciled. +Only then fsync a return-to-SQLite marker and commit the corresponding control receipt. + +Ship signed spool segments directly, without waiting for SQLite replay. +Their sealing metadata and upload progress must remain durable outside the failed SQLite sink. +Preserve the marker and pending segments if replay cannot finish. +Identical existing records are idempotent; conflicting positions halt recovery. +Delete nothing until replay, verified shipping, and retention requirements permit it. + +Bound bytes, count, and age using owner-approved quotas. +Reserve intent, outcome, stream-terminal, and control capacity before admission. +Overflow rejects new work with 503 and never evicts evidence. +Tests must remove spool files while retaining the marker and interrupt every transition, replay, and upload boundary. +Acceptance: no SQLite restart can conceal a recorded spool suffix, and direct shipping works while SQLite remains unavailable. ## Related ADRs -- [0001: Chains](0001-per-replica-chains.md) defines unique positions. -- [0002: Two-phase receipts](0002-two-phase-receipts.md) defines durability boundaries. -- [0003: Shipping](0003-durable-shipping.md) governs deletion and acknowledgment. -- [0005: Keys](0005-root-certified-keys.md) defines signing availability. +- [0001: Chain positions](0001-per-replica-chains.md). +- [0002: Durability boundaries](0002-two-phase-receipts.md). +- [0003: Shipping and deletion](0003-durable-shipping.md). +- [0005: Signing authority](0005-root-certified-keys.md). diff --git a/docs/design/0007-kubernetes-lifecycle.md b/docs/design/0007-kubernetes-lifecycle.md index 738878f..c7193c5 100644 --- a/docs/design/0007-kubernetes-lifecycle.md +++ b/docs/design/0007-kubernetes-lifecycle.md @@ -10,86 +10,95 @@ Proposed ## Context -This baseline includes [Compose](../../docker-compose.yaml) and [configuration YAML](../../configs/agentgate.yaml), not Helm or Kubernetes workload manifests. +This baseline has [Compose](../../docker-compose.yaml) and [configuration YAML](../../configs/agentgate.yaml), not Kubernetes workload manifests. The [health endpoint](../../internal/gateway/gateway.go) reports service count, not ledger readiness. [HTTP shutdown](../../cmd/agentgw/main.go) currently allows 10 seconds. -Deferred closure then stops the ledger, clears signer keys, and closes SQLite. -[Ledger closure](../../internal/receipt/ledger.go) finishes an already submitted transaction but rejects pending requests. -It does not seal segments. +Then [ledger closure](../../internal/receipt/ledger.go) finishes submitted transactions but rejects pending requests, without sealing. +Signer and database closure follow. -The same SQLite database also stores authorization, vault, and organization state. -Receipt isolation does not replicate that state. +SQLite also stores authorization, vault, and organization state. Receipt isolation does not replicate it. ## Decision Drivers -- Give every replica exclusive, restart-stable storage. -- Stop admission before draining work. -- Distinguish storage pressure from process failure. -- Preserve runtime-independent governance. +- Preserve identity and exclusive storage. +- Drain safely without restart loops. +- Keep governance runtime-independent. ## Considered Options | Option | Pros | Cons | | --- | --- | --- | -| StatefulSet with per-pod PVC | Stable attachment and explicit replica lifecycle | Volume fencing and zone recovery still need operations | -| Deployment with PVC per replica | Familiar rollout controller | Dynamic claim ownership needs extra identity machinery | -| Deployment with shared writable PVC | Simple manifest | Violates isolated writer and signer-cache assumptions | -| Ephemeral Deployment storage | Easy replacement | Loses committed, unshipped evidence | +| StatefulSet with per-pod PVC | Stable attachment and identity | Storage fencing and zone recovery | +| Deployment with PVC per replica | Familiar controller | Additional claim-ownership machinery | +| Shared or ephemeral storage | Simple scheduling | Writer conflicts or evidence loss | ## Decision Propose a StatefulSet with 1 persistent data volume per replica. Use `volumeClaimTemplates` and `ReadWriteOncePod` where supported. -Otherwise require storage-level single-writer fencing beyond `ReadWriteOnce`. -Test filesystem locking and fsync behavior. +Otherwise require storage fencing beyond `ReadWriteOnce`. Test filesystem locking and fsync. Never share a writable SQLite database between replicas. Persist the provisioned replica UUID and epoch alongside SQLite. -The StatefulSet ordinal locates its PVC but is not the cryptographic replica ID. -Replacement PVCs receive new UUIDs; intact restarts resume their epoch. +Ordinals locate PVCs; they are not cryptographic identities. +Empty PVCs stay unready until explicit intermediate enrolment. +Never silently initialize replacement identity. +The signed registry binds ordinal, new UUID, and replacement of the prior UUID. +Report the old chain lost from its last shipped or checkpointed head, with unknown suffix. +Without coverage, report it lost with no known head. +Intact restarts require recovery and lease checks before resuming. Retain PVCs on scale-down. Audit explicit deletion. +Require [0001's external lease and token](0001-per-replica-chains.md) before admission and each batch. +Kubernetes Lease expiry alone does not fence a partitioned node. +Force-detach requires proven old-node storage fencing, then lease takeover with a newer token. +Unproven fencing blocks takeover. + The only proposed gateway-owned writable state is its WAL data volume. -That includes the main SQLite file, WAL, SHM, encrypted keys, outbox, sealed files, and optional spool. +It includes SQLite, WAL, SHM, encrypted keys, outbox, sealed files, and any future spool. Existing authorization and vault rows remain on that volume. Configuration and mounted secrets stay read-only. Object storage and optional Tessera publication require separate durable infrastructure. -Consistent authorization and vault provisioning across replicas remains a rollout prerequisite. +Multi-replica AgentGate cannot ship until per-replica auth/vault state is solved. +Test provisioning, rotation, and revocation propagation before enabling replicas beyond 1. Never copy live WAL files to provision replicas. Propose separate `/livez` and `/readyz` endpoints plus a startup probe. -Liveness checks process progress, not upstream reachability or object-store availability. -Readiness requires completed recovery, exclusive volume ownership, an authorized signer, and admission capacity. -Spool mode requires healthy reserved capacity. +Liveness checks process progress, not upstream or object storage. +Readiness checks recovery, exclusive volume ownership, signer authority, and capacity. +The initial deployment is fail-closed. The deferred spool is neither required nor enabled. +Only later spool-enabled deployments require [0006's spool protocol](0006-receipt-failure-modes.md). Upload outages preserve readiness until local quotas block admission. Probes must not append synthetic receipts. -Propose an authenticated local drain control used by both `preStop` and SIGTERM. -First fail readiness and stop accepting new actions. +Use authenticated local drain control for `preStop` and SIGTERM. +Fail readiness and stop admission. Allow endpoint propagation, then drain HTTP work. Bound long-lived SSE streams and write their terminal or unknown records. -Wait for outstanding ledger append acknowledgments before calling `Ledger.Close`. +Wait for append acknowledgments before `Ledger.Close`. Seal the final segment and persist its outbox state. Never wait indefinitely for upload. -Finally close the ledger, clear signer keys, and close SQLite. +Close ledger, clear signer keys after descriptor durability, then close SQLite. Recover interrupted drain from that volume. -Propose `terminationGracePeriodSeconds: 60` as a starting budget. -Budget 5 seconds for endpoint propagation, 40 for requests and streams, and 10 for ledger completion and sealing. -Reserve 5 seconds for closure. -This requires changing the current shutdown timeout. -Hook time counts within the grace period. +Propose `terminationGracePeriodSeconds: 60`, including hooks. +Allocate 5 seconds for propagation, 40 for requests/streams, 10 for ledger/sealing, and 5 for closure. +Change the current shutdown timeout accordingly. +The 40-second stream budget cuts longer SSE streams. +The expected `stream_result` has `termination = unknown` and reason `drain_deadline`. +Any digest covers only the observed prefix. +Document these rollout cuts; unlimited stream continuity is not promised. -For installations with at least 2 replicas, propose a PDB with `maxUnavailable: 1`. +For 2 or more replicas, propose a PDB with `maxUnavailable: 1`. Spread across hostnames; prefer zones when available. Use rolling updates with at most 1 unavailable replica. Document single-replica maintenance downtime. PDBs neither prevent node failures nor replicate unshipped evidence. -Clawdlinux's operator applies sandbox labels and egress policy independently of the agent runtime. -Runtime selection belongs to the operator's `pkg/runtime.Registry`, not gateway branches. -Keep the gVisor label injector and configure runtime classes through that integration. -No runtime adapter should reimplement sandbox or egress policy. +Clawdlinux's operator applies sandbox labels and egress policy independently of runtime. +Select runtimes through its `pkg/runtime.Registry`, not gateway branches. +Keep its gVisor label injector and runtime-class integration. +Adapters must not reimplement sandbox or egress policy. Air-gapped profiles require no public-cloud endpoint. ## Consequences @@ -102,31 +111,37 @@ Drain preserves local evidence without requiring remote availability. ### Negative Stateful storage complicates rescheduling and disaster recovery. -Replica authorization provisioning remains separate work. +Unsolved replica authorization state blocks scale-out release. ### Risks and Mitigations -A split-brain volume attachment can fork evidence. Require fencing and test forced node recovery. -Slow streams can exhaust shutdown time. Enforce drain deadlines and recover missing terminal records. -Topology spread can leave pods unschedulable. Use installation-specific constraints and document degraded availability. +Split-brain attachments can fork evidence. Require lease and storage fencing. +PVC loss loses unshipped evidence. Preserve registry history and report loss. +Slow streams exhaust shutdown time. Cut them and record unknown termination. +Spread constraints can block scheduling. Document installation-specific degraded availability. ## Implementation Notes -1. Add recovery-aware probes and idempotent drain control before publishing manifests. - Acceptance: unhealthy storage removes readiness without triggering a restart loop. +1. Add recovery-aware probes and idempotent drain. + Acceptance: unhealthy storage removes readiness without restart loops. 2. Test a 2-replica rolling update with continuous traffic and SSE. - Acceptance: no new attempts enter a draining pod; committed receipts survive rescheduling. -3. Kill a pod during drain and force a node replacement. - Acceptance: outbox recovery resumes, and no two writers acquire the same replica identity. -4. Disconnect external networking and the optional upload destination. + Acceptance: draining pods reject new attempts; committed receipts survive rescheduling. +3. Partition a node with active upstream work, expire its lease, then request CSI force-detach. + Attempt takeover before and after proving storage fencing. + Acceptance: unfenced takeover fails; replacement tokens increase; the old writer cannot admit or publish. +4. Replace its PVC with an empty one. + Acceptance: enrolment gates readiness; registry records replacement and old-chain loss. +5. Drain an SSE stream longer than 40 seconds. + Acceptance: stream cuts produce unknown termination, never completion. +6. Disconnect external networking and the optional upload destination. Acceptance: local governance and offline verification work; capacity exhaustion fails closed. -5. Validate equivalent sandbox labels and egress policy across supported operator runtimes. +7. Validate equivalent sandbox labels and egress policy across supported operator runtimes. Acceptance: changing runtime selection does not change receipt or sealing guarantees. ## Related ADRs -- [0001: Identity](0001-per-replica-chains.md) governs volume loss and epochs. -- [0002: Lifecycle](0002-two-phase-receipts.md) supplies drain recovery records. -- [0003: Segments](0003-durable-shipping.md) defines sealing and retention. -- [0005: Rotation](0005-root-certified-keys.md) defines key expiry and drain interactions. -- [0006: Failure modes](0006-receipt-failure-modes.md) determines readiness capacity. +- [0001: Fencing and volume loss](0001-per-replica-chains.md). +- [0002: Drain recovery](0002-two-phase-receipts.md). +- [0003: Sealing](0003-durable-shipping.md). +- [0005: Enrolment and rotation](0005-root-certified-keys.md). +- [0006: Dependency only for enabled spool](0006-receipt-failure-modes.md). diff --git a/docs/design/README.md b/docs/design/README.md index e1eb73c..c008550 100644 --- a/docs/design/README.md +++ b/docs/design/README.md @@ -19,9 +19,9 @@ The [v2 outline](../spec/RECEIPT-v2.md) is a proposal, not an interoperable spec | [0001](0001-per-replica-chains.md) | Separate replica epochs with authenticated v2 chain IDs | Proposed | 2026-10-18 | | [0002](0002-two-phase-receipts.md) | Commit intent before dispatch and outcome before response | Proposed | 2026-10-18 | | [0003](0003-durable-shipping.md) | Ship signed segments from replica-local SQLite | Proposed | 2026-10-18 | -| [0004](0004-merkle-checkpoints.md) | Evaluate Tessera with C2SP tiles and signed checkpoints | Proposed | 2026-10-18 | -| [0005](0005-root-certified-keys.md) | Certify short-lived replica keys with an organization root | Proposed | 2026-10-18 | -| [0006](0006-receipt-failure-modes.md) | Default closed; permit bounded durable spooling by opt-in | Proposed | 2026-10-18 | +| [0004](0004-merkle-checkpoints.md) | Publish signed head vectors first; defer Tessera proofs | Proposed | 2026-10-18 | +| [0005](0005-root-certified-keys.md) | Delegate an offline root to a scoped local epoch-key issuer | Proposed | 2026-10-18 | +| [0006](0006-receipt-failure-modes.md) | Implement fail-closed first; defer spooling pending a named failure class | Proposed | 2026-10-18 | | [0007](0007-kubernetes-lifecycle.md) | Use StatefulSet identities and isolated WAL volumes | Proposed | 2026-10-18 | ## Current implementation @@ -40,9 +40,37 @@ The [Phase 0 benchmark PR](https://github.com/Clawdlinux/agentgate/pull/50) cont Its reported latency gate is not met. Phase 2 must not claim that extra durable writes meet that gate. +## Open decisions + +The owner must approve these choices before dependent features are enabled. +Defaults below are recommendations, not shipped configuration. +All certificate and epoch changes remain subject to the frozen v1 compatibility boundary. + +| Decision | Recommended default | Related ADRs | +| --- | --- | --- | +| V2 protocol approval | Assign a protocol owner; approve binary preimages and fixtures before emitting v2. | [0001](0001-per-replica-chains.md), [v2](../spec/RECEIPT-v2.md) | +| Fencing backend and timing | Kubernetes Lease with registry-allocated tokens; renew every 5 seconds within a 15-second lease. Require storage fencing before takeover. | [0001](0001-per-replica-chains.md), [0007](0007-kubernetes-lifecycle.md) | +| Issuer custody and renewal | Offline root; local HSM or Vault transit for the intermediate. Renew annually; target epoch rotation every 24 hours. | [0005](0005-root-certified-keys.md) | +| Revocation scope and freshness | Intermediate-signed status every 6 hours, valid for freshness checks for 24 hours. Use explicit epoch or sequence cutoffs. | [0005](0005-root-certified-keys.md) | +| SQLite storage model | File per epoch, enabling whole-file archival. A single database instead needs an approved append-only-preserving reclamation migration. | [0003](0003-durable-shipping.md) | +| Evidence retention | Propose 30 days locally and 365 days in verified archive. Never delete unshipped evidence or required trust material. | [0003](0003-durable-shipping.md) | +| Publication profile | Signed head vectors and inventories first. Enable Tessera only after compact-proof demand and offline backend qualification. | [0004](0004-merkle-checkpoints.md) | +| Publisher custody and failover | Separate protected publisher keys; 1 active writer with storage-fenced standby. Refuse unproven takeover. | [0004](0004-merkle-checkpoints.md) | +| Accepted unshipped-loss risk | Target 10-second healthy shipping. Require explicit acceptance of asynchronous suffix loss; do not claim zero-loss durability. | [0003](0003-durable-shipping.md) | +| Replica authorization and vault state | Keep replicas at 1 until local provisioning, credential rotation, and revocation propagation pass cross-replica tests. | [0007](0007-kubernetes-lifecycle.md) | +| Client retry semantics | Bind client request IDs now. Defer retry deduplication and provider idempotency-key mapping; never automatically replay unknown outcomes. | [0002](0002-two-phase-receipts.md) | +| Spool failure class | Keep `fail_closed` and reject `fail_open_spool`. Reconsider only after a named fault and independent durability test exist. | [0006](0006-receipt-failure-modes.md) | +| Added-latency budget | Retain the under-1-ms p99 objective pending owner review. Claim no pass without dedicated Linux measurements. | [0002](0002-two-phase-receipts.md) | +| Long-stream rollout policy | Keep the 40-second drain budget. Document stream cuts and expected unknown termination instead of promising uninterrupted SSE. | [0007](0007-kubernetes-lifecycle.md) | +| Witness policy | Disabled by default. Require explicit local quorum policy before making witness-backed anti-equivocation claims. | [0004](0004-merkle-checkpoints.md) | + +Issuer software and certificate-format approval do not solve replica-state distribution. +Multi-replica AgentGate remains blocked until that separate state problem is solved. +Untrusted clocks cannot prove status freshness; report unknown unless verifier time or a status revision is independently trusted. + ## Review boundaries Every decision remains proposed until owner review. Implementation notes describe future acceptance tests, not completed validation. Each ADR records tradeoffs and links its dependencies. -The sequencing document lists remaining owner decisions. +The [sequencing document](0000-phase2-sequencing.md) separates prerequisites from deferred features. diff --git a/docs/spec/RECEIPT-v2.md b/docs/spec/RECEIPT-v2.md index 22c1cf3..9cf9943 100644 --- a/docs/spec/RECEIPT-v2.md +++ b/docs/spec/RECEIPT-v2.md @@ -23,7 +23,8 @@ Do not update v1 encoding locks to accommodate v2. Never re-encode historical v1 receipts during migration or export. Retain original v1 JSONL exports and their independently trusted key history. -A proposed root-signed migration descriptor may bind their exact payload digest and terminal head. +A migration descriptor binds original export digests, terminal head, legacy key history, and destination v2 chain. +The offline root and final v1 key co-sign the same preimage. It does not rewrite old receipts or manufacture authenticated v1 chain IDs. V2 and legacy manifests require separate explicit version dispatch. @@ -33,7 +34,7 @@ Propose required authenticated `chain_id = :`. `replica-id` is a lowercase provisioned UUID. `epoch` is positive decimal without leading zeros. Persist both; never derive cryptographic identity solely from a reusable Kubernetes ordinal. -Each chain starts at sequence 1 with a zero previous hash. +The epoch-open control record occupies sequence 1 with a zero previous hash. Within that chain, sequence and previous-hash checks remain mandatory. No global ordering between replica chains is implied. @@ -41,17 +42,23 @@ Proposed epoch-open records reference the preceding terminal hash and segment di Key rotation creates a new epoch. An intact restart resumes its current epoch. Lost or restored storage requires fencing and a new replica UUID. +An external epoch lease and monotonic fencing token are checked before admission and each writer batch. +Bind that token into signed records and descriptors. +The issuer refuses duplicate `(UUID, epoch)` certification, but a copied certificate still requires lease-based fencing. +Offline coverage compares inventories with the intermediate-signed replica registry and issued-certificate log. +Registered chains with neither epoch-close nor inventory coverage are reported lost, even when no receipt survives. +Empty replacement PVCs require explicit issuer enrolment and a registry entry replacing the earlier UUID for that ordinal. ## 3. Proposed Record Kinds -All records authenticate their kind, chain identity, sequence, previous hash, timestamp, and signing key. +All records authenticate their kind, chain identity, fencing token, sequence, previous hash, timestamp, and signing key. Required action fields and optional result fields must be specified per kind. Do not copy v1's required HTTP status into an intent with a fabricated value. | Kind | Proposed authenticated content | | --- | --- | -| `intent` | Attempt ID, principal, agent, delegation, destination action, params digest, policy | -| `outcome` | Full intent reference, observation class, known status, latency, bounded error code | +| `intent` | Attempt ID, optional client request ID, principal, agent, delegation, destination action, params digest, policy | +| `outcome` | Full intent reference, observation, upstream attempt count, known status, latency, bounded error code | | `stream_result` | Header-outcome reference, available byte digest and count, termination classification | | `decision` | Authenticated denied or rate-limited attempt without upstream dispatch | | `control` | Typed mode changes or epoch boundaries with actor, reason, and bound configuration | @@ -60,7 +67,12 @@ An intent reference includes chain ID, sequence, entry hash, and attempt ID. Every attempt ID is gateway-generated and unique within its replica identity. Specify its exact size and encoding before implementation. Unknown outcomes use `outcome_unknown`, without invented provider status or successful completion. +Runtime-certain `not_dispatched` requires a known zero action-dispatch count. +Crash recovery cannot infer that state from an orphan intent. +Count action transport retries in `upstream_attempt_count`; unknown counts remain explicitly absent. Propose a distinct observation field rather than extending v1's frozen policy enum. +Bind any validated client request ID into the intent. +Client retry deduplication and upstream `Idempotency-Key` mapping remain deferred. SSE outcomes attest header acceptance before headers are forwarded. Supplemental terminal records bind SHA-256 of bytes accepted by downstream writes. @@ -68,6 +80,8 @@ They do not prove client delivery. Recovery emits unknown outcomes or unknown stream termination without replaying upstream calls. References may cross epoch boundaries. Duplicate or contradictory terminal records must be rejected. +Draining streams after 40 seconds may produce unknown termination with reason `drain_deadline`. +That expected interruption is not a successful stream outcome. The [earlier stream proposal](proposals/receipt-stream-digest.md) remains unimplemented. This outline proposes replacing its single-action-receipt rule for v2 lifecycle records. @@ -88,20 +102,64 @@ No placeholder encoding in this outline authorizes production emission. ## 5. Keys and Export Artifacts -Propose root-certified replica keys scoped to exactly 1 replica UUID and epoch. -Certificates bind purpose, public key, and validity under a versioned minimal signed statement. -A separately scoped log certificate authorizes checkpoint signatures for a specific origin. -Preserve v1 legacy trust behavior outside an explicit migration envelope. +Propose an offline organization root delegating to a longer-lived, in-cluster intermediate issuer. +The root delegation binds organization, installation, allowed purposes, and no further intermediate delegation. +The issuer certifies epoch keys and records each issuance in the signed replica registry. +Use v1-style length-prefixed binary certificate preimages, not canonical JSON. +Freeze domain strings, field order, `U32LE` lengths, `U64LE` integers, raw keys, and absence markers. +Ed25519 signs the SHA-256 certificate preimage digest. + +| Purpose | Authorized scope and signer | +| --- | --- | +| `certificate_issuance` | Offline-root-delegated intermediate; permitted installation and child purposes only | +| `replica_registry` | Intermediate; registrations, issued certificates, fencing generations, replacements, and losses | +| `receipt` | Epoch key; action lifecycle records in exactly 1 UUID and epoch | +| `control` | Epoch key; epoch boundaries, authenticated mode changes, and future sink-transition markers | +| `segment_descriptor` | Epoch key; its epoch's signed segment ranges | +| `legacy_segment` | Intermediate-certified archival publisher; migrated v1 export scope | +| `inventory` | Intermediate-certified singleton publisher; cross-replica inventories and head vectors | +| `checkpoint` | Separately certified log key; installation and checkpoint origin | +| `trust_status` | Intermediate; delegated-key status and explicit sequence or epoch revocation cutoffs | +| `migration` | Offline root plus final v1 signing key; mandatory co-signatures | + +Epoch and sequence bounds govern validity, not receipt clock values. +The certificate binds exactly 1 epoch starting at sequence 1. +Signed epoch-close fixes its inclusive terminal sequence. +Verifiers reject later records and require signed inventory coverage of close and final descriptor for terminal claims. +When Merkle mode is enabled, also prove epoch-close inclusion against its checkpoint. +Without that coverage, report an unclosed prefix or missing terminal evidence, not a complete chain. +Time windows are operational renewal hygiene only. +Recommend annual intermediate renewal and 24-hour epoch rotation. +Do not reject historical evidence merely because its time window elapsed. + +Rotate at a writer batch boundary without draining admission or inflight calls. +Durably seal the old epoch's final descriptor before destroying its key. +The new epoch starts with epoch-open; later outcomes may reference prior-epoch intents. +Registry and activation recovery must prevent duplicate issuance or competing active epochs. + +Recommend intermediate-signed trust-status statements every 6 hours, with a 24-hour freshness limit. +Bind revision, predecessor digest, affected certificates, and explicit revocation cutoffs. +Freshness requires trusted verifier time or a separately pinned revision; otherwise report freshness unknown. +Stale or missing status cannot produce a current-trust success claim. +Root-signed status handles intermediate replacement. +Preserve v1 trust behavior outside the co-signed migration envelope. Proposed sealed segments contain original receipt bytes and signed range descriptors. Descriptors bind payload digest, chain scope, anchor, terminal hash, predecessor segment, and certificate digests. Specify export-container versions independently of receipt versions. Offline bundles include required certificates and signed trust-status information. +They also include registry history and the signed inventory or head vector. Their trust anchor is an independently pinned organization root, never an embedded self-asserted root. +Descriptors and inventories require conditional immutable creation; filesystem export requires exclusive creation and fsync. +The initial failure mode is `fail_closed`, returning 503 when required durable recording fails. +Spooling remains deferred until its covered failure class and external sink-transition marker protocol are approved. ## 6. Transparency Proof Profile -Propose C2SP [tlog-checkpoint](https://c2sp.org/tlog-checkpoint@v1.1.0) +Recommend signed head vectors and inventories first. They do not provide compact Merkle inclusion proofs. +The fenced singleton publisher validates segment certificate paths against the independently pinned root. +Its durable predecessor state and storage fencing are required during failover. +For the later Tessera profile, propose C2SP [tlog-checkpoint](https://c2sp.org/tlog-checkpoint@v1.1.0) and [tlog-tiles](https://c2sp.org/tlog-tiles@v1.0.0) with SHA-256. Freeze leaf bytes binding receipt version, authenticated chain scope, sequence, and entry hash. Legacy scope comes from its signed descriptor, not modified v1 receipts. @@ -115,11 +173,11 @@ Witness quorum remains optional. 1. Approve a complete byte-level specification and deterministic vectors for each record and artifact kind. 2. Preserve all v1 vector outcomes and byte locks under dual-version verification. 3. Reject mismatched chain identity, duplicate terminal references, conflicting segments, and unauthorized certificate purposes. -4. Run crash recovery across intent, outcome, spool, sealing, and epoch rotation boundaries. -5. Verify inclusion and consistency proofs with no network or private keys. -6. Document clock limitations, revocation freshness, retention, and owner-approved signing lifetimes. +4. Run crash recovery across intent, outcome, fencing, sealing, and batch-boundary epoch rotation. +5. Verify vectors offline; gate later Merkle mode on offline inclusion and consistency fixtures. +6. Document registry coverage, revocation freshness, storage-model retention, and certificate renewal targets. -Open questions include certificate renewal policy, accepted publication lag, and archival retention. +The [open decisions](../design/README.md#open-decisions) record recommended defaults. Proof freshness requires a separately trusted checkpoint or time policy. No artifact alone proves that every external side effect was recorded. @@ -129,6 +187,6 @@ No artifact alone proves that every external side effect was recorded. - [0002: Intent and outcome](../design/0002-two-phase-receipts.md) - [0003: Sealed segments](../design/0003-durable-shipping.md) - [0004: Merkle checkpoints](../design/0004-merkle-checkpoints.md) -- [0005: Root-certified keys](../design/0005-root-certified-keys.md) +- [0005: Scoped issuer and epoch keys](../design/0005-root-certified-keys.md) - [0006: Failure modes](../design/0006-receipt-failure-modes.md) - [0007: Kubernetes lifecycle](../design/0007-kubernetes-lifecycle.md) From 966fd107f0787fc91c36dbc9e91023185d47bbb3 Mon Sep 17 00:00:00 2001 From: Shreyansh Sancheti <43677304+shreyanshjain7174@users.noreply.github.com> Date: Sun, 18 Oct 2026 06:46:47 +0530 Subject: [PATCH 3/3] docs(design): clarify lease and loss cutoffs Constrain takeover evidence to covered heads. Keep lease checks local and leave latency gate selection to the owner. Signed-off-by: Shreyansh Sancheti <43677304+shreyanshjain7174@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/design/0001-per-replica-chains.md | 20 +++++++++++++++----- docs/design/0005-root-certified-keys.md | 8 ++++++++ docs/design/0007-kubernetes-lifecycle.md | 7 ++++++- docs/design/README.md | 3 ++- docs/spec/RECEIPT-v2.md | 10 +++++++++- 5 files changed, 40 insertions(+), 8 deletions(-) diff --git a/docs/design/0001-per-replica-chains.md b/docs/design/0001-per-replica-chains.md index 0e15d82..a1cc304 100644 --- a/docs/design/0001-per-replica-chains.md +++ b/docs/design/0001-per-replica-chains.md @@ -40,20 +40,29 @@ Persist a positive, monotonically increasing `epoch` with replica identity. The signed epoch-open control record is exactly `seq = 1`, with a zero previous hash. The uniqueness key becomes `(chain_id, seq)`. -Intact restarts resume their epoch. Rotation starts another. +Intact restarts resume their epoch unless a loss or takeover cutoff requires another. +Rotation starts another epoch. Lost storage requires a new UUID, even for reused ordinals. There is no total cross-replica execution order. ### Fencing and Replica Authority -Propose an external epoch lease, checked before admission and every writer batch. +Propose an external epoch lease with a cached ownership view checked before admission and every writer batch. Use a Kubernetes Lease by default. Non-Kubernetes installations may use conditional object updates on an epoch-open lease. +Renew in the background. Never call the API server per batch. +Use a monotonic self-fencing deadline, approximately 10 seconds with 5-second renewals and a 15-second lease. +The deadline must be shorter than lease duration minus maximum clock drift. +Measure from the start of the last successful renewal attempt, not its delayed acknowledgment. +Deadline expiry stops admission and signing, including batches still waiting for SQLite. The authoritative registry allocates monotonically increasing fencing tokens through compare-and-swap. Lease acquisition requires the current workload identity, not merely a copied receipt certificate. Include the token in epoch-open, subsequent records, and segment descriptors. -Lease loss or an uncertain renewal stops admission and further signing. -Registry handoffs bind the recovered head and token cutover boundary after storage fencing. -Publishers reject stale writes beyond that boundary, not legitimate historical receipts from the earlier token. +Confirmed lease loss fences immediately; failed renewals never extend the cached deadline. +Loss declarations and lease takeovers for the same UUID require an intermediate-signed `trust_status` cutoff at the last covered head. +Bind chain ID, sequence, and head hash; without coverage, use sequence 0 and the zero hash. +Verifiers reject every old-epoch record beyond that cutoff, including an unshipped local suffix. +Persist the cutoff before activating takeover; continue under a newly certified epoch, never beyond the old cutoff. +Registry handoffs bind that covered head and token boundary after storage fencing. Already-dispatched upstream calls cannot be recalled by a lease. The intermediate issuer refuses to certify the same `(replica UUID, epoch)` twice. @@ -118,6 +127,7 @@ Epoch resets could hide missing history. Require explicit predecessor references Acceptance: concurrent replicas can both emit sequence 1 without collision. 4. Test concurrent lease acquisition, stale tokens, renewal loss, and copied certificates. Acceptance: only the fenced owner admits work; duplicate certificate issuance is refused. + Acceptance: cached checks make no per-batch API calls; monotonic deadline expiry fences despite wall-clock changes. 5. Remove every segment of a registered chain. Acceptance: offline coverage reports it lost, rather than silently omitting it. 6. Interrupt migration and verify both co-signatures. diff --git a/docs/design/0005-root-certified-keys.md b/docs/design/0005-root-certified-keys.md index 6698770..48a6802 100644 --- a/docs/design/0005-root-certified-keys.md +++ b/docs/design/0005-root-certified-keys.md @@ -89,6 +89,9 @@ Elapsed time alone cannot invalidate historical evidence. Propose `agentgate keys rotate` at a batch boundary. Keep admission open; bounded queues may briefly wait. +Obtain the next certificate before closing the current epoch. +If the intermediate is unavailable, remain in the current epoch and alert instead of beginning rotation. +Continued signing still requires valid lease ownership and no applicable trust-status cutoff. After the current batch, append epoch-close and durably seal the final segment descriptor. Retain the old key until descriptor and directory fsync. Durably activate the next certified epoch and write epoch-open at sequence 1 before queued work. @@ -103,6 +106,10 @@ Current `Rotate(atSeq)` needs epoch-aware changes; calling `Rotate(1)` is not va Revocation policy remains an [owner decision](README.md#open-decisions). Recommend intermediate-signed status every 6 hours; `next_update` is 24 hours after issuance. Bind revision, predecessor digest, certificate IDs, and revoked epoch or sequence cutoffs. +Every registry loss declaration or same-UUID lease takeover must issue a cutoff at the last covered head. +Bind the affected epoch, terminal sequence, and head hash; verifiers reject records beyond that bound. +Without coverage, sequence 0 rejects the entire affected epoch. +Publish the cutoff before takeover activation; any continuation requires a newly certified epoch. Offline verification reports bundled status as-of time. Freshness needs trusted verifier time or a separately pinned status revision. Otherwise report freshness unknown, not unseen revocation checks. @@ -142,6 +149,7 @@ Coordinate manifest hardening without changing frozen v1 encoding. Acceptance: out-of-bounds records fail; missing terminal coverage cannot pass as complete. 4. Test stale, missing, and revoked trust-status bundles without networking. Acceptance: report exact revocation scope and unknown freshness without fabricating clock-based trust. + Acceptance: loss and takeover reject old-epoch suffixes beyond covered heads; issuer outages preserve the current epoch. ## Related ADRs diff --git a/docs/design/0007-kubernetes-lifecycle.md b/docs/design/0007-kubernetes-lifecycle.md index c7193c5..3b09b0d 100644 --- a/docs/design/0007-kubernetes-lifecycle.md +++ b/docs/design/0007-kubernetes-lifecycle.md @@ -49,9 +49,13 @@ Without coverage, report it lost with no known head. Intact restarts require recovery and lease checks before resuming. Retain PVCs on scale-down. Audit explicit deletion. -Require [0001's external lease and token](0001-per-replica-chains.md) before admission and each batch. +Check [0001's cached lease view and token](0001-per-replica-chains.md) before admission and each batch, without per-batch API calls. +Background renewal sets a monotonic self-fencing deadline, approximately 10 seconds with the 5-second/15-second defaults. +Keep that deadline shorter than lease duration minus maximum clock drift; self-fence when it expires. Kubernetes Lease expiry alone does not fence a partitioned node. Force-detach requires proven old-node storage fencing, then lease takeover with a newer token. +Loss declarations and same-UUID takeovers require a `trust_status` cutoff at the last covered head before activation. +Verifiers reject the affected epoch's suffix beyond that head; a takeover must use a newly certified epoch. Unproven fencing blocks takeover. The only proposed gateway-owned writable state is its WAL data volume. @@ -129,6 +133,7 @@ Spread constraints can block scheduling. Document installation-specific degraded 3. Partition a node with active upstream work, expire its lease, then request CSI force-detach. Attempt takeover before and after proving storage fencing. Acceptance: unfenced takeover fails; replacement tokens increase; the old writer cannot admit or publish. + Acceptance: renewal failure self-fences on monotonic deadline; takeover status rejects records beyond the last covered head. 4. Replace its PVC with an empty one. Acceptance: enrolment gates readiness; registry records replacement and old-chain loss. 5. Drain an SSE stream longer than 40 seconds. diff --git a/docs/design/README.md b/docs/design/README.md index c008550..a1fb86d 100644 --- a/docs/design/README.md +++ b/docs/design/README.md @@ -50,6 +50,7 @@ All certificate and epoch changes remain subject to the frozen v1 compatibility | --- | --- | --- | | V2 protocol approval | Assign a protocol owner; approve binary preimages and fixtures before emitting v2. | [0001](0001-per-replica-chains.md), [v2](../spec/RECEIPT-v2.md) | | Fencing backend and timing | Kubernetes Lease with registry-allocated tokens; renew every 5 seconds within a 15-second lease. Require storage fencing before takeover. | [0001](0001-per-replica-chains.md), [0007](0007-kubernetes-lifecycle.md) | +| Registry storage backend | Recommend a CRD backed by cluster etcd for air-gapped Kubernetes. Conditional object storage is an alternative; filesystem export supports offline inspection, not live lease authority. | [0001](0001-per-replica-chains.md) | | Issuer custody and renewal | Offline root; local HSM or Vault transit for the intermediate. Renew annually; target epoch rotation every 24 hours. | [0005](0005-root-certified-keys.md) | | Revocation scope and freshness | Intermediate-signed status every 6 hours, valid for freshness checks for 24 hours. Use explicit epoch or sequence cutoffs. | [0005](0005-root-certified-keys.md) | | SQLite storage model | File per epoch, enabling whole-file archival. A single database instead needs an approved append-only-preserving reclamation migration. | [0003](0003-durable-shipping.md) | @@ -60,7 +61,7 @@ All certificate and epoch changes remain subject to the frozen v1 compatibility | Replica authorization and vault state | Keep replicas at 1 until local provisioning, credential rotation, and revocation propagation pass cross-replica tests. | [0007](0007-kubernetes-lifecycle.md) | | Client retry semantics | Bind client request IDs now. Defer retry deduplication and provider idempotency-key mapping; never automatically replay unknown outcomes. | [0002](0002-two-phase-receipts.md) | | Spool failure class | Keep `fail_closed` and reject `fail_open_spool`. Reconsider only after a named fault and independent durability test exist. | [0006](0006-receipt-failure-modes.md) | -| Added-latency budget | Retain the under-1-ms p99 objective pending owner review. Claim no pass without dedicated Linux measurements. | [0002](0002-two-phase-receipts.md) | +| Added-latency budget | The owner must re-set the gate. Phase 0 single-phase p99 already exceeds 1 ms at c16: 3.42 ms ([PR #50](https://github.com/Clawdlinux/agentgate/pull/50)). Recommend a measured, concurrency-qualified budget after the tail-latency investigation lands. | [0002](0002-two-phase-receipts.md) | | Long-stream rollout policy | Keep the 40-second drain budget. Document stream cuts and expected unknown termination instead of promising uninterrupted SSE. | [0007](0007-kubernetes-lifecycle.md) | | Witness policy | Disabled by default. Require explicit local quorum policy before making witness-backed anti-equivocation claims. | [0004](0004-merkle-checkpoints.md) | diff --git a/docs/spec/RECEIPT-v2.md b/docs/spec/RECEIPT-v2.md index 9cf9943..504ebad 100644 --- a/docs/spec/RECEIPT-v2.md +++ b/docs/spec/RECEIPT-v2.md @@ -40,9 +40,12 @@ No global ordering between replica chains is implied. Proposed epoch-open records reference the preceding terminal hash and segment digest when available. Key rotation creates a new epoch. -An intact restart resumes its current epoch. +An intact restart resumes its current epoch unless a loss or takeover cutoff requires another. Lost or restored storage requires fencing and a new replica UUID. An external epoch lease and monotonic fencing token are checked before admission and each writer batch. +Use cached ownership and a monotonic renewal deadline, not a per-batch API-server request. +Self-fence after approximately 10 seconds with 5-second renewals and a 15-second lease. +The deadline must be shorter than lease duration minus maximum clock drift. Bind that token into signed records and descriptors. The issuer refuses duplicate `(UUID, epoch)` certification, but a copied certificate still requires lease-based fencing. Offline coverage compares inventories with the intermediate-signed replica registry and issued-certificate log. @@ -133,12 +136,17 @@ Recommend annual intermediate renewal and 24-hour epoch rotation. Do not reject historical evidence merely because its time window elapsed. Rotate at a writer batch boundary without draining admission or inflight calls. +Obtain the next certificate first; intermediate outages leave the replica in its current epoch and trigger alerts. +Existing lease and trust-status bounds still govern continued signing. Durably seal the old epoch's final descriptor before destroying its key. The new epoch starts with epoch-open; later outcomes may reference prior-epoch intents. Registry and activation recovery must prevent duplicate issuance or competing active epochs. Recommend intermediate-signed trust-status statements every 6 hours, with a 24-hour freshness limit. Bind revision, predecessor digest, affected certificates, and explicit revocation cutoffs. +Registry loss declarations and same-UUID lease takeovers require a cutoff at the last covered epoch head. +Verifiers reject every old-epoch record beyond its sequence and hash boundary; uncovered epochs use a sequence-0 cutoff. +Persist that status before takeover activation and use a newly certified epoch for continuation. Freshness requires trusted verifier time or a separately pinned revision; otherwise report freshness unknown. Stale or missing status cannot produce a current-trust success claim. Root-signed status handles intermediate replacement.