diff --git a/.gitignore b/.gitignore index 87f7bb0..f5ca8fc 100644 --- a/.gitignore +++ b/.gitignore @@ -58,3 +58,6 @@ functions/.env.* # Personal Claude Code permissions with absolute local paths .claude/settings.local.json + +# Generated by the periodic repo-scout agent +NEXT-STEPS.md diff --git a/NEXT-STEPS.md b/NEXT-STEPS.md deleted file mode 100644 index 10a7f1b..0000000 --- a/NEXT-STEPS.md +++ /dev/null @@ -1,11 +0,0 @@ -# Next steps - -**As of 2026-10-03 UTC, item 2 updated 2026-10-10.** Derived from [the canonical roadmap](metacortexplan.md). - -1. Close BASE #68 and VERIFY #69 on the board. DOC #67 is accepted, with the evidence in [the acceptance record](metacortexplan.md#2026-10-03-utc-foundation-acceptance-doc-67). The baseline is on `release` at `97bc073`, [hosted CI passed](https://github.com/chrishonson/Metacortex/actions/runs/37097980532), and a fresh clone passed all three gates. BASE's preserved-work inventory is in [the baseline record](docs/operations/2026-10-01-adoption-baseline.md). Preserve completed work and the independent gate definitions. -2. CORE #70 landed in PR #14 (`4e2d337`) and is deployed in revision `metacortexmcp-00031-dig`; see [the CORE record](metacortexplan.md#2026-10-10-utc-core-70-acceptance-record). RECOVERY #71 follows BASE/VERIFY. TIERING #83 stays deferred. -3. Continue the dependency graph in the roadmap, recording exact source, gate and live evidence as each card completes. - -Card 66 is **done**: PR #12 merged, the new write contract deployed, and all nine scoped profiles plus admin passed save/search/fetch and old-name rejection. See [the rollout record](docs/operations/2026-10-03-card66-rollout.md). Long-running clients may need a tool-catalog refresh. - -Execute selected work in this session; do not activate unrelated workers. Publication #84 retains its required review block. Product deployment, release, recovery and UI acceptance remain separate evidence requirements for their respective cards. diff --git a/README.md b/README.md index 509d9ba..daa6c53 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ MetaCortex is a serverless MCP memory service backed by Firestore vector search ## Status and roadmap -MetaCortex is already released (repository tag `v0.3.0`). This checkout contains a reconciled local baseline; publication and production deployment are separate operations. See the [baseline record](docs/operations/2026-10-01-adoption-baseline.md), [unified roadmap](metacortexplan.md), and [immediate next steps](NEXT-STEPS.md). +MetaCortex is already released (repository tag `v0.3.0`). This checkout contains a reconciled local baseline; publication and production deployment are separate operations. See the [unified roadmap](metacortexplan.md). The browser setup wizard, owner login/OAuth, management dashboard, and ChatGPT extension are planned, not available in this baseline. Software is MIT-licensed; owners pay their own Firebase/model usage and agent subscriptions. One installation holds one owner's shared corpus; profiles do not provide tenant isolation. diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 3bcdceb..cc65c6f 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -2,7 +2,7 @@ ## Current implementation -MetaCortex is an already-released, serverless user-memory service: Express and the MCP SDK run in Firebase Cloud Functions 2nd Gen; Firestore stores canonical memories and vectors. The [roadmap](../metacortexplan.md) is authoritative for future work. The [baseline record](operations/2026-10-01-adoption-baseline.md) distinguishes local integration from deployment evidence. +MetaCortex is an already-released, serverless user-memory service: Express and the MCP SDK run in Firebase Cloud Functions 2nd Gen; Firestore stores canonical memories and vectors. The [roadmap](../metacortexplan.md) is authoritative for future work. ```mermaid flowchart TD diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index 49ad2dd..fe2cc5a 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -1,6 +1,6 @@ # Deployment and operator playbook -This guide describes the current local baseline, not the planned browser installer. See the [unified roadmap](../metacortexplan.md), [baseline evidence](operations/2026-10-01-adoption-baseline.md), and [security boundaries](SECURITY.md). MetaCortex is already released. Card 66 subsequently deployed on 2026-10-03 UTC; see the [rollout record](operations/2026-10-03-card66-rollout.md). +This guide describes the current local baseline, not the planned browser installer. See the [unified roadmap](../metacortexplan.md) and [security boundaries](SECURITY.md). MetaCortex is already released. ## Current contract @@ -82,7 +82,7 @@ firebase deploy --project --only functions The Functions predeploy hook builds the source. The alias selects `.env.`; passing a raw project ID may load a different dotenv file. Always verify effective non-secret settings. -Existing production Secret Manager migration is complete: [card-23 record](operations/2026-09-09-secret-migration.md). Do not repeat the plaintext migration or rotate credentials merely because deployment docs changed. +Existing production Secret Manager migration completed on 2026-09-09. Do not repeat the plaintext migration or rotate credentials merely because deployment docs changed. ## TTL and retention diff --git a/docs/SECURITY.md b/docs/SECURITY.md index 5e02a66..757aef6 100644 --- a/docs/SECURITY.md +++ b/docs/SECURITY.md @@ -7,7 +7,7 @@ This document describes current implementation and accepted policy. Planned cont - Dedicated client tokens, tool allowlists, readable-state allowlists, and per-profile origin rules. - Separate admin endpoint and three-tool ordinary-agent profiles; `list_context` requires an explicit grant. - Firestore rules deny direct client access to memory, audit, fingerprint, and evaluation collections. Admin SDK access uses IAM. -- Runtime secrets are bound through Secret Manager. The [2026-09-09 migration](operations/2026-09-09-secret-migration.md) is complete; it did not rotate credentials. +- Runtime secrets are bound through Secret Manager. The 2026-09-09 migration is complete; it did not rotate credentials. - Requests have a 1 MB JSON limit, including base64 image input. Raw image assets are not stored. - Full retrieval-query telemetry is opt-in. Audit summaries can still include a query preview and error metadata; treat them as private data. diff --git a/docs/operations/2026-09-09-secret-migration.md b/docs/operations/2026-09-09-secret-migration.md deleted file mode 100644 index 18341af..0000000 --- a/docs/operations/2026-09-09-secret-migration.md +++ /dev/null @@ -1,37 +0,0 @@ -# Card-23: production secret migration - -As of 2026-09-09 UTC, production revision `metacortexmcp-00028-pul` is ACTIVE -and binds version 1 of `MCP_ADMIN_TOKEN`, `GEMINI_API_KEY`, and -`MCP_CLIENT_PROFILES_JSON` from Secret Manager in `my-brain-88870`. -None of those keys remains in `serviceConfig.environmentVariables`. - -The existing values were transferred in memory directly to Secret Manager; -no credential bytes were printed or written to a temporary file. The original -checkout's `functions/.env.prod` had the three secret entries removed after -production verification. The isolated deployment used the existing non-secret -production settings, preserving model and collection configuration. - -## Verification - -- 82 tests passed with coverage; TypeScript build passed. -- Deployment preflight passed, including enabled Secret Manager versions. -- Synthetic preflight check rejected a plaintext dotenv credential without - printing the synthetic value. -- Authenticated MCP `tools/list` succeeded for admin and all nine scoped clients: - chatgpt-web, claude-web, openclaw, antigravity-ide, claude-code-windows, - claude-code-mac, gemini-spark, grok-bot, grok. -- A read-only `search_context` request through the grok client succeeded, - exercising the injected Gemini key and retrieval path. No memory was created. -- Production function metadata confirmed all three secret references and no - remaining plaintext secret keys. - -## Completed scope - -Nick explicitly removed rotation from card-23 on 2026-09-09 UTC ("no rotation"). -The completed scope is storage migration with existing credential values -preserved. Credential rotation and client reconfiguration are not outstanding -requirements for this card. No rotation was performed. - -Future deployments must retain the secret bindings in this branch. Do not rerun -the original plaintext migration against the already-migrated deployment. -Rollback must not restore ordinary environment variables containing credentials. diff --git a/docs/operations/2026-10-01-adoption-baseline.md b/docs/operations/2026-10-01-adoption-baseline.md deleted file mode 100644 index 170be10..0000000 --- a/docs/operations/2026-10-01-adoption-baseline.md +++ /dev/null @@ -1,61 +0,0 @@ -# Self-hosted adoption baseline - -Started 2026-10-01; continued 2026-10-02. Scope: reconcile documentation, record the dependency-linked backlog without activation, and establish a reviewed local baseline. The [canonical roadmap](../../metacortexplan.md) governs future implementation. - -## Preserved and integrated work - -| Evidence | Disposition | -|---|---| -| Released `v0.3.0` and release branch `08d144f` | Existing product retained; no invented 1.0 milestone. | -| Initial tracked working-tree patch | Preserved privately before integration; local QA configuration retained and ignored. | -| `dde9099`, card 66 | Local write contract renamed to `save_context` without an alias. Deployment/profile migration remains unverified. | -| Archive worktree `932f730`, cards 58–65; integration `d54ca20` | Portable archives, dynamic top-level backup, restore, inventory helpers and tests integrated. Historical QA evidence retained. No live restore repeated. | -| Card 62 followed by cards 64–65 | The earlier four-collection backup was superseded by dynamic top-level discovery; neither proves recursive or content-equal recovery. | -| Card 23 and the secret-migration record | Completed Secret Manager migration retained. Credential rotation was explicitly excluded. | -| Evaluation worktree `c3d17d2`, card 45 | Separate work preserved with its publication restrictions. Existing retrieval evaluation remains in the baseline. | -| Partial trace work, card 30 | Historical incomplete work preserved; no new trace feature introduced. | -| Studies and personal settings committed in `d54ca20` | Preserved in Git under the tag `archive/ci-recurrence-data` (the baseline branch was deleted on 2026-10-10); excluded from the distributable Journey allowlist. | - -## Incoming review before pull - -Reviewed the three upstream documentation commits between `d54ca20` and `82cb302`, then fast-forwarded the clean branch. No executable code changed in those incoming commits. - -Resolved findings: - -- `NEXT-STEPS.md` excluded card posting despite the authorized backlog scope; restored the roadmap's posting rules. -- Journey setup entered secrets before selecting the target and copied credential placeholders into production dotenv. Instructions now select a project first, omit credential keys entirely, and specify the project on secret commands. -- Journey metadata still recommended URL credentials and its README retained the old preview-model default. Aligned both with the current code and documented legacy-auth transition. -- Slides overstated read-state allowlists as restrictions on all memory operations. Clarified the current write-state gap. -- Deployment docs referenced a missing baseline record and script check. Added this record and a runnable check. - -## Verification and distribution baseline - -`verification.json` and `scripts/run-gate.py` declare and execute the typecheck, test, and package gates. CI uses those gate IDs on Node 22, matching the Functions engine. The script compiler includes operator TypeScript without emitting it into the deployed function. - -The new check exposed accesses to an SDK project-ID property absent from its public TypeScript declaration. A checked accessor preserves the target guard and refuses an unverifiable target. Regression coverage checks the installed SDK, production-write refusal, and missing-project refusal without network operations. - -The Journey allowlist now includes imported runtime modules, package-referenced scripts, tests, the lockfile, and current operating docs. Its package gate extracts only those files, installs dependencies independently, and runs tests/build/script checks. Live smoke credentials are removed from that check's environment. No artifact is published by verification. - -Verified on 2026-10-02: - -- 117 tests passed across 10 files, including MCP loopback integration tests and three target-guard regression cases. -- Functions build and operator TypeScript checks passed. -- Independent Journey extraction installed 428 locked packages and passed the same 117 tests, build, and script checks under Node 22.22.2 (69 allowlisted source/document files, two examples). A preliminary local Node 20 run also passed; Node 22 is the supported baseline. -- Maintained-document relative links resolved and `git diff --check` passed. -- Hosted CI, live deployment, model validation, recovery and registry publication were not run. - -## Remaining limits - -Local tests and packaging checks do not verify production, live models, OAuth/client compatibility, an actual cloud restore, or first-time-owner usability. CI configuration is local until pushed and observed running. - -Portable archives preserve known fields and may commit/push when run without protective flags. Full backups discover top-level collections, not subcollections; counts do not prove content equality or snapshot consistency. Write-mode restore can prune and can fail partway through. Existing hardcoded maintainer guards/defaults are not portable installation protections. These remain RECOVERY and DEPLOY work. - -All adoption cards were requested in backlog; 18 remain there. DISTRIBUTE #84 is automatically review-blocked by its outward-facing policy flag. No ready/claimed cards exist, and the review safeguard remains intact. Foundation cards record delivered local evidence and remaining acceptance rather than requesting duplicate implementation. Card 66 retains its remaining deployed-client verification; no production claim is inferred from a local rename. - -## Board evidence - -On 2026-10-02, read back all 19 unique adoption cards (#67–85), their dependency IDs, declared gates, and null leases. Board totals after posting: 84 cards; 24 backlog, 3 blocked, 43 done, 14 abandoned, zero ready/claimed/failed-out. The extra blocked item is publication #84's mandatory review escalation. No existing queue ordering or completed-card states changed. - -## Subsequent execution, 2026-10-03 UTC - -The user authorized pushing and deployments. Baseline `9c3feac` was pushed and [hosted CI passed](https://github.com/chrishonson/Metacortex/actions/runs/37096977416) all declared gates. Card 66 then landed and deployed; see the [completed rollout](2026-10-03-card66-rollout.md). Earlier local-only and pending-rollout statements above are historical, not current blockers. Release integration preserves study/personal data on the original branch while including only the intended product/baseline changes. diff --git a/docs/operations/2026-10-03-card66-rollout.md b/docs/operations/2026-10-03-card66-rollout.md deleted file mode 100644 index ad2c78d..0000000 --- a/docs/operations/2026-10-03-card66-rollout.md +++ /dev/null @@ -1,41 +0,0 @@ -# Card 66: save_context rollout - -Completed 2026-10-03 UTC after explicit authorization to push, merge, update the existing profiles, deploy, and run live checks. - -## Landed source - -[PR #12](https://github.com/chrishonson/Metacortex/pull/12) merged into `release` at `ac0b78fd479b4bf0eb5b551acad1efdee7a59498`. Its tree matches focused source `c89c57c58497638ac0cef60ea732ca0834d1c749`: the rename from `dde9099` plus three remaining roadmap references. No compatibility alias was added. - -The focused checkout passed 82 tests and the TypeScript build on Node 22. The encompassing adoption baseline `9c3feac` separately passed [all three hosted verification gates](https://github.com/chrishonson/Metacortex/actions/runs/37096977416), including 117 tests and clean Journey installation. The optional Claude review action failed without producing a review; manual diff review completed, and no required merge gate was bypassed. - -## Deployment - -- Project/function: `my-brain-88870`, `us-central1/metaCortexMcp`. -- Previous revision: `metacortexmcp-00029-max`. -- Verified ACTIVE revision: `metacortexmcp-00030-zaj`. -- `MCP_CLIENT_PROFILES_JSON` version 2 replaces only the write-tool name in all nine profile allowlists. Tokens, origins, lifecycle grants and other tools are unchanged. Version 1 remains available for deliberate rollback with the old source. -- Admin and Gemini secret bindings remain version 1. No credentials were rotated or written into local files. -- Deployment used an isolated copy of the tested source and the deployed non-secret settings. Model, region, collection and 768-dimensional embedding settings were preserved. No data migration, index change, restore or memory deletion was performed. - -## Live acceptance - -Each endpoint passed fresh MCP discovery, `save_context`, `search_context`, `fetch_context`, and rejection of `remember_context`: - -| Profile | Result | -|---|---| -| chatgpt-web | Passed; configured browser origin checked | -| claude-web | Passed; configured browser origin checked | -| openclaw | Passed | -| antigravity-ide | Passed | -| claude-code-windows | Passed | -| claude-code-mac | Passed | -| gemini-spark | Passed; configured browser origin checked | -| grok-bot | Passed | -| grok | Passed | -| admin | Passed; full six-tool surface | - -All ten identical writes resolved to one operational verification memory, `9cGoQNZXzScCkSRTAsNI`, under topic `metacortex-operations-card66`. The first created it; the remaining nine were duplicates. Every endpoint searched and fetched that same record. No private existing memory content was printed. - -These checks verify fresh MCP sessions at the configured endpoints. Long-running agents may need to reconnect or refresh their tool catalogs after this breaking rename; the previous name is not an alias. They do not establish the future OAuth or ChatGPT UI acceptance matrix. - -Card 66 was released as **done** with merge, gate and deployment evidence. The adoption roadmap continues with foundation acceptance and CORE. diff --git a/docs/superpowers/plans/2026-08-17-memory-checkpoint.md b/docs/superpowers/plans/2026-08-17-memory-checkpoint.md deleted file mode 100644 index 011888a..0000000 --- a/docs/superpowers/plans/2026-08-17-memory-checkpoint.md +++ /dev/null @@ -1,1568 +0,0 @@ -> Historical archive design/implementation record, retained from the completed archive branch. The current roadmap is [metacortexplan.md](../../../metacortexplan.md). Current behavior and limitations are in [MEMORY_ARCHIVE.md](../../MEMORY_ARCHIVE.md) and [FULL_BACKUP.md](../../FULL_BACKUP.md). Old task checkboxes are not the active backlog. - -# Memory Checkpoint Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Archive the MetaCortex memory store to deterministic NDJSON in a private git repository outside GCP, and restore it into Firestore with document ids and retrieval behavior intact. - -**Architecture:** Pure logic lives in `functions/src/archive.ts` (serialization, manifest, guardrails) and `functions/src/archiveFirestore.ts` (Firestore adapters), so vitest covers it under the existing `src/**/*.ts` coverage rule. Two thin `tsx` CLI drivers in `functions/scripts/` handle env loading, argument parsing, and git. Embeddings are never archived — restore regenerates them from `retrieval_text`. - -**Tech Stack:** TypeScript (ES2022, NodeNext), Node 22, firebase-admin 13, vitest 3, tsx. - -**Spec:** [docs/superpowers/specs/2026-08-17-memory-checkpoint-design.md](../specs/2026-08-17-memory-checkpoint-design.md) - -## Global Constraints - -- **Document ids must be preserved on restore.** `metadata.superseded_by` holds document ids; restore writes via `.doc(id).set(...)` with the archived id and never allocates a new one. -- **Restore must not go through `MetaCortexService.remember()`.** That path allocates ids, applies write-fingerprint dedup, and re-derives metadata defaults. -- **Embeddings are never archived.** Restore re-embeds `retrieval_text` with `taskType: "RETRIEVAL_DOCUMENT"` and `title: metadata.module_name`, matching [service.ts:55-59](../../../functions/src/service.ts) exactly. Any deviation changes the vector. -- **Serialization must be byte-deterministic.** Documents sorted by id; object keys emitted in a fixed order at every level; `undefined` fields omitted rather than serialized as `null`. -- **Truncation guardrail:** refuse to write when the new document count is more than 10% below the previous manifest's, unless `--force`. -- **Staleness threshold:** 3 days. -- **`schema_version` is `1`.** -- **Excluded collections:** `memory_events`, `memory_vectors_write_fingerprints`. -- **Flag convention:** writes to the local archive happen by default (`--dry-run` to inspect); writes to Firestore require an explicit `--write`, matching `backfill-firestore-provenance.mjs`. -- New source files are TypeScript under `functions/src/`; CLI drivers are TypeScript under `functions/scripts/` run via `tsx`, following `scripts/retrieval-eval.ts`. -- Tests are `functions/test/*.test.ts` (vitest `include: ["test/**/*.test.ts"]`). - -## File Structure - -| File | Responsibility | -|------|----------------| -| `functions/src/archive.ts` | Create. Pure logic: archive types, canonical key ordering, NDJSON serialize/parse, manifest construction, truncation guardrail, embedding-mismatch description, restore orchestration over a port. | -| `functions/src/archiveFirestore.ts` | Create. Firestore adapters: read collection into `ArchivedMemory[]` (dropping `embedding`), and a `RestoreTarget` that writes by explicit id. | -| `functions/test/archive.test.ts` | Create. Covers everything in `archive.ts`. | -| `functions/test/archiveFirestore.test.ts` | Create. Covers the pure document-mapping function. | -| `functions/scripts/support/cli.ts` | Create. Shared CLI helpers: argument parsing, `.env`/`.env.prod` loading, `.firebaserc` project resolution. | -| `functions/scripts/backup-memories.ts` | Create. CLI: read Firestore, write NDJSON + manifest, commit and push. | -| `functions/scripts/restore-memories.ts` | Create. CLI: read NDJSON, re-embed, write to a target collection. | -| `functions/package.json` | Modify. Add `backup:memories` and `restore:memories` scripts. | -| `functions/.env.example` | Modify. Document `METACORTEX_ARCHIVE_DIR`. | -| `scripts/deploy-session-preflight.sh` | Modify. Add a `== Memory archive ==` staleness section. | -| `docs/MEMORY_ARCHIVE.md` | Create. Runbook: setup, launchd agent, restore procedure. | -| `CLAUDE.md` | Modify. Add the two commands to Common Commands. | - ---- - -### Task 1: Archive serialization core - -Deterministic NDJSON is the foundation everything else rests on. If serialization is not byte-stable, the git diffs that justify this whole design become noise. - -**Files:** -- Create: `functions/src/archive.ts` -- Test: `functions/test/archive.test.ts` - -**Interfaces:** -- Consumes: `MemoryMetadata`, `MemoryMedia` from `functions/src/types.ts` -- Produces: - - `ARCHIVE_SCHEMA_VERSION: 1` - - `interface ArchivedMemory { id: string; content: string; retrieval_text: string; metadata: MemoryMetadata; media?: MemoryMedia }` - - `canonicalizeArchivedMemory(memory: ArchivedMemory): Record` - - `serializeArchive(memories: readonly ArchivedMemory[]): string` - - `parseArchive(ndjson: string): ArchivedMemory[]` - -- [ ] **Step 1: Write the failing test** - -Create `functions/test/archive.test.ts`: - -```typescript -import { describe, expect, it } from "vitest"; - -import { - parseArchive, - serializeArchive, - type ArchivedMemory -} from "../src/archive.js"; - -function memory(overrides: Partial = {}): ArchivedMemory { - return { - id: "doc-b", - content: "Nick prefers MVI for Android", - retrieval_text: "Nick prefers MVI for Android", - metadata: { - module_name: "android", - branch_state: "active", - created_at: 1_700_000_000_000, - updated_at: 1_700_000_000_000, - modality: "text" - }, - ...overrides - }; -} - -describe("serializeArchive", () => { - it("emits one JSON object per line, sorted by document id", () => { - const output = serializeArchive([ - memory({ id: "doc-c" }), - memory({ id: "doc-a" }), - memory({ id: "doc-b" }) - ]); - - const ids = output - .trimEnd() - .split("\n") - .map(line => JSON.parse(line).id); - - expect(ids).toEqual(["doc-a", "doc-b", "doc-c"]); - }); - - it("terminates the final line with a newline", () => { - expect(serializeArchive([memory()]).endsWith("\n")).toBe(true); - }); - - it("is byte-identical regardless of key insertion order", () => { - const straightforward = serializeArchive([memory()]); - - const scrambled = serializeArchive([ - { - metadata: { - modality: "text", - updated_at: 1_700_000_000_000, - branch_state: "active", - created_at: 1_700_000_000_000, - module_name: "android" - }, - retrieval_text: "Nick prefers MVI for Android", - id: "doc-b", - content: "Nick prefers MVI for Android" - } as ArchivedMemory - ]); - - expect(scrambled).toBe(straightforward); - }); - - it("omits absent optional fields rather than emitting null", () => { - const line = serializeArchive([memory()]).trimEnd(); - - expect(line).not.toContain("null"); - expect(line).not.toContain("media"); - expect(line).not.toContain("superseded_by"); - }); - - it("round-trips every optional metadata field", () => { - const full = memory({ - id: "doc-full", - metadata: { - module_name: "metacortex-product", - branch_state: "deprecated", - created_at: 1_700_000_000_000, - updated_at: 1_700_000_100_000, - modality: "mixed", - artifact_refs: ["gs://bucket/diagram.png"], - superseded_by: "doc-successor", - valid_from: 1_699_000_000_000, - valid_until: 1_701_000_000_000, - supersession_reason: "corrected", - initiator: "user", - provenance: { - origin: "user_asserted", - source_session: "session-42", - derived_from: ["doc-a", "doc-b"], - confidence: 0.97 - } - }, - media: { kind: "inline_image", mime_type: "image/png" } - }); - - expect(parseArchive(serializeArchive([full]))).toEqual([full]); - }); -}); - -describe("parseArchive", () => { - it("ignores blank lines and tolerates CRLF", () => { - const ndjson = serializeArchive([memory()]).trimEnd(); - - expect(parseArchive(`\r\n${ndjson}\r\n\r\n`)).toHaveLength(1); - }); - - it("reports the line number of malformed content", () => { - const ndjson = `${serializeArchive([memory()]).trimEnd()}\nnot-json\n`; - - expect(() => parseArchive(ndjson)).toThrow(/line 2/); - }); -}); -``` - -- [ ] **Step 2: Run the test to verify it fails** - -Run: `npx --prefix functions vitest run test/archive.test.ts` -Expected: FAIL — cannot resolve `../src/archive.js`. - -- [ ] **Step 3: Write the implementation** - -Create `functions/src/archive.ts`: - -```typescript -import type { MemoryMedia, MemoryMetadata } from "./types.js"; - -export const ARCHIVE_SCHEMA_VERSION = 1; - -export interface ArchivedMemory { - id: string; - content: string; - retrieval_text: string; - metadata: MemoryMetadata; - media?: MemoryMedia; -} - -const MEMORY_KEY_ORDER = [ - "id", - "content", - "retrieval_text", - "metadata", - "media" -] as const; - -const METADATA_KEY_ORDER = [ - "module_name", - "branch_state", - "created_at", - "updated_at", - "modality", - "artifact_refs", - "superseded_by", - "valid_from", - "valid_until", - "supersession_reason", - "initiator", - "provenance" -] as const; - -const PROVENANCE_KEY_ORDER = [ - "origin", - "source_session", - "derived_from", - "confidence" -] as const; - -const MEDIA_KEY_ORDER = ["kind", "mime_type"] as const; - -function orderKeys( - source: Record, - keyOrder: readonly string[] -): Record { - const ordered: Record = {}; - - for (const key of keyOrder) { - if (typeof source[key] !== "undefined" && source[key] !== null) { - ordered[key] = source[key]; - } - } - - return ordered; -} - -export function canonicalizeArchivedMemory( - memory: ArchivedMemory -): Record { - const metadata = orderKeys( - memory.metadata as unknown as Record, - METADATA_KEY_ORDER - ); - - if (memory.metadata.provenance) { - metadata.provenance = orderKeys( - memory.metadata.provenance as unknown as Record, - PROVENANCE_KEY_ORDER - ); - } - - const canonical = orderKeys( - memory as unknown as Record, - MEMORY_KEY_ORDER - ); - - canonical.metadata = metadata; - - if (memory.media) { - canonical.media = orderKeys( - memory.media as unknown as Record, - MEDIA_KEY_ORDER - ); - } - - return canonical; -} - -export function serializeArchive( - memories: readonly ArchivedMemory[] -): string { - const sorted = [...memories].sort((left, right) => - left.id < right.id ? -1 : left.id > right.id ? 1 : 0 - ); - - return sorted - .map(memory => JSON.stringify(canonicalizeArchivedMemory(memory))) - .map(line => `${line}\n`) - .join(""); -} - -export function parseArchive(ndjson: string): ArchivedMemory[] { - const memories: ArchivedMemory[] = []; - const lines = ndjson.split(/\r?\n/); - - for (let index = 0; index < lines.length; index += 1) { - const line = lines[index].trim(); - - if (!line) { - continue; - } - - try { - memories.push(JSON.parse(line) as ArchivedMemory); - } catch { - throw new Error(`archive is malformed at line ${index + 1}`); - } - } - - return memories; -} -``` - -- [ ] **Step 4: Run the test to verify it passes** - -Run: `npx --prefix functions vitest run test/archive.test.ts` -Expected: PASS, 7 tests. - -- [ ] **Step 5: Commit** - -```bash -git add functions/src/archive.ts functions/test/archive.test.ts -git commit -m "feat(archive): deterministic NDJSON serialization for memory archive" -``` - ---- - -### Task 2: Manifest and truncation guardrail - -The manifest is what makes the archive self-describing; the guardrail is what stops a partial read from overwriting a good archive. - -**Files:** -- Modify: `functions/src/archive.ts` (append) -- Test: `functions/test/archive.test.ts` (append) - -**Interfaces:** -- Consumes: `ArchivedMemory`, `ARCHIVE_SCHEMA_VERSION` from Task 1 -- Produces: - - `interface ArchiveManifest { generated_at: number; project_id: string; collection: string; document_count: number; state_histogram: Record; embedding_model: string; embedding_dimensions: number; schema_version: number }` - - `buildManifest(params: BuildManifestParams): ArchiveManifest` - - `interface BuildManifestParams { memories: readonly ArchivedMemory[]; generatedAt: number; projectId: string; collection: string; embeddingModel: string; embeddingDimensions: number }` - - `checkTruncation(previousCount: number | undefined, nextCount: number): TruncationVerdict` - - `interface TruncationVerdict { allowed: boolean; reason?: string }` - - `describeEmbeddingMismatch(manifest: ArchiveManifest, current: { embeddingModel: string; embeddingDimensions: number }): string | null` - -- [ ] **Step 1: Write the failing test** - -Append to `functions/test/archive.test.ts` (and extend the existing import from `../src/archive.js` to include `buildManifest`, `checkTruncation`, and `describeEmbeddingMismatch`): - -```typescript -describe("buildManifest", () => { - it("counts documents and tallies branch states", () => { - const manifest = buildManifest({ - memories: [ - memory({ id: "a" }), - memory({ - id: "b", - metadata: { ...memory().metadata, branch_state: "deprecated" } - }), - memory({ - id: "c", - metadata: { ...memory().metadata, branch_state: "deprecated" } - }) - ], - generatedAt: 1_771_286_400_000, - projectId: "my-brain-88870", - collection: "memory_vectors", - embeddingModel: "text-embedding-004", - embeddingDimensions: 768 - }); - - expect(manifest.document_count).toBe(3); - expect(manifest.state_histogram).toEqual({ active: 1, deprecated: 2 }); - expect(manifest.schema_version).toBe(1); - expect(manifest.generated_at).toBe(1_771_286_400_000); - }); - - it("orders histogram keys so the manifest serializes deterministically", () => { - const manifest = buildManifest({ - memories: [ - memory({ - id: "a", - metadata: { ...memory().metadata, branch_state: "wip" } - }), - memory({ - id: "b", - metadata: { ...memory().metadata, branch_state: "active" } - }) - ], - generatedAt: 1, - projectId: "p", - collection: "c", - embeddingModel: "m", - embeddingDimensions: 768 - }); - - expect(Object.keys(manifest.state_histogram)).toEqual(["active", "wip"]); - }); -}); - -describe("checkTruncation", () => { - it("allows the first run when there is no previous manifest", () => { - expect(checkTruncation(undefined, 0).allowed).toBe(true); - }); - - it("allows growth", () => { - expect(checkTruncation(100, 143).allowed).toBe(true); - }); - - it("allows a drop within 10 percent", () => { - expect(checkTruncation(100, 90).allowed).toBe(true); - }); - - it("blocks a drop beyond 10 percent and explains why", () => { - const verdict = checkTruncation(100, 89); - - expect(verdict.allowed).toBe(false); - expect(verdict.reason).toMatch(/89/); - expect(verdict.reason).toMatch(/100/); - }); - - it("blocks an empty read against a non-empty previous archive", () => { - expect(checkTruncation(143, 0).allowed).toBe(false); - }); -}); - -describe("describeEmbeddingMismatch", () => { - const manifest = { - generated_at: 1, - project_id: "p", - collection: "c", - document_count: 1, - state_histogram: { active: 1 }, - embedding_model: "text-embedding-004", - embedding_dimensions: 768, - schema_version: 1 - }; - - it("returns null when the vector space matches", () => { - expect( - describeEmbeddingMismatch(manifest, { - embeddingModel: "text-embedding-004", - embeddingDimensions: 768 - }) - ).toBeNull(); - }); - - it("describes a dimension change", () => { - const message = describeEmbeddingMismatch(manifest, { - embeddingModel: "text-embedding-004", - embeddingDimensions: 1536 - }); - - expect(message).toMatch(/768/); - expect(message).toMatch(/1536/); - }); - - it("describes a model change", () => { - const message = describeEmbeddingMismatch(manifest, { - embeddingModel: "gemini-embedding-001", - embeddingDimensions: 768 - }); - - expect(message).toMatch(/text-embedding-004/); - expect(message).toMatch(/gemini-embedding-001/); - }); -}); -``` - -- [ ] **Step 2: Run the test to verify it fails** - -Run: `npx --prefix functions vitest run test/archive.test.ts` -Expected: FAIL — `buildManifest` is not exported. - -- [ ] **Step 3: Write the implementation** - -Append to `functions/src/archive.ts`: - -```typescript -const TRUNCATION_TOLERANCE = 0.9; - -export interface ArchiveManifest { - generated_at: number; - project_id: string; - collection: string; - document_count: number; - state_histogram: Record; - embedding_model: string; - embedding_dimensions: number; - schema_version: number; -} - -export interface BuildManifestParams { - memories: readonly ArchivedMemory[]; - generatedAt: number; - projectId: string; - collection: string; - embeddingModel: string; - embeddingDimensions: number; -} - -export function buildManifest(params: BuildManifestParams): ArchiveManifest { - const tally = new Map(); - - for (const memory of params.memories) { - const state = memory.metadata.branch_state; - tally.set(state, (tally.get(state) ?? 0) + 1); - } - - const stateHistogram: Record = {}; - - for (const state of [...tally.keys()].sort()) { - stateHistogram[state] = tally.get(state)!; - } - - return { - generated_at: params.generatedAt, - project_id: params.projectId, - collection: params.collection, - document_count: params.memories.length, - state_histogram: stateHistogram, - embedding_model: params.embeddingModel, - embedding_dimensions: params.embeddingDimensions, - schema_version: ARCHIVE_SCHEMA_VERSION - }; -} - -export interface TruncationVerdict { - allowed: boolean; - reason?: string; -} - -export function checkTruncation( - previousCount: number | undefined, - nextCount: number -): TruncationVerdict { - if (typeof previousCount !== "number" || previousCount === 0) { - return { allowed: true }; - } - - if (nextCount >= previousCount * TRUNCATION_TOLERANCE) { - return { allowed: true }; - } - - return { - allowed: false, - reason: - `read ${nextCount} documents but the previous archive held ${previousCount}; ` + - "a drop beyond 10% looks like a partial read. Re-run with --force if this is expected." - }; -} - -export function describeEmbeddingMismatch( - manifest: ArchiveManifest, - current: { embeddingModel: string; embeddingDimensions: number } -): string | null { - const differences: string[] = []; - - if (manifest.embedding_model !== current.embeddingModel) { - differences.push( - `model ${manifest.embedding_model} -> ${current.embeddingModel}` - ); - } - - if (manifest.embedding_dimensions !== current.embeddingDimensions) { - differences.push( - `dimensions ${manifest.embedding_dimensions} -> ${current.embeddingDimensions}` - ); - } - - if (differences.length === 0) { - return null; - } - - return ( - `archive was taken from a different vector space (${differences.join(", ")}); ` + - "restored vectors will not match the archived ones" - ); -} -``` - -- [ ] **Step 4: Run the test to verify it passes** - -Run: `npx --prefix functions vitest run test/archive.test.ts` -Expected: PASS, 17 tests. - -- [ ] **Step 5: Commit** - -```bash -git add functions/src/archive.ts functions/test/archive.test.ts -git commit -m "feat(archive): manifest, truncation guardrail, and vector-space mismatch check" -``` - ---- - -### Task 3: Firestore read adapter and backup CLI - -**Files:** -- Create: `functions/src/archiveFirestore.ts` -- Create: `functions/test/archiveFirestore.test.ts` -- Create: `functions/scripts/support/cli.ts` -- Create: `functions/scripts/backup-memories.ts` -- Modify: `functions/package.json` -- Modify: `functions/.env.example` - -**Interfaces:** -- Consumes: `ArchivedMemory`, `serializeArchive`, `buildManifest`, `checkTruncation`, `ArchiveManifest` from Tasks 1-2 -- Produces: - - `toArchivedMemory(id: string, data: Record): ArchivedMemory` - - `readMemoriesForArchive(firestore: Firestore, collectionName: string): Promise` - - `readArg(args: string[], name: string): string | undefined` - - `loadEnvironment(directory: string): void` - - `readFirebaseProject(rootDir: string): string | undefined` - -- [ ] **Step 1: Write the failing test** - -Create `functions/test/archiveFirestore.test.ts`: - -```typescript -import { describe, expect, it } from "vitest"; - -import { toArchivedMemory } from "../src/archiveFirestore.js"; - -describe("toArchivedMemory", () => { - const data = { - content: "Nick prefers MVI for Android", - retrieval_text: "Nick prefers MVI for Android", - embedding: { _values: [0.1, 0.2, 0.3] }, - metadata: { - module_name: "android", - branch_state: "active", - created_at: 1_700_000_000_000, - updated_at: 1_700_000_000_000, - modality: "text" - } - }; - - it("keeps the document id", () => { - expect(toArchivedMemory("doc-a", data).id).toBe("doc-a"); - }); - - it("drops the embedding field", () => { - expect(toArchivedMemory("doc-a", data)).not.toHaveProperty("embedding"); - }); - - it("falls back to content when retrieval_text is missing", () => { - const { retrieval_text: _omitted, ...withoutRetrievalText } = data; - - expect(toArchivedMemory("doc-a", withoutRetrievalText).retrieval_text).toBe( - "Nick prefers MVI for Android" - ); - }); - - it("preserves media when present", () => { - const withMedia = { - ...data, - media: { kind: "inline_image", mime_type: "image/png" } - }; - - expect(toArchivedMemory("doc-a", withMedia).media).toEqual({ - kind: "inline_image", - mime_type: "image/png" - }); - }); - - it("omits media when absent", () => { - expect(toArchivedMemory("doc-a", data).media).toBeUndefined(); - }); -}); -``` - -- [ ] **Step 2: Run the test to verify it fails** - -Run: `npx --prefix functions vitest run test/archiveFirestore.test.ts` -Expected: FAIL — cannot resolve `../src/archiveFirestore.js`. - -- [ ] **Step 3: Write the adapter** - -Create `functions/src/archiveFirestore.ts`: - -```typescript -import type { Firestore } from "firebase-admin/firestore"; - -import type { ArchivedMemory } from "./archive.js"; -import type { MemoryMedia, MemoryMetadata } from "./types.js"; - -export function toArchivedMemory( - id: string, - data: Record -): ArchivedMemory { - const content = typeof data.content === "string" ? data.content : ""; - const retrievalText = - typeof data.retrieval_text === "string" ? data.retrieval_text : content; - - return { - id, - content, - retrieval_text: retrievalText, - metadata: data.metadata as MemoryMetadata, - ...(data.media ? { media: data.media as MemoryMedia } : {}) - }; -} - -export async function readMemoriesForArchive( - firestore: Firestore, - collectionName: string -): Promise { - const snapshot = await firestore.collection(collectionName).get(); - - return snapshot.docs.map(doc => - toArchivedMemory(doc.id, doc.data() as Record) - ); -} -``` - -- [ ] **Step 4: Run the test to verify it passes** - -Run: `npx --prefix functions vitest run test/archiveFirestore.test.ts` -Expected: PASS, 5 tests. - -- [ ] **Step 5: Write the shared CLI helpers** - -Both CLI drivers need identical env loading and argument parsing. Extract once rather than duplicating. - -Create `functions/scripts/support/cli.ts`: - -```typescript -import fs from "node:fs"; -import path from "node:path"; - -export function readArg(args: string[], name: string): string | undefined { - const index = args.findIndex(arg => arg === `--${name}`); - - return index === -1 ? undefined : args[index + 1]; -} - -export function loadEnvironment(directory: string): void { - const explicitKeys = new Set(Object.keys(process.env)); - - for (const fileName of [".env", ".env.prod"]) { - const filePath = path.join(directory, fileName); - - if (!fs.existsSync(filePath)) { - continue; - } - - for (const rawLine of fs.readFileSync(filePath, "utf8").split(/\r?\n/)) { - const line = rawLine.trim(); - - if (!line || line.startsWith("#")) { - continue; - } - - const separatorIndex = line.indexOf("="); - - if (separatorIndex === -1) { - continue; - } - - const key = line.slice(0, separatorIndex).trim(); - - if (explicitKeys.has(key)) { - continue; - } - - let value = line.slice(separatorIndex + 1).trim(); - - if ( - (value.startsWith('"') && value.endsWith('"')) || - (value.startsWith("'") && value.endsWith("'")) - ) { - value = value.slice(1, -1); - } - - process.env[key] = value; - } - } -} - -export function readFirebaseProject(rootDir: string): string | undefined { - const firebaseRcPath = path.join(rootDir, ".firebaserc"); - - if (!fs.existsSync(firebaseRcPath)) { - return undefined; - } - - const firebaseRc = JSON.parse(fs.readFileSync(firebaseRcPath, "utf8")); - const project = firebaseRc.projects?.prod ?? firebaseRc.projects?.default; - - return typeof project === "string" && project.trim() - ? project.trim() - : undefined; -} -``` - -- [ ] **Step 6: Write the backup CLI** - -Create `functions/scripts/backup-memories.ts`: - -```typescript -import { execFileSync } from "node:child_process"; -import fs from "node:fs"; -import path from "node:path"; -import { fileURLToPath } from "node:url"; - -import { getApp, getApps, initializeApp } from "firebase-admin/app"; -import { getFirestore } from "firebase-admin/firestore"; - -import { - buildManifest, - checkTruncation, - serializeArchive, - type ArchiveManifest -} from "../src/archive.js"; -import { readMemoriesForArchive } from "../src/archiveFirestore.js"; -import { loadEnvironment, readArg, readFirebaseProject } from "./support/cli.js"; - -const scriptDirectory = path.dirname(fileURLToPath(import.meta.url)); -const functionsDirectory = path.resolve(scriptDirectory, ".."); -const repoRoot = path.resolve(functionsDirectory, ".."); - -loadEnvironment(functionsDirectory); - -const args = process.argv.slice(2); -const dryRun = args.includes("--dry-run"); -const force = args.includes("--force"); -const noPush = args.includes("--no-push"); -const archiveDir = - readArg(args, "archive-dir") ?? process.env.METACORTEX_ARCHIVE_DIR; -const projectId = - readArg(args, "project") ?? - process.env.GOOGLE_CLOUD_PROJECT ?? - process.env.GCLOUD_PROJECT ?? - readFirebaseProject(repoRoot) ?? - "my-brain-88870"; -const collectionName = - readArg(args, "memory-collection") ?? - process.env.MEMORY_COLLECTION?.trim() ?? - "memory_vectors"; -const embeddingModel = - process.env.GEMINI_EMBEDDING_MODEL?.trim() || "text-embedding-004"; -const embeddingDimensions = Number.parseInt( - process.env.GEMINI_EMBEDDING_DIMENSIONS?.trim() || "768", - 10 -); - -if (!archiveDir) { - console.error( - "METACORTEX_ARCHIVE_DIR is not set. Point it at your private archive repository, or pass --archive-dir." - ); - process.exit(1); -} - -if (!fs.existsSync(archiveDir)) { - console.error(`archive directory does not exist: ${archiveDir}`); - process.exit(1); -} - -const archivePath = path.join(archiveDir, "memories.ndjson"); -const manifestPath = path.join(archiveDir, "manifest.json"); - -const app = getApps().length === 0 ? initializeApp({ projectId }) : getApp(); -const firestore = getFirestore(app); - -console.log(`project: ${projectId}`); -console.log(`collection: ${collectionName}`); -console.log(`archive: ${archiveDir}`); -console.log(`mode: ${dryRun ? "dry-run" : "write"}`); - -const memories = await readMemoriesForArchive(firestore, collectionName); -const previousManifest = readManifest(manifestPath); -const verdict = checkTruncation(previousManifest?.document_count, memories.length); - -console.log( - `documents: ${memories.length}` + - (previousManifest ? ` (previous: ${previousManifest.document_count})` : "") -); - -if (!verdict.allowed && !force) { - console.error(`refusing to write: ${verdict.reason}`); - process.exit(1); -} - -if (!verdict.allowed) { - console.warn(`--force overriding guardrail: ${verdict.reason}`); -} - -const manifest = buildManifest({ - memories, - generatedAt: Date.now(), - projectId, - collection: collectionName, - embeddingModel, - embeddingDimensions -}); - -console.log(`states: ${JSON.stringify(manifest.state_histogram)}`); - -if (dryRun) { - console.log("Dry run complete. Nothing written."); - process.exit(0); -} - -fs.writeFileSync(archivePath, serializeArchive(memories), "utf8"); -fs.writeFileSync(manifestPath, `${JSON.stringify(manifest, null, 2)}\n`, "utf8"); - -const status = git(["status", "--porcelain"], archiveDir).trim(); - -if (!status) { - console.log("Archive unchanged; nothing to commit."); - process.exit(0); -} - -git(["add", "memories.ndjson", "manifest.json"], archiveDir); -git( - [ - "commit", - "-m", - `chore(archive): ${manifest.document_count} memories from ${collectionName}` - ], - archiveDir -); - -if (noPush) { - console.log("Committed. Skipping push (--no-push)."); - process.exit(0); -} - -git(["push"], archiveDir); -console.log("Committed and pushed."); - -function git(gitArgs: string[], cwd: string): string { - return execFileSync("git", gitArgs, { cwd, encoding: "utf8" }); -} - -function readManifest(filePath: string): ArchiveManifest | undefined { - if (!fs.existsSync(filePath)) { - return undefined; - } - - return JSON.parse(fs.readFileSync(filePath, "utf8")) as ArchiveManifest; -} -``` - -- [ ] **Step 7: Register the npm script** - -In `functions/package.json`, add after the `backfill:provenance` line: - -```json - "backup:memories": "tsx scripts/backup-memories.ts", -``` - -- [ ] **Step 8: Document the environment variable** - -Append to `functions/.env.example`: - -``` -# Absolute path to the private git repository holding the memory archive. -# Required by `npm run backup:memories`. -METACORTEX_ARCHIVE_DIR= -``` - -- [ ] **Step 9: Verify the CLI runs end to end against production, without writing** - -```bash -mkdir -p /tmp/metacortex-archive-check && npm --prefix functions run backup:memories -- --dry-run --archive-dir /tmp/metacortex-archive-check -``` - -Expected: prints project, collection, a document count in the low hundreds, a state histogram containing `active` and `deprecated`, then "Dry run complete. Nothing written." Confirm `/tmp/metacortex-archive-check` is still empty. - -- [ ] **Step 10: Run the full suite** - -Run: `npm --prefix functions test` -Expected: PASS, coverage reported. - -- [ ] **Step 11: Commit** - -```bash -git add functions/src/archiveFirestore.ts functions/test/archiveFirestore.test.ts functions/scripts/support/cli.ts functions/scripts/backup-memories.ts functions/package.json functions/.env.example -git commit -m "feat(archive): export memory store to NDJSON in a private git repo" -``` - ---- - -### Task 4: Restore path - -Where correctness matters most. Restore must reproduce ids exactly, or the supersession graph is silently destroyed. - -**Files:** -- Modify: `functions/src/archive.ts` (append) -- Modify: `functions/src/archiveFirestore.ts` (append) -- Modify: `functions/test/archive.test.ts` (append) -- Create: `functions/scripts/restore-memories.ts` -- Modify: `functions/package.json` - -**Interfaces:** -- Consumes: `ArchivedMemory`, `parseArchive` from Task 1; `EmbeddingClient` from `functions/src/embeddings.ts` -- Produces: - - `interface RestoreTarget { writeMemory(memory: ArchivedMemory, embedding: number[]): Promise }` - - `restoreMemories(params: RestoreParams): Promise` - - `interface RestoreParams { memories: readonly ArchivedMemory[]; embeddings: EmbeddingClient; target: RestoreTarget }` - - `interface RestoreResult { restored: number }` - - `class FirestoreRestoreTarget implements RestoreTarget` (constructor: `(firestore: Firestore, collectionName: string)`) - -- [ ] **Step 1: Write the failing test** - -Append to `functions/test/archive.test.ts` (extend the `../src/archive.js` import with `restoreMemories`, and add `import type { EmbeddingClient, EmbeddingRequest } from "../src/embeddings.js";`): - -```typescript -class RecordingEmbeddingClient implements EmbeddingClient { - readonly requests: EmbeddingRequest[] = []; - - async embed(request: EmbeddingRequest): Promise { - this.requests.push(request); - - return [request.text.length, 0, 0]; - } -} - -class RecordingRestoreTarget { - readonly written: { memory: ArchivedMemory; embedding: number[] }[] = []; - - async writeMemory(memory: ArchivedMemory, embedding: number[]): Promise { - this.written.push({ memory, embedding }); - } -} - -describe("restoreMemories", () => { - it("preserves archived document ids", async () => { - const target = new RecordingRestoreTarget(); - - await restoreMemories({ - memories: [memory({ id: "DSFBIyUni6OGhySMkbHw" })], - embeddings: new RecordingEmbeddingClient(), - target - }); - - expect(target.written[0].memory.id).toBe("DSFBIyUni6OGhySMkbHw"); - }); - - it("keeps the supersession graph resolvable after restore", async () => { - const target = new RecordingRestoreTarget(); - - await restoreMemories({ - memories: [ - memory({ id: "successor" }), - memory({ - id: "original", - metadata: { - ...memory().metadata, - branch_state: "deprecated", - superseded_by: "successor" - } - }) - ], - embeddings: new RecordingEmbeddingClient(), - target - }); - - const ids = new Set(target.written.map(entry => entry.memory.id)); - const deprecated = target.written.find( - entry => entry.memory.id === "original" - ); - - expect(ids.has(deprecated!.memory.metadata.superseded_by!)).toBe(true); - }); - - it("embeds retrieval_text with the same request shape the service uses", async () => { - const embeddings = new RecordingEmbeddingClient(); - - await restoreMemories({ - memories: [ - memory({ - id: "a", - content: "canonical content", - retrieval_text: "normalized retrieval text", - metadata: { ...memory().metadata, module_name: "android" } - }) - ], - embeddings, - target: new RecordingRestoreTarget() - }); - - expect(embeddings.requests[0]).toEqual({ - text: "normalized retrieval text", - taskType: "RETRIEVAL_DOCUMENT", - title: "android" - }); - }); - - it("reports how many documents were restored", async () => { - const result = await restoreMemories({ - memories: [memory({ id: "a" }), memory({ id: "b" })], - embeddings: new RecordingEmbeddingClient(), - target: new RecordingRestoreTarget() - }); - - expect(result.restored).toBe(2); - }); - - it("round-trips an archive without losing metadata", async () => { - const original = memory({ - id: "doc-full", - metadata: { - module_name: "metacortex-product", - branch_state: "deprecated", - created_at: 1_700_000_000_000, - updated_at: 1_700_000_100_000, - modality: "mixed", - artifact_refs: ["gs://bucket/diagram.png"], - superseded_by: "doc-successor", - valid_from: 1_699_000_000_000, - valid_until: 1_701_000_000_000, - supersession_reason: "corrected", - initiator: "user", - provenance: { - origin: "user_asserted", - source_session: "session-42", - derived_from: ["doc-a"], - confidence: 0.97 - } - }, - media: { kind: "inline_image", mime_type: "image/png" } - }); - - const target = new RecordingRestoreTarget(); - - await restoreMemories({ - memories: parseArchive(serializeArchive([original])), - embeddings: new RecordingEmbeddingClient(), - target - }); - - expect(target.written[0].memory).toEqual(original); - expect(target.written[0].embedding).toHaveLength(3); - }); -}); -``` - -- [ ] **Step 2: Run the test to verify it fails** - -Run: `npx --prefix functions vitest run test/archive.test.ts` -Expected: FAIL — `restoreMemories` is not exported. - -- [ ] **Step 3: Write the restore logic** - -Append to `functions/src/archive.ts` (and add `import type { EmbeddingClient } from "./embeddings.js";` at the top): - -```typescript -export interface RestoreTarget { - writeMemory(memory: ArchivedMemory, embedding: number[]): Promise; -} - -export interface RestoreParams { - memories: readonly ArchivedMemory[]; - embeddings: EmbeddingClient; - target: RestoreTarget; -} - -export interface RestoreResult { - restored: number; -} - -export async function restoreMemories( - params: RestoreParams -): Promise { - let restored = 0; - - for (const memory of params.memories) { - const embedding = await params.embeddings.embed({ - text: memory.retrieval_text, - taskType: "RETRIEVAL_DOCUMENT", - title: memory.metadata.module_name - }); - - await params.target.writeMemory(memory, embedding); - restored += 1; - } - - return { restored }; -} -``` - -- [ ] **Step 4: Run the test to verify it passes** - -Run: `npx --prefix functions vitest run test/archive.test.ts` -Expected: PASS, 22 tests. - -- [ ] **Step 5: Write the Firestore restore target** - -Append to `functions/src/archiveFirestore.ts` (extend the imports to `import { FieldValue, type Firestore } from "firebase-admin/firestore";` and add `ArchivedMemory`, `RestoreTarget` to the `./archive.js` import): - -```typescript -export class FirestoreRestoreTarget implements RestoreTarget { - constructor( - private readonly firestore: Firestore, - private readonly collectionName: string - ) {} - - async writeMemory( - memory: ArchivedMemory, - embedding: number[] - ): Promise { - await this.firestore - .collection(this.collectionName) - .doc(memory.id) - .set({ - content: memory.content, - retrieval_text: memory.retrieval_text, - embedding: FieldValue.vector(embedding), - metadata: memory.metadata, - ...(memory.media ? { media: memory.media } : {}) - }); - } -} -``` - -- [ ] **Step 6: Write the restore CLI** - -Create `functions/scripts/restore-memories.ts`: - -```typescript -import fs from "node:fs"; -import path from "node:path"; -import { fileURLToPath } from "node:url"; - -import { getApp, getApps, initializeApp } from "firebase-admin/app"; -import { getFirestore } from "firebase-admin/firestore"; - -import { - describeEmbeddingMismatch, - parseArchive, - restoreMemories, - type ArchiveManifest, - type ArchivedMemory, - type RestoreTarget -} from "../src/archive.js"; -import { FirestoreRestoreTarget } from "../src/archiveFirestore.js"; -import { loadConfig } from "../src/config.js"; -import { GeminiEmbeddingClient } from "../src/embeddings.js"; -import { loadEnvironment, readArg, readFirebaseProject } from "./support/cli.js"; - -const scriptDirectory = path.dirname(fileURLToPath(import.meta.url)); -const functionsDirectory = path.resolve(scriptDirectory, ".."); -const repoRoot = path.resolve(functionsDirectory, ".."); - -loadEnvironment(functionsDirectory); - -const args = process.argv.slice(2); -const write = args.includes("--write"); -const archiveDir = - readArg(args, "archive-dir") ?? process.env.METACORTEX_ARCHIVE_DIR; -const projectId = - readArg(args, "project") ?? - process.env.GOOGLE_CLOUD_PROJECT ?? - process.env.GCLOUD_PROJECT ?? - readFirebaseProject(repoRoot) ?? - "my-brain-88870"; -const targetCollection = readArg(args, "collection"); - -if (!archiveDir) { - console.error("METACORTEX_ARCHIVE_DIR is not set. Pass --archive-dir."); - process.exit(1); -} - -if (!targetCollection) { - console.error( - "--collection is required. Restore into a scratch collection first and diff it before touching memory_vectors." - ); - process.exit(1); -} - -const config = loadConfig(process.env); -const memories = parseArchive( - fs.readFileSync(path.join(archiveDir, "memories.ndjson"), "utf8") -); -const manifest = JSON.parse( - fs.readFileSync(path.join(archiveDir, "manifest.json"), "utf8") -) as ArchiveManifest; - -console.log(`archive: ${archiveDir}`); -console.log(`archived at: ${new Date(manifest.generated_at).toISOString()}`); -console.log(`documents: ${memories.length}`); -console.log(`target: ${projectId}/${targetCollection}`); -console.log(`mode: ${write ? "write" : "dry-run"}`); - -if (memories.length !== manifest.document_count) { - console.warn( - `warning: archive holds ${memories.length} documents but the manifest records ${manifest.document_count}` - ); -} - -const mismatch = describeEmbeddingMismatch(manifest, { - embeddingModel: config.embeddingModel, - embeddingDimensions: config.embeddingDimensions -}); - -if (mismatch) { - console.warn(`warning: ${mismatch}`); -} - -if (!write) { - console.log("Dry run complete. Re-run with --write to restore."); - process.exit(0); -} - -const app = getApps().length === 0 ? initializeApp({ projectId }) : getApp(); -const firestore = getFirestore(app); -const embeddings = new GeminiEmbeddingClient({ - apiKey: config.geminiApiKey, - model: config.embeddingModel, - dimensions: config.embeddingDimensions -}); -const target: RestoreTarget = new FirestoreRestoreTarget( - firestore, - targetCollection -); - -let completed = 0; -const result = await restoreMemories({ - memories, - embeddings, - target: { - async writeMemory(memory: ArchivedMemory, embedding: number[]) { - await target.writeMemory(memory, embedding); - completed += 1; - - if (completed % 25 === 0) { - console.log(`restored ${completed}/${memories.length}`); - } - } - } -}); - -console.log(`Restored ${result.restored} documents into ${targetCollection}.`); -``` - -- [ ] **Step 7: Register the npm script** - -In `functions/package.json`, add after `backup:memories`: - -```json - "restore:memories": "tsx scripts/restore-memories.ts", -``` - -- [ ] **Step 8: Run the full suite and typecheck** - -Run: `npm --prefix functions test && npm --prefix functions run build` -Expected: PASS, and a clean `tsc`. Note that `tsconfig.json` includes only `src/**/*.ts`, so the build type-checks `archive.ts` and `archiveFirestore.ts` but not the CLI drivers — same as the existing `scripts/retrieval-eval.ts`. The drivers are exercised by actually running them in Task 3 Step 9 and the final Verification section. - -- [ ] **Step 9: Commit** - -```bash -git add functions/src/archive.ts functions/src/archiveFirestore.ts functions/test/archive.test.ts functions/scripts/restore-memories.ts functions/package.json -git commit -m "feat(archive): restore path preserving document ids and vector reproduction" -``` - ---- - -### Task 5: Staleness detection and runbook - -A backup depending on a laptop being awake fails silently, and a silent backup failure is indistinguishable from a working one until the day it matters. - -**Files:** -- Modify: `scripts/deploy-session-preflight.sh` -- Create: `docs/MEMORY_ARCHIVE.md` -- Modify: `CLAUDE.md` - -- [ ] **Step 1: Add the staleness section to the preflight script** - -In `scripts/deploy-session-preflight.sh`, insert before the `== Client profiles ==` section: - -```bash -echo -echo "== Memory archive ==" -ARCHIVE_DIR="${METACORTEX_ARCHIVE_DIR:-$(read_env_key functions/.env METACORTEX_ARCHIVE_DIR)}" - -if [[ -z "$ARCHIVE_DIR" ]]; then - echo "warning: METACORTEX_ARCHIVE_DIR is not set; the memory store has no off-GCP archive" -elif [[ ! -f "$ARCHIVE_DIR/manifest.json" ]]; then - echo "warning: no manifest.json in $ARCHIVE_DIR; run 'npm --prefix functions run backup:memories'" -else - node - "$ARCHIVE_DIR/manifest.json" <<'NODE' -const fs = require("fs"); - -const manifest = JSON.parse(fs.readFileSync(process.argv[1], "utf8")); -const ageDays = (Date.now() - manifest.generated_at) / 86400000; - -console.log( - `archive: ${manifest.document_count} documents, ${ageDays.toFixed(1)} days old` -); - -if (ageDays > 3) { - console.log( - `warning: memory archive is ${ageDays.toFixed(1)} days old; run 'npm --prefix functions run backup:memories'` - ); -} -NODE -fi -``` - -- [ ] **Step 2: Verify the preflight section runs** - -Run: `./scripts/deploy-session-preflight.sh 2>&1 | grep -A 2 "Memory archive"` -Expected: the section prints, warning that `METACORTEX_ARCHIVE_DIR` is unset (or reporting archive age once it is). The script must still exit 0 — this check warns, it does not fail. - -- [ ] **Step 3: Write the runbook** - -Create `docs/MEMORY_ARCHIVE.md`: - -````markdown -# Memory Archive - -The memory store's only copy outside Firestore. Deterministic NDJSON in a -private git repository, restorable into any Firebase project. - -## What is archived - -`content`, `retrieval_text`, and the full `metadata` object for every document -in `memory_vectors`, including deprecated ones. - -Embeddings are **not** archived. They are a pure function of `retrieval_text` -plus the configured model and dimensions, so restore regenerates them — which -keeps the archive small, keeps diffs readable, and keeps the cold copy -independent of any single vector space. - -Image bytes are not archived because they are never stored: images are -normalized to text at write time and only `{kind, mime_type}` persists. - -`memory_events` and `memory_vectors_write_fingerprints` are out of scope. - -## Setup - -1. Create a **private** repository for the archive. It contains everything - MetaCortex knows, in plaintext. -2. Clone it locally and set `METACORTEX_ARCHIVE_DIR` in `functions/.env` to its - absolute path. -3. Run a first backup: - - ```bash - npm --prefix functions run backup:memories - ``` - -## Daily run - -Install a launchd agent at `~/Library/LaunchAgents/com.metacortex.backup.plist`: - -```xml - - - - - Label - com.metacortex.backup - ProgramArguments - - /bin/zsh - -lc - cd ~/git/metacortex && npm --prefix functions run backup:memories - - StartCalendarInterval - - Hour - 9 - Minute - 30 - - RunAtLoad - - StandardOutPath - /tmp/metacortex-backup.log - StandardErrorPath - /tmp/metacortex-backup.log - - -``` - -Load it: - -```bash -launchctl load ~/Library/LaunchAgents/com.metacortex.backup.plist -``` - -`deploy-session-preflight.sh` warns when the archive is more than 3 days old, -so rot surfaces at a moment attention is already on the system. - -## Restore - -Restore never targets `memory_vectors` directly. Land in a scratch collection, -verify, then decide. - -```bash -npm --prefix functions run restore:memories -- --collection memory_vectors_restore --write -``` - -The restore preserves archived document ids, which is what keeps -`superseded_by` links intact. It re-embeds every document, so it needs a -working `GEMINI_API_KEY` and takes a few minutes. - -If the manifest's `embedding_model` or `embedding_dimensions` differ from the -current configuration, restore warns and proceeds — a legitimate migration -path, but the resulting vectors occupy a different space than the archive. - -Restoring into a live collection requires a matching vector index. See -`firestore.indexes.json`. - -## Guardrails - -The backup refuses to write when the document count drops more than 10% below -the previous manifest, since a partial read committed over a good archive is -the failure mode that turns the backup into the problem. Override with -`--force` after confirming the drop is real. - -Inspect without writing anything: - -```bash -npm --prefix functions run backup:memories -- --dry-run -``` - -## Test your restores - -An archive that has never been restored is a hypothesis. Restore into a scratch -collection periodically and confirm the document count and a few known ids. -```` - -- [ ] **Step 4: Add the commands to CLAUDE.md** - -In the "Common Commands" section of `CLAUDE.md`, after the deploy block, add: - -````markdown -Memory archive (see `docs/MEMORY_ARCHIVE.md`): -```bash -npm --prefix functions run backup:memories # Archive memory_vectors to $METACORTEX_ARCHIVE_DIR -npm --prefix functions run backup:memories -- --dry-run # Inspect without writing -npm --prefix functions run restore:memories -- --collection memory_vectors_restore --write -``` -```` - -- [ ] **Step 5: Run the full suite one last time** - -Run: `npm --prefix functions test && npm --prefix functions run build` -Expected: PASS, clean build. - -- [ ] **Step 6: Commit** - -```bash -git add scripts/deploy-session-preflight.sh docs/MEMORY_ARCHIVE.md CLAUDE.md -git commit -m "feat(archive): staleness detection and archive runbook" -``` - ---- - -## Verification - -After Task 5, confirm the archive works end to end against a real private repo: - -1. `npm --prefix functions run backup:memories` — commits and pushes an archive. -2. Re-run it immediately — reports "Archive unchanged; nothing to commit," proving serialization is deterministic against live data, not just fixtures. -3. `npm --prefix functions run restore:memories -- --collection memory_vectors_restore --write` — then compare document counts against the manifest and spot-check that a deprecated document's `superseded_by` still resolves to a document that exists in the restored collection. diff --git a/docs/superpowers/specs/2026-08-17-memory-checkpoint-design.md b/docs/superpowers/specs/2026-08-17-memory-checkpoint-design.md deleted file mode 100644 index de7336a..0000000 --- a/docs/superpowers/specs/2026-08-17-memory-checkpoint-design.md +++ /dev/null @@ -1,153 +0,0 @@ -> Historical archive design/implementation record, retained from the completed archive branch. The current roadmap is [metacortexplan.md](../../../metacortexplan.md). Current behavior and limitations are in [MEMORY_ARCHIVE.md](../../MEMORY_ARCHIVE.md) and [FULL_BACKUP.md](../../FULL_BACKUP.md). Old task checkboxes are not the active backlog. - -# Memory Checkpoint: Portable Off-GCP Archive - -Date: 2026-08-17 -Status: Approved design, pending implementation plan - -## Problem - -Firestore is the only copy of the MetaCortex memory store. The collection holds roughly 150 documents (~52 active, ~91 deprecated as of the 2026-07-13 coherence review) accumulated over months, and none of it is reproducible. Loss of the `my-brain-88870` project — deletion, billing lapse, compromised service account, or Google-side failure — is total and permanent. - -The store is also, by design, append-only in spirit: `deprecate_context` is a soft supersession and nothing is ever hard-deleted. That protects against bad *logical* writes, but offers nothing against loss of the underlying database. - -## Goals - -Produce a portable, self-describing, version-controlled archive of the memory store that lives outside GCP entirely and can be restored into a fresh Firebase project — or read by something that is not MetaCortex at all. - -## Non-goals - -- **Firestore-native backups and PITR.** Explicitly out of scope. The archive is the only recovery layer. -- **Backing up `memory_events`.** An unbounded observability audit trail, not durable state. -- **Backing up `memory_vectors_write_fingerprints`.** Ephemeral TTL data with a 30-day expiry; regenerating it is a no-op. -- **Point-in-time granularity finer than a run.** Recovery granularity is "last successful archive run." See Risks. - -## Accepted consequence - -With no Firestore-native layer, recovery granularity is the archive cadence. A bad `consolidate_context` on Tuesday is unrecoverable if the last run was Sunday. This is why the cadence is daily rather than weekly, and why the staleness check is part of the deliverable rather than a nice-to-have. - -## Architecture - -Two scripts plus a manifest, following the established `functions/scripts/*.mjs` pattern (`backfill-firestore-provenance.mjs` is the reference: firebase-admin over ADC, `.env` / `.env.prod` loading, `--write` style flags). - -``` -functions/scripts/backup-memories.mjs → $METACORTEX_ARCHIVE_DIR/memories.ndjson - $METACORTEX_ARCHIVE_DIR/manifest.json -functions/scripts/restore-memories.mjs ← reads both, re-embeds, writes to Firestore -``` - -The archive directory is a **separate private git repository**, not this one. `metacortex` is shared and carries PRs; memory content is personal. Git history is the retention policy — no lifecycle rules, no bucket configuration, and diffs that show exactly which memories changed between runs. - -### Why the embedding is not archived - -`service.ts:55` embeds `preparedContent.retrieval_text`, and `retrieval_text` is persisted verbatim on the document. The embedding is therefore a pure function of archived data plus the configured model and dimensions. Re-embedding on restore reproduces the same vector when the model is unchanged, so retrieval behavior is preserved exactly. - -Archiving 768 floats per document would multiply file size roughly fivefold, make git diffs unreadable, and pin the cold archive to a single vector space — the one thing the project's own constraints say never to mix. Re-embedding is also mandatory on any model or dimension change, so restore needs the capability regardless. - -Cost of the tradeoff: restore requires a working `GEMINI_API_KEY` and takes minutes rather than seconds, for roughly 150 embed calls. - -### Why image bytes are not archived - -`FirestoreMemoryRepository.store` persists `media` as `{kind, mime_type}` only (`memoryRepository.ts:110`). Source images are normalized to text by `GeminiMultimodalPreparer` at write time and the bytes are discarded before storage. Text plus metadata is therefore complete fidelity — there is nothing else to lose. - -## Data format - -`memories.ndjson` — one JSON object per line, sorted by document id, with keys emitted in a fixed order at every level. Determinism is a hard requirement: an unchanged store must produce a byte-identical file, or the git diff stops carrying signal. - -```json -{ - "id": "DSFBIyUni6OGhySMkbHw", - "content": "...", - "retrieval_text": "...", - "metadata": { - "module_name": "archive", - "branch_state": "active", - "created_at": 1768329964448, - "updated_at": 1768329964448, - "modality": "text", - "provenance": { "origin": "agent_inferred", "confidence": 0.97 } - }, - "media": { "kind": "inline_image", "mime_type": "image/png" } -} -``` - -Optional fields are omitted when absent rather than emitted as `null`, matching how the documents are actually stored. Every field of `MemoryMetadata` round-trips: `artifact_refs`, `superseded_by`, `valid_from`, `valid_until`, `supersession_reason`, `initiator`, and the full `provenance` object. - -`manifest.json` — what makes the archive self-describing and drives the staleness check: - -```json -{ - "generated_at": 1771286400000, - "project_id": "my-brain-88870", - "collection": "memory_vectors", - "document_count": 143, - "state_histogram": { "active": 52, "deprecated": 91 }, - "embedding_model": "text-embedding-004", - "embedding_dimensions": 768, - "schema_version": 1 -} -``` - -`embedding_model` and `embedding_dimensions` record the vector space the archive was taken *from*, so a restore into a different configuration is a visible, deliberate act rather than a silent one. - -## Export behavior - -Read the whole collection in one pass, drop `embedding`, normalize key order, write both files, then commit and push from the archive directory. - -**Truncation guardrail.** If `document_count` is more than 10% below the previous manifest's, refuse to write and exit non-zero unless `--force` is passed. A partial read committed over a good archive is the one failure mode that turns the backup into the problem. The count is monotonic in normal operation because deprecation is soft and nothing is deleted, so any decrease is worth a human look. - -**Dry run.** `--dry-run` reports what would be written and the delta against the previous manifest without touching the working tree, mirroring the `--write` opt-in convention of the existing backfill scripts. - -## Restore behavior - -`restore-memories.mjs` reads the NDJSON, re-embeds each `retrieval_text` with the currently configured embedding client, and writes to Firestore. - -**Document ids must be preserved.** `metadata.superseded_by` holds document ids, so the supersession graph is only meaningful if restore writes to `.doc(id).set(...)` with the archived id. Generating fresh ids would silently destroy every correction and consolidation link in the store. This is the single most important correctness property of the restore path. - -For the same reason restore writes documents directly rather than going through `MetaCortexService.remember()`: the service path allocates new ids, applies write-fingerprint deduplication, and re-derives metadata defaults. Restore must reproduce archived state exactly, not re-infer it. - -`--collection ` targets an arbitrary collection so a restore can land in a scratch collection and be diffed against the live one before anything is trusted. `--project` follows the same resolution order as the existing backfill scripts. - -Restore warns loudly when the manifest's `embedding_model` or `embedding_dimensions` differ from the current configuration, since that means the restored vectors occupy a different space than the archive was taken from. It proceeds — that is a legitimate migration path — but it says so. - -## Staleness detection - -A backup that depends on a laptop being awake fails silently, and a silent backup failure is indistinguishable from a working one until the day it matters. - -- A launchd agent runs the export daily. -- `scripts/deploy-session-preflight.sh` gains a `== Memory archive ==` section that reads `manifest.json` and warns when `generated_at` is more than 3 days old, following the existing warning-not-failure convention of that script. - -The preflight check is deliberately a warning rather than a hard failure: it should surface rot at a moment attention is already on the system, not block unrelated deploys. - -## Testing - -Both scripts factor their logic into importable modules so behavior is testable without hitting Firestore or Gemini, matching how the existing suite uses in-memory fakes. - -- **Round-trip.** Seed `InMemoryMemoryRepository` with documents covering every optional metadata field, export, restore into a second repository, assert content, `retrieval_text`, ids, and all metadata are identical and embeddings were regenerated. -- **Supersession graph.** A deprecated document whose `superseded_by` points at another archived document still resolves after restore. This is the regression test for the id-preservation requirement. -- **Determinism.** Exporting the same fixture twice yields byte-identical output; exporting with keys inserted in a different order yields the same bytes. -- **Truncation guardrail.** An export reporting materially fewer documents than the previous manifest exits non-zero without writing, and `--force` overrides it. -- **Model mismatch.** A manifest recording different embedding dimensions than the current config produces a warning on restore. - -Coverage target follows the project minimum of 60%. - -## Configuration - -| Variable | Purpose | -|----------|---------| -| `METACORTEX_ARCHIVE_DIR` | Path to the private archive repository. Required by the export script. | - -New npm scripts in `functions/package.json`, alongside the existing `backfill:*` entries: - -``` -"backup:memories": "node scripts/backup-memories.mjs", -"restore:memories": "node scripts/restore-memories.mjs" -``` - -## Risks - -**Cadence gap.** Recovery is only as good as the last run, and the run depends on a laptop being awake. Mitigated by daily scheduling plus the preflight staleness warning, not eliminated. If this proves insufficient in practice, the escalation path is a scheduled Cloud Function writing the same NDJSON to an off-project bucket — the exporter logic would be reusable as-is. - -**Untested restores.** A backup never restored is a hypothesis, not a backup. The scratch-collection restore path exists so this can be exercised cheaply; doing so periodically should be an explicit habit, not an assumption. - -**Archive repository confidentiality.** The archive contains everything MetaCortex knows, in plaintext, in git. The repository must be private. At-rest encryption of the dump was considered and rejected for v1: it would defeat the readable-diff property that motivates a git-backed archive, and the threat model here is loss of availability, not disclosure. Revisit if the archive ever leaves a personal machine. diff --git a/metacortexplan.md b/metacortexplan.md index 8d302ac..4e4582d 100644 --- a/metacortexplan.md +++ b/metacortexplan.md @@ -8,7 +8,7 @@ Extend the already-released MetaCortex into a free, approachable, self-hosted me The target experience includes browser-based installation into the owner’s Firebase project, secure agent connections, a memory-management web UI, and a ChatGPT extension. Do not assign “1.0” or another release number before assessing the actual compatibility changes. -**Status (2026-10-02):** planning and supporting-document reconciliation complete. Baseline integrated on `codex/self-hosted-adoption-baseline` (`save_context` rename in `dde9099`, backup/archive/restore in `d54ca20`), followed by review and fast-forward of upstream documentation through `82cb302`. Local verification and distribution checks are established; see the [baseline evidence](docs/operations/2026-10-01-adoption-baseline.md). All 19 cards are posted with verified dependencies: 18 in backlog and DISTRIBUTE review-blocked automatically by the control plane. No ready/claimed work was created. Card 66 is now merged and deployed; see the [rollout evidence](docs/operations/2026-10-03-card66-rollout.md). Product milestones B–F remain unimplemented. +**Status (2026-10-02):** planning and supporting-document reconciliation complete. Baseline integrated on `codex/self-hosted-adoption-baseline` (`save_context` rename in `dde9099`, backup/archive/restore in `d54ca20`), followed by review and fast-forward of upstream documentation through `82cb302`. Local verification and distribution checks are established. All 19 cards are posted with verified dependencies: 18 in backlog and DISTRIBUTE review-blocked automatically by the control plane. No ready/claimed work was created. Card 66 is now merged and deployed. Product milestones B–F remain unimplemented. ### One canonical plan @@ -25,7 +25,6 @@ Supporting documents retain distinct roles: | Document | Role | |---|---| -| `NEXT-STEPS.md` | Short, dated view of immediate roadmap priorities and verified blockers. No independent backlog. | | `README.md` | Released capabilities, setup entry point, costs, limitations, and links to the roadmap and runbooks. | | Architecture guide | Current components, data flow, and trust boundaries; clearly distinguish proposed changes. | | Deployment guide | Installation, configuration, verification, upgrades, rollback, and recovery procedures. Separate reusable instructions from maintainer-specific operations. | @@ -229,7 +228,7 @@ Use the version-neutral initiative **MetaCortex self-hosted adoption**. Planning - Use `software` cards for implementation and `task` cards for reconciliation, external acceptance, publication, and operational handoff. - Preserve unrelated queue ordering. Leave placement unspecified unless a verified gate requires it. - Read the board back to verify backlog state (or mandatory policy review block), dependencies, gate references, and duplicate absence. -- Write actual card numbers into the canonical roadmap and derive `NEXT-STEPS.md` from the immediate prerequisites. +- Write actual card numbers into the canonical roadmap. ## 5. Verification and completion @@ -262,8 +261,6 @@ All claims must be supported by evidence from the exact release candidate. Passi | `README.md` | Retained. States six tools and legacy URL-token status. | | `CLAUDE.md`, `AGENTS.md` | Corrected tool count to six. | | `journey-kit/` (`kit.md`, `README.md`, `examples/browser-client-setup.md`) | Corrected. Secrets moved from `.env.prod` to Secret Manager, tokenized ChatGPT URL no longer the recommended path, "secure" and "one-click" claims removed, model name matched to code default. Kit version number not changed. | -| `docs/superpowers/` plan and spec | Retained as dated records. | -| `docs/operations/2026-09-09-secret-migration.md` | Retained as historical evidence. | | `slides.html` | Corrected. Removed $0 cost, infinite-scaling, near-zero-latency, one-click, "secure", daily-cron, and unverified native-client claims. Tool count now six. Deletion wording now matches soft-deprecation. | | `studies/ci-recurrence/` | Archived 2026-10-10 as a dead end, not on `release`. Study code is tag `archive/ci-recurrence-study` and its retained data is tag `archive/ci-recurrence-data`. It has no MetaCortex claims. | @@ -287,7 +284,7 @@ The strategic plan at `08d144f:metacortexplan.md` is preserved in Git. Its subst | Competitive matrix and reliability claims about other products | Removed from the active roadmap as unsupported and unnecessary for self-hosted acceptance. Historical text remains in Git. | | Old scouting suggestions to prune/discard/publish work | Superseded by the preservation inventory and per-card evidence in the baseline record. No age-based deletion or publication. | -The prior NEXT-STEPS material was not tracked at release commit `08d144f`; its current replacement is a dated view of this roadmap. Dated archive designs retain their historical role. Screenshots and presentation assets are explanatory material, not release acceptance evidence. +Screenshots and presentation assets are explanatory material, not release acceptance evidence. ### Board reconciliation, 2026-10-02 @@ -297,7 +294,7 @@ DOC/BASE/VERIFY retain delivered local evidence and remaining review/CI/deployme ### 2026-10-03 UTC execution update -Card 66 landed through PR #12 and is done: production revision `metacortexmcp-00030-zaj`, profile secret version 2, nine scoped clients and admin verified. The [rollout record](docs/operations/2026-10-03-card66-rollout.md) distinguishes endpoint checks from long-running client catalog refreshes. Baseline `9c3feac` is pushed and its hosted typecheck/test/package gates passed. The earlier local-only and undeployed statements in dated records describe the state before this rollout. +Card 66 landed through PR #12 and is done: production revision `metacortexmcp-00030-zaj`, profile secret version 2, nine scoped clients and admin verified. Baseline `9c3feac` is pushed and its hosted typecheck/test/package gates passed. The earlier local-only and undeployed statements in dated records describe the state before this rollout. The release integration includes the intended roadmap, archive/backup code, verification contract and packaging corrections. Unrelated study data and personal settings were preserved on `codex/self-hosted-adoption-baseline`, which was deleted on 2026-10-10 after its tip was tagged `archive/ci-recurrence-data`. They are not part of the release integration or Journey package. DOC/BASE/VERIFY acceptance precedes CORE #70; remaining cards stay queued until selected. diff --git a/scripts/build-journey-kit.mjs b/scripts/build-journey-kit.mjs index 79ef945..1cf7e01 100644 --- a/scripts/build-journey-kit.mjs +++ b/scripts/build-journey-kit.mjs @@ -334,11 +334,6 @@ const SOURCE_FILE_MANIFEST = [ role: "documentation", description: "Bundled metacortexplan.md." }, - { - path: "NEXT-STEPS.md", - role: "documentation", - description: "Bundled NEXT-STEPS.md." - }, { path: "docs/DEPLOYMENT.md", role: "documentation", @@ -374,21 +369,6 @@ const SOURCE_FILE_MANIFEST = [ role: "documentation", description: "Bundled docs/OPENCLAW_MEMORY_OPS.md." }, - { - path: "docs/operations/2026-09-09-secret-migration.md", - role: "documentation", - description: "Bundled docs/operations/2026-09-09-secret-migration.md." - }, - { - path: "docs/operations/2026-10-03-card66-rollout.md", - role: "documentation", - description: "Card 66 deployed contract and live verification evidence." - }, - { - path: "docs/operations/2026-10-01-adoption-baseline.md", - role: "documentation", - description: "Bundled docs/operations/2026-10-01-adoption-baseline.md." - }, { path: "journey-kit/examples/browser-client-setup.md", role: "documentation",