diff --git a/AGENTS.md b/AGENTS.md index 9b2cff5d2..f679d90d3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -48,11 +48,17 @@ One entry point, and every arm is the exact command CI runs: ./run.sh all # the release gate: lint, docs, identity, full, gnn ``` -`lint` and `docs` are offline and take seconds. `fast` and `full` materialize the -pinned Journal-Utilities revision from -[`.github/journal-utilities-pin.json`](.github/journal-utilities-pin.json); pass -`--utilities PATH` to reuse an existing tree, which is refused unless it sits on -the pin. There is no `types` arm: this repository ships no typed source. Never +`lint`, `docs` and `identity` are offline. `fast` and `full` execute a fresh, +context-managed snapshot of the Journal-Utilities commit in +[`.github/journal-utilities-pin.json`](.github/journal-utilities-pin.json). +`--utilities PATH` supplies Git objects, never executable working-tree bytes: +HEAD may differ, but the exact commit must exist. A `.gate-pin` stamp alone is +insufficient; caller bytes are preserved and the snapshot is removed on exit. +An invalid explicit source fails without acquisition; default cold acquisition +can need network. `all` validates every option before running arms, then stops +at the first nonzero result in lint/docs/identity/full/gnn order. Its `--utilities` +option goes only to `full`, and `--cold` only to `gnn`. +There is no `types` arm: this repository ships no typed source. Never add a command to a workflow without adding the same command to [`scripts/gate/cli.py`](scripts/gate/cli.py). diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 247c95a4a..a2f94d2a3 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -51,11 +51,18 @@ call `./run.sh`, they do not carry a second copy of any command. ./run.sh all # everything a pull request must pass ``` -`lint` and `docs` run offline in seconds; `fast`/`full` use the Journal-Utilities -revision pinned in `.github/journal-utilities-pin.json` (an explicit -`--utilities PATH` is accepted only when it sits on that pin). What each arm -actually proves, plus the current job layout, is in -[`docs/OPERATIONS.md`](docs/OPERATIONS.md). +`lint`, `docs` and `identity` run offline. `fast`/`full` execute a fresh, +context-managed snapshot of the Journal-Utilities revision pinned in +`.github/journal-utilities-pin.json`. `--utilities PATH` supplies Git objects +containing that commit; HEAD and dirty working-tree bytes are not executable +authority. A `.gate-pin` stamp alone is insufficient. Caller content is preserved +and the snapshot is removed on exit. Invalid explicit sources fail without +acquisition; default cold acquisition can require network. + +`all` validates its complete option list before running lint, docs, identity, +full and gnn, stopping at the first nonzero result. Its `--utilities PATH` goes +only to `full`, and `--cold` only to `gnn`. See +[`docs/OPERATIONS.md`](docs/OPERATIONS.md) for each arm's limits and CI layout. ## Citation diff --git a/README.md b/README.md index 3a37e4831..12b389872 100644 --- a/README.md +++ b/README.md @@ -23,9 +23,12 @@ Tooling: https://github.com/ActiveInferenceInstitute/Journal_Utilities ## Status -The content on `main` is **complete and current** against the Institute channel: every -non-duplicate channel video is a part in exactly one item, and the read-only integrity -gate passes in CI. The repository is **generated**: the per-item `metadata.json` core +The coverage target is that every non-duplicate video in the supplied channel +manifest belongs to exactly one canonical item. Validation uses the pinned +Journal-Utilities manifest; it does not enumerate the live channel or establish +current hosted CI status. Billing-lock failures were recorded on 2026-11-12 +(see [`TO-DO.md`](TO-DO.md)); those dated receipts do not establish today's hosted +availability or results. The repository is **generated**: the per-item `metadata.json` core keys, `transcript.*`, `captions/`, `translations/`, `INDEX.json`/`INDEX.md` and the item/series `README.md` files are produced by [Journal-Utilities](https://github.com/ActiveInferenceInstitute/Journal_Utilities) and @@ -36,8 +39,9 @@ channel ingest run from a Journal-Utilities checkout (see - **Works offline:** cloning, browsing, grepping and editing the hand-editable files (see [`CONTRIBUTING.md`](CONTRIBUTING.md)). -- **Needs the sibling Journal-Utilities checkout:** index/transcript validation and the - static-site build. +- **Uses pinned Journal-Utilities:** index/transcript validation executes a fresh + exact-commit snapshot. A sibling working tree is not required as executable + source; the static-site workflow uses its separately pinned producer checkout. - **Needs network:** the weekly ingest workflow (YouTube enumeration via `yt-dlp`). - **Not on `main`:** audio — check out the `audio` branch for the 64 kbps media. @@ -62,8 +66,9 @@ INDEX.md # human index, grouped by series Current item and video totals are generated in [`INDEX.json`](INDEX.json). Deliberate duplicate records are marked with `duplicate_of`; the index reports -both indexed video records and unique video IDs. Every Institute channel video -is represented (uncategorized videos live under `Other/`). +both indexed video records and unique video IDs. Coverage is assessed against the +supplied manifest, not guaranteed against the live channel. Indexed videos without +a more specific series live under `Other/`. ## Branches @@ -85,13 +90,13 @@ or [`INDEX.md`](INDEX.md) (human), then read each item's `metadata.json` and `transcript.txt`. The static reader site is published at https://activeinferenceinstitute.github.io/ActiveInferenceJournal/. -There is nothing to install or build here. To **regenerate or validate** the repository, -run the gate from this checkout (`./run.sh` brings up the pinned Journal-Utilities -revision by itself), or run the Journal-Utilities scripts directly from that checkout -when you are changing the generator (enrichment is dry-run unless `--apply` is passed): +Browsing requires nothing to install or build. Validation runs from this checkout; +cold Journal-Utilities acquisition and GNN provisioning can require network and +an appropriate runtime. Run Journal-Utilities directly when changing the generator +(enrichment is dry-run unless `--apply` is passed): ```bash -./run.sh all # lint, docs, identity, full, gnn — the same commands CI runs +./run.sh all # lint, docs, identity, full, gnn; stop at first failure ``` ```bash @@ -116,8 +121,7 @@ See [Operations](docs/OPERATIONS.md) for every arm, workflow and pin. ## Provenance -Transcripts and metadata are pulled completely and idempotently from the Institute -YouTube channel by +Transcripts and metadata are produced from Institute YouTube content by [Journal-Utilities](https://github.com/ActiveInferenceInstitute/Journal_Utilities). Most transcripts are WhisperX-diarized: `transcript.json` holds the immutable raw segments (`SPEAKER_NN`), human speaker identifications are recorded in diff --git a/TO-DO.md b/TO-DO.md index 9f865ebe3..87a1954cb 100644 --- a/TO-DO.md +++ b/TO-DO.md @@ -190,13 +190,13 @@ overhaul, cross-cutting refactors. ## Open / deferred -- **Hosted GitHub Actions is locked, org-wide.** Every run dies before its first - step with "The job was not started because your account is locked due to a - billing issue" (run `43252922624` for `estate/sidecar-defaults`; the CodeQL - default-setup run `43264122709` fails the same way, 2026-11-12). No pull - request can show a green check until the organisation's billing is restored, so - `./run.sh all` locally is the only evidence a PR can carry. Owner: Institute - GitHub organisation admin; the fix is outside this repository. +- **Hosted billing-lock failures were recorded on 2026-11-12.** Run + `43252922624` for `estate/sidecar-defaults` and CodeQL default-setup run + `43264122709` failed before their first step with "The job was not started + because your account is locked due to a billing issue". These are dated + receipts, not a fresh observation of hosted availability. Local gate evidence + is separate and cannot establish hosted success. Owner: Institute GitHub + organisation admin; billing administration is outside this repository. - **Legacy translation triage (I16).** 423 `.srt` files remain in capital-T `Translations/` across 30 items, and 600 quarantined files sit under `translations/conflicts/` across 44 items (counts re-derived from @@ -205,24 +205,22 @@ overhaul, cross-cutting refactors. Each remaining file needs a human to confirm its `video_id` and language before the two-step `git mv` — no script can settle a guess, which is why the M2 pass stopped here. Owner: volunteers, coordinated from that page. -- **Weekly ingest is still a dispatch-only draft, and M5 does not exist.** - `weekly-ingest.yml` has no `schedule:` and its guard fails any non-dispatch - run; correspondingly `arm_manifest.py --strict` (manifest age and duplicate - ids as failures) is documented as landing "when that lands" - ([`docs/OPERATIONS.md`](docs/OPERATIONS.md), "Weekly ingest (draft)"). - Owner: maintainers; blocked on reviewing Journal-Utilities `docs/m3-ingest.md` - and removing the guard, which is outside this repository. +- **Weekly ingest remains a dispatch-only draft; scheduling/policy is pending.** + `weekly-ingest.yml` has no `schedule:` and rejects non-dispatch runs. + `arm_manifest.py --strict` already exists and fails age/duplicate-ID warnings; + `full` does not select it. Owner: maintainers, after reviewing the producer's + `docs/m3-ingest.md` and deliberately deciding scheduling and strictness. - **Coordinate member adoption of the Journal's GNN pin.** The lead pin advances to `v5.1.0` (`75c834bd`). Each of the eight member copies moves with its own conformance evidence after the lead revision is accepted. Follow the [Publishing contract](docs/workflows/publishing.md#the-drift-test) for the canonical bytes and digest. Owner: the Publishing group maintainers. -- **`estate/sidecar-defaults` is unmerged with no PR.** One commit ahead of - `main` (`a93f103c`), changing only `.aii/config.yaml` (drops - `relation: references` from the Active_Inference_Ontology relation). Land it - through the sidecar generator or delete the branch. Owner: whoever ran the - estate sidecar pass; nothing in this repository blocks it except the hosted - gate being unavailable. +- **Sidecar branch disposition remains owner-held.** The 2026-11-12 observation + recorded `estate/sidecar-defaults` as one commit ahead (`a93f103c`), unmerged + with no PR, changing `.aii/config.yaml`'s ontology `references` relation. + That is historical branch/PR evidence, not a fresh inquiry or deletion + authorization. Owner: the estate sidecar curator; use canonical generated + projections for any approved adoption. - **`INDEX.md` line width** — generator output: the summary line is 143 characters, item lines exactly 140. `generate_journal_indexes.py --check` compares the file byte-for-byte with generator output, so it cannot be diff --git a/data/video/activeinferenceinstitute/Courses/PhysicsAsInformationProcessing_ChrisFields/assets/html/TO-DO_and_Transcript_Links.html b/data/video/activeinferenceinstitute/Courses/PhysicsAsInformationProcessing_ChrisFields/assets/html/TO-DO_and_Transcript_Links.html index e2b569aa6..501aaa1dc 100644 --- a/data/video/activeinferenceinstitute/Courses/PhysicsAsInformationProcessing_ChrisFields/assets/html/TO-DO_and_Transcript_Links.html +++ b/data/video/activeinferenceinstitute/Courses/PhysicsAsInformationProcessing_ChrisFields/assets/html/TO-DO_and_Transcript_Links.html @@ -4,14 +4,12 @@

TO-DO, Prose, and Slide links in Coda and Github


Coda Prose and Slides Links

-Github Session Links
+Github Session Links

-

-

- +
\ No newline at end of file diff --git a/data/video/activeinferenceinstitute/Courses/PhysicsAsInformationProcessing_ChrisFields/assets/notes/TO-DO_and_Transcript_Links.md b/data/video/activeinferenceinstitute/Courses/PhysicsAsInformationProcessing_ChrisFields/assets/notes/TO-DO_and_Transcript_Links.md index 11e77b931..c15408494 100644 --- a/data/video/activeinferenceinstitute/Courses/PhysicsAsInformationProcessing_ChrisFields/assets/notes/TO-DO_and_Transcript_Links.md +++ b/data/video/activeinferenceinstitute/Courses/PhysicsAsInformationProcessing_ChrisFields/assets/notes/TO-DO_and_Transcript_Links.md @@ -3,4 +3,4 @@ [Coda Prose and Slides Links](https://coda.io/d/ActInf-Journal_dwYsKMwppRN/Chris-Fields-Physics-Livestreams-Status_su7Pa#_lu6bc) -[Github Session Links](https://github.com/ActiveInferenceInstitute/ActiveInferenceJournal/blob/main/Courses/PhysicsAsInformationProcessing_ChrisFields) +[Github Session Links](../../) diff --git a/data/video/activeinferenceinstitute/Courses/PhysicsAsInformationProcessing_ChrisFields/assets/notes/TO-DO_and_Transcript_Links_Lecture-1.md b/data/video/activeinferenceinstitute/Courses/PhysicsAsInformationProcessing_ChrisFields/assets/notes/TO-DO_and_Transcript_Links_Lecture-1.md index a48e6dd58..0564808c1 100644 --- a/data/video/activeinferenceinstitute/Courses/PhysicsAsInformationProcessing_ChrisFields/assets/notes/TO-DO_and_Transcript_Links_Lecture-1.md +++ b/data/video/activeinferenceinstitute/Courses/PhysicsAsInformationProcessing_ChrisFields/assets/notes/TO-DO_and_Transcript_Links_Lecture-1.md @@ -6,13 +6,13 @@ Editorial status: (unknown) -[Prose link](https://github.com/ActiveInferenceInstitute/ActiveInferenceJournal/tree/main/Courses/PhysicsAsInformationProcessing_ChrisFields/Lecture_1/Transcripts/Prose) +[Prose link](../prose/cFPIP-01L.md) -[Session files](https://github.com/ActiveInferenceInstitute/ActiveInferenceJournal/tree/main/Courses/PhysicsAsInformationProcessing_ChrisFields/Lecture_1) +[Session files](../../) ## Course-level information [Coda TO-DO Links](https://coda.io/d/AIJ-Physics-course_d4OKzETEXqj/TO-DO_suEjI#_luD5m) [Coda Prose and Slides Links](https://coda.io/d/ActInf-Journal_dwYsKMwppRN/Chris-Fields-Physics-Livestreams-Status_su7Pa#_lu6bc) -[All Github Sessions](https://github.com/ActiveInferenceInstitute/ActiveInferenceJournal/blob/main/Courses/PhysicsAsInformationProcessing_ChrisFields) +[All Github Sessions](../../) diff --git a/docs/OPERATIONS.md b/docs/OPERATIONS.md index be6278a1d..3e23acfe7 100644 --- a/docs/OPERATIONS.md +++ b/docs/OPERATIONS.md @@ -22,17 +22,18 @@ is duplicated in YAML. | Arm | Cost | What it proves | | --- | --- | --- | | `lint` | seconds | Every `uses:` is pinned to a 40-hex commit, every workflow declares `name`/`on`/`permissions`, the Journal-Utilities checkout takes its ref from the pin file rather than a literal SHA, both pin files match their schema, and shipped JSON parses. | -| `docs` | seconds | Every relative link in every tracked Markdown file resolves with the exact case GitHub serves, in-page anchors exist, each doc under `docs/` is reachable from [`documentation-index.md`](documentation-index.md), the timings table covers exactly the arms, and every documented pin prefix matches a pin file. | -| `identity` | ~0.1 s | The publishing group contract: [`workflows/record-identity.schema.json`](workflows/record-identity.schema.json) parses and declares `record-identity/v1`, the digest recorded in [`workflows/publishing.md`](workflows/publishing.md) matches the file (the same `--verify-copy` path a member runs), every `item` names its own directory, every `series` names a directory above it, and every `parts[].video_id` is an 11-character id. | +| `docs` | seconds | Relative Markdown links, exact-case paths, anchors, docs-index reachability, timing-table coverage and documented pins; isolated real-file regressions preserve inline-code heading words and duplicate suffixes while ignoring code links, comments and fences. | +| `identity` | unmeasured for this source cohort | Contained corpus paths; schema-declared INDEX/metadata series, item, ordered parts and duplicate projection; Boolean transcript presence; session-to-part membership; local artifact digests and real-file selftests. This is not a general JSON Schema validator or live-channel check. | | `fast` | ~1 s | `generate_journal_indexes.py --journal . --check` against the pinned Journal-Utilities: `INDEX.json`/`INDEX.md` are not stale. | | `full` | ~30 s | The split-transcript check, the read-only `validate_journal.py` gate, and the manifest staleness/reconciliation report (E13). | | `gnn` | minutes, network | The pinned GNN parser installs at the pinned revision, reports the pinned version, and parses/type-checks every `*.gnn.md` sketch (or the pinned upstream exemplar when the repository has none). The arm accepts only a literal `True` from the type checker's three-valued sentinel (`True`/`2`/`False`) and runs its own verdict selftest so a truthiness test cannot regress. | There is no `types` arm: this repository ships no typed source. Arms are -stdlib-only Python, so there is no environment to install for `lint`, `docs`, -`fast` or `full`. `gnn` builds its own isolated environment because the pinned -package declares `requires-python >=3.11,<3.15`; on a host without such an -interpreter, set `GATE_PYTHON` or run `uv python install 3.12`. +stdlib-only Python for the offline `lint`, `docs` and `identity` arms. `fast` and +`full` also need the pinned Journal-Utilities objects and their admitted runtime. +`gnn` builds an isolated environment because its pinned package declares +`requires-python >=3.11,<3.15`; provisioning requires separate authority and an +appropriate interpreter. That environment is keyed by repository, **revision** and interpreter, and the installed parser is bound to the pin by the commit id pip records in @@ -58,12 +59,15 @@ one-file, reviewable change. Measured wall time per arm on this host, before and after the gate existed, is generated in [`_generated/gate-timings.md`](_generated/gate-timings.md). +Those are historical measurements, not timings for the newly qualified identity +and docs implementations. Generated timing artifacts remain unchanged until a +separately admitted real measurement refresh. ## Workflows | Workflow | File | Trigger | Purpose | | --- | --- | --- | --- | -| Validate journal integrity | [`journal-integrity.yml`](../.github/workflows/journal-integrity.yml) | `push`, `pull_request`, `workflow_call` | Release gate: `lint`, `docs`, `fast`, `full` against the pinned Journal-Utilities. | +| Validate journal integrity | [`journal-integrity.yml`](../.github/workflows/journal-integrity.yml) | `push`, `pull_request`, `workflow_call` | `lint`, `docs`, `identity`, `fast`, `full` against the pinned Journal-Utilities; GNN is separate. | | GNN conformance | [`gnn-conformance.yml`](../.github/workflows/gnn-conformance.yml) | `push`, `pull_request` | The `gnn-conformance` job, display name `gnn conformance (pinned)`: `./run.sh gnn`. | | Deploy GitHub Pages | [`deploy-pages.yml`](../.github/workflows/deploy-pages.yml) | `push` to `main` touching `data/**` or `INDEX.json`; `workflow_dispatch` | Runs the integrity gate, then builds and deploys the static site bundle. | | Weekly ingest | [`weekly-ingest.yml`](../.github/workflows/weekly-ingest.yml) | `workflow_dispatch` only (draft) | Secret-free channel enumeration, new-video delta, scaffold preview, reconciliation report. | @@ -80,15 +84,18 @@ name (`gnn conformance (pinned)`) are unchanged; only its workflow file moved. ### Validate journal integrity The `validate` job checks out the journal and the pinned Journal-Utilities -revision, then runs the four gate arms. The steps that used to be inline Python +revision, then runs lint, docs, identity, fast and full. The steps formerly inline heredocs in the workflow are now `scripts/gate/arm_manifest.py` and the `fast`/`full` arms, so the locally reproduced command and the CI command are literally the same file. -The reconciliation step reports manifest age and any channel/manifest/INDEX -disagreement; a manifest older than 7 days and duplicate ids are warnings, not -failures, until the scheduled refresh (M5) exists. `arm_manifest.py --strict` -turns them into failures when that lands. +The reconciliation script reports manifest age, duplicate IDs and differences +between manifest and INDEX part IDs. In the current `full` invocation, whole-day +`age_days > 7` and duplicate IDs warn without failing this reporting step. +The existing `arm_manifest.py --strict` makes either warning return 1; `full` +does not pass it. Set differences remain printed diagnostics even in strict +mode; producer integrity validation is a separate check. Strict-policy adoption +and scheduled refresh remain maintainer decisions. ### Deploy GitHub Pages @@ -125,14 +132,21 @@ reviewed changes to the pin files rather than automatic updates. `./run.sh all` is the release gate, and it is what a pull request should be screened with. Two notes about running it here: -- `fast` and `full` need the pinned Journal-Utilities tree. With no arguments the - gate exports it from a local clone (`$GATE_JU_CLONE`, else `../Journal_Utilities`) - into `$GATE_CACHE` (default: the system temp directory) and reuses it; if the - clone lacks the pinned commit it fetches it. Pass `--utilities PATH` to use an - existing tree - it is refused unless it is at the pin. +- `fast` and `full` execute fresh context-managed snapshots of pinned Git objects. + Warm objects can be reused, but executable snapshots are never reused, including + in CI. `--utilities PATH` or `$GATE_JU_CLONE` is an explicit object source: + it must already contain the exact commit, regardless of HEAD; invalid explicit + sources fail without fetching. Without an explicit source, local/cache objects + are preferred and cold acquisition uses a new owned bare store. A `.gate-pin` + stamp alone is insufficient. Caller working-tree bytes are preserved. - `gnn` needs network on a cold cache and a Python in the pinned package's range. +- `all` validates every option before effects, then runs lint, docs, identity, + full and gnn, stopping at the first nonzero result. Its `--utilities PATH` + goes only to full and `--cold` only to gnn. Direct fast/full `--utilities`, + direct gnn `--cold` and bench `--cold` remain available. -Offline, `lint` and `docs` are the whole gate, and both are seconds. +`lint`, `docs` and `identity` are offline subsets, not release, scientific, +hosted-CI or live-channel acceptance. ## Branch model diff --git a/docs/SCHEMA.md b/docs/SCHEMA.md index dcfaf3ba9..77d41c3d6 100644 --- a/docs/SCHEMA.md +++ b/docs/SCHEMA.md @@ -130,6 +130,10 @@ corresponding display fields without losing the source text: to its standalone per-talk upload when one exists; that upload's own item carries `duplicate_of` back to the full recording. `title` may be a segment label ("Roundtable") when the talk has no distinct title. + `session_name` captures a `video_id` belonging to this record's `parts`; + optional `sessions[].video_id` is a separate standalone-upload link and need + not equal a part ID. Identity checking does not add timestamp, title or + session-numbering policy beyond the schema grammar. **Ownership: this repo.** The enrichment script *seeds* `sessions[]` — from YouTube chapter lists (timestamped video descriptions, cached by `fetch_chapters.py`) or curated data — only when an item has none, and @@ -149,6 +153,11 @@ corresponding display fields without losing the source text: home is the full-recording item). The duplicate intentionally repeats the canonical video's ID; coverage reconciliation excludes the secondary record and `INDEX.json` exposes both record and unique video counts. + Its value is the canonical path with `data///` removed, not a + full INDEX path. If projected into INDEX, the optional marker must agree with + metadata and resolve to a contained indexed record. Raw transcript blocks, + URL agreement, canonical-video uniqueness and manifest reconciliation remain + the pinned Journal-Utilities validator's responsibility. - **`data/video/activeinferenceinstitute/private_videos.json`** documents private/unlisted channel videos known to the Institute (from the legacy session database) that are deliberately absent from this public corpus. diff --git a/docs/workflows/publishing.md b/docs/workflows/publishing.md index a1495aba2..71d6f1a65 100644 --- a/docs/workflows/publishing.md +++ b/docs/workflows/publishing.md @@ -6,11 +6,12 @@ implementations of everything else. This page is the contract between them: whic repository owns which concern, what a record's identity is, and the one step each member takes to stop carrying its own copy of a solved problem. -The contract is enforceable, not aspirational. Every rule below is checked by -`./run.sh identity` in this repository against -[`record-identity.schema.json`](record-identity.schema.json), and a member can run the -same check against its own copy with +`./run.sh identity` checks the schema-declared record relationships and local +artifact agreement described below, using +[`record-identity.schema.json`](record-identity.schema.json) and [`scripts/gate/identity.py`](../../scripts/gate/identity.py). +It is not a general JSON Schema validator, channel-freshness check, hosted-CI +receipt, scientific qualification or authenticated upstream release. ## Evidence rule @@ -70,7 +71,8 @@ schema, not a second definition of it. | --- | --- | --- | --- | --- | | `item` | item id | this repository | `metadata.json` `item`; `INDEX.json` `path` | equals the item directory name | | `video` | 11-character YouTube id | this repository records; YouTube assigns | `metadata.json` `parts[].video_id`; `INDEX.json` `parts[]` | `^[A-Za-z0-9_-]{11}$` | -| `session` | `_sessNN` | this repository | `metadata.json` `sessions[].session_name`; `transcript.json` blocks | the leading 11 characters are a part's `video_id`, `NN` is two digits | +| `session` | `_sessNN` | this repository | `metadata.json` `sessions[].session_name`; `transcript.json` blocks | the leading 11 characters are a part's `video_id`, `NN` is a positive ordinal zero-padded to at least two digits (`01`–`99`, `100+`) | +| `session upload` | optional standalone video id | this repository | `metadata.json` `sessions[].video_id` | a separate upload link, not required to equal a part ID | | `paper` | `doi:`/`arxiv:`/`s2:`/`openalex:`/`title:` id | act_inf_metaanalysis | `output/data/corpus.jsonl` `paper_id`; this repository's `metadata.json` `doi` | prefix decides the scheme | | `doc` | `document_name` | ISSS | `archived_documents/metadata.json` `document_name` | the basename of the source URL | | `pub` | DOI + version | act_inf_metaanalysis | `manuscript/config.yaml` `doi` + `version` | one deposited publication | @@ -96,6 +98,9 @@ the first three path components (`data///`) are dropped from the r path. The value is therefore **not** an `INDEX.json` path and **not** a bare item id. A consumer that concatenates `duplicate_of` onto an `INDEX.json` path gets a path that does not exist; a consumer that matches it as a suffix gets the right record. +When `duplicate_of` is present in INDEX, its optional projection must agree +with metadata and resolve to an admitted indexed record under the same source +prefix. Identity checking adds no cycle, self-reference or video-equality policy. ### What carries no identity @@ -138,8 +143,11 @@ line per copy. `./run.sh identity` recomputes every one, so an edited artifact w digest fails this repository's gate: ```text -record-identity.schema.json 84f6c4df6efc3f15f0d8a9a05c8aabd0cbdc08b0f4feac5fcc8ae860a81e306b -identity.py 847328ff163fa90c869495164869caf4e83313748e8a444dbab8d4e459b11074 +record-identity.schema.json e1e51b19fc3e6e90f478a0138ceef5ce53a57489573412aaf5128c933c509147 +identity.py a7258f6a92e95437091332de222066f4bdf89f78df16dc82867f29d751654100 +identity_selftest.py 8021a807d62ece306e4076ca5b418f6c11356967f553f749f80eb0be3f1577cf +arm_docs.py fdab3423cc0ef72f38f7f1e3037d7f71a74aa81f25a10b1adf006c73c3b816b2 +docs_selftest.py 2d5c7b24f22ad78dbd0d3d799d790909b5e519968a820f7e02756f5ad396f5d8 gnn-pin.json 397e5f2803fadb6dfde336dbacf50ed195090effff1fdd13f54d0a8813cc1fa9 ``` @@ -155,6 +163,21 @@ compares it to the file. A copy that has drifted, or a digest that was updated w the file, fails. The lead repository runs exactly this command as part of `./run.sh identity`, so the path members use is exercised by the lead's gate rather than assumed to work. +These digests bind source artifacts, not their Git provenance. An adopting +owner must select an actual immutable producer revision and update selected +copies and their digest contract together; a dirty base revision cannot +identify new working-tree bytes. `--verify-copy` +checks exactly the caller-selected file against its local contract: it does not +fetch or authenticate a producer, establish freshness, or validate a member's +research corpus. + +Knowledge-Engineering's archive consumer verifies copies, not a fabricated local +Journal corpus. Garden-of-Iris retains its own Markdown inventory, adapter and +generated-facts check. Its direct `check_links()`/`check_docs_index()` path needs +the selected `arm_docs.py` and existing dependencies; adopting/exposing the +native docs arm also requires `docs_selftest.py`. Adopting the new identity +arm/selftests requires the coherent identity module, selftest and unchanged +schema. No member copies or pins are changed by lead source qualification. ## Migration step for every member @@ -176,8 +199,9 @@ private copy of a shared concern; point at the canonical one and pin it by diges - `scripts/gate/identity.py` takes no configuration; it reads the schema. - add ` ` for every copy you keep to your contract copy, and run `python3 scripts/gate/identity.py --verify-copy ` from your gate. - `identity` itself runs a selftest of that check, so a passing run means both - directions were exercised: a matching `.py` copy passes, a diverged one fails. + Identity real-file selftests exercise matching and independently diverged + copies plus corpus/schema/path relationships; docs selftests exercise actual + Markdown anchors, masked links and index reachability. 3. Replace a prose mirror with a pointer. Journal_Utilities' `docs/JOURNAL_SCHEMA.md` and this repository's [`../SCHEMA.md`](../SCHEMA.md) must not both define the schema; the mirror keeps its "How the pipeline writes this" chapters and links here for identity. @@ -186,10 +210,16 @@ private copy of a shared concern; point at the canonical one and pin it by diges enforces the `x-identity` block directly. Where a member's own gate already proves something this contract mentions, keep that and -do not duplicate it. `validate_journal.py` in the pinned Journal-Utilities revision is the -authority for the raw transcript layer (blank session ids, part URL agreement, -`duplicate_of` resolution, manifest reconciliation); this repository's `identity` arm -deliberately does not re-check it. +do not duplicate it. This identity arm admits corpus-derived paths inside the +selected root before content reads; INDEX series/item/ordered part IDs and the +optional duplicate marker agree with metadata, the Boolean transcript flag +matches its regular artifact, and each schema-parsed session name belongs to a +part. Series ancestry stops at the corpus boundary. Contained symlinks and an +explicit symlinked root remain supported; this is not descriptor-level defense +against an adversarial concurrent writer. The pinned Journal-Utilities +`validate_journal.py` remains responsible for raw transcript blocks, URL +agreement, canonical-video uniqueness and manifest reconciliation. Caption and +translation filenames carry no inferred identity. ## Tools this host does not have @@ -203,10 +233,15 @@ that is an extra check, not the contract. ## Keeping this page honest -`./run.sh identity` fails when the schema does not parse, when its `schema_version` is not -`record-identity/v1`, when its `x-identity` block is incomplete, when any recorded digest -above does not match its file or names no single artifact, when the check's own selftest -does not prove both directions, when a record's `item` does not name its own directory, when -its `series` names no directory above it, or when a part's `video_id` is not an -11-character id. `./run.sh docs` fails when a link on this page does not resolve with the -exact case GitHub serves. +`./run.sh identity` fails for an invalid/incomplete schema contract, missing or +ambiguous artifacts, escaping discovered artifact paths, stale digests, or +real-file selftest failures. Corpus admission rejects unsafe/escaping INDEX, +record, metadata and transcript paths before content reads. It reconciles +metadata/path identity, the complete required INDEX projection and ordered +parts, strict Boolean transcript presence, optional session membership and +optional source-relative duplicate targets. The schema version remains +`record-identity/v1`; legitimate null category/episode values and empty parts +without sessions/transcripts remain supported. `./run.sh docs` checks links, +anchors, index coverage and isolated real-Markdown regressions. Neither arm +establishes immutable delivery or substitutes for the remaining producer/GNN +and member-owned acceptance. diff --git a/docs/workflows/record-identity.schema.json b/docs/workflows/record-identity.schema.json index 46096c1fb..e8773e01f 100644 --- a/docs/workflows/record-identity.schema.json +++ b/docs/workflows/record-identity.schema.json @@ -86,7 +86,7 @@ }, "patterns": { "video_id": "^[A-Za-z0-9_-]{11}$", - "session_id": "^(?P[A-Za-z0-9_-]{11})_sess[0-9]{2}$", + "session_id": "^(?P[A-Za-z0-9_-]{11})_sess(?:0[1-9]|[1-9][0-9]+)$", "paper_id": "^(?:doi|arxiv|s2|openalex|title):.+$" }, "namespaces": { diff --git a/scripts/gate/arm_bench.py b/scripts/gate/arm_bench.py index 0cdadde4b..2252a1d95 100644 --- a/scripts/gate/arm_bench.py +++ b/scripts/gate/arm_bench.py @@ -69,15 +69,14 @@ def render(measured: dict[str, float], baseline: dict) -> str: lines += [ "", - "`gnn` is measured **cold**: `./run.sh bench --cold` discards the cached parser", - "environment first, because a warm cache hides the install that dominated the old", - "inline CI step (pip's own wheel cache is *not* cleared, so a first-ever network", - "install is slower than the number here). `fast` is a subset of `full`, and `all`", - "runs `lint`, `docs`, `full` and `gnn`, so nothing is checked twice.", + "`gnn` is measured cold only when `./run.sh bench --cold` requests it;", + "that option discards the parser environment, not pip's wheel cache.", + "`fast` is a subset of `full`. `all` runs `lint`, `docs`, `identity`,", + "`full` and `gnn` in order and stops at the first nonzero result.", "", "`full` is dominated by `validate_journal.py`, which varies by tens of percent", "between runs on the same host; treat a single-run difference of that size as", - "noise, not as a regression. `gnn` is the only arm whose cost is network-bound.", + "noise, not as a regression. Cold Journal-Utilities acquisition and GNN setup can require network.", "", "## What each arm runs", "", @@ -100,10 +99,10 @@ def render(measured: dict[str, float], baseline: dict) -> str: lines.append(f"| {step} | {seconds:.2f}s |") lines += [ "", - "Journal-Utilities is materialized once from the pin into the gate cache and", - "reused by `fast` and `full`; in CI the checkout step already provides it, so", - "that row is a local-only cost, paid inside the first local `fast`/`full` run and", - "reused afterwards; CI does not pay it. The remaining rows are paid by `gnn`.", + "Warm Journal-Utilities Git objects can be reused, but every `fast`/`full`", + "invocation executes a fresh context-managed exact-commit snapshot.", + "A CI checkout supplies objects, not executable working-tree authority;", + "snapshot materialization still occurs in CI. The remaining setup rows are paid by `gnn`.", "", f"Measured {datetime.now(timezone.utc).strftime('%Y-%m-%dT%H:%M:%SZ')} with " f"Python {platform.python_version()} on {platform.system()} {platform.machine()}.", diff --git a/scripts/gate/arm_docs.py b/scripts/gate/arm_docs.py index 39105099f..10c4c0c53 100644 --- a/scripts/gate/arm_docs.py +++ b/scripts/gate/arm_docs.py @@ -19,8 +19,9 @@ FENCE = re.compile(r"^ {0,3}(?:```|~~~)") COMMENT = re.compile(r"", re.DOTALL) -INLINE_CODE = re.compile(r"`[^`\n]*`") +INLINE_CODE = re.compile(r"(?]*>|[^)\s]+)(?:\s+[\"'][^\"']*[\"'])?\s*\)") +HEADING_TEXT = re.compile(INLINE_CODE.pattern + r"|!?\[([^\]]*)\]\([^)]*\)") REFDEF = re.compile(r"^ {0,3}\[[^\]]+\]:\s*(\S+)", re.MULTILINE) SCHEME = re.compile(r"^[a-zA-Z][a-zA-Z0-9+.-]*:") HEADING = re.compile(r"^ {0,3}(#{1,6})\s+(.*?)\s*#*\s*$") @@ -46,8 +47,8 @@ def fail(message: str) -> None: failures.append(message) -def strip_noise(text: str) -> str: - """Drop HTML comments, fenced blocks and inline code, keeping line numbers.""" +def strip_noise(text: str, *, keep_inline_code: bool = False) -> str: + """Suppress comments/fences; retain code words only for heading extraction.""" text = COMMENT.sub(lambda m: "\n" * m.group(0).count("\n"), text) kept: list[str] = [] fenced = False @@ -56,22 +57,22 @@ def strip_noise(text: str) -> str: fenced = not fenced kept.append("") continue - kept.append("" if fenced else INLINE_CODE.sub("``", line)) + kept.append("" if fenced else line if keep_inline_code else INLINE_CODE.sub("``", line)) return "\n".join(kept) def slugify(title: str) -> str: - title = re.sub(r"!?\[([^\]]*)\]\([^)]*\)", r"\1", title) + title = HEADING_TEXT.sub(lambda match: match.group(2) if match.group(1) else match.group(3), title) title = title.replace("`", "").strip().lower() title = re.sub(r"[^\w\s-]", "", title, flags=re.UNICODE) - return re.sub(r"\s+", "-", title) + return re.sub(r"\s", "-", title) def anchors(path: Path) -> set[str]: if path not in _anchors: found: set[str] = set() seen: dict[str, int] = {} - for line in strip_noise(path.read_text(encoding="utf-8")).splitlines(): + for line in strip_noise(path.read_text(encoding="utf-8"), keep_inline_code=True).splitlines(): match = HEADING.match(line) if not match: continue @@ -105,8 +106,8 @@ def links_in(text: str) -> list[tuple[int, str]]: return found -def check_links() -> None: - for path in tracked_markdown(): +def check_links(paths: list[Path] | None = None) -> None: + for path in tracked_markdown() if paths is None else paths: if not path.exists(): continue rel = path.relative_to(ROOT) @@ -192,6 +193,9 @@ def run(argv: list[str]) -> int: check_docs_index() check_timings() check_documented_pins() + import docs_selftest + + failures.extend(docs_selftest.run()) if failures: for message in failures: print(f"docs: FAIL {message}", file=sys.stderr) diff --git a/scripts/gate/arm_identity.py b/scripts/gate/arm_identity.py index 264589cdf..e48d876fc 100644 --- a/scripts/gate/arm_identity.py +++ b/scripts/gate/arm_identity.py @@ -1,13 +1,10 @@ """Arm: identity - the publishing group's record-identity contract. -Three things, and nothing that another arm already proves: - -1. the canonical schema is complete and parses, and the contract's recorded - digest of it is current (``--verify-copy`` exercises the exact code path a - member repository runs against its own copy); -2. every record's item id names its own directory and its series names a - directory above it, so a path built from ``metadata.json`` resolves; -3. every part's ``video_id`` matches the 11-character pattern consumers join on. +The schema declares contained corpus paths, the INDEX/metadata identity +projection, transcript presence and session-to-part membership. The arm checks +those relationships, local artifact digests and real-file regressions. +Series ancestry stops at the selected corpus boundary. This is not a general +JSON Schema validator, live freshness check or authenticated upstream release. The raw transcript layer is deliberately out of scope: the pinned ``validate_journal.py`` in the ``full`` arm already refuses a split record with @@ -41,7 +38,7 @@ def run(argv: list[str]) -> int: if problem: failures.append(problem) - # The members pin Python copies too, so prove that path on every run. + # Real-file corpus admission and matching/diverged artifact-copy regressions. failures += identity_selftest.run() if failures: diff --git a/scripts/gate/cli.py b/scripts/gate/cli.py index db01fd885..79af3f55f 100644 --- a/scripts/gate/cli.py +++ b/scripts/gate/cli.py @@ -2,11 +2,13 @@ ``./run.sh `` execs this module, so the arm a developer runs and the arm CI runs are the same command line. Arms are independent: each one runs alone, and -``all`` runs them in a fixed order. +``all`` validates its options before running them in a fixed order, stopping at +the first failure and returning that arm's status. """ from __future__ import annotations +import argparse import importlib import sys from pathlib import Path @@ -36,7 +38,8 @@ def usage() -> None: for name, description in ARMS.items(): print(f" {name:<{width}} {description}") print( - "\noptions: --utilities PATH use an existing Journal-Utilities tree" + "\noptions: --utilities PATH fast/full/all: snapshot a local Git object source" + "\n --cold gnn/bench/all: rebuild the GNN environment" "\n --write | --check bench only: refresh or verify the timings doc" ) @@ -52,12 +55,33 @@ def main(argv: list[str]) -> int: raise GateError(f"unknown arm {arm!r}") if arm == "all": + parser = argparse.ArgumentParser( + prog="./run.sh all", + description="Run lint, docs, identity, full and gnn; stop at the first failure.", + allow_abbrev=False, + ) + parser.add_argument("--utilities", help="Journal-Utilities Git object source") + parser.add_argument("--cold", action="store_true", help="rebuild the GNN environment") + args = parser.parse_args(rest) + if args.utilities is not None and not args.utilities.strip(): + parser.error("--utilities requires a nonempty path") for name in ORDER: - importlib.import_module(f"arm_{name}").run([]) + options: list[str] = [] + if name == "full" and args.utilities is not None: + options = ["--utilities", args.utilities] + elif name == "gnn" and args.cold: + options = ["--cold"] + result = importlib.import_module(f"arm_{name}").run(options) + if result != 0: + return result return 0 return importlib.import_module(f"arm_{arm}").run(rest) if __name__ == "__main__": - sys.exit(main(sys.argv[1:])) + try: + sys.exit(main(sys.argv[1:])) + except GateError as exc: + print(str(exc), file=sys.stderr) + sys.exit(exc.code) diff --git a/scripts/gate/cli_selftest.py b/scripts/gate/cli_selftest.py new file mode 100644 index 000000000..a35e5e110 --- /dev/null +++ b/scripts/gate/cli_selftest.py @@ -0,0 +1,228 @@ +"""Offline dispatcher regressions using real subprocess exits and real offline arms. + +The process fixtures exercise aggregation, not GNN or producer qualification. +Run explicitly with the existing stdlib Python; no downloads or installations. +""" + +from __future__ import annotations + +import hashlib +import json +import os +import shutil +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + +SOURCE = Path(__file__).resolve().parent +ROOT = SOURCE.parents[1] +ORDER = ["lint", "docs", "identity", "full", "gnn"] + +PROCESS_ARM = '''import subprocess +import sys +from pathlib import Path + +def run(argv): + root = Path(__file__).resolve().parents[2] + return subprocess.run( + [sys.executable, "-B", str(root / "arm_process.py"), + Path(__file__).stem.removeprefix("arm_"), *argv], cwd=root, + ).returncode +''' +PROCESS_BODY = '''import argparse +import json +import os +import sys +from pathlib import Path + +arm, options = sys.argv[1], sys.argv[2:] +parser = argparse.ArgumentParser(prog=arm, allow_abbrev=False) +if arm == "full": + parser.add_argument("--utilities") +elif arm == "gnn": + parser.add_argument("--cold", action="store_true") +parser.parse_args(options) +with Path("executed.jsonl").open("a", encoding="utf-8") as stream: + stream.write(json.dumps({"arm": arm, "options": options}) + "\\n") +sys.exit(json.loads(os.environ["ARM_FAILURES"]).get(arm, 0)) +''' +AUDITED_DISPATCH = '''import runpy +import sys +from pathlib import Path + +def deny_acquisition(event, args): + if event.startswith("socket."): + raise RuntimeError("FORBIDDEN_ACQUISITION") + if event == "subprocess.Popen": + command = args[1] + if isinstance(command, (list, tuple)): + git_fetch = Path(command[0]).name == "git" and any( + word in command for word in ("fetch", "clone", "pull", "push") + ) + install = "-m" in command and any(word in command for word in ("pip", "venv")) + if git_fetch or install: + raise RuntimeError("FORBIDDEN_ACQUISITION") +sys.addaudithook(deny_acquisition) +cli = Path(__file__).resolve().parent / "scripts/gate/cli.py" +sys.argv = [str(cli), *sys.argv[1:]] +runpy.run_path(str(cli), run_name="__main__") +''' + + +class AggregateProcessTests(unittest.TestCase): + def setUp(self): + temporary = tempfile.TemporaryDirectory(prefix="aij-cli-selftest-") + self.addCleanup(temporary.cleanup) + self.root = Path(temporary.name) + self.gate = self.root / "scripts" / "gate" + self.gate.mkdir(parents=True) + for source in SOURCE.glob("*.py"): + shutil.copy2(source, self.gate / source.name) + self.driver = self.root / "dispatch.py" + self.driver.write_text(AUDITED_DISPATCH, encoding="utf-8") + self.env = { + key: value for key, value in os.environ.items() + if not key.startswith(("GIT_", "GATE_", "PYTHON")) + } + self.env.update( + GIT_CONFIG_GLOBAL=os.devnull, + GIT_CONFIG_NOSYSTEM="1", + GIT_ALLOW_PROTOCOL="", + GIT_TERMINAL_PROMPT="0", + GATE_CACHE=str(self.root / "cache"), + ) + + def dispatch(self, options=(), failures=None): + env = {**self.env, "ARM_FAILURES": json.dumps(failures or {})} + return subprocess.run( + [sys.executable, "-I", "-B", str(self.driver), "all", *options], + cwd=self.root, env=env, capture_output=True, text=True, timeout=30, + ) + + def process_arms(self): + (self.root / "arm_process.py").write_text(PROCESS_BODY, encoding="utf-8") + for arm in ORDER: + (self.gate / f"arm_{arm}.py").write_text(PROCESS_ARM, encoding="utf-8") + + def executed(self): + trace = self.root / "executed.jsonl" + return [json.loads(line) for line in trace.read_text().splitlines()] if trace.exists() else [] + + def test_each_nonzero_process_status_stops_the_aggregate(self): + self.process_arms() + for position, arm in enumerate(ORDER): + with self.subTest(arm=arm): + (self.root / "executed.jsonl").unlink(missing_ok=True) + code = 11 + position + result = self.dispatch(failures={arm: code, **{later: 41 for later in ORDER[position + 1:]}}) + self.assertEqual(result.returncode, code, result.stdout + result.stderr) + self.assertEqual([event["arm"] for event in self.executed()], ORDER[:position + 1]) + + def test_all_success_and_options_reach_only_their_owners(self): + self.process_arms() + path = str(self.root / "source with spaces") + result = self.dispatch(["--utilities", path, "--cold"]) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertEqual(self.executed(), [ + {"arm": arm, "options": ["--utilities", path] if arm == "full" else ["--cold"] if arm == "gnn" else []} + for arm in ORDER + ]) + + def test_all_success_without_options(self): + self.process_arms() + result = self.dispatch() + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertEqual(self.executed(), [{"arm": arm, "options": []} for arm in ORDER]) + + def test_invalid_options_have_no_arm_side_effects(self): + self.process_arms() + cases = [ + ["--unknown"], ["--write"], ["--check"], ["--verify-only"], + ["--utilities"], ["--utilities", ""], ["--utilities", " "], + ["--util", "somewhere"], ["--cold", "unexpected"], + ] + for options in cases: + with self.subTest(options=options): + result = self.dispatch(options) + self.assertEqual(result.returncode, 2, result.stdout + result.stderr) + self.assertEqual(self.executed(), []) + + def test_help_does_not_run_an_arm(self): + self.process_arms() + result = self.dispatch(["--help"]) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertEqual(self.executed(), []) + + def offline_corpus(self): + github = self.root / ".github" + github.mkdir() + for name in ("journal-utilities-pin.json", "gnn-pin.json"): + shutil.copy2(ROOT / ".github" / name, github / name) + workflows = self.root / "docs" / "workflows" + workflows.mkdir(parents=True) + schema = workflows / "record-identity.schema.json" + shutil.copy2(ROOT / "docs" / "workflows" / schema.name, schema) + digest = hashlib.sha256(schema.read_bytes()).hexdigest() + (workflows / "publishing.md").write_text(f"{schema.name} {digest}\n", encoding="utf-8") + (self.root / "docs" / "documentation-index.md").write_text( + "[Publishing](workflows/publishing.md)\n", encoding="utf-8", + ) + generated = self.root / "docs" / "_generated" + generated.mkdir() + (generated / "gate-timings.md").write_text( + "\n".join(f"| `{arm}` | fixture |" for arm in ["lint", "docs", "identity", "fast", "full", "gnn"]), + encoding="utf-8", + ) + relative = "data/video/source/Series/Item" + record = self.root / relative + record.mkdir(parents=True) + metadata = {"item": "Item", "series": "Series", "parts": [{"video_id": "abcdefghijk"}]} + (record / "metadata.json").write_text(json.dumps(metadata), encoding="utf-8") + index_record = { + "path": relative, + "series": metadata["series"], + "item": metadata["item"], + "parts": [part["video_id"] for part in metadata["parts"]], + "has_transcript": False, + } + (self.root / "INDEX.json").write_text(json.dumps({"items": [index_record]}), encoding="utf-8") + return record / "metadata.json" + + def assert_offline_failure(self, expected): + result = self.dispatch() + self.assertEqual(result.returncode, 1, result.stdout + result.stderr) + self.assertIn(expected, result.stdout + result.stderr) + self.assertNotIn("FORBIDDEN_ACQUISITION", result.stdout + result.stderr) + + def test_real_malformed_json_stops_before_docs_or_providers(self): + self.offline_corpus() + (self.root / "INDEX.json").write_text("{", encoding="utf-8") + self.assert_offline_failure("INDEX.json: invalid JSON") + + def test_real_unresolved_link_stops_before_identity_or_providers(self): + self.offline_corpus() + (self.root / "README.md").write_text("[missing](absent.md)\n", encoding="utf-8") + self.assert_offline_failure("link `absent.md` does not resolve") + + def test_real_full_receives_explicit_source_and_refuses_before_gnn(self): + self.offline_corpus() + missing = self.root / "missing object source" + result = self.dispatch(["--utilities", str(missing), "--cold"]) + self.assertEqual(result.returncode, 1, result.stdout + result.stderr) + self.assertIn("does not provide exact Git commit", result.stderr) + self.assertIn(str(missing), result.stderr) + self.assertNotIn("FORBIDDEN_ACQUISITION", result.stdout + result.stderr) + self.assertIn("identity: ok", result.stdout) + + def test_real_record_identity_failure_stops_before_providers(self): + metadata = self.offline_corpus() + record = json.loads(metadata.read_text()) + record["item"] = "another-item" + metadata.write_text(json.dumps(record), encoding="utf-8") + self.assert_offline_failure("the item id must name its own directory") + + +if __name__ == "__main__": + unittest.main() diff --git a/scripts/gate/common.py b/scripts/gate/common.py index 6b1066971..ee88ef84c 100644 --- a/scripts/gate/common.py +++ b/scripts/gate/common.py @@ -10,14 +10,16 @@ import io import json import os +import re import shutil import subprocess import sys -import tarfile import tempfile import time +from collections.abc import Iterator +from contextlib import contextmanager from dataclasses import dataclass -from pathlib import Path +from pathlib import Path, PurePosixPath ROOT = Path(__file__).resolve().parents[2] @@ -89,8 +91,37 @@ def git(args: list[str], cwd: Path | None = None) -> subprocess.CompletedProcess ) +def _object_git( + args: list[str], source: Path, *, input: bytes | None = None, network: bool = False +) -> subprocess.CompletedProcess: + """Read literal objects without replacement refs, lazy fetches or caller Git env.""" + env = {key: value for key, value in os.environ.items() if not key.startswith("GIT_")} + env.update( + GIT_CONFIG_GLOBAL=os.devnull, + GIT_CONFIG_NOSYSTEM="1", + GIT_TERMINAL_PROMPT="0", + GIT_NO_REPLACE_OBJECTS="1", + GIT_NO_LAZY_FETCH="1", + GIT_OPTIONAL_LOCKS="0", + GIT_ALLOW_PROTOCOL="https" if network else "", + ) + return subprocess.run( + ["git", "-c", "core.fsmonitor=false", "-c", f"core.hooksPath={os.devnull}", *args], + cwd=source, + env=env, + input=input, + capture_output=True, + ) + + def _hash_object(clone: Path, revision: str) -> bool: - return git(["cat-file", "-e", f"{revision}^{{commit}}"], cwd=clone).returncode == 0 + # Do not inherit an enclosing repository when passed an exported subdirectory. + if not (clone / ".git").exists() and not ( + (clone / "HEAD").is_file() and (clone / "objects").is_dir() + ): + return False + result = _object_git(["cat-file", "-t", revision], clone) + return result.returncode == 0 and result.stdout.strip() == b"commit" def _local_clone() -> Path | None: @@ -103,88 +134,140 @@ def _local_clone() -> Path | None: return None -def _extract_archive(clone: Path, revision: str, destination: Path) -> None: - archive = subprocess.run( - ["git", "-C", str(clone), "archive", revision], - capture_output=True, - check=True, +def _extract_tree(source: Path, revision: str, destination: Path) -> None: + """Materialize every committed blob, not archive/checkout-transformed bytes.""" + listed = _object_git(["ls-tree", "-rz", "--full-tree", revision], source) + if listed.returncode: + raise GateError(f"cannot read the committed tree {revision} from {source}") + entries: list[tuple[bytes, bytes, Path]] = [] + for record in listed.stdout.split(b"\0"): + if not record: + continue + metadata, name = record.split(b"\t", 1) + mode, kind, oid = metadata.split() + relative = PurePosixPath(os.fsdecode(name)) + if ( + relative.is_absolute() + or any(part in {"", ".", ".."} or part.lower() == ".git" for part in os.fsdecode(name).split("/")) + or kind != b"blob" + or mode not in {b"100644", b"100755", b"120000"} + ): + raise GateError(f"cannot bind committed entry {os.fsdecode(name)!r} ({mode.decode()})") + entries.append((mode, oid, destination / relative)) + + objects = _object_git( + ["cat-file", "--batch"], source, + input=b"".join(oid + b"\n" for _, oid, _ in entries), ) - destination.mkdir(parents=True, exist_ok=True) - with tarfile.open(fileobj=io.BytesIO(archive.stdout)) as tar: - tar.extractall(destination, filter="data") + if objects.returncode: + raise GateError(f"cannot read all blobs for {revision} from {source}") + stream = io.BytesIO(objects.stdout) + links: list[tuple[Path, str]] = [] + for mode, oid, target in entries: + header = stream.readline().split() + if len(header) != 3 or header[:2] != [oid, b"blob"]: + raise GateError(f"missing exact blob {oid.decode()} for {target.relative_to(destination)}") + size = int(header[2]) + body = stream.read(size) + if len(body) != size or stream.read(1) != b"\n": + raise GateError(f"incomplete blob {oid.decode()}") + parent = destination + for part in target.relative_to(destination).parts[:-1]: + child = parent / part + if child.exists() and part not in os.listdir(parent): + raise GateError(f"committed directory names collide at {child.relative_to(destination)}") + child.mkdir(exist_ok=True) + parent = child + if target.exists() or target.is_symlink(): + raise GateError(f"committed paths collide at {target.relative_to(destination)}") + if mode == b"120000": + links.append((target, os.fsdecode(body))) + else: + target.write_bytes(body) + target.chmod(0o755 if mode == b"100755" else 0o644) + for target, link in links: + if os.path.isabs(link): + raise GateError(f"committed symlink escapes the snapshot: {target.relative_to(destination)}") + target.symlink_to(link) + for target, _ in links: + try: + target.resolve().relative_to(destination.resolve()) + except (ValueError, RuntimeError): + raise GateError(f"committed symlink escapes or cycles: {target.relative_to(destination)}") from None def _clone_at_pin(repository: str, revision: str, destination: Path) -> None: - destination.mkdir(parents=True, exist_ok=True) - env = os.environ.copy() - env.setdefault("GIT_TERMINAL_PROMPT", "0") + """Fetch into a newly owned bare object store; never check out or reset a caller.""" for args in ( - ["init", "--quiet"], - ["remote", "add", "origin", f"https://github.com/{repository}.git"], - ["fetch", "--quiet", "--depth", "1", "origin", revision], - ["checkout", "--quiet", "--detach", "FETCH_HEAD"], + ["init", "--bare", "--quiet"], + ["fetch", "--quiet", "--depth", "1", "--no-tags", f"https://github.com/{repository}.git", revision], ): - completed = subprocess.run(["git", *args], cwd=str(destination), env=env) + completed = _object_git(args, destination, network=True) if completed.returncode != 0: - raise GateError( - f"cannot materialize {repository}@{revision[:12]} into {destination}" - ) + raise GateError(f"cannot acquire {repository}@{revision} into {destination}") + if not _hash_object(destination, revision): + raise GateError(f"acquisition did not provide exact commit {revision}") -def journal_utilities(override: str | None = None) -> Path: - """Return a Journal-Utilities tree at the pinned revision. +def _producer_source(repository: str, revision: str, override: str | None) -> Path: + if override is not None: + source = Path(override).expanduser().resolve() + if not override.strip() or not _hash_object(source, revision): + raise GateError( + f"{source} does not provide exact Git commit {revision}; " + "a mutable .gate-pin export is not an object source" + ) + return source + + local = _local_clone() + if local is not None: + if _hash_object(local, revision): + return local + if os.environ.get("GATE_JU_CLONE"): + raise GateError(f"GATE_JU_CLONE {local} does not provide exact Git commit {revision}") + + cache = cache_dir() + # Old caches may supply objects, but neither their stamp nor their worktree is trusted. + candidates = [cache / f"ju-{revision[:12]}", *sorted(cache.glob(f"ju-objects-{revision}-*"))] + for candidate in candidates: + if _hash_object(candidate, revision): + return candidate - An explicit ``--utilities`` path is trusted but checked: a Git tree must sit - at the pinned commit, and an exported tree must carry the materialization - stamp written below. Anything else is refused, because running the gate - against an unpinned engine silently changes what the result means. + source = Path(tempfile.mkdtemp(prefix=f"ju-objects-{revision}-", dir=cache)) + try: + log(f"journal-utilities: fetching {repository}@{revision}") + _clone_at_pin(repository, revision, source) + except BaseException: + shutil.rmtree(source) + raise + return source + + +@contextmanager +def journal_utilities(override: str | None = None) -> Iterator[Path]: + """Yield an owned exact-commit snapshot, valid only inside the ``with`` block. + + Explicit sources must already contain the pinned commit; their HEAD, index, + dirty files and stamps are irrelevant and untouched. Without an explicit + source, local objects or object caches are preferred before a cold fetch. + Snapshots are never reused and are removed on success or failure. """ pin = read_pin(JU_PIN) - repository = pin.get("repository", "") - revision = pin.get("revision", "") - if not repository or not revision: - raise GateError(f"{JU_PIN.relative_to(ROOT)} lacks repository/revision") - - if override: - path = Path(override).expanduser().resolve() - if not (path / "scripts" / "validate_journal.py").exists(): - raise GateError(f"{path} is not a Journal-Utilities checkout") - if (path / ".git").exists(): - head = git(["rev-parse", "HEAD"], cwd=path).stdout.strip() - if head != revision: - raise GateError( - f"{path} is at {head[:12]}, but {JU_PIN.relative_to(ROOT)} pins " - f"{revision[:12]}; check out the pin or update the pin file" - ) - else: - stamp = path / ".gate-pin" - if not stamp.exists() or stamp.read_text(encoding="utf-8").strip() != revision: - raise GateError( - f"{path} carries no .gate-pin stamp for {revision[:12]}; " - "run the gate without --utilities to materialize the pin" - ) - log(f"journal-utilities: {path} @ {revision[:12]}") - return path - - destination = cache_dir() / f"ju-{revision[:12]}" - stamp = destination / ".gate-pin" - if stamp.exists() and stamp.read_text(encoding="utf-8").strip() == revision: - log(f"journal-utilities: cached {destination}") - return destination - - if destination.exists(): - shutil.rmtree(destination) - - clone = _local_clone() - if clone is not None and _hash_object(clone, revision): - log(f"journal-utilities: exporting {revision[:12]} from {clone}") - _extract_archive(clone, revision, destination) - else: - log(f"journal-utilities: fetching {repository}@{revision[:12]}") - _clone_at_pin(repository, revision, destination) - - stamp.write_text(revision + "\n", encoding="utf-8") - return destination + repository = pin.get("repository") + revision = pin.get("revision") + if repository != "ActiveInferenceInstitute/Journal_Utilities" or not isinstance(revision, str) or not re.fullmatch(r"[0-9a-f]{40}", revision): + raise GateError(f"{JU_PIN.relative_to(ROOT)} needs the Journal-Utilities repository and a full commit id") + source = _producer_source(repository, revision, override) + with tempfile.TemporaryDirectory(prefix=f"ju-snapshot-{revision[:12]}-", dir=cache_dir()) as tmp: + tree = Path(tmp) + try: + _extract_tree(source, revision, tree) + except (OSError, ValueError) as exc: + raise GateError(f"cannot materialize exact producer snapshot: {exc}") from None + if not (tree / "scripts" / "validate_journal.py").is_file(): + raise GateError(f"{revision} has no Journal-Utilities validator") + log(f"journal-utilities: snapshot {tree} @ {revision} from {source}") + yield tree def find_python(candidates: list[str]) -> str: diff --git a/scripts/gate/common_selftest.py b/scripts/gate/common_selftest.py new file mode 100644 index 000000000..4f14d938b --- /dev/null +++ b/scripts/gate/common_selftest.py @@ -0,0 +1,393 @@ +"""Exact-object snapshot regressions with real, temporary Git repositories. + +Fixture scripts validate a small input record; they are object/execution probes, +not substitutes for qualification of the pinned Journal-Utilities corpus tools. +No network, dependency installation, user checkout mutation or provider runtime. +""" + +from __future__ import annotations + +import json +import os +import shutil +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + +SOURCE = Path(__file__).resolve().parent + +HELPER = '''import json +from pathlib import Path + +TREE = Path(__file__).resolve().parents[1] + +def accepts(journal): + expected = json.loads((TREE / "data" / "expected record.json").read_text()) + actual = json.loads((journal / "record.json").read_text()) + return actual == expected +''' +PROBE = '''#!/usr/bin/env python3 +import argparse +import json +import sys +from pathlib import Path +from helper import TREE, accepts + +parser = argparse.ArgumentParser() +parser.add_argument("--journal", type=Path, default=Path.cwd()) +parser.add_argument("--check", action="store_true") +parser.add_argument("--utilities", type=Path) +parser.add_argument("--manifest", type=Path) +args = parser.parse_args() +if not accepts(args.journal): + print("fixture record rejected", file=sys.stderr) + sys.exit(23) +if args.utilities is not None and args.utilities.resolve() != TREE: + sys.exit(24) +if args.manifest is not None: + if args.manifest.resolve() != TREE / "data/output/channel_videos.json": + sys.exit(25) + json.loads(args.manifest.read_text()) +with (args.journal / "executed.jsonl").open("a", encoding="utf-8") as stream: + stream.write(json.dumps({"script": Path(__file__).name, "tree": str(TREE)}) + "\\n") +''' +DRIVER = '''import json +import os +import subprocess +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +def deny_network(event, args): + if event.startswith("socket."): + raise RuntimeError("FORBIDDEN_NETWORK_ATTEMPT") + if event == "subprocess.Popen": + command = args[1] + if isinstance(command, (list, tuple)) and Path(command[0]).name == "git": + if any(word in command for word in ("fetch", "clone", "pull", "push")): + raise RuntimeError("FORBIDDEN_NETWORK_ATTEMPT") +sys.addaudithook(deny_network) + +import common +import utilities + +mode, source = sys.argv[1:3] +source = None if source == "AUTO" else source + +def inventory(root): + result = {} + for path in sorted(root.rglob("*")): + name = path.relative_to(root).as_posix() + if path.is_symlink(): + result[name] = ["link", os.readlink(path)] + elif path.is_file(): + result[name] = ["file", path.read_bytes().hex(), bool(path.stat().st_mode & 0o111)] + return result + +try: + if mode in {"fast", "full"}: + sys.exit(utilities.main([] if source is None else ["--utilities", source], mode)) + if mode == "overlap": + with common.journal_utilities(source) as first: + (first / "scripts/helper.py").write_text("raise RuntimeError('dirty snapshot')\\n") + with common.journal_utilities(source) as second: + result = subprocess.run([sys.executable, "-B", str(second / "scripts/validate_journal.py")]) + distinct = first != second and first.exists() and second.exists() + second_removed = not second.exists() + print("SNAPSHOT_RESULT=" + json.dumps({"distinct": distinct, "cleaned": second_removed and not first.exists()})) + sys.exit(result.returncode) + with common.journal_utilities(source) as tree: + contents = inventory(tree) + if mode == "raise": + raise RuntimeError("consumer failure") + result = subprocess.run([sys.executable, "-B", str(tree / "scripts/validate_journal.py")]) + print("SNAPSHOT_RESULT=" + json.dumps({"tree": str(tree), "cleaned": not tree.exists(), "contents": contents})) + sys.exit(result.returncode) +except common.GateError as exc: + print(str(exc), file=sys.stderr) + sys.exit(exc.code) +except RuntimeError as exc: + if str(exc) != "consumer failure": + raise + print("SNAPSHOT_RESULT=" + json.dumps({"cleaned": not tree.exists()})) + sys.exit(55) +''' + + +def inventory(root: Path, *, include_git: bool = True): + result = {} + for path in sorted(root.rglob("*")): + relative = path.relative_to(root) + if not include_git and ".git" in relative.parts: + continue + if path.is_symlink(): + result[relative.as_posix()] = ["link", os.readlink(path)] + elif path.is_file(): + result[relative.as_posix()] = ["file", path.read_bytes().hex(), bool(path.stat().st_mode & 0o111)] + return result + + +class ExactSnapshotTests(unittest.TestCase): + def setUp(self): + temporary = tempfile.TemporaryDirectory(prefix="aij-object-selftest-") + self.addCleanup(temporary.cleanup) + self.root = Path(temporary.name) + self.consumer = self.root / "consumer" + self.gate = self.consumer / "scripts/gate" + self.gate.mkdir(parents=True) + for name in ("common.py", "utilities.py", "arm_manifest.py"): + shutil.copy2(SOURCE / name, self.gate / name) + (self.gate / "probe_driver.py").write_text(DRIVER, encoding="utf-8") + self.cache = self.root / "cache" + self.cache.mkdir() + self.env = { + key: value for key, value in os.environ.items() + if not key.startswith(("GIT_", "GATE_", "PYTHON")) + } + self.env.update( + GIT_CONFIG_GLOBAL=os.devnull, + GIT_CONFIG_NOSYSTEM="1", + GIT_TERMINAL_PROMPT="0", + GIT_ALLOW_PROTOCOL="", + GATE_CACHE=str(self.cache), + ) + self.producer = self.root / "producer with spaces" + scripts = self.producer / "scripts" + scripts.mkdir(parents=True) + (scripts / "helper.py").write_text(HELPER, encoding="utf-8") + for name in ("validate_journal.py", "generate_journal_indexes.py", "repair_split_transcripts.py"): + (scripts / name).write_text(PROBE, encoding="utf-8") + (scripts / name).chmod(0o755) + (scripts / "helper-link.py").symlink_to("helper.py") + data = self.producer / "data" + (data / "output").mkdir(parents=True) + (data / "expected record.json").write_text('{"record": "committed"}', encoding="utf-8") + (data / "format-é.txt").write_text("$Format:%H$\n", encoding="utf-8") + (data / "output/channel_videos.json").write_text( + json.dumps({"enumerated_at": "2000-01-01T00:00:00+00:00", "videos": []}), encoding="utf-8", + ) + (self.producer / ".gitattributes").write_text( + 'scripts/helper.py export-ignore\n"data/format-é.txt" export-subst\n', encoding="utf-8", + ) + self.git("init", "--quiet") + self.git("add", ".") + self.git("commit", "--quiet", "-m", "owned fixture") + self.revision = self.git("rev-parse", "HEAD").stdout.decode().strip() + self.committed = inventory(self.producer, include_git=False) + self.pin = self.consumer / ".github/journal-utilities-pin.json" + self.pin.parent.mkdir() + self.write_pin(self.revision) + (self.consumer / "record.json").write_text('{"record": "committed"}', encoding="utf-8") + (self.consumer / "INDEX.json").write_text('{"items": []}', encoding="utf-8") + + def git(self, *args, input=None): + return subprocess.run( + ["git", "-c", "core.fsmonitor=false", "-c", f"core.hooksPath={os.devnull}", + "-c", "commit.gpgsign=false", "-c", "user.name=Gate fixture", + "-c", "user.email=gate-fixture@example.invalid", *args], + cwd=self.producer, env=self.env, input=input, capture_output=True, check=True, + ) + + def write_pin(self, revision): + self.pin.write_text(json.dumps({ + "repository": "ActiveInferenceInstitute/Journal_Utilities", "revision": revision, + }), encoding="utf-8") + + def invoke(self, mode="snapshot", *, source=None, auto=False, env=None): + return subprocess.run( + [sys.executable, "-I", "-B", str(self.gate / "probe_driver.py"), mode, + "AUTO" if auto else str(source or self.producer)], + cwd=self.consumer, env={**self.env, **(env or {})}, + capture_output=True, text=True, timeout=30, + ) + + def result(self, completed, code=0): + self.assertEqual(completed.returncode, code, completed.stdout + completed.stderr) + self.assertNotIn("FORBIDDEN_NETWORK_ATTEMPT", completed.stdout + completed.stderr) + lines = [line for line in completed.stdout.splitlines() if line.startswith("SNAPSHOT_RESULT=")] + self.assertEqual(len(lines), 1, completed.stdout) + return json.loads(lines[0].partition("=")[2]) + + def assert_no_snapshots(self): + self.assertEqual(list(self.cache.glob("ju-snapshot-*")), []) + + def test_clean_pin_materializes_every_blob_link_and_mode_then_cleans(self): + before = inventory(self.producer) + result = self.result(self.invoke()) + self.assertEqual(result["contents"], self.committed) + self.assertNotEqual(Path(result["tree"]), self.producer) + self.assertTrue(result["cleaned"]) + self.assertEqual(inventory(self.producer), before) + self.assert_no_snapshots() + + def test_dirty_validator_cannot_turn_a_rejected_record_green(self): + (self.consumer / "record.json").write_text('{"record": "invalid"}') + (self.producer / "scripts/validate_journal.py").write_text("raise SystemExit(0)\n") + (self.producer / "untracked.txt").write_text("preserve this dirty tree") + before = inventory(self.producer) + result = self.result(self.invoke(), 23) + self.assertEqual(result["contents"], self.committed) + self.assertEqual(inventory(self.producer), before) + self.assertFalse((self.consumer / "executed.jsonl").exists()) + + def test_staged_imported_helper_cannot_bypass_record_validation(self): + (self.consumer / "record.json").write_text('{"record": "invalid"}') + (self.producer / "scripts/helper.py").write_text( + "from pathlib import Path\nTREE = Path(__file__).resolve().parents[1]\ndef accepts(journal): return True\n", + ) + self.git("add", "scripts/helper.py") + (self.producer / "scripts/untracked.py").write_text("raise RuntimeError('untracked')\n") + before = inventory(self.producer) + result = self.result(self.invoke(), 23) + self.assertEqual(result["contents"], self.committed) + self.assertEqual(inventory(self.producer), before) + + def test_source_head_may_differ_but_pin_and_dirty_data_are_preserved(self): + (self.producer / "data/expected record.json").write_text('{"record": "later"}') + self.git("add", ".") + self.git("commit", "--quiet", "-m", "later source head") + (self.producer / "scripts/helper.py").write_text("raise RuntimeError('dirty')\n") + before = inventory(self.producer) + result = self.result(self.invoke()) + self.assertEqual(result["contents"], self.committed) + self.assertEqual(inventory(self.producer), before) + + def test_git_replacement_objects_do_not_override_pinned_blobs(self): + old = self.git("rev-parse", f"{self.revision}:scripts/helper.py").stdout.decode().strip() + new = self.git("hash-object", "-w", "--stdin", input=b"raise RuntimeError('replacement')\n").stdout.decode().strip() + self.git("replace", old, new) + before = inventory(self.producer) + result = self.result(self.invoke()) + self.assertEqual(result["contents"], self.committed) + self.assertEqual(inventory(self.producer), before) + + def test_linked_worktree_is_an_object_source_not_executed_content(self): + worktree = self.root / "linked source" + self.git("worktree", "add", "--quiet", "--detach", str(worktree), self.revision) + (worktree / "scripts/helper.py").write_text("raise RuntimeError('worktree dirty')\n") + before = inventory(worktree) + source_before = inventory(self.producer) + result = self.result(self.invoke(source=worktree)) + self.assertEqual(result["contents"], self.committed) + self.assertEqual(inventory(worktree), before) + self.assertEqual(inventory(self.producer), source_before) + + def test_mutable_stamped_export_is_refused_without_touching_it(self): + exported = self.root / "export" + shutil.copytree(self.producer, exported, symlinks=True, ignore=shutil.ignore_patterns(".git")) + (exported / ".gate-pin").write_text(self.revision + "\n") + (exported / "scripts/helper.py").write_text("raise RuntimeError('export dirty')\n") + before = inventory(exported) + completed = self.invoke(source=exported) + self.assertEqual(completed.returncode, 1, completed.stdout + completed.stderr) + self.assertIn("not an object source", completed.stderr) + self.assertNotIn("FORBIDDEN_NETWORK_ATTEMPT", completed.stderr) + self.assertEqual(inventory(exported), before) + self.assert_no_snapshots() + + def test_dirty_stamped_git_cache_supplies_objects_only(self): + cached = self.cache / f"ju-{self.revision[:12]}" + self.producer.rename(cached) + self.producer = cached + (cached / ".gate-pin").write_text(self.revision + "\n") + (cached / "scripts/validate_journal.py").write_text("raise SystemExit(42)\n") + (cached / "data/expected record.json").write_text('{"record": "cache dirty"}') + before = inventory(cached) + result = self.result(self.invoke(auto=True)) + self.assertEqual(result["contents"], self.committed) + self.assertEqual(inventory(cached), before) + self.assert_no_snapshots() + + def test_stamped_export_cache_is_preserved_while_local_objects_are_used(self): + exported = self.cache / f"ju-{self.revision[:12]}" + exported.mkdir() + (exported / ".gate-pin").write_text(self.revision + "\n") + (exported / "dirty.txt").write_text("do not overwrite or delete") + before = inventory(self.cache) + result = self.result(self.invoke(auto=True, env={"GATE_JU_CLONE": str(self.producer)})) + self.assertEqual(result["contents"], self.committed) + self.assertEqual(inventory(self.cache), before) + + def test_missing_exact_commit_fails_without_network_or_source_changes(self): + self.write_pin("f" * 40) + before = inventory(self.producer) + completed = self.invoke() + self.assertEqual(completed.returncode, 1, completed.stdout + completed.stderr) + self.assertIn("does not provide exact Git commit", completed.stderr) + self.assertNotIn("FORBIDDEN_NETWORK_ATTEMPT", completed.stderr) + self.assertEqual(inventory(self.producer), before) + self.assert_no_snapshots() + + def test_escaping_symlink_and_gitlink_fail_instead_of_partial_execution(self): + outside = self.root / "outside.txt" + outside.write_text("untouched") + (self.producer / "escape").symlink_to(outside) + self.git("add", "escape") + self.git("commit", "--quiet", "-m", "escaping link") + self.write_pin(self.git("rev-parse", "HEAD").stdout.decode().strip()) + before = inventory(self.producer) + completed = self.invoke() + self.assertEqual(completed.returncode, 1, completed.stdout + completed.stderr) + self.assertIn("symlink escapes", completed.stderr) + self.assertEqual(outside.read_text(), "untouched") + self.assertEqual(inventory(self.producer), before) + self.assert_no_snapshots() + self.git("update-index", "--add", "--cacheinfo", f"160000,{self.revision},submodule") + self.git("commit", "--quiet", "-m", "gitlink") + self.write_pin(self.git("rev-parse", "HEAD").stdout.decode().strip()) + completed = self.invoke() + self.assertEqual(completed.returncode, 1, completed.stdout + completed.stderr) + self.assertIn("cannot bind committed entry 'submodule'", completed.stderr) + self.assert_no_snapshots() + + def test_overlapping_contexts_do_not_share_mutable_execution_trees(self): + result = self.result(self.invoke("overlap")) + self.assertTrue(result["distinct"]) + self.assertTrue(result["cleaned"]) + self.assert_no_snapshots() + + def test_consumer_exception_removes_only_its_snapshot(self): + before = inventory(self.producer) + result = self.result(self.invoke("raise"), 55) + self.assertTrue(result["cleaned"]) + self.assertEqual(inventory(self.producer), before) + self.assert_no_snapshots() + + def test_fast_and_full_keep_the_snapshot_through_all_real_child_processes(self): + before = inventory(self.producer) + for mode, scripts in ( + ("fast", ["generate_journal_indexes.py"]), + ("full", ["generate_journal_indexes.py", "repair_split_transcripts.py", "validate_journal.py"]), + ): + with self.subTest(mode=mode): + trace = self.consumer / "executed.jsonl" + trace.unlink(missing_ok=True) + completed = self.invoke(mode) + self.assertEqual(completed.returncode, 0, completed.stdout + completed.stderr) + self.assertNotIn("FORBIDDEN_NETWORK_ATTEMPT", completed.stderr) + events = [json.loads(line) for line in trace.read_text().splitlines()] + self.assertEqual([event["script"] for event in events], scripts) + self.assertEqual(len({event["tree"] for event in events}), 1) + self.assertFalse(Path(events[0]["tree"]).exists()) + if mode == "full": + self.assertIn("manifest: reconciliation complete", completed.stdout) + self.assertEqual(inventory(self.producer), before) + self.assert_no_snapshots() + + def test_failed_producer_process_cleans_snapshot_and_preserves_source(self): + (self.consumer / "record.json").write_text('{"record": "invalid"}') + before = inventory(self.producer) + completed = self.invoke("full") + self.assertEqual(completed.returncode, 1, completed.stdout + completed.stderr) + self.assertIn("fixture record rejected", completed.stderr) + self.assertIn("derived indexes failed (exit 23)", completed.stderr) + self.assertFalse((self.consumer / "executed.jsonl").exists()) + self.assertEqual(inventory(self.producer), before) + self.assert_no_snapshots() + + +if __name__ == "__main__": + unittest.main() diff --git a/scripts/gate/docs_selftest.py b/scripts/gate/docs_selftest.py new file mode 100644 index 000000000..f4969148f --- /dev/null +++ b/scripts/gate/docs_selftest.py @@ -0,0 +1,190 @@ +"""Offline Markdown regressions using real documents and the owning checker. + +run() is called by the docs arm. Temporary roots replace only checker configuration; +link resolution, heading extraction and docs-index coverage use production code. +""" + +from __future__ import annotations + +import tempfile +import unittest +from pathlib import Path + +import arm_docs + + +class DocsTests(unittest.TestCase): + def setUp(self) -> None: + temporary = tempfile.TemporaryDirectory() + self.addCleanup(temporary.cleanup) + self.root = Path(temporary.name).resolve() + self.docs = self.root / "docs" + self.docs.mkdir() + self.index = self.docs / "documentation-index.md" + configured = { + "ROOT": self.root, + "DOCS": self.docs, + "INDEX": self.index, + "failures": [], + "_anchors": {}, + } + original = {name: getattr(arm_docs, name) for name in configured} + self.addCleanup(self.restore_configuration, original) + for name, value in configured.items(): + setattr(arm_docs, name, value) + + @staticmethod + def restore_configuration(original: dict) -> None: + for name, value in original.items(): + setattr(arm_docs, name, value) + + def document(self, relative: str, text: str) -> Path: + path = self.root / relative + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(text, encoding="utf-8") + return path + + def test_garden_heading_and_repeated_inline_words_resolve(self) -> None: + architecture = self.document( + "docs/ARCHITECTURE.md", + "# Architecture\n\n" + "### `discord_bot.py` — the Iris Discord bot\n" + "### `discord_bot.py` — the Iris Discord bot\n" + "### discord_bot.py — the Iris Discord bot\n" + "[First](#discord_botpy--the-iris-discord-bot)\n" + "[Second](#discord_botpy--the-iris-discord-bot-1)\n" + "[Third](#discord_botpy--the-iris-discord-bot-2)\n", + ) + readme = self.document( + "README.md", + "[Discord](docs/ARCHITECTURE.md#discord_botpy--the-iris-discord-bot)\n" + "[Repeated](docs/ARCHITECTURE.md#discord_botpy--the-iris-discord-bot-2)\n", + ) + arm_docs.check_links([readme, architecture]) + self.assertEqual(arm_docs.failures, []) + self.assertEqual( + arm_docs.anchors(architecture), + { + "architecture", + "discord_botpy--the-iris-discord-bot", + "discord_botpy--the-iris-discord-bot-1", + "discord_botpy--the-iris-discord-bot-2", + }, + ) + + def test_different_code_headings_do_not_share_duplicate_numbers(self) -> None: + path = self.document( + "README.md", + "## `alpha.py`\n## `beta.py`\n## alpha.py\n" + "[Alpha](#alphapy)\n[Beta](#betapy)\n[Again](#alphapy-1)\n", + ) + arm_docs.check_links([path]) + self.assertEqual(arm_docs.failures, []) + self.assertEqual(arm_docs.anchors(path), {"alphapy", "betapy", "alphapy-1"}) + + def test_links_inside_inline_code_are_not_checked(self) -> None: + path = self.document( + "README.md", + "# Real\n" + "`[missing](absent.md)`\n" + "`![missing image](absent.png)`\n" + "`[reference]: absent.md`\n" + "[Real](#real)\n", + ) + arm_docs.check_links([path]) + self.assertEqual(arm_docs.failures, []) + + def test_code_links_do_not_make_a_doc_reachable(self) -> None: + self.document("docs/hidden.md", "# Hidden\n") + self.index.write_text( + "# Index\n`[Hidden](hidden.md)`\n`[hidden]: hidden.md`\n", + encoding="utf-8", + ) + arm_docs.check_docs_index() + self.assertEqual( + arm_docs.failures, + ["docs/hidden.md is not reachable from docs/documentation-index.md"], + ) + arm_docs.failures.clear() + self.index.write_text("# Index\n[Hidden](hidden.md)\n", encoding="utf-8") + arm_docs.check_docs_index() + self.assertEqual(arm_docs.failures, []) + + def test_fences_and_comments_do_not_claim_headings_or_links(self) -> None: + path = self.document( + "README.md", + "# Visible\n" + "\n" + "```md\n## `same`\n[Missing](fenced.md)\n```\n" + "~~~md\n## `same`\n[Missing](tilde.md)\n~~~\n" + "## `same`\n[Visible](#same)\n", + ) + arm_docs.check_links([path]) + self.assertEqual(arm_docs.failures, []) + self.assertEqual(arm_docs.anchors(path), {"visible", "same"}) + + def test_missing_anchor_and_real_link_still_fail_at_original_lines(self) -> None: + path = self.document( + "README.md", + "# `alpha.py`\n" + "\n" + "```md\n[Ignored](nope.md)\n```\n" + "`[Ignored](also-nope.md)`\n" + "[Wrong](#betapy)\n" + "[Missing](real-missing.md)\n", + ) + arm_docs.check_links([path]) + self.assertEqual( + arm_docs.failures, + [ + "README.md:8: missing anchor `#betapy` in this file", + "README.md:9: link `real-missing.md` does not resolve", + ], + ) + + def test_docs_index_ignores_fenced_and_commented_links(self) -> None: + self.document("docs/hidden.md", "# Hidden\n") + self.index.write_text( + "# Index\n\n" + "```md\n[Hidden](hidden.md)\n```\n", + encoding="utf-8", + ) + arm_docs.check_docs_index() + self.assertEqual( + arm_docs.failures, + ["docs/hidden.md is not reachable from docs/documentation-index.md"], + ) + + + def test_literal_code_links_retain_words_without_becoming_links(self) -> None: + path = self.document( + "README.md", + "## `[label](missing.md)`\n" + "## ``[label](other.md)``\n" + "## [Visible `label`](README.md)\n" + "[First](#labelmissingmd)\n" + "[Second](#labelothermd)\n" + "[Third](#visible-label)\n" + "``[Ignored](absent.md) with a `literal` tick``\n", + ) + arm_docs.check_links([path]) + self.assertEqual(arm_docs.failures, []) + self.assertEqual( + arm_docs.anchors(path), + {"labelmissingmd", "labelothermd", "visible-label"}, + ) + + +def run() -> list[str]: + result = unittest.TestResult() + unittest.defaultTestLoader.loadTestsFromTestCase(DocsTests).run(result) + return [f"selftest: {test.id()}\n{detail}" for test, detail in result.errors + result.failures] + + +if __name__ == "__main__": + import sys + + problems = run() + for problem in problems: + print(problem, file=sys.stderr) + sys.exit(1 if problems else 0) diff --git a/scripts/gate/identity.py b/scripts/gate/identity.py index c5dd98e30..343dfa327 100644 --- a/scripts/gate/identity.py +++ b/scripts/gate/identity.py @@ -21,9 +21,10 @@ import hashlib import json import re +import stat import sys from dataclasses import dataclass -from pathlib import Path +from pathlib import Path, PureWindowsPath SCHEMA_VERSION = "record-identity/v1" # A pinned copy is any artifact kind a member can carry, not just a schema: the @@ -55,6 +56,7 @@ class Spec: index_file: str derived_prefix_components: int transcript_flag: dict + index_record_keys: tuple[str, ...] def digest(self) -> str: return hashlib.sha256(self.path.read_bytes()).hexdigest() @@ -72,6 +74,36 @@ def _require(mapping: dict, key: str, where: str) -> object: return mapping[key] +def _relative_path(value: object, where: str) -> Path: + """Admit one unambiguous, corpus-relative POSIX path before filesystem access.""" + if ( + not isinstance(value, str) + or not value + or "\0" in value + or "\\" in value + or PureWindowsPath(value).drive + or any(part in {"", ".", ".."} for part in value.split("/")) + ): + raise IdentityError(f"{where}: expected a relative path without traversal") + return Path(value) + + +def _contained_path(root: Path, value: object, where: str) -> Path: + relative = _relative_path(value, where) + try: + resolved = (root / relative).resolve() + except (OSError, RuntimeError, ValueError) as exc: + raise IdentityError(f"{where}: cannot resolve path: {exc}") from None + if not resolved.is_relative_to(root): + raise IdentityError(f"{where}: path escapes the corpus") + return resolved + + +def _read_json(path: Path) -> object: + if not stat.S_ISREG(path.stat().st_mode): + raise IdentityError("expected a regular JSON file") + return json.loads(path.read_text(encoding="utf-8")) + def load_schema(path: Path) -> Spec: """Parse and compile the canonical schema, refusing an incomplete contract.""" try: @@ -106,25 +138,55 @@ def load_schema(path: Path) -> Spec: for required in ("video_id", "session_id"): if required not in patterns: raise IdentityError(f"{path}: x-identity.patterns lacks `{required}`") + if "video_id" not in patterns["session_id"].groupindex: + raise IdentityError(f"{path}: session_id pattern must capture `video_id`") flag = _require(block, "transcript_flag", f"{path}: x-identity") if not isinstance(flag, dict): raise IdentityError(f"{path}: x-identity.transcript_flag must be an object") + for key in ("key", "artifact"): + value = _require(flag, key, f"{path}: x-identity.transcript_flag") + if not isinstance(value, str) or not value: + raise IdentityError(f"{path}: transcript_flag.{key} must be a nonempty string") + _relative_path(flag["artifact"], f"{path}: transcript_flag.artifact") + + keys: dict[str, str] = {} + for key in ("leaf_key", "ancestor_key", "duplicate_key", "index_file"): + value = _require(block, key, f"{path}: x-identity") + if not isinstance(value, str) or not value: + raise IdentityError(f"{path}: x-identity.{key} must be a nonempty string") + keys[key] = value + _relative_path(keys["index_file"], f"{path}: x-identity.index_file") + index_keys = _require(block, "index_record_keys", f"{path}: x-identity") + expected_keys = { + "path", keys["leaf_key"], keys["ancestor_key"], "parts", + flag["key"], keys["duplicate_key"], + } + if ( + not isinstance(index_keys, list) + or not all(isinstance(key, str) for key in index_keys) + or len(index_keys) != len(expected_keys) + or set(index_keys) != expected_keys + ): + raise IdentityError( + f"{path}: index_record_keys must name {', '.join(sorted(expected_keys))}" + ) components = _require(block, "derived_prefix_components", f"{path}: x-identity") - if not isinstance(components, int) or components < 1: + if type(components) is not int or components < 1: raise IdentityError(f"{path}: derived_prefix_components must be a positive integer") return Spec( path=path, document=document, patterns=patterns, - leaf_key=str(_require(block, "leaf_key", f"{path}: x-identity")), - ancestor_key=str(_require(block, "ancestor_key", f"{path}: x-identity")), - duplicate_key=str(_require(block, "duplicate_key", f"{path}: x-identity")), - index_file=str(_require(block, "index_file", f"{path}: x-identity")), + leaf_key=keys["leaf_key"], + ancestor_key=keys["ancestor_key"], + duplicate_key=keys["duplicate_key"], + index_file=keys["index_file"], derived_prefix_components=components, transcript_flag=flag, + index_record_keys=tuple(index_keys), ) @@ -154,7 +216,9 @@ def resolve_artifact(root: Path, name: str) -> Path | None: """Find the file a recorded digest names, or None when it is not unique.""" matches: list[Path] = [] for location in ARTIFACT_LOCATIONS: - candidate = root / location / name if location else root / name + relative = f"{location}/{name}" if location else name + _contained_path(root.resolve(), relative, f"artifact {name}") + candidate = root / relative if candidate.is_file(): matches.append(candidate) return matches[0] if len(matches) == 1 else None @@ -171,14 +235,24 @@ def check_contract(contract: Path, schema: Spec) -> list[str]: return failures root = repository_root(contract) for name, expected in sorted(recorded.items()): - path = schema.path if name == schema.path.name else resolve_artifact(root, name) - if path is None: - failures.append( - f"{contract.name}: records {name}, which is not one file under " - f"{root} at {', '.join(location or '.' for location in ARTIFACT_LOCATIONS)}" - ) + try: + if name == schema.path.name: + relative = schema.path.absolute().relative_to(root).as_posix() + path = _contained_path(root, relative, f"artifact {name}") + else: + path = resolve_artifact(root, name) + if path is None: + failures.append( + f"{contract.name}: records {name}, which is not one file under " + f"{root} at {', '.join(location or '.' for location in ARTIFACT_LOCATIONS)}" + ) + continue + if not stat.S_ISREG(path.stat().st_mode): + raise IdentityError(f"artifact {name}: expected a regular file") + digest = hashlib.sha256(path.read_bytes()).hexdigest() + except (IdentityError, OSError, ValueError) as exc: + failures.append(f"{contract.name}: cannot read artifact {name}: {exc}") continue - digest = hashlib.sha256(path.read_bytes()).hexdigest() if digest != expected: failures.append( f"{contract.name}: records {name} as {expected[:12]}, " @@ -192,9 +266,13 @@ def check_contract(contract: Path, schema: Spec) -> list[str]: def verify_copy(copy: Path, contract: Path) -> str | None: """Return a failure message when a member's artifact copy diverges.""" try: + if not stat.S_ISREG(copy.stat().st_mode): + return f"{copy}: expected a regular file" digest = hashlib.sha256(copy.read_bytes()).hexdigest() except FileNotFoundError: return f"{copy}: no such file" + except OSError as exc: + return f"{copy}: cannot read artifact: {exc}" recorded = recorded_digests(contract) expected = recorded.get(copy.name) if expected is None: @@ -208,95 +286,161 @@ def verify_copy(copy: Path, contract: Path) -> str | None: def check_corpus(root: Path, spec: Spec) -> tuple[list[str], dict[str, int]]: - """Enforce the identity grammar over one corpus; return failures and counts.""" + """Check contained record identities, not the producer's raw transcript layer.""" failures: list[str] = [] counts = {"records": 0, "parts": 0, "duplicates": 0} - index_path = root / spec.index_file try: - index = json.loads(index_path.read_text(encoding="utf-8")) - except FileNotFoundError: - return [f"missing {spec.index_file} under {root}"], counts - except json.JSONDecodeError as exc: - return [f"{spec.index_file} is not valid JSON: {exc}"], counts + root = root.resolve(strict=True) + if not root.is_dir(): + raise IdentityError("corpus root must be a directory") + index_path = _contained_path(root, spec.index_file, "index_file") + index = _read_json(index_path) + except (IdentityError, OSError, RuntimeError, ValueError) as exc: + return [f"{spec.index_file}: {exc}"], counts records = index.get("items") if isinstance(index, dict) else None if not isinstance(records, list): return [f"{spec.index_file}: `items` must be a list"], counts - known_paths = { - str(item.get("path", "")) for item in records if isinstance(item, dict) - } - seen: set[str] = set() + # Admit every record path before any metadata read or duplicate resolution. + admitted: list[tuple[dict, str]] = [] + known_paths: set[str] = set() for position, record in enumerate(records): + where = f"{spec.index_file}: items[{position}]" if not isinstance(record, dict): - failures.append(f"{spec.index_file}: items[{position}] is not an object") + failures.append(f"{where} is not an object") continue - relative = str(record.get("path", "")) - if not relative: - failures.append(f"{spec.index_file}: items[{position}] has no path") + relative = record.get("path") + try: + record_dir = _contained_path(root, relative, f"{where}.path") + if not record_dir.is_dir(): + raise IdentityError(f"{where}: record path is not a directory") + except (IdentityError, OSError) as exc: + failures.append(str(exc)) continue - if relative in seen: + if relative in known_paths: failures.append(f"{spec.index_file}: duplicate record path {relative}") continue - seen.add(relative) - record_dir = root / relative - metadata_path = record_dir / "metadata.json" + known_paths.add(relative) + admitted.append((record, relative)) + + for record, relative in admitted: try: - metadata = json.loads(metadata_path.read_text(encoding="utf-8")) - except FileNotFoundError: - failures.append(f"{relative}: no metadata.json") - continue - except json.JSONDecodeError as exc: - failures.append(f"{relative}: metadata.json is not valid JSON: {exc}") + metadata_path = _contained_path( + root, f"{relative}/metadata.json", f"{relative}: metadata.json" + ) + metadata = _read_json(metadata_path) + except (IdentityError, OSError, ValueError) as exc: + failures.append(f"{relative}: metadata.json: {exc}") continue if not isinstance(metadata, dict): failures.append(f"{relative}: metadata.json must hold an object") continue counts["records"] += 1 + for key in spec.index_record_keys: + if key != spec.duplicate_key and key not in record: + failures.append(f"{relative}: INDEX record lacks `{key}`") + + # Use the admitted logical path, never parents of the host's corpus root. + path = Path(relative) leaf = metadata.get(spec.leaf_key) - if leaf != record_dir.name: + if leaf != path.name: failures.append( f"{relative}: {spec.leaf_key} is {leaf!r}, but the directory is " - f"{record_dir.name!r}; the item id must name its own directory" + f"{path.name!r}; the item id must name its own directory" ) - ancestors = {parent.name for parent in record_dir.parents if parent.name} series = metadata.get(spec.ancestor_key) - if series not in ancestors: + if not isinstance(series, str) or series not in path.parts[:-1]: failures.append( f"{relative}: {spec.ancestor_key} is {series!r}, which names no " - "directory above the item" + "directory above the item inside the corpus" ) + for key in (spec.leaf_key, spec.ancestor_key): + if record.get(key) != metadata.get(key): + failures.append(f"{relative}: INDEX `{key}` disagrees with metadata.json") parts = metadata.get("parts") + video_ids: list[object] = [] + valid_video_ids: set[str] = set() if not isinstance(parts, list): failures.append(f"{relative}: `parts` must be a list") - continue - for position, part in enumerate(parts): - if not isinstance(part, dict): - failures.append(f"{relative}: parts[{position}] is not an object") - continue - video_id = part.get("video_id") - counts["parts"] += 1 - if not isinstance(video_id, str) or not spec.patterns["video_id"].match(video_id): + else: + for position, part in enumerate(parts): + if not isinstance(part, dict): + failures.append(f"{relative}: parts[{position}] is not an object") + continue + video_id = part.get("video_id") + video_ids.append(video_id) + counts["parts"] += 1 + if ( + not isinstance(video_id, str) + or not spec.patterns["video_id"].fullmatch(video_id) + ): + failures.append( + f"{relative}: parts[{position}].video_id is {video_id!r}, which is " + "not an 11-character YouTube id" + ) + else: + valid_video_ids.add(video_id) + if not isinstance(record.get("parts"), list) or record["parts"] != video_ids: + failures.append(f"{relative}: INDEX `parts` disagrees with metadata video IDs") + + flag_key = spec.transcript_flag["key"] + artifact = spec.transcript_flag["artifact"] + try: + transcript = _contained_path(root, f"{relative}/{artifact}", f"{relative}: {artifact}") + try: + mode = transcript.stat().st_mode + except FileNotFoundError: + has_transcript = False + else: + if not stat.S_ISREG(mode): + raise IdentityError(f"{artifact} must be a regular file") + has_transcript = True + if type(record.get(flag_key)) is not bool or record[flag_key] != has_transcript: failures.append( - f"{relative}: parts[{position}].video_id is {video_id!r}, which is " - "not an 11-character YouTube id" + f"{relative}: INDEX `{flag_key}` must be {has_transcript} " + f"to match {artifact}" ) - - duplicate = metadata.get(spec.duplicate_key) - if duplicate is not None: + except (IdentityError, OSError) as exc: + failures.append(f"{relative}: {artifact}: {exc}") + + sessions = metadata.get("sessions", []) + if not isinstance(sessions, list): + failures.append(f"{relative}: `sessions` must be a list") + else: + for position, session in enumerate(sessions): + where = f"{relative}: sessions[{position}].session_name" + name = session.get("session_name") if isinstance(session, dict) else None + match = spec.patterns["session_id"].fullmatch(name) if isinstance(name, str) else None + if match is None: + failures.append(f"{where} is {name!r}, which does not match session_id") + elif match.group("video_id") not in valid_video_ids: + failures.append(f"{where} {name!r} does not belong to a record part") + # sessions[].video_id can name a separate per-talk upload, not this part. + + duplicate_key = spec.duplicate_key + if ( + (duplicate_key in record) != (duplicate_key in metadata) + or record.get(duplicate_key) != metadata.get(duplicate_key) + ): + failures.append(f"{relative}: INDEX `{duplicate_key}` disagrees with metadata.json") + if duplicate_key in metadata: counts["duplicates"] += 1 - if not isinstance(duplicate, str): - failures.append(f"{relative}: {spec.duplicate_key} must be a string") - else: - prefix = "/".join(relative.split("/")[: spec.derived_prefix_components]) - target = f"{prefix}/{duplicate}" + duplicate = metadata[duplicate_key] + try: + suffix = _relative_path(duplicate, f"{relative}: {duplicate_key}") + prefix = Path(*path.parts[: spec.derived_prefix_components]) + target = (prefix / suffix).as_posix() + _contained_path(root, target, f"{relative}: {duplicate_key}") if target not in known_paths: failures.append( - f"{relative}: {spec.duplicate_key} {duplicate!r} resolves to " + f"{relative}: {duplicate_key} {duplicate!r} resolves to " f"{target}, which is not a record path" ) + except IdentityError as exc: + failures.append(str(exc)) return failures, counts diff --git a/scripts/gate/identity_selftest.py b/scripts/gate/identity_selftest.py index 9ed6bf56f..9156aca6b 100644 --- a/scripts/gate/identity_selftest.py +++ b/scripts/gate/identity_selftest.py @@ -1,19 +1,16 @@ -"""Selftest for the digest check the contract tells members to run. +"""Offline identity regressions over real temporary corpora and artifact copies. -The migration step asks each member to pin a `.py` copy, not only a schema, so -the drift test has to prove that path. This builds a fake checkout in a temp -directory — a module, a contract recording its sha256, and a prose line that -must not be mistaken for a digest — and asserts both directions: a matching copy -passes, a diverged copy fails with a drift message. - -Deterministic, offline, no repository state touched. Run from the `identity` -arm so it cannot rot. +The identity arm calls run(); direct execution uses the same cases. Fixtures +exercise the canonical schema and checker without reading or changing the corpus. """ from __future__ import annotations import hashlib +import json import tempfile +import unittest +from dataclasses import replace from pathlib import Path import identity @@ -26,6 +23,400 @@ ) +class CorpusIdentityTests(unittest.TestCase): + def setUp(self) -> None: + temporary = tempfile.TemporaryDirectory() + self.addCleanup(temporary.cleanup) + self.base = Path(temporary.name).resolve() + self.root = self.base / "corpus" + self.root.mkdir() + self.spec = identity.load_schema( + Path(__file__).resolve().parents[2] / "docs/workflows/record-identity.schema.json" + ) + self.relative = "data/video/source/Series/2024/Item" + self.record_dir = self.root / self.relative + self.record_dir.mkdir(parents=True) + self.metadata = { + "series": "Series", + "item": "Item", + "source": "youtube", + "channel": "source", + "category": None, + "episode": None, + "parts": [ + {"video_id": "abcdefghijk", "url": "https://www.youtube.com/watch?v=abcdefghijk", "title": "First"}, + {"video_id": "lmnopqrstuv", "url": "https://www.youtube.com/watch?v=lmnopqrstuv", "title": "Second"}, + ], + "sessions": [ + {"session_name": "abcdefghijk_sess01", "video_id": "wxyz0123456"}, + {"session_name": "lmnopqrstuv_sess02"}, + {"session_name": "lmnopqrstuv_sess100"}, + ], + } + self.record = { + "path": self.relative, + "series": "Series", + "item": "Item", + "parts": ["abcdefghijk", "lmnopqrstuv"], + "has_transcript": True, + } + self.items = [self.record] + self.save_metadata() + self.save_index() + (self.record_dir / "transcript.txt").write_text("Derived transcript.\n", encoding="utf-8") + + def save_metadata(self) -> None: + (self.record_dir / "metadata.json").write_text(json.dumps(self.metadata), encoding="utf-8") + + def save_index(self) -> None: + (self.root / "INDEX.json").write_text(json.dumps({"items": self.items}), encoding="utf-8") + + def assert_rejected(self, fragment: str, spec: identity.Spec | None = None) -> dict[str, int]: + problems, counts = identity.check_corpus(self.root, spec or self.spec) + self.assertTrue(any(fragment in problem for problem in problems), problems) + return counts + + def test_nested_series_and_standalone_session_upload(self) -> None: + for directory, name in ( + ("captions", "A human title.srt"), + ("translations", "not-a-video-id.fr.srt"), + ): + target = self.record_dir / directory / name + target.parent.mkdir() + target.write_text("1\n00:00:00,000 --> 00:00:01,000\nWords\n", encoding="utf-8") + # Raw transcript validation belongs to Journal-Utilities, not this checker. + (self.record_dir / "transcript.json").write_text("raw-layer sentinel", encoding="utf-8") + self.assertEqual( + identity.check_corpus(self.root, self.spec), + ([], {"records": 1, "parts": 2, "duplicates": 0}), + ) + + def test_source_relative_duplicate(self) -> None: + canonical = self.root / "data/video/source/Series/Canonical" + canonical.mkdir() + metadata = dict(self.metadata, item="Canonical") + (canonical / "metadata.json").write_text(json.dumps(metadata), encoding="utf-8") + self.metadata["duplicate_of"] = "Series/Canonical" + self.record["duplicate_of"] = "Series/Canonical" + self.items.append(dict(self.record, path=canonical.relative_to(self.root).as_posix(), + item="Canonical", has_transcript=False)) + del self.items[1]["duplicate_of"] + self.save_metadata() + self.save_index() + self.assertEqual( + identity.check_corpus(self.root, self.spec), + ([], {"records": 2, "parts": 4, "duplicates": 1}), + ) + + def test_internal_symlinks_and_explicit_symlinked_root(self) -> None: + stored_index = self.root / "stored-index.json" + (self.root / "INDEX.json").rename(stored_index) + (self.root / "INDEX.json").symlink_to(stored_index.name) + stored_record = self.root / "stored-record" + self.record_dir.rename(stored_record) + self.record_dir.symlink_to(stored_record, target_is_directory=True) + linked_root = self.base / "corpus-link" + linked_root.symlink_to(self.root, target_is_directory=True) + self.assertEqual( + identity.check_corpus(linked_root, self.spec), + ([], {"records": 1, "parts": 2, "duplicates": 0}), + ) + + def test_index_path_admission_precedes_read(self) -> None: + outside = self.base / "outside.json" + outside.write_bytes(b"\xffoutside") + for name in (str(outside), "../outside.json", "folder/../INDEX.json", "C:/outside.json"): + with self.subTest(path=name): + counts = self.assert_rejected("relative path without traversal", replace(self.spec, index_file=name)) + self.assertEqual(counts["records"], 0) + + def test_index_symlink_escape_precedes_read(self) -> None: + outside = self.base / "outside.json" + outside.write_bytes(b"\xffoutside") + index = self.root / "INDEX.json" + index.unlink() + index.symlink_to(outside) + self.assertEqual(self.assert_rejected("path escapes the corpus")["records"], 0) + + def test_record_path_admission_precedes_metadata_read(self) -> None: + outside = self.base / "Outside" + outside.mkdir() + (outside / "metadata.json").write_bytes(b"\xffoutside") + for value in ( + str(outside), "../Outside", "data/../Outside", "/missing", "C:/Outside", + "", ".", None, 7, ["data"], "data\\Outside", + ): + with self.subTest(path=value): + self.record["path"] = value + self.save_index() + self.assertEqual( + self.assert_rejected("relative path without traversal")["records"], 0 + ) + + def test_record_symlink_escape_precedes_metadata_read(self) -> None: + outside = self.base / "Outside" + self.record_dir.rename(outside) + (outside / "metadata.json").write_bytes(b"\xffoutside") + self.record_dir.symlink_to(outside, target_is_directory=True) + self.assertEqual(self.assert_rejected("path escapes the corpus")["records"], 0) + + def test_metadata_symlink_escape_precedes_read(self) -> None: + outside = self.base / "outside.json" + outside.write_bytes(b"\xffoutside") + metadata = self.record_dir / "metadata.json" + metadata.unlink() + metadata.symlink_to(outside) + self.assertEqual(self.assert_rejected("path escapes the corpus")["records"], 0) + + def test_series_cannot_name_corpus_or_host_ancestor(self) -> None: + for name in (self.root.name, self.base.name): + with self.subTest(series=name): + self.metadata["series"] = self.record["series"] = name + self.save_metadata() + self.save_index() + self.assert_rejected("inside the corpus") + + def test_index_identity_mismatches(self) -> None: + for key, wrong in ( + ("series", "Other"), ("item", "Other"), + ("parts", ["lmnopqrstuv", "abcdefghijk"]), + ("parts", ["abcdefghijk"]), ("parts", [{"video_id": "abcdefghijk"}]), + ): + with self.subTest(key=key, value=wrong): + original = self.record[key] + self.record[key] = wrong + self.save_index() + self.assert_rejected(f"INDEX `{key}` disagrees") + self.record[key] = original + + def test_index_requires_identity_fields(self) -> None: + for key in ("series", "item", "parts", "has_transcript"): + with self.subTest(key=key): + original = self.record.pop(key) + self.save_index() + self.assert_rejected(f"INDEX record lacks `{key}`") + self.record[key] = original + + def test_metadata_identity_must_match_path(self) -> None: + self.metadata["item"] = self.record["item"] = "Different" + self.save_metadata() + self.save_index() + self.assert_rejected("item id must name its own directory") + + def test_duplicate_record_path(self) -> None: + self.items.append(dict(self.record)) + self.save_index() + self.assert_rejected("duplicate record path") + + def test_transcript_flag_binds_regular_file(self) -> None: + for value in (False, 1, "true", None): + with self.subTest(flag=value): + self.record["has_transcript"] = value + self.save_index() + self.assert_rejected("INDEX `has_transcript` must be True") + transcript = self.record_dir / "transcript.txt" + transcript.unlink() + self.record["has_transcript"] = True + self.save_index() + self.assert_rejected("INDEX `has_transcript` must be False") + self.record["has_transcript"] = False + self.save_index() + self.assertEqual(identity.check_corpus(self.root, self.spec)[0], []) + transcript.mkdir() + self.assert_rejected("must be a regular file") + + def test_transcript_symlink_escape_even_when_flag_false(self) -> None: + outside = self.base / "outside.txt" + outside.write_text("Outside transcript", encoding="utf-8") + transcript = self.record_dir / "transcript.txt" + transcript.unlink() + transcript.symlink_to(outside) + for flag in (True, False): + with self.subTest(flag=flag): + self.record["has_transcript"] = flag + self.save_index() + self.assert_rejected("path escapes the corpus") + + def test_session_names_bind_part_ids_not_standalone_upload(self) -> None: + for value, fragment in ( + ("abcdefghijk_sess00", "does not match session_id"), + ("abcdefghijk_sess1", "does not match session_id"), + ("abcdefghijk_sess001", "does not match session_id"), + ("abcdefghijk_sess01\n", "does not match session_id"), + ("wxyz0123456_sess01", "does not belong to a record part"), + (None, "does not match session_id"), + ): + with self.subTest(session=value): + self.metadata["sessions"] = [{"session_name": value}] + self.save_metadata() + self.assert_rejected(fragment) + for sessions in (None, "abcdefghijk_sess01", {}, [None], [{}]): + with self.subTest(sessions=sessions): + self.metadata["sessions"] = sessions + self.save_metadata() + self.assert_rejected("sessions") + del self.metadata["sessions"] + self.save_metadata() + self.assertEqual(identity.check_corpus(self.root, self.spec)[0], []) + + def test_invalid_part_identity_and_shape(self) -> None: + for value in ("short", "abcdefghijk\n", None, []): + with self.subTest(video_id=value): + self.metadata["parts"][0]["video_id"] = value + self.save_metadata() + self.assert_rejected("not an 11-character YouTube id") + self.metadata["parts"] = [None] + self.save_metadata() + self.assert_rejected("parts[0] is not an object") + self.metadata["parts"] = None + self.save_metadata() + self.assert_rejected("`parts` must be a list") + + def test_duplicate_index_presence_and_value(self) -> None: + self.record["duplicate_of"] = "Series/Canonical" + self.save_index() + self.assert_rejected("INDEX `duplicate_of` disagrees") + del self.record["duplicate_of"] + self.metadata["duplicate_of"] = "Series/Canonical" + self.save_metadata() + self.save_index() + self.assert_rejected("INDEX `duplicate_of` disagrees") + self.record["duplicate_of"] = "Series/Different" + self.save_index() + self.assert_rejected("INDEX `duplicate_of` disagrees") + + def test_duplicate_path_is_source_relative_and_contained(self) -> None: + for value, fragment in ( + ("Series/Missing", "not a record path"), + (self.relative, "not a record path"), + ("../Item", "relative path without traversal"), + (str(self.record_dir), "relative path without traversal"), + (None, "relative path without traversal"), + ): + with self.subTest(duplicate=value): + self.metadata["duplicate_of"] = self.record["duplicate_of"] = value + self.save_metadata() + self.save_index() + self.assert_rejected(fragment) + outside = self.base / "external-record" + outside.mkdir() + (self.root / "data/video/source/Series/Escape").symlink_to(outside, target_is_directory=True) + self.metadata["duplicate_of"] = self.record["duplicate_of"] = "Series/Escape" + self.save_metadata() + self.save_index() + self.assert_rejected("path escapes the corpus") + + def test_malformed_json_and_nonregular_metadata(self) -> None: + metadata = self.record_dir / "metadata.json" + for value in (b"{", b"\xff", b"[]"): + with self.subTest(content=value): + metadata.write_bytes(value) + self.assert_rejected("metadata.json") + metadata.unlink() + metadata.mkdir() + self.assert_rejected("regular JSON file") + + def test_schema_drives_transcript_key_and_artifact(self) -> None: + document = json.loads(self.spec.path.read_text(encoding="utf-8")) + block = document["x-identity"] + block["transcript_flag"] = {"key": "transcribed", "artifact": "derived/view.txt"} + block["index_record_keys"].remove("has_transcript") + block["index_record_keys"].append("transcribed") + schema = self.root / "schema.json" + schema.write_text(json.dumps(document), encoding="utf-8") + spec = identity.load_schema(schema) + del self.record["has_transcript"] + self.record["transcribed"] = True + self.save_index() + self.assert_rejected("INDEX `transcribed` must be False", spec) + artifact = self.record_dir / "derived/view.txt" + artifact.parent.mkdir() + artifact.write_text("Derived words", encoding="utf-8") + self.assertEqual(identity.check_corpus(self.root, spec)[0], []) + + def test_schema_refuses_unsafe_paths_and_missing_relationships(self) -> None: + schema = self.root / "schema.json" + for key, value in ( + ("index_file", "../INDEX.json"), + ("index_file", str(self.root / "INDEX.json")), + ("transcript_flag", {"key": "has_transcript", "artifact": "../transcript.txt"}), + ("transcript_flag", {"key": "has_transcript"}), + ("index_record_keys", ["path"]), + ("patterns", {"video_id": "^[a-z]{11}$", "session_id": "^[a-z]{11}_sess[0-9]{2}$"}), + ): + with self.subTest(key=key, value=value): + document = json.loads(self.spec.path.read_text(encoding="utf-8")) + document["x-identity"][key] = value + schema.write_text(json.dumps(document), encoding="utf-8") + with self.assertRaises(identity.IdentityError): + identity.load_schema(schema) + + + def test_empty_parts_and_absent_sessions_are_valid(self) -> None: + self.metadata["parts"] = self.record["parts"] = [] + del self.metadata["sessions"] + (self.record_dir / "transcript.txt").unlink() + self.record["has_transcript"] = False + self.save_metadata() + self.save_index() + self.assertEqual( + identity.check_corpus(self.root, self.spec), + ([], {"records": 1, "parts": 0, "duplicates": 0}), + ) + + def artifact_contract(self) -> tuple[Path, identity.Spec, Path]: + # Root discovery needs only this real marker, not a Git process/store. + (self.root / ".git").mkdir() + schema = self.root / "docs/workflows/record-identity.schema.json" + schema.parent.mkdir(parents=True) + schema.write_bytes(self.spec.path.read_bytes()) + module = self.root / "scripts/gate" / MODULE + module.parent.mkdir(parents=True) + module.write_text(BODY, encoding="utf-8") + contract = schema.parent / "publishing.md" + contract.write_text( + f"{schema.name} {hashlib.sha256(schema.read_bytes()).hexdigest()}\n" + f"{MODULE} {hashlib.sha256(module.read_bytes()).hexdigest()}\n", + encoding="utf-8", + ) + return contract, identity.load_schema(schema), module + + def test_contract_refuses_escaping_module_and_schema_symlinks(self) -> None: + contract, spec, module = self.artifact_contract() + self.assertEqual(identity.check_contract(contract, spec), []) + for artifact in (module, spec.path): + with self.subTest(artifact=artifact.name): + original = artifact.read_bytes() + outside = self.base / artifact.name + outside.write_bytes(original) + artifact.unlink() + artifact.symlink_to(outside) + problems = identity.check_contract(contract, spec) + self.assertTrue(any("path escapes the corpus" in p for p in problems), problems) + artifact.unlink() + artifact.write_bytes(original) + + def test_contract_refuses_ambiguous_artifact(self) -> None: + contract, spec, module = self.artifact_contract() + duplicate = self.root / ".github" / MODULE + duplicate.parent.mkdir() + duplicate.write_bytes(module.read_bytes()) + self.assertIsNone(identity.resolve_artifact(self.root, MODULE)) + problems = identity.check_contract(contract, spec) + self.assertTrue(any("not one file under" in p for p in problems), problems) + + def test_explicit_copy_is_not_replaced_by_discovered_artifact(self) -> None: + contract, _, module = self.artifact_contract() + selected = self.base / MODULE + selected.write_bytes(module.read_bytes()) + self.assertIsNone(identity.verify_copy(selected, contract)) + selected.write_text(BODY + "# changed selected copy\n", encoding="utf-8") + self.assertIn("drifted", identity.verify_copy(selected, contract)) + selected.unlink() + selected.mkdir() + self.assertIn("regular file", identity.verify_copy(selected, contract)) + + def run() -> list[str]: failures: list[str] = [] with tempfile.TemporaryDirectory() as tmp: @@ -56,6 +447,9 @@ def run() -> list[str]: failures.append("selftest: a diverged .py copy was accepted") elif "drifted" not in problem: failures.append(f"selftest: diverged copy failed without a drift message: {problem}") + result = unittest.TestResult() + unittest.defaultTestLoader.loadTestsFromTestCase(CorpusIdentityTests).run(result) + failures.extend(f"selftest: {test.id()}\n{detail}" for test, detail in result.errors + result.failures) return failures diff --git a/scripts/gate/utilities.py b/scripts/gate/utilities.py index 3e78b0f99..02f677d52 100644 --- a/scripts/gate/utilities.py +++ b/scripts/gate/utilities.py @@ -2,8 +2,8 @@ ``fast`` is the derived-index check alone (seconds). ``full`` adds the split transcript check, the read-only integrity gate, and manifest reconciliation. -Both run the pinned revision; an explicit ``--utilities`` tree is accepted only -when it sits on the pin (see ``common.journal_utilities``). +Both execute an owned snapshot of the pinned Git objects. An explicit +``--utilities`` source need only contain the commit; its working tree is not run. """ from __future__ import annotations @@ -15,28 +15,28 @@ def main(argv: list[str], arm: str) -> int: parser = argparse.ArgumentParser(prog=f"./run.sh {arm}", description=__doc__) - parser.add_argument("--utilities", help="existing Journal-Utilities tree at the pin") + parser.add_argument("--utilities", help="local Git object source containing the Journal-Utilities pin") args = parser.parse_args(argv) - tree = journal_utilities(args.utilities) - scripts = tree / "scripts" - manifest = tree / "data" / "output" / "channel_videos.json" - python = "python3" - - run("derived indexes", [python, str(scripts / "generate_journal_indexes.py"), "--journal", ".", "--check"]) - if arm == "fast": - return 0 - - run( - "split transcript artifacts", - [python, str(scripts / "repair_split_transcripts.py"), "--journal", ".", "--utilities", str(tree), "--check"], - ) - run( - "journal integrity", - [python, str(scripts / "validate_journal.py"), "--journal", ".", "--manifest", str(manifest)], - ) - run( - "manifest reconciliation", - [python, str(ROOT / "scripts" / "gate" / "arm_manifest.py"), "--manifest", str(manifest)], - ) + with journal_utilities(args.utilities) as tree: + scripts = tree / "scripts" + manifest = tree / "data" / "output" / "channel_videos.json" + python = "python3" + + run("derived indexes", [python, "-B", str(scripts / "generate_journal_indexes.py"), "--journal", ".", "--check"]) + if arm == "fast": + return 0 + + run( + "split transcript artifacts", + [python, "-B", str(scripts / "repair_split_transcripts.py"), "--journal", ".", "--utilities", str(tree), "--check"], + ) + run( + "journal integrity", + [python, "-B", str(scripts / "validate_journal.py"), "--journal", ".", "--manifest", str(manifest)], + ) + run( + "manifest reconciliation", + [python, "-B", str(ROOT / "scripts" / "gate" / "arm_manifest.py"), "--manifest", str(manifest)], + ) return 0