diff --git a/README.md b/README.md index dbb65f5..0a68f8a 100644 --- a/README.md +++ b/README.md @@ -40,6 +40,8 @@ it is doing and what to do if it stops; a run that refuses with a reason is work intended. The live path needs a running Docker daemon because SkillTree owns quizzes, completion, points, and ranking. The offline path needs neither Docker nor a network. +Help, readiness, and artifact-only exits do not initialize passwords. Live launchers share the password owner; bootstrap may skip seeding only with `--skip-seed-if-live-content-ready` after semantic content verification, not matching counts. Offline artifacts and controlled regressions do not prove a live official SkillTree display. + ## Read it without installing anything Generated outputs are committed, so the artifacts are browsable straight from the diff --git a/docs/gates.md b/docs/gates.md index 60ad4e7..42ee10f 100644 --- a/docs/gates.md +++ b/docs/gates.md @@ -90,6 +90,10 @@ between them fails instead of silently changing the resolution. The live half of the gate set (Docker, a real SkillTree, browser proofs, captured screenshots) stays local and explicit. No arm here simulates it. +Strict live proof requires semantic content parity and actual official SDK project/subject/quiz views with visible same-origin frames, ready/accessibility evidence, and matching project/instance/export digest. Capture consumes a successful v2 `--proof-path` and matching PNG associations. Pending/fallback/unchecked views and controlled loopback fixtures are not live proof. + +The 2026-11-13 full non-browser census failed: its native source binding changed, seven tests failed, and coverage was below the unchanged 90% floor. The separate seven-node continuation and appended coverage database are narrower observations, not a fresh immutable whole-source certificate. The complete per-module census still records under-floor modules with no approved exceptions. Script smoke tests now run real generators in a disposable copy of public source inputs instead of rewriting the audited checkout's tracked output; final qualification must still verify the native unchanged-source receipt. Source-bound machine reports remain authoritative. Keep science K4, the pinned GNN origin, real stack/browser, and roadmap-owner holds separate; generated `tasks.yaml`/TODO/readiness projections are not hand-updated. + ## Measured numbers Per-arm wall times and the test/coverage counts are generated, not written by hand: diff --git a/docs/learner_dashboard.md b/docs/learner_dashboard.md index b0c9765..61717c1 100644 --- a/docs/learner_dashboard.md +++ b/docs/learner_dashboard.md @@ -95,14 +95,9 @@ uv run python scripts/validate_full_app_stack.py --keep-running Use this when reporting that the app is live. It starts or reuses the Docker Compose SkillTree stack, regenerates project artifacts, bootstraps and seeds local SkillTree auth, reconciles missing live prerequisite edges, starts the dashboard, requires `/healthz`, -requires live admin count parity, requires live dependency parity, checks -figure-registry/dashboard chart-contract parity, proves the raw SkillTree `FORM` login -through `chrome-devtools-axi`, and runs the Chrome console gate. Docker daemon -availability is a hard precondition: if Docker or Colima is not running, this command -fails rather than silently downgrading to artifact-only mode. `--app-ready` is the faster -local usability proof and is explicitly not a full dependency-parity proof. -`--keep-running` leaves the dashboard process open for manual inspection at -`http://127.0.0.1:8765`. +requires semantic admin content parity and complete live dependency parity, checks figure-registry/dashboard chart contracts, and proves real FORM login plus the official SDK's project, subject, and quiz views. Visible matching-origin frames, ready callback/accessibility snapshots, and matching project/instance/export digest are required; pending, fallback, unchecked, or count-only states are not green. `chrome-devtools-axi` and a real Docker/SkillTree stack are prerequisites, not substituted boundaries. + +Successful v2 browser/full-stack proof supplies `--proof-path` to screenshot capture. Use explicit task-owned screenshot output with matching PNG associations. `--keep-running` retains only a successfully admitted dashboard server; failure must settle it. `--app-ready` remains a narrower usability scope, not full dependency parity. Artifact-only mode: @@ -163,7 +158,7 @@ lesson. are absolute, credential-free HTTP(S) URLs; unsafe schemes remain plain text. - SkillTree deployment matrix: exposes only browser-safe deployment mode status for local FORM bootstrap, ignored local state, token-backed admin seeding, and externally operated - SkillTree. Admin secrets, client secrets, and bearer tokens remain server-side. + SkillTree. Root credentials and trusted-client secrets remain server-side. The temporary learner bearer token reaches official browser-client memory, never persistent browser storage; it is not accurate to call all bearer tokens server-only. - Official display host: imports `@skilltree/skills-client-js` and configures `SkillsConfiguration` with `serviceUrl`, `projectId=activeInference`, and an absolute dashboard URL for the local `/api/skilltree/token` authenticator when bootstrap is @@ -176,6 +171,8 @@ lesson. proof that the live SkillTree process is currently running or accepting browser passwords. +Public artifact/status projections remove quiz answer options and correctness keys recursively; the raw validated export and semantic digest remain server-side authorities. `/api/status.project.project` supplies the nested public identity used by live validation. No public projection or fixture response is a scientific, security-review, or live-service qualification by itself. + ## Accessibility The shell targets WCAG 2.1 Level A keyboard and structure basics: a visually-hidden diff --git a/docs/output_inventory.md b/docs/output_inventory.md index fc0eabb..112c053 100644 --- a/docs/output_inventory.md +++ b/docs/output_inventory.md @@ -8,6 +8,8 @@ Paths in the table are relative to the project root (`Active_Skillference/`). +The complete registered figure population includes captured artifacts; it is not the deterministic rendered subset. Registry-owned counts and generated provenance must describe the right population without inventing a fixed total. + ## File inventory | File | Role | Generated by | Step | diff --git a/docs/pilot/README.md b/docs/pilot/README.md index ec3a830..ad7e676 100644 --- a/docs/pilot/README.md +++ b/docs/pilot/README.md @@ -17,6 +17,8 @@ The contract requires consent and data minimization, prohibits direct identifier analysis export, names the unit of analysis and missing-data policy, and keeps SkillTree as the system of record for progress and quiz state. +Four nonempty linked populations establish schema readiness only. They do not establish recruitment, participant data, demographics, collection completion, or educational efficacy. + ## Protocol sequence 1. Pre-register recruitment, inclusion/exclusion, outcomes, observation windows, stopping diff --git a/docs/publication_readiness.md b/docs/publication_readiness.md index 0013d9a..1ee30d5 100644 --- a/docs/publication_readiness.md +++ b/docs/publication_readiness.md @@ -29,11 +29,7 @@ roadmap tasks are complete. - License, third-party notice, citation metadata, DOI, and public-repository boundaries are signposted in `LICENSE`, `THIRD_PARTY_NOTICES.md`, `docs/ip_license_status.md`, `CITATION.cff`, `.zenodo.json`, `codemeta.json`, and `manuscript/config.yaml`. -- Renderer-facing repository and release-date metadata remain blank or omitted until a real - published release exists, so rendered PDFs, CFF, and CodeMeta do not advertise an - unavailable release endpoint. Probed on 2026-08-09, - `https://github.com/ActiveInferenceInstitute/Active_Skillference` exists and is public but - holds zero refs, so the target is reachable and still unpublished. +- The 2026-08-09 observation found a reachable public target with zero refs; that is historical evidence, not today's remote state. Later v1.0.0/DOI records below remain authoritative for their own identifiers. Current metadata must agree with actual records and a newly admitted observation, never a guessed blanket unpublished state. ## Release-Allowed Claims @@ -43,12 +39,7 @@ roadmap tasks are complete. - SkillTree is cited and discussed as an NSA-origin open-source training platform, with explicit non-endorsement and deployment-boundary language. - Project code/content metadata currently declares MIT, while third-party materials are either inventoried as vendored notices or cited/linked without relicensing. -- The owner-confirmed public target is - `ActiveInferenceInstitute/Active_Skillference`, and that target now exists as an empty - public repository; the claim becomes release-allowed only after the repository carries - published content and a real release, and the release inventory is refreshed. -- Platform-facing source-repository links and release dates are not release-allowed while - that target holds no published content. +- Public repository/release claims must match the actual selected release record and newly admitted remote observation. A reachable target, an empty target, advertised refs, a failed probe, and an unrefreshed report are different states. Do not substitute historical reachability for current publication or infer current emptiness from a failed command. ## Claims To Avoid Until Future Evidence @@ -69,11 +60,8 @@ roadmap tasks are complete. ## Current Blockers -- DOI/Zenodo release is blocked until an actual release tag and deposit exist. -- Public repository release is blocked until the confirmed target - `ActiveInferenceInstitute/Active_Skillference` holds published content and a real release - tag, and repository metadata is refreshed. The target's existence cleared the - reachability question only; the publication question is still open. +- Subsequent DOI/Zenodo reservation, streamed upload, and publication each require explicit authorization and actual receipts. Existing v1.0.0 identifiers remain preserved; this work authorizes no new deposit, tag, upload, or release. +- A current repository/readiness projection requires its owning generator and an admitted fresh probe. AS-O14's first-page/title identity and product-selection hold, plus legal/date/DOI decisions, remain owner obligations. - Learner-outcome claims are blocked until a real study produces evidence. - Live deployment claims are blocked until Docker SkillTree, dashboard handoff, privacy, security, trademark, and access-control reviews are complete. @@ -101,11 +89,10 @@ uv run python scripts/validate_web_math.py uv run python scripts/validate_release_readiness_inventory.py ``` -Also re-check the public-target blocker. An empty result means the repository exists but -nothing is published there, which keeps the blocker in force: +An explicitly admitted future remote check may inspect advertised refs; empty success is not the same as failed or unrefreshed evidence. No new remote observation is claimed here: ```bash -git ls-remote --heads https://github.com/ActiveInferenceInstitute/Active_Skillference.git HEAD +git ls-remote --refs https://github.com/ActiveInferenceInstitute/Active_Skillference.git ``` For a DOI-backed release or deployment publication rather than a static public-review diff --git a/docs/quickstart.md b/docs/quickstart.md index f16c477..38d8180 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -13,6 +13,8 @@ screen-reader affordances. Use `./run.sh --offline` when Docker is unavailable a `./run.sh --check` to test readiness without starting anything. The step-by-step commands below are what `run.sh` runs, for when you want to drive them individually. +Help/readiness/artifact-only paths never initialize passwords. Live launchers use the shared password owner. `--skip-seed-if-live-content-ready` admits actual semantic content readiness; the removed count-only shortcut is not sufficient proof. A successful artifact build is not a running official SkillTree stack. + Get Active Skillference running in 5 minutes: validate the curriculum DAG, run the kernels that produce every claim, export a provenance-gated SkillTree project, and run diff --git a/docs/rendering_pipeline.md b/docs/rendering_pipeline.md index 9521ada..62fc76d 100644 --- a/docs/rendering_pipeline.md +++ b/docs/rendering_pipeline.md @@ -13,13 +13,9 @@ the manuscript's own configuration, preamble, and bibliography. It needs `pandoc `pandoc-crossref`, `xelatex` and `bibtex` on `PATH` and nothing from any other repository; the arm is `./gates.sh render` ([`gates.md`](gates.md)). -After finalizing the PDF, the renderer writes the tracked -`output/pdf/Active_Skillference_combined.receipt.json`: schema -`active-skillference-pdf/v1`, repository-relative output path, exact byte count, -SHA-256, and the checkout's Git `HEAD` source revision. Commit source changes before -rendering so that revision identifies the source used; commit the receipt alongside -the PDF. The render arm finishes with `validate_rendered_freshness.py`, which rejects -stale hydrated sections, graph counts, or publication-bundle roadmap text. +After finalizing the PDF, the renderer writes an `active-skillference-pdf/v2` receipt binding actual configuration, preamble, bibliography, ordered hydrated sections, literal TeX assets, and finalized output bytes. HEAD alone is not source identity. Legacy receipts require a new admitted render, never relabeling or restamping. Unknown prose placeholders fail while literal code examples survive. + +The configured atlas is physical page two, before the table of contents; its placement has a real-render regression. Input/content custody, retained-publication identity, and parent visual inspection remain separate requirements. The freshness validator still rejects stale sections, graph counts, and roadmap text. ## The flow diff --git a/run.sh b/run.sh index 80eb6f7..e39856f 100755 --- a/run.sh +++ b/run.sh @@ -22,22 +22,6 @@ DASHBOARD_URL="http://${DASHBOARD_HOST}:${DASHBOARD_PORT}" SKILLTREE_WAIT_ATTEMPTS="${SKILLTREE_WAIT_ATTEMPTS:-240}" ACTIVE_SKILLFERENCE_LOCAL_STATE_DIR="${ACTIVE_SKILLFERENCE_LOCAL_STATE_DIR:-.active_skillference_local}" ACTIVE_SKILLFERENCE_LEARNER_ID="${ACTIVE_SKILLFERENCE_LEARNER_ID:-local-learner}" -# Compose interpolates this; it must match the value the stack was created -# with, so reuse the stored one and generate it only on first run. This mirrors -# src/integration/full_app_stack.py::ensure_postgres_password. -_pg_file="$ACTIVE_SKILLFERENCE_LOCAL_STATE_DIR/postgres_password" -if [ -z "${ACTIVE_SKILLFERENCE_POSTGRES_PASSWORD:-}" ]; then - if [ -s "$_pg_file" ]; then - ACTIVE_SKILLFERENCE_POSTGRES_PASSWORD="$(tr -d '\n' < "$_pg_file")" - else - mkdir -p "$ACTIVE_SKILLFERENCE_LOCAL_STATE_DIR" - chmod 700 "$ACTIVE_SKILLFERENCE_LOCAL_STATE_DIR" 2>/dev/null || true - ACTIVE_SKILLFERENCE_POSTGRES_PASSWORD="$(LC_ALL=C tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 40)" - printf '%s\n' "$ACTIVE_SKILLFERENCE_POSTGRES_PASSWORD" > "$_pg_file" - chmod 600 "$_pg_file" 2>/dev/null || true - fi -fi -export ACTIVE_SKILLFERENCE_POSTGRES_PASSWORD MODE="full" for arg in "$@"; do @@ -121,6 +105,14 @@ if [ "$MODE" = "offline" ]; then --host "$DASHBOARD_HOST" --port "$DASHBOARD_PORT" fi +# Live launchers share one private password owner; help/check/offline never enter it. +if [ -z "${ACTIVE_SKILLFERENCE_POSTGRES_PASSWORD:-}" ]; then + ACTIVE_SKILLFERENCE_POSTGRES_PASSWORD="$(uv run python -c \ + 'import sys; from src.integration.full_app_stack import ensure_postgres_password; print(ensure_postgres_password(sys.argv[1]))' \ + "$ACTIVE_SKILLFERENCE_LOCAL_STATE_DIR")" +fi +export ACTIVE_SKILLFERENCE_POSTGRES_PASSWORD + # ------------------------------------------------------------- live SkillTree # Run a long command with a live elapsed-time heartbeat, so a slow step is # visibly slow rather than indistinguishable from a hang. @@ -172,9 +164,9 @@ curl -fsS "$SKILLTREE_URL/public/status" >/dev/null 2>&1 || fail \ "Or run without it: ./run.sh --offline" step "Seeding the curriculum into SkillTree" -note "This is the slow step, and it is idempotent: if SkillTree already reports" -note "the exported subject and skill counts, seeding is skipped entirely and" -note "this finishes in seconds." +note "Seeding is skipped only when the actual project, subjects, skills, quizzes" +note "and answer keys match the export. Missing resources can be created;" +note "changed or extra content is refused rather than silently overwritten." note "A first seed writes every subject, skill, and quiz over REST. Budget" note "10-40 minutes on Apple/ARM Docker emulation; it is working, not stuck." note "Prerequisite edges are NOT seeded here. They are far slower and the" @@ -183,7 +175,7 @@ if ! run_with_heartbeat "seeding" uv run python scripts/bootstrap_local_skilltre --base-url "$SKILLTREE_URL" \ --state-dir "$ACTIVE_SKILLFERENCE_LOCAL_STATE_DIR" \ --learner-id "$ACTIVE_SKILLFERENCE_LEARNER_ID" \ - --skip-seed-if-live-counts-ready \ + --skip-seed-if-live-content-ready \ --skip-dependencies; then fail "Seeding did not complete." \ "The output above ends with the failing call." \ diff --git a/scripts/README.md b/scripts/README.md index ded6b87..77e401d 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -72,8 +72,8 @@ ls -la output/figures/ # generated PNGs plus figure_registry.json | `validate_web_math.py` | Fail if dashboard or rendered web HTML math surfaces lack MathJax support | `src.dashboard/assets/`, `output/web/` | none (exit 0/1) | | `publish_zenodo_deposit.py` | Upload the release artifact to the reserved Zenodo draft and publish it, minting a permanent DOI | `ZENODO_PROD_TOKEN`, `data/zenodo_reservation.json`, a release file | published deposit JSON on stdout | | `reserve_zenodo_doi.py` | Reserve a Zenodo DOI on an unpublished draft deposition; never publishes | `ZENODO_PROD_TOKEN` from a dotenv file, `.zenodo.json` | JSON reservation summary on stdout | -| `capture_dashboard_screenshots.py` | Capture the registered learner-dashboard evidence screenshots from a running local stack and write their provenance record | Chrome via `chrome-devtools-axi`, running dashboard, `src.screenshots` manifest | PNGs plus `screenshot_provenance.json` under `manuscript/assets/screenshots/` | -| `validate_dashboard_console.py` | Browser-console proof for the dashboard, with optional required-browser and required-dashboard modes | Chrome via `chrome-devtools-axi`, dashboard URL | none (exit 0/1) | +| `capture_dashboard_screenshots.py` | Capture from successful v2 browser/full-stack `--proof-path`, with matching project/instance/export digest and PNG associations | Real Chrome/official SDK stack and explicit task-owned output | PNGs plus bound provenance | +| `validate_dashboard_console.py` | Strict official project/subject/quiz view proof: `--require-browser --require-dashboard`, visible matching-origin frames and ready/accessibility evidence; no fallback success | Existing `chrome-devtools-axi` and actual dashboard/SkillTree | Explicit v2 proof artifact | | `validate_skilltree_form_login.py` | Browser proof that the raw local SkillTree `FORM` login reaches the authenticated administrator surface without leaking local secrets | Chrome via `chrome-devtools-axi`, `.active_skillference_local/skilltree_auth.json`, SkillTree login URL | JSON proof summary on stdout | | `validate_live_skilltree_dashboard.py` | CLI wrapper around `src.dashboard.live_validation`; fails if the local dashboard is not wired to a reachable SkillTree service, current export counts, complete artifacts, ready local auth, token exchange, and optional live parity checks | `src.dashboard.live_validation`, `output/data/skilltree_project.json`, dashboard `/api/status`, SkillTree `/public/status` | none (exit 0/1) | | `validate_full_app_stack.py` | CLI wrapper around `src.integration.full_app_stack` for the strict Docker + SkillTree + dashboard + browser proof, including dependency reconciliation unless `--app-ready` is selected | `src.integration.full_app_stack`, Docker/Compose, local SkillTree, dashboard, generated artifacts | JSON proof summary on stdout plus a redacted proof artifact under the state directory | @@ -85,7 +85,7 @@ ls -la output/figures/ # generated PNGs plus figure_registry.json | `z_generate_gate_metrics.py` | Render `docs/gate_metrics.md` from the recorded arm-timing receipt, or `--check` that it is current | `docs/_generated/arm_timings.json` | `docs/gate_metrics.md` with `--write`; none with `--check` | | `export_skilltree.py` | Provenance-gate the curriculum and export a SkillTree-native project JSON | `src.curriculum`, `src.kernels`, `src.provenance`, `src.skilltree` | `output/data/skilltree_project.json` | | `seed_skilltree.py` | Build + render the project and seed it into a live SkillTree server over REST | `src.curriculum`, `src.kernels`, `src.provenance`, `src.skilltree` | network (a running SkillTree instance) | -| `bootstrap_local_skilltree.py` | Bootstrap local SkillTree `FORM` auth, seed the project, and store ignored trusted-client state | `src.curriculum`, `src.kernels`, `src.provenance`, `src.skilltree.local_auth` | network + `.active_skillference_local/` | +| `bootstrap_local_skilltree.py` | Bootstrap shared live password owner, seed semantic content, store ignored trusted-client state; `--skip-seed-if-live-content-ready` replaces count-only readiness | Actual local SkillTree and current export | network + ignored private state | | `serve_learner_dashboard.py` | Serve the local learner dashboard shell | `src.dashboard` | local HTTP server at `http://127.0.0.1:8765` | | `validate_gnn_spec.py` | Validate the discrete-agent GNN specification against the pinned GNN revision and the kernel | `gnn/`, `src.gnn_conformance` | none (exit 0 pass, 1 conformance failure, 2 GNN unavailable) | | `run_skilltree.sh` | One command: bring up Docker SkillTree + Postgres, wait for the API, then seed | `docker-compose.yml`, `vendor/` | a running instance at `http://localhost:8080` | diff --git a/scripts/bootstrap_local_skilltree.py b/scripts/bootstrap_local_skilltree.py index fb7ddc8..4feb901 100755 --- a/scripts/bootstrap_local_skilltree.py +++ b/scripts/bootstrap_local_skilltree.py @@ -43,13 +43,13 @@ PROJECT_NAME = SKILLTREE_PROJECT_NAME -def build_project(): # type: ignore[no-untyped-def] +def build_project(project_id: str = PROJECT_ID, project_name: str = PROJECT_NAME): # type: ignore[no-untyped-def] """Build the provenance-rendered SkillTree project (no network).""" graph = build_curriculum() registry = build_claim_registry() validate_curriculum(graph.skills, registry) rendered = render_graph(graph, registry) - return export_project(rendered, PROJECT_ID, PROJECT_NAME) + return export_project(rendered, project_id, project_name) def main() -> int: @@ -85,11 +85,11 @@ def main() -> int: ), ) parser.add_argument( - "--skip-seed-if-live-counts-ready", + "--skip-seed-if-live-content-ready", action="store_true", help=( - "After login, skip idempotent core seeding when live SkillTree already " - "reports the exported subject and skill counts." + "Skip core seeding only after live identities, content and quiz answer " + "correctness match the current export." ), ) parser.add_argument( @@ -173,7 +173,9 @@ def main() -> int: ) return 0 - project = None if args.no_seed else build_project() + project = ( + None if args.no_seed else build_project(args.project_id, args.project_name) + ) try: result = ensure_local_skilltree_auth( base_url=args.base_url, @@ -183,7 +185,7 @@ def main() -> int: project=project, seed_project=not args.no_seed, seed_dependencies=not args.skip_dependencies, - skip_seed_if_live_counts_ready=args.skip_seed_if_live_counts_ready, + skip_seed_if_live_content_ready=args.skip_seed_if_live_content_ready, verify_token=True, ) except (LocalSkillTreeAuthError, SkillTreeClientError) as exc: diff --git a/scripts/capture_dashboard_screenshots.py b/scripts/capture_dashboard_screenshots.py index 0fa0a3e..a92b0de 100755 --- a/scripts/capture_dashboard_screenshots.py +++ b/scripts/capture_dashboard_screenshots.py @@ -37,6 +37,11 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace: help="tracked directory receiving the captured PNGs and provenance record", ) parser.add_argument("--browser-command", default=DEFAULT_BROWSER_COMMAND) + parser.add_argument( + "--proof-path", + type=Path, + help="Explicit current v2 full-stack proof; older/unmatched proofs remain unassociated.", + ) parser.add_argument("--viewport-width", type=int, default=VIEWPORT_WIDTH) parser.add_argument("--viewport-height", type=int, default=VIEWPORT_HEIGHT) return parser.parse_args(argv) @@ -51,6 +56,7 @@ def main(argv: list[str] | None = None) -> int: browser_command=args.browser_command, viewport_width=args.viewport_width, viewport_height=args.viewport_height, + proof_path=args.proof_path, ) try: provenance = capture_screenshots(settings) diff --git a/scripts/render_manuscript.py b/scripts/render_manuscript.py index 6d619dd..116c0da 100755 --- a/scripts/render_manuscript.py +++ b/scripts/render_manuscript.py @@ -31,8 +31,7 @@ from __future__ import annotations import argparse -import hashlib -import json +import os import shutil import subprocess import sys @@ -43,15 +42,25 @@ MANUSCRIPT = ROOT / "manuscript" OUTPUT_MANUSCRIPT = ROOT / "output" / "manuscript" OUTPUT_PDF = ROOT / "output" / "pdf" -PDF_NAME = "Active_Skillference_combined.pdf" -# Files the hydrator writes beside the sections that are not manuscript sections. -SECTION_EXCLUDE = {"config.yaml", "preamble.md", "README.md", "AGENTS.md", "SYNTAX.md"} # Log lines that mean the PDF would ship with broken references or citations. A LaTeX # "Font Warning: ... undefined" line must not be read as a broken reference, so a match # needs both the subject ("Citation"/"Reference") and the word "undefined". BROKEN_LOG_SUBJECTS = ("Citation", "Reference") +sys.path.insert(0, str(ROOT)) + +from src.manuscript.pdf_receipt import ( + PDF_NAME, + add_pdf_assets, + capture_pdf_inputs, + hydrated_section_files, + latex_asset_paths, + validate_pdf_inputs, + write_pdf_receipt, +) + + def _require(tool: str) -> str: found = shutil.which(tool) if found is None: @@ -167,12 +176,50 @@ def _titlepage(config: dict[str, object]) -> str: return "\n".join(lines) +def _page_two_visual(config: dict[str, object]) -> str: + front = config.get("front_matter") + if not isinstance(front, dict) or "page_two_visual" not in front: + return "" + visual = front["page_two_visual"] + if not isinstance(visual, dict): + raise SystemExit("front_matter.page_two_visual must be a mapping") + image_name, title, caption = ( + visual.get(key) for key in ("image", "title", "caption") + ) + if not all( + isinstance(value, str) and value.strip() + for value in (image_name, title, caption) + ): + raise SystemExit( + "front_matter.page_two_visual requires image, title and caption" + ) + image = (OUTPUT_MANUSCRIPT / str(image_name)).resolve() + if not image.is_file(): + raise SystemExit( + f"front_matter.page_two_visual.image does not resolve to a file: {image}" + ) + return "\n".join( + [ + "```{=latex}", + r"\thispagestyle{empty}", + r"\begin{center}", + r"{\Large\bfseries " + _latex_escape(str(title)) + r"\par}", + r"\vspace{0.6em}", + r"\includegraphics[width=\textwidth,height=0.75\textheight,keepaspectratio]{" + + image.as_posix() + + "}", + r"\par\vspace{0.6em}", + _latex_escape(str(caption)), + r"\end{center}", + r"\clearpage", + "```", + "", + ] + ) + + def _section_files() -> list[Path]: - return [ - path - for path in sorted(OUTPUT_MANUSCRIPT.glob("*.md")) - if path.name not in SECTION_EXCLUDE - ] + return hydrated_section_files(OUTPUT_MANUSCRIPT) def _assemble(config: dict[str, object], destination: Path) -> list[Path]: @@ -182,7 +229,11 @@ def _assemble(config: dict[str, object], destination: Path) -> list[Path]: f"no hydrated sections in {OUTPUT_MANUSCRIPT}; run " "`uv run python scripts/z_generate_manuscript_variables.py`" ) - parts = [_titlepage(config)] + parts = [ + _titlepage(config), + _page_two_visual(config), + "```{=latex}\n\\setcounter{tocdepth}{2}\n\\tableofcontents\n\\clearpage\n```\n", + ] for index, path in enumerate(sections): if index: parts.append("```{=latex}\n\\newpage\n```\n") @@ -241,8 +292,8 @@ def _check_log(log: Path) -> None: ) -def _write_pdf_receipt(target: Path, *, root: Path = ROOT) -> Path: - """Record the finalized PDF bytes and the checkout's source revision.""" +def _write_pdf_receipt(target: Path, *, root: Path, inputs: dict[str, object]) -> Path: + """Record final PDF bytes and the actual pre-render input manifest.""" revision = subprocess.run( ["git", "rev-parse", "HEAD"], cwd=root, @@ -250,26 +301,20 @@ def _write_pdf_receipt(target: Path, *, root: Path = ROOT) -> Path: capture_output=True, text=True, ).stdout.strip() - content = target.read_bytes() - receipt = target.with_suffix(".receipt.json") - receipt.write_text( - json.dumps( - { - "schema": "active-skillference-pdf/v1", - "output_path": target.relative_to(root).as_posix(), - "size_bytes": len(content), - "sha256": hashlib.sha256(content).hexdigest(), - "source_revision": revision, - }, - indent=2, - ) - + "\n", - encoding="utf-8", + return write_pdf_receipt( + target, + root=root, + source_revision=revision, + inputs=inputs, ) - return receipt -def _render_pdf(scratch: Path, config: dict[str, object]) -> Path: +def _render_pdf( + scratch: Path, + config: dict[str, object], + *, + inputs: dict[str, object], +) -> Path: pandoc = _require("pandoc") crossref = _require("pandoc-crossref") xelatex = _require("xelatex") @@ -289,8 +334,6 @@ def _render_pdf(scratch: Path, config: dict[str, object]) -> Path: str(source), "--to=latex", "--standalone", - "--toc", - "--toc-depth=2", "--number-sections", "--filter", crossref, @@ -302,12 +345,19 @@ def _render_pdf(scratch: Path, config: dict[str, object]) -> Path: "--metadata", "documentclass=article", *_metadata(config), + "--variable", + "title=", "-o", str(tex), ], cwd=scratch, what="pandoc", ) + inputs = add_pdf_assets( + ROOT, + inputs, + latex_asset_paths(tex, build_directory=scratch, root=ROOT), + ) latex = [xelatex, "-interaction=nonstopmode", "-halt-on-error", tex.name] _run(latex, cwd=scratch, what="the first xelatex pass") @@ -315,14 +365,27 @@ def _render_pdf(scratch: Path, config: dict[str, object]) -> Path: _run(latex, cwd=scratch, what="the second xelatex pass") _run(latex, cwd=scratch, what="the final xelatex pass") _check_log(scratch / "combined.log") + validate_pdf_inputs(ROOT, inputs) produced = scratch / "combined.pdf" if not produced.is_file() or produced.stat().st_size == 0: raise SystemExit(f"xelatex reported success but wrote no PDF ({produced})") OUTPUT_PDF.mkdir(parents=True, exist_ok=True) target = OUTPUT_PDF / PDF_NAME - shutil.copy2(produced, target) - _write_pdf_receipt(target) + temporary: Path | None = None + try: + with tempfile.NamedTemporaryFile( + dir=OUTPUT_PDF, prefix=f".{PDF_NAME}.", delete=False + ) as stream: + temporary = Path(stream.name) + with produced.open("rb") as source_stream: + shutil.copyfileobj(source_stream, stream) + shutil.copystat(produced, temporary) + os.replace(temporary, target) + finally: + if temporary is not None: + temporary.unlink(missing_ok=True) + _write_pdf_receipt(target, root=ROOT, inputs=inputs) print(f"rendered {len(sections)} sections -> {target.relative_to(ROOT)}") return target @@ -341,10 +404,11 @@ def main(argv: list[str] | None = None) -> int: _require("xelatex") _require("bibtex") _hydrate(args.skip_kernels) + inputs = capture_pdf_inputs(ROOT, _section_files()) config = _config() with tempfile.TemporaryDirectory(dir=ROOT, prefix=".render-") as scratch: - _render_pdf(Path(scratch), config) + _render_pdf(Path(scratch), config, inputs=inputs) return 0 diff --git a/scripts/run_kernels.py b/scripts/run_kernels.py index d84e940..1e83eb2 100755 --- a/scripts/run_kernels.py +++ b/scripts/run_kernels.py @@ -20,7 +20,10 @@ def main() -> list[Path]: - bundle = build_generated_outputs(_ROOT) + try: + bundle = build_generated_outputs(_ROOT) + except ValueError as exc: + raise SystemExit(str(exc)) from None print(f"registered {bundle.provenance_claims} provenance claims") print(f"validated scholarship coverage: {bundle.cited_sources} cited sources") print( diff --git a/scripts/run_learning_dashboard.sh b/scripts/run_learning_dashboard.sh index acecd8a..9d2074d 100755 --- a/scripts/run_learning_dashboard.sh +++ b/scripts/run_learning_dashboard.sh @@ -13,7 +13,6 @@ DASHBOARD_AUTHENTICATOR="${DASHBOARD_AUTHENTICATOR:-${SKILLTREE_AUTHENTICATOR:-} ACTIVE_SKILLFERENCE_LOCAL_AUTH="${ACTIVE_SKILLFERENCE_LOCAL_AUTH:-1}" ACTIVE_SKILLFERENCE_LEARNER_ID="${ACTIVE_SKILLFERENCE_LEARNER_ID:-local-learner}" ACTIVE_SKILLFERENCE_LOCAL_STATE_DIR="${ACTIVE_SKILLFERENCE_LOCAL_STATE_DIR:-.active_skillference_local}" -POSTGRES_PASSWORD_FILE="$ACTIVE_SKILLFERENCE_LOCAL_STATE_DIR/postgres_password" SEED_ARGS=(--base-url "$SKILLTREE_URL" --require-live) if ! command -v docker >/dev/null 2>&1; then @@ -34,19 +33,9 @@ if ! command -v curl >/dev/null 2>&1; then fi if [ -z "${ACTIVE_SKILLFERENCE_POSTGRES_PASSWORD:-}" ]; then - mkdir -p "$ACTIVE_SKILLFERENCE_LOCAL_STATE_DIR" - chmod 700 "$ACTIVE_SKILLFERENCE_LOCAL_STATE_DIR" - if [ ! -s "$POSTGRES_PASSWORD_FILE" ]; then - ( - umask 077 - uv run python - <<'PY' > "$POSTGRES_PASSWORD_FILE" -import secrets - -print(secrets.token_urlsafe(32)) -PY - ) - fi - ACTIVE_SKILLFERENCE_POSTGRES_PASSWORD="$(cat "$POSTGRES_PASSWORD_FILE")" + ACTIVE_SKILLFERENCE_POSTGRES_PASSWORD="$(uv run python -c \ + 'import sys; from src.integration.full_app_stack import ensure_postgres_password; print(ensure_postgres_password(sys.argv[1]))' \ + "$ACTIVE_SKILLFERENCE_LOCAL_STATE_DIR")" fi export ACTIVE_SKILLFERENCE_POSTGRES_PASSWORD diff --git a/scripts/run_skilltree.sh b/scripts/run_skilltree.sh index 7d37407..9c3f9b3 100755 --- a/scripts/run_skilltree.sh +++ b/scripts/run_skilltree.sh @@ -16,7 +16,6 @@ SKILLTREE_WAIT_ATTEMPTS="${SKILLTREE_WAIT_ATTEMPTS:-240}" ACTIVE_SKILLFERENCE_LOCAL_AUTH="${ACTIVE_SKILLFERENCE_LOCAL_AUTH:-}" ACTIVE_SKILLFERENCE_LEARNER_ID="${ACTIVE_SKILLFERENCE_LEARNER_ID:-local-learner}" ACTIVE_SKILLFERENCE_LOCAL_STATE_DIR="${ACTIVE_SKILLFERENCE_LOCAL_STATE_DIR:-.active_skillference_local}" -POSTGRES_PASSWORD_FILE="$ACTIVE_SKILLFERENCE_LOCAL_STATE_DIR/postgres_password" if [ -z "$ACTIVE_SKILLFERENCE_LOCAL_AUTH" ]; then if [ -n "${SKILLTREE_TOKEN:-}" ]; then @@ -50,19 +49,9 @@ if [ "$ACTIVE_SKILLFERENCE_LOCAL_AUTH" != "1" ] && [ -z "${SKILLTREE_TOKEN:-}" ] fi if [ -z "${ACTIVE_SKILLFERENCE_POSTGRES_PASSWORD:-}" ]; then - mkdir -p "$ACTIVE_SKILLFERENCE_LOCAL_STATE_DIR" - chmod 700 "$ACTIVE_SKILLFERENCE_LOCAL_STATE_DIR" - if [ ! -s "$POSTGRES_PASSWORD_FILE" ]; then - ( - umask 077 - uv run python - <<'PY' > "$POSTGRES_PASSWORD_FILE" -import secrets - -print(secrets.token_urlsafe(32)) -PY - ) - fi - ACTIVE_SKILLFERENCE_POSTGRES_PASSWORD="$(cat "$POSTGRES_PASSWORD_FILE")" + ACTIVE_SKILLFERENCE_POSTGRES_PASSWORD="$(uv run python -c \ + 'import sys; from src.integration.full_app_stack import ensure_postgres_password; print(ensure_postgres_password(sys.argv[1]))' \ + "$ACTIVE_SKILLFERENCE_LOCAL_STATE_DIR")" fi export ACTIVE_SKILLFERENCE_POSTGRES_PASSWORD diff --git a/scripts/serve_learner_dashboard.py b/scripts/serve_learner_dashboard.py index c2d6e86..0d34e63 100755 --- a/scripts/serve_learner_dashboard.py +++ b/scripts/serve_learner_dashboard.py @@ -34,6 +34,9 @@ def main() -> int: parser.add_argument("--skilltree-url", default="http://localhost:8080") parser.add_argument("--host", default="127.0.0.1") parser.add_argument("--port", type=int, default=8765) + parser.add_argument( + "--instance-id", default="", help="Public identity for an owned proof run." + ) parser.add_argument( "--authenticator", default=os.environ.get("SKILLTREE_AUTHENTICATOR", ""), @@ -87,6 +90,7 @@ def main() -> int: local_auth_enabled=args.local_auth_enabled, local_state_dir=Path(args.local_state_dir), learner_id=args.learner_id, + instance_id=args.instance_id, ) if args.check: print(json.dumps(dashboard_payload(config), indent=2, sort_keys=True)) diff --git a/scripts/validate_dashboard_console.py b/scripts/validate_dashboard_console.py index b1dfd0b..7b7ed7b 100755 --- a/scripts/validate_dashboard_console.py +++ b/scripts/validate_dashboard_console.py @@ -1,42 +1,34 @@ #!/usr/bin/env python3 -"""Opt-in browser smoke gate: fail on unexpected dashboard console errors. +"""Opt-in real-browser proof of official project, subject and quiz views. -The server-side dashboard tests cannot see what the learner's browser logs. This gate -drives a real browser (chrome-devtools-axi) against a running dashboard and fails if any -console error appears EXCEPT the one known, cosmetic upstream probe: the -``@skilltree/skills-client-js`` SDK checks service status against SkillTree's built-in -default ``Inception`` project, which returns 401 under our project-scoped authenticator. -That probe is non-blocking (the real ``activeInference`` learner view still renders) and -is not fixable from our configuration, so it is allow-listed and everything else fails. - -Requires a running dashboard and the ``chrome-devtools-axi`` CLI. By default, if either -is absent the gate reports ``skipped`` (not ``clean``) and exits 0, matching the project -convention that optional local browser tooling must not break offline checks. Strict -callers can pass ``--require-browser`` and ``--require-dashboard`` to fail closed. +The gate waits for the official SDK's actual route callback, a visible frame +on the configured SkillTree origin, and matching browser-visible content. +Fallback and unresolved pending states fail even when the console is empty. +Only the established cosmetic upstream Inception probe is allow-listed. +Absent tooling/dashboard remains an explicit optional skip unless required. """ from __future__ import annotations import argparse -import re +import json import shutil -import subprocess -import time +import sys import urllib.error import urllib.request +from pathlib import Path from urllib.parse import urlparse -# Console errors that are known, cosmetic, and outside our control are allow-listed here. -_ALLOWED_ERROR_PATTERNS = ( - re.compile(r"projects/Inception/.*token.*401", re.IGNORECASE), - re.compile(r"Unable to authenticate.*Inception", re.IGNORECASE), -) - -# This script deliberately stays import-free from the project's src/ package (no -# sys.path bootstrap), so the http(s)-only guard is inlined rather than shared via -# src.net_guard; keep both checks in sync (see also src/net_guard.py). _ALLOWED_URL_SCHEMES = frozenset({"http", "https"}) +_ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(_ROOT)) + +from src.dashboard.browser_validation import ( + BrowserDisplayError, + validate_dashboard_browser, +) + def _dashboard_reachable(url: str, timeout: float) -> bool: if urlparse(url).scheme not in _ALLOWED_URL_SCHEMES: @@ -49,42 +41,14 @@ def _dashboard_reachable(url: str, timeout: float) -> bool: return False -def _console_errors(url: str, settle_seconds: float) -> list[str]: - subprocess.run( - ["chrome-devtools-axi", "open", url], - check=True, - capture_output=True, - text=True, - timeout=60, - ) - time.sleep(settle_seconds) - result = subprocess.run( - ["chrome-devtools-axi", "console", "--type", "error"], - check=True, - capture_output=True, - text=True, - timeout=60, - ) - errors: list[str] = [] - for line in result.stdout.splitlines(): - if "[error]" in line: - errors.append(line.strip()) - return errors - - -def unexpected_errors(errors: list[str]) -> list[str]: - """Return console errors that are not on the cosmetic-upstream allow-list.""" - return [ - error - for error in errors - if not any(pattern.search(error) for pattern in _ALLOWED_ERROR_PATTERNS) - ] - - def main() -> int: parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--dashboard-url", default="http://127.0.0.1:8765/") parser.add_argument("--settle-seconds", type=float, default=5.0) + parser.add_argument("--timeout", type=float, default=45.0) + parser.add_argument( + "--json", action="store_true", help="Emit the structured browser proof." + ) parser.add_argument( "--require-browser", action="store_true", @@ -114,18 +78,24 @@ def main() -> int: print(message) return 0 - errors = _console_errors(args.dashboard_url, args.settle_seconds) - unexpected = unexpected_errors(errors) - if unexpected: - print("dashboard console gate FAILED — unexpected console errors:") - for error in unexpected: - print(f" - {error}") + try: + summary = validate_dashboard_browser( + args.dashboard_url, + timeout=args.timeout, + settle_seconds=args.settle_seconds, + ) + except BrowserDisplayError as exc: + if args.json: + print(json.dumps({"ok": False, "checked": False, "error": str(exc)})) + else: + print(f"dashboard console gate FAILED: {exc}") return 1 - allowed = len(errors) - len(unexpected) - print( - f"dashboard console gate OK (no unexpected errors; {allowed} known " - "upstream Inception probe message(s) allow-listed)" - ) + if args.json: + print(json.dumps(summary, sort_keys=True)) + else: + print( + "dashboard console gate OK (official project, subject and quiz views ready)" + ) return 0 diff --git a/scripts/validate_rendered_freshness.py b/scripts/validate_rendered_freshness.py index 32e89c4..90afe60 100755 --- a/scripts/validate_rendered_freshness.py +++ b/scripts/validate_rendered_freshness.py @@ -5,8 +5,8 @@ Count-bearing prose is substituted from ``output/data/manuscript_variables.json`` at render time, so if the curriculum is regenerated but the manuscript is not re-rendered, the rendered ``.md`` (and the PDF) silently keep stale numbers while every other gate -stays green. This gate re-renders each source file from the *current* variables and -asserts the committed rendered copy is byte-identical, catching exactly that drift. +stays green. This gate re-renders each source file from the current variables, +then validates finalized PDF bytes and the complete bound render-input manifest. """ from __future__ import annotations @@ -24,6 +24,8 @@ import z_generate_manuscript_variables as zg +from src.manuscript.pdf_receipt import pdf_receipt_issues + ROADMAP_OPEN_RE = re.compile( r"(?P\d+)\s+open items across\s+(?P[^.]+)", re.IGNORECASE, @@ -174,6 +176,12 @@ def main() -> int: for item in stale: print(f" - {item}") return 1 + pdf_stale = pdf_receipt_issues(ROOT) + if pdf_stale: + print("PDF freshness FAILED (a genuine re-render is required):") + for item in pdf_stale: + print(f" - {item}") + return 1 bundle_stale = check_publication_bundle_roadmap_freshness() if bundle_stale: print("publication bundle freshness FAILED (re-render before publishing):") @@ -183,7 +191,7 @@ def main() -> int: rendered = sorted(zg.OUTPUT_MANUSCRIPT.glob("*.md")) print( f"rendered manuscript freshness OK ({len(rendered)} files match current " - "artifacts; publication roadmap text matches)" + "artifacts; PDF bytes/inputs and publication roadmap text match)" ) return 0 diff --git a/scripts/z_generate_manuscript_variables.py b/scripts/z_generate_manuscript_variables.py index 55411d2..9031c4b 100755 --- a/scripts/z_generate_manuscript_variables.py +++ b/scripts/z_generate_manuscript_variables.py @@ -40,7 +40,9 @@ def replace_claim(match: re.Match[str]) -> str: def replace_variable(match: re.Match[str]) -> str: key = match.group(1) - return variables.get(key, match.group(0)) + if key not in variables: + raise KeyError(f"unresolved manuscript variable: {key}") + return variables[key] return VARIABLE_TOKEN_RE.sub( replace_variable, CLAIM_TOKEN_RE.sub(replace_claim, text) @@ -52,15 +54,13 @@ def _render_markdown(text: str, variables: dict[str, str]) -> str: cursor = 0 for match in CODE_SPAN_RE.finditer(text): parts.append(_render_prose(text[cursor : match.start()], variables)) - parts.append(match.group(0)) + # Only code examples lose placeholder syntax; prose must resolve above. + code = CLAIM_TOKEN_RE.sub(lambda item: f"claim:{item.group(1)}", match.group(0)) + parts.append(VARIABLE_TOKEN_RE.sub(lambda item: item.group(1), code)) cursor = match.end() parts.append(_render_prose(text[cursor:], variables)) rendered = "".join(parts) - # Code-span examples explain token syntax. Keep them readable without - # leaving literal double-brace placeholders in rendered public surfaces. - rendered = CLAIM_TOKEN_RE.sub(lambda m: f"claim:{m.group(1)}", rendered) - rendered = VARIABLE_TOKEN_RE.sub(lambda m: m.group(1), rendered) leftovers = sorted(set(ANY_TOKEN_RE.findall(rendered))) if leftovers: raise ValueError(f"unresolved manuscript tokens: {', '.join(leftovers)}") diff --git a/src/artifacts/json_io.py b/src/artifacts/json_io.py index 0f31039..f664ad7 100644 --- a/src/artifacts/json_io.py +++ b/src/artifacts/json_io.py @@ -1,6 +1,9 @@ from __future__ import annotations import json +import os +import stat +import tempfile from pathlib import Path from typing import Any, cast @@ -30,17 +33,29 @@ def write_json_artifact( ensure_ascii: bool = True, ) -> Path: path = Path(path) - path.parent.mkdir(parents=True, exist_ok=True) - path.write_text( - json.dumps( - payload, - indent=2, - sort_keys=True, - ensure_ascii=ensure_ascii, - ) - + "\n", - encoding="utf-8", + content = ( + json.dumps(payload, indent=2, sort_keys=True, ensure_ascii=ensure_ascii) + "\n" ) + path.parent.mkdir(parents=True, exist_ok=True) + mode = stat.S_IMODE(path.stat().st_mode) if path.exists() else None + temporary: Path | None = None + try: + with tempfile.NamedTemporaryFile( + mode="w", + encoding="utf-8", + dir=path.parent, + prefix=f".{path.name}.", + suffix=".tmp", + delete=False, + ) as stream: + temporary = Path(stream.name) + stream.write(content) + if mode is not None: + temporary.chmod(mode) + os.replace(temporary, path) + finally: + if temporary is not None: + temporary.unlink(missing_ok=True) return path diff --git a/src/curriculum/_ext_dynamical_geometry.py b/src/curriculum/_ext_dynamical_geometry.py index f797b95..b996c54 100644 --- a/src/curriculum/_ext_dynamical_geometry.py +++ b/src/curriculum/_ext_dynamical_geometry.py @@ -178,12 +178,13 @@ "Measure local distinguishability of distributions with the Fisher metric.", "The Fisher information matrix " "$g_{ij}(\\theta)=\\mathbb{E}_{p_\\theta}[\\partial_i\\log p_\\theta\\,\\partial_j\\log p_\\theta]$ " - "is the unique (up to scale) Riemannian metric on a statistical manifold invariant under " - "reparameterisation. It tells you how far apart two nearby distributions are in information " - "terms.", + "defines a Riemannian metric for a regular model with nonsingular Fisher information. " + "Its line element is invariant under smooth invertible reparameterisation, but this " + "coordinate invariance alone does not establish uniqueness. The metric measures " + "the local distinguishability of nearby distributions.", ("geo_statistical_manifold",), _q( - "The Fisher information matrix plays the role of what on a statistical manifold?", + "For a regular model with nonsingular Fisher information, the Fisher matrix plays which role?", ["A Riemannian metric", "A prior", "A loss function"], 0, ), diff --git a/src/curriculum/_ext_learning_hierarchical.py b/src/curriculum/_ext_learning_hierarchical.py index 7cbd27d..9205d58 100644 --- a/src/curriculum/_ext_learning_hierarchical.py +++ b/src/curriculum/_ext_learning_hierarchical.py @@ -387,7 +387,7 @@ "Prediction-Error Message Passing Across Levels", _AI, "Propagate precision-weighted prediction errors upward and predictions downward to perform inference over the whole hierarchy.", - "Hierarchical inference is implemented by message passing: each level sends top-down predictions to the level below and returns bottom-up prediction errors to the level above. Under a Gaussian (Laplace) approximation these messages are precision-weighted prediction errors $\\xi^{(\\ell)} = \\Pi^{(\\ell)}\\big(\\mu^{(\\ell)} - g(\\mu^{(\\ell+1)})\\big)$, and belief updating ascends the free-energy gradient until errors are minimised across all levels simultaneously. Only the residual that a level cannot explain is forwarded upward, so well-predicted signals are 'explained away' and do not propagate. This reciprocal exchange is the computational core shared by predictive coding and discrete hierarchical active inference.", + "Hierarchical inference is implemented by message passing: each level sends top-down predictions to the level below and returns bottom-up prediction errors to the level above. Under a Gaussian (Laplace) approximation these messages are precision-weighted prediction errors $\\xi^{(\\ell)} = \\Pi^{(\\ell)}\\big(\\mu^{(\\ell)} - g(\\mu^{(\\ell+1)})\\big)$, and belief updating follows negative free-energy gradients to reduce the joint variational objective. Only the residual that a level cannot explain is forwarded upward, so well-predicted signals are 'explained away' and do not propagate. This reciprocal exchange is the computational core shared by predictive coding and discrete hierarchical active inference.", ("hier_empirical_priors", "vm_message_passing", "ai_predictive_coding"), _q( "What is forwarded UP the hierarchy during inference?", diff --git a/src/curriculum/_ext_precision_exploration.py b/src/curriculum/_ext_precision_exploration.py index 57022a9..7bb93bc 100644 --- a/src/curriculum/_ext_precision_exploration.py +++ b/src/curriculum/_ext_precision_exploration.py @@ -112,7 +112,7 @@ "Guidance Functions and Risk-Sensitive Policies", _AI, "Distinguish risk (divergence from preferences) from ambiguity in policy scoring.", - "The guidance view decomposes the pragmatic part of expected free energy into a risk " + "The guidance view decomposes total expected free energy into a risk " "term, the divergence between predicted and preferred outcomes, and an ambiguity term, " "the expected uncertainty of the likelihood. Shaping the preference distribution C is " "therefore how a designer steers behaviour: tightening C makes the policy risk-averse " @@ -146,9 +146,13 @@ "Rate-distortion theory asks for the least information an agent must keep to hold " "expected distortion below a bound; expected free energy makes the mirror trade-off, " "balancing the epistemic value of an observation against the pragmatic value of a " - "preferred outcome. Both say a bounded agent should pay for information only to the " - "extent it buys reduced cost, which is why precision behaves like the slope that " - "sets the exchange rate between bits gathered and value pursued.", + "preferred outcome. A rate-distortion multiplier can set a model's relative trade-off. " + "Policy precision is different: in $q(\\pi)=\\sigma(-\\gamma G(\\pi))$, $\\gamma$ " + "scales total expected free energy and controls policy-selection sharpness for fixed " + "scores, not the relative coefficients of epistemic and pragmatic components. " + "Under the kernel's signed-value convention, pragmatic cost is the negative of " + "pragmatic value. This policy-precision statement does not identify every model or " + "sensory precision with the same role.", ("ai_expected_free_energy", "im_rate_distortion", "ai_epistemic_value"), _q( "Expected free energy and rate-distortion theory share which structure?", @@ -160,11 +164,11 @@ 0, ), _q( - "In this reading, precision plays the role of:", + "For fixed EFE scores, increasing policy precision $\\gamma$ does what?", [ - "The slope setting the exchange rate between information and value", - "A fixed constant unrelated to the trade-off", - "The number of policies considered", + "Concentrates probability on policies with lower total expected free energy", + "Changes only the epistemic coefficient relative to the pragmatic coefficient", + "Leaves relative policy probabilities unchanged", ], 0, ), diff --git a/src/curriculum/catalog_data/core_strata.py b/src/curriculum/catalog_data/core_strata.py index c5c6d64..bcff3e3 100644 --- a/src/curriculum/catalog_data/core_strata.py +++ b/src/curriculum/catalog_data/core_strata.py @@ -84,8 +84,9 @@ "Eigenvalues & Positive-Definiteness", _M, "Interpret eigenvalues and positive-definite matrices.", - "For $A\\mathbf{v}=\\lambda\\mathbf{v}$, $\\lambda$ is an eigenvalue; covariance and " - "precision matrices are symmetric positive-definite.", + "For $A\\mathbf{v}=\\lambda\\mathbf{v}$, $\\lambda$ is an eigenvalue. Covariance matrices " + "are symmetric positive-semidefinite; a nonsingular covariance matrix is " + "positive-definite, and its inverse precision matrix is also symmetric positive-definite.", ("m_matrix",), _q( "A covariance matrix is always:", @@ -147,7 +148,7 @@ "partials and certifies local curvature/convexity.", ("m_chain_rule", "m_eigen"), _q( - "A function is locally convex where its Hessian is:", + "For a twice continuously differentiable function, which Hessian condition is sufficient for local strict convexity?", ["Positive-definite", "Negative-definite", "Singular"], 0, ), @@ -157,8 +158,12 @@ "Convexity", _M, "Recognise convex functions and sets.", - "A convex function lies below its chords: $f(\\lambda x+(1-\\lambda)y)\\le \\lambda f(x)+(1-\\lambda)f(y)$. " - "Convexity guarantees a unique minimum — and powers Jensen's inequality later.", + "A convex function lies below its chords for $0\\le\\lambda\\le1$: " + "$f(\\lambda x+(1-\\lambda)y)\\le \\lambda f(x)+(1-\\lambda)f(y)$. " + "Every local minimum is global, but convexity alone guarantees neither attainment " + "nor uniqueness. Strict convexity gives at most one minimizer; existence requires " + "additional conditions, such as continuity on a nonempty compact feasible set. " + "Convexity also powers Jensen's inequality.", ("m_jacobian_hessian",), _q( "A convex function's graph lies:", diff --git a/src/curriculum/graph.py b/src/curriculum/graph.py index 65bf895..957320c 100644 --- a/src/curriculum/graph.py +++ b/src/curriculum/graph.py @@ -13,6 +13,7 @@ from __future__ import annotations from dataclasses import dataclass +from heapq import heapify, heappop, heappush from .model import ALL_STRATA, CurriculumError, Skill, Stratum, Subject @@ -147,9 +148,10 @@ def _check_subject_identity(self) -> None: "SkillTree resolves skills and subjects in one id namespace" ) - def _check_prerequisites_exist(self) -> None: - ids = self.skill_ids() - for skill in self.skills: + def _check_prerequisites_exist(self, skills: list[Skill] | None = None) -> None: + skills = self.skills if skills is None else skills + ids = {skill.skill_id for skill in skills} + for skill in skills: for prereq in skill.prerequisites: if prereq not in ids: raise CurriculumError( @@ -211,25 +213,26 @@ def _check_acyclic(self) -> None: # -- ordering / depth --------------------------------------------------- def topological_order(self) -> list[Skill]: - """Kahn's algorithm — every skill appears after all its prerequisites.""" - self._check_prerequisites_exist() - by_id = {s.skill_id: s for s in self.skills} + """Kahn's algorithm, choosing the smallest currently ready skill id.""" + skills = self.skills + self._check_prerequisites_exist(skills) + by_id = {skill.skill_id: skill for skill in skills} indegree = {sid: 0 for sid in by_id} dependents: dict[str, list[str]] = {sid: [] for sid in by_id} - for prereq, dependent in self.prerequisite_edges(): - indegree[dependent] += 1 - dependents[prereq].append(dependent) - # Deterministic: process ready nodes in sorted id order. - ready = sorted(sid for sid, d in indegree.items() if d == 0) + for skill in skills: + for prereq in skill.prerequisites: + indegree[skill.skill_id] += 1 + dependents[prereq].append(skill.skill_id) + ready = [sid for sid, degree in indegree.items() if degree == 0] + heapify(ready) order: list[Skill] = [] while ready: - sid = ready.pop(0) + sid = heappop(ready) order.append(by_id[sid]) for dep in dependents[sid]: indegree[dep] -= 1 if indegree[dep] == 0: - ready.append(dep) - ready.sort() + heappush(ready, dep) if len(order) != len(by_id): raise CurriculumError("topological_order failed — graph is cyclic") return order @@ -256,15 +259,16 @@ def core_max_depth(self) -> int: implementation and cognitive-security families — giving the depth of the actual concept chain a learner must climb, separate from review sequencing. """ + ordered = self.topological_order() core = { skill.skill_id - for skill in self.skills + for skill in ordered if skill.skill_id.split("_")[0] not in _SCAFFOLD_SKILL_PREFIXES } if not core: return 0 level: dict[str, int] = {} - for skill in self.topological_order(): + for skill in ordered: if skill.skill_id not in core: continue core_prereqs = [p for p in skill.prerequisites if p in core] @@ -284,12 +288,16 @@ def reachable_from_foundations(self) -> set[str]: correctly reported as disconnected, rather than being trivially reachable from that illegal root. """ - by_id = {s.skill_id: s for s in self.skills} + skills = self.skills + by_id = {skill.skill_id: skill for skill in skills} forward: dict[str, list[str]] = {sid: [] for sid in by_id} - for prereq, dependent in self.prerequisite_edges(): - forward[prereq].append(dependent) + for skill in skills: + for prereq in skill.prerequisites: + forward[prereq].append(skill.skill_id) roots = [ - s.skill_id for s in self.skills if s.is_root and s.stratum.is_foundational + skill.skill_id + for skill in skills + if skill.is_root and skill.stratum.is_foundational ] seen: set[str] = set() stack = list(roots) @@ -330,19 +338,41 @@ def _check_all_skills_reachable_from_foundations(self) -> None: # -- reporting ---------------------------------------------------------- def stats(self) -> dict[str, object]: - levels = self.levels() + ordered = self.topological_order() + levels: dict[str, int] = {} + core_levels: dict[str, int] = {} + core_ids = { + skill.skill_id + for skill in ordered + if skill.skill_id.split("_")[0] not in _SCAFFOLD_SKILL_PREFIXES + } + skills_per_stratum = {stratum.value: 0 for stratum in ALL_STRATA} + for skill in ordered: + levels[skill.skill_id] = ( + 1 + max(levels[prereq] for prereq in skill.prerequisites) + if skill.prerequisites + else 0 + ) + skills_per_stratum[skill.stratum.value] += 1 + if skill.skill_id in core_ids: + core_prereqs = [ + prereq for prereq in skill.prerequisites if prereq in core_ids + ] + core_levels[skill.skill_id] = ( + 1 + max(core_levels[prereq] for prereq in core_prereqs) + if core_prereqs + else 0 + ) + strata = {subject.stratum for subject in self.subjects} return { "subjects": len(self.subjects), - "skills": len(self.skills), - "prerequisite_edges": len(self.prerequisite_edges()), - "strata_present": sorted(s.value for s in self.strata_present()), - "strata_count": len(self.strata_present()), - "all_strata_present": self.strata_present() == set(ALL_STRATA), - "max_depth": max(levels.values()) if levels else 0, - "core_max_depth": self.core_max_depth(), - "roots": sum(1 for s in self.skills if s.is_root), - "skills_per_stratum": { - stratum.value: sum(1 for s in self.skills if s.stratum is stratum) - for stratum in ALL_STRATA - }, + "skills": len(ordered), + "prerequisite_edges": sum(len(skill.prerequisites) for skill in ordered), + "strata_present": sorted(stratum.value for stratum in strata), + "strata_count": len(strata), + "all_strata_present": strata == set(ALL_STRATA), + "max_depth": max(levels.values(), default=0), + "core_max_depth": max(core_levels.values(), default=0), + "roots": sum(skill.is_root for skill in ordered), + "skills_per_stratum": skills_per_stratum, } diff --git a/src/curriculum/materials/builders.py b/src/curriculum/materials/builders.py index 3b16b8f..7ed0542 100644 --- a/src/curriculum/materials/builders.py +++ b/src/curriculum/materials/builders.py @@ -40,6 +40,7 @@ def build_course_material_index( registry: ClaimRegistry | None = None, ) -> dict[str, object]: """Build one modular material record per skill in the curriculum graph.""" + registry = registry if registry is not None else ClaimRegistry() source_index = { str(source["source_id"]): source for source in source_payload.get("sources", []) @@ -53,14 +54,19 @@ def build_course_material_index( for skill in graph.topological_order(): subject = subject_by_skill[skill.skill_id] - pointers = _pointers_for_skill(skill, source_payload, source_index) + pointers = _pointers_for_skill(skill, source_payload, source_index, registry) prerequisites = [ - _skill_ref(skill_by_id[prereq], subject_by_skill[prereq], levels[prereq]) + _skill_ref( + skill_by_id[prereq], subject_by_skill[prereq], levels[prereq], registry + ) for prereq in skill.prerequisites ] dependents = [ _skill_ref( - skill_by_id[dependent], subject_by_skill[dependent], levels[dependent] + skill_by_id[dependent], + subject_by_skill[dependent], + levels[dependent], + registry, ) for dependent in downstream.get(skill.skill_id, ()) ] @@ -72,9 +78,9 @@ def build_course_material_index( ) skills[skill.skill_id] = { "skillId": skill.skill_id, - "name": skill.name, + "name": _render_text(skill.name, registry), "subjectId": subject.subject_id, - "subjectName": subject.name, + "subjectName": _render_text(subject.name, registry), "stratum": skill.stratum.value, "depth": levels[skill.skill_id], "objective": _render_text(skill.objective, registry), @@ -87,7 +93,11 @@ def build_course_material_index( ), "prerequisites": prerequisites, "relatedSkills": _related_skills( - skill, skill_by_id, subject_by_skill, levels + skill, + skill_by_id, + subject_by_skill, + levels, + registry, ), "downstreamSkills": dependents, "assessmentSummary": _assessment_summary(skill), @@ -110,7 +120,7 @@ def build_course_material_index( "externalMaterialPolicy": source_payload.get("policy", {}), }, "summary": {}, - "sources": _source_summaries(source_index), + "sources": _source_summaries(source_index, registry), "skills": skills, } normalized = normalize_course_material_index(payload) diff --git a/src/curriculum/materials/github_sources.py b/src/curriculum/materials/github_sources.py index 54fa8be..3fad5fc 100644 --- a/src/curriculum/materials/github_sources.py +++ b/src/curriculum/materials/github_sources.py @@ -11,39 +11,10 @@ from src.net_guard import require_http_url +from .helpers import _iter_configured_pointers, _required_string from .models import CourseMaterialIssue -def _required_string(mapping: dict[str, Any], key: str) -> str: - value = mapping.get(key) - return value if isinstance(value, str) else "" - - -def _as_pointer_list(value: Any) -> list[dict[str, Any]]: - if not isinstance(value, list): - return [] - return [item for item in value if isinstance(item, dict)] - - -def _iter_configured_pointers( - payload: dict[str, Any], -) -> list[tuple[str, dict[str, Any]]]: - pointers: list[tuple[str, dict[str, Any]]] = [] - stratum_mappings = payload.get("stratum_mappings") - if isinstance(stratum_mappings, dict): - for stratum_key, raw in stratum_mappings.items(): - for pointer in _as_pointer_list(raw): - pointers.append((f"stratum_mappings.{stratum_key}", pointer)) - topic_mappings = payload.get("topic_mappings") - if isinstance(topic_mappings, list): - for topic_idx, topic in enumerate(topic_mappings): - if not isinstance(topic, dict): - continue - for pointer in _as_pointer_list(topic.get("pointers")): - pointers.append((f"topic_mappings[{topic_idx}].pointers", pointer)) - return pointers - - def _validate_online_trees( source_index: dict[str, dict[str, Any]], payload: dict[str, Any], @@ -60,7 +31,7 @@ def _validate_online_trees( continue try: tree_paths = _fetch_github_tree_paths(parts[0], parts[1], commit, timeout) - except (OSError, urllib.error.URLError, ValueError) as exc: + except (OSError, urllib.error.URLError, ValueError, TypeError) as exc: issues.append( CourseMaterialIssue( "github-tree", diff --git a/src/curriculum/materials/helpers.py b/src/curriculum/materials/helpers.py index 8d52230..d8db8f4 100644 --- a/src/curriculum/materials/helpers.py +++ b/src/curriculum/materials/helpers.py @@ -209,6 +209,7 @@ def _pointers_for_skill( skill: Skill, payload: dict[str, Any], source_index: dict[str, dict[str, Any]], + registry: ClaimRegistry, ) -> list[dict[str, object]]: raw_pointers: list[tuple[dict[str, Any], str, tuple[str, ...]]] = [] stratum_mappings = payload.get("stratum_mappings", {}) @@ -254,13 +255,17 @@ def _pointers_for_skill( repo_url = _required_string(source, "repo_url").rstrip("/") merged[key] = { "sourceId": source_id, - "sourceTitle": _required_string(source, "title"), - "title": _required_string(pointer, "title"), - "role": role, + "sourceTitle": _render_text( + _required_string(source, "title"), registry + ), + "title": _render_text(_required_string(pointer, "title"), registry), + "role": _render_text(role, registry), "path": path, "commit": commit, "url": f"{repo_url}/blob/{commit}/{path}", - "licenseStatus": _required_string(source, "license_status"), + "licenseStatus": _render_text( + _required_string(source, "license_status"), registry + ), } scope_sets[key].add(match_scope) term_sets[key].update(matched_terms) @@ -277,7 +282,7 @@ def _sections_for_skill( prerequisites: list[dict[str, object]], dependents: list[dict[str, object]], pointers: list[dict[str, object]], - registry: ClaimRegistry | None, + registry: ClaimRegistry, ) -> list[dict[str, str]]: prereq_body = ( "This root skill can be entered without a prior local prerequisite." @@ -308,11 +313,11 @@ def _sections_for_skill( }, { "heading": "Prerequisite Context", - "body": prereq_body, + "body": _render_text(prereq_body, registry), }, { "heading": "Downstream Role", - "body": downstream_body, + "body": _render_text(downstream_body, registry), }, { "heading": "Assessment Contract", @@ -366,6 +371,7 @@ def _related_skills( skill_by_id: dict[str, Skill], subject_by_skill: dict[str, Subject], levels: dict[str, int], + registry: ClaimRegistry, ) -> list[dict[str, object]]: related_ids = list(skill.related) for prereq in skill.prerequisites: @@ -381,17 +387,23 @@ def _related_skills( related_skill, subject_by_skill[related_skill.skill_id], levels[related_skill.skill_id], + registry, ) ) return rows -def _skill_ref(skill: Skill, subject: Subject, depth: int) -> dict[str, object]: +def _skill_ref( + skill: Skill, + subject: Subject, + depth: int, + registry: ClaimRegistry, +) -> dict[str, object]: return { "skillId": skill.skill_id, - "name": skill.name, + "name": _render_text(skill.name, registry), "subjectId": subject.subject_id, - "subjectName": subject.name, + "subjectName": _render_text(subject.name, registry), "stratum": skill.stratum.value, "depth": depth, } @@ -410,18 +422,25 @@ def _downstream_map(graph: CurriculumGraph) -> dict[str, tuple[str, ...]]: } -def _source_summaries(source_index: dict[str, dict[str, Any]]) -> list[dict[str, str]]: +def _source_summaries( + source_index: dict[str, dict[str, Any]], + registry: ClaimRegistry, +) -> list[dict[str, str]]: rows: list[dict[str, str]] = [] for source_id in sorted(source_index): source = source_index[source_id] rows.append( { "sourceId": source_id, - "title": _required_string(source, "title"), + "title": _render_text(_required_string(source, "title"), registry), "repoUrl": _required_string(source, "repo_url"), "commit": _required_string(source, "commit"), - "licenseStatus": _required_string(source, "license_status"), - "statusNote": _required_string(source, "status_note"), + "licenseStatus": _render_text( + _required_string(source, "license_status"), registry + ), + "statusNote": _render_text( + _required_string(source, "status_note"), registry + ), } ) return rows @@ -502,5 +521,5 @@ def _write_json(path: Path, payload: object) -> None: write_json_artifact(path, payload) -def _render_text(text: str, registry: ClaimRegistry | None) -> str: - return render(text, registry) if registry is not None else text +def _render_text(text: str, registry: ClaimRegistry) -> str: + return render(text, registry) diff --git a/src/curriculum/materials/validation.py b/src/curriculum/materials/validation.py index 3398efb..8aebbd7 100644 --- a/src/curriculum/materials/validation.py +++ b/src/curriculum/materials/validation.py @@ -8,6 +8,8 @@ from src.curriculum.graph import CurriculumGraph from src.curriculum.model import ALL_STRATA from src.curriculum.source_contracts import GITHUB_COMMIT_RE +from src.provenance.binding import ProvenanceError, render +from src.provenance.claims import ClaimRegistry from .github_sources import _github_repo_parts, _validate_online_trees from .helpers import ( @@ -217,6 +219,27 @@ def audit_course_material_index( ) -> CourseMaterialAudit: """Check that every skill has non-empty material and external pointers.""" issues: list[CourseMaterialIssue] = [] + registry = ClaimRegistry() + pending: list[tuple[str, object]] = [("materials", index)] + while pending: + location, value = pending.pop() + if isinstance(value, str): + try: + render(value, registry) + except ProvenanceError: + issues.append( + CourseMaterialIssue( + "unresolved-claim-token", + location, + "learner-facing material contains an unresolved or malformed claim token", + ) + ) + elif isinstance(value, dict): + pending.extend((f"{location}.{key}", child) for key, child in value.items()) + elif isinstance(value, (list, tuple)): + pending.extend( + (f"{location}[{idx}]", child) for idx, child in enumerate(value) + ) normalized = normalize_course_material_index(index) if not normalized.skills_is_mapping: issues.append( @@ -282,14 +305,6 @@ def audit_course_material_index( ) ) continue - if "{{claim:" in str(section.get("body", "")): - issues.append( - CourseMaterialIssue( - "unresolved-claim-token", - f"skills.{skill_id}.sections[{section_idx}]", - "section body contains an unresolved claim token", - ) - ) if not isinstance(pointers, list) or not pointers: issues.append( CourseMaterialIssue( diff --git a/src/curriculum/model.py b/src/curriculum/model.py index f3615c0..11cc9d5 100644 --- a/src/curriculum/model.py +++ b/src/curriculum/model.py @@ -15,6 +15,7 @@ from __future__ import annotations import re +import unicodedata from dataclasses import dataclass, field from enum import Enum from typing import Literal @@ -22,6 +23,18 @@ # ``{{claim:NAME}}`` token used in skill descriptions for provenance-bound values. CLAIM_TOKEN_RE = re.compile(r"\{\{claim:([A-Za-z0-9_.\-]+)\}\}") + +def _strip_format_chars(text: str) -> str: + """Normalize invisible Unicode format characters before claim admission/collection. + + Format characters can split an otherwise identical token or result number. + Rendering, validation, linting and original-token provenance share this policy. + """ + return "".join( + character for character in text if unicodedata.category(character) != "Cf" + ) + + SelfReportingType = Literal["HonorSystem", "Approval", "Quiz"] _VALID_SELF_REPORTING = ("HonorSystem", "Approval", "Quiz") QuestionType = Literal["SingleChoice", "MultipleChoice", "TextInput"] @@ -148,7 +161,7 @@ def claim_tokens(self) -> set[str]: texts.extend(question.answers) tokens: set[str] = set() for text in texts: - tokens.update(CLAIM_TOKEN_RE.findall(text)) + tokens.update(CLAIM_TOKEN_RE.findall(_strip_format_chars(text))) return tokens @property diff --git a/src/curriculum/quality.py b/src/curriculum/quality.py index 5dfee98..8250dbc 100644 --- a/src/curriculum/quality.py +++ b/src/curriculum/quality.py @@ -72,7 +72,8 @@ def _word_count(text: str) -> int: def audit_curriculum_quality(graph: CurriculumGraph) -> list[CurriculumQualityIssue]: """Return learner-quality issues that structural validation does not cover.""" issues: list[CurriculumQualityIssue] = [] - by_id = {skill.skill_id: skill for skill in graph.skills} + skills = graph.skills + by_id = {skill.skill_id: skill for skill in skills} for subject in graph.subjects: if _word_count(subject.description) < MIN_SUBJECT_DESCRIPTION_WORDS: @@ -84,7 +85,7 @@ def audit_curriculum_quality(graph: CurriculumGraph) -> list[CurriculumQualityIs ) ) - for skill in graph.skills: + for skill in skills: if len(skill.assessments) < MIN_ASSESSMENTS_PER_SKILL: issues.append( CurriculumQualityIssue( @@ -334,24 +335,13 @@ def _matched_placeholder_distractor(answer: str) -> str | None: def transitive_prerequisites(graph: CurriculumGraph) -> dict[str, set[str]]: - """Return each skill id -> the set of all skills reachable through prerequisites.""" - direct = {s.skill_id: set(s.prerequisites) for s in graph.skills} + """Return transitive prerequisite sets without recursion on a validated DAG.""" closure: dict[str, set[str]] = {} - - def resolve(sid: str, seen: frozenset[str]) -> set[str]: - if sid in closure: - return closure[sid] - acc: set[str] = set() - for prereq in direct.get(sid, set()): - if prereq in seen: # guard (graph is a validated DAG, but be defensive) - continue - acc.add(prereq) - acc |= resolve(prereq, seen | {prereq}) - closure[sid] = acc - return acc - - for skill in graph.skills: - resolve(skill.skill_id, frozenset({skill.skill_id})) + for skill in graph.topological_order(): + ancestors = set(skill.prerequisites) + for prereq in skill.prerequisites: + ancestors.update(closure[prereq]) + closure[skill.skill_id] = ancestors return closure @@ -360,7 +350,8 @@ def semantic_prerequisite_coverage( ) -> tuple[SemanticPrerequisiteCoverage, ...]: """Return concept-level reachability counts used by the semantic coverage figure.""" closure = transitive_prerequisites(graph) - by_id = {s.skill_id: s for s in graph.skills} + skills = graph.skills + by_id = {skill.skill_id: skill for skill in skills} rows: list[SemanticPrerequisiteCoverage] = [] for concept, patterns, definers in _CONCEPT_PREREQUISITES: compiled = [re.compile(pattern, re.IGNORECASE) for pattern in patterns] @@ -370,7 +361,7 @@ def semantic_prerequisite_coverage( if any(definer in prereqs for definer in definers) ) assessed = 0 - for skill in graph.skills: + for skill in skills: if skill.skill_id in definers: continue text = " ".join( @@ -417,8 +408,9 @@ def audit_semantic_prerequisites( for concept, pats, definers in _CONCEPT_PREREQUISITES ] closure = transitive_prerequisites(graph) - by_id = {s.skill_id: s for s in graph.skills} - for skill in graph.skills: + skills = graph.skills + by_id = {skill.skill_id: skill for skill in skills} + for skill in skills: text = " ".join( tuple(q.prompt for q in skill.assessments) + tuple(a for q in skill.assessments for a in q.answers) diff --git a/src/curriculum/recommendations.py b/src/curriculum/recommendations.py index d3f5d9b..5e903c4 100644 --- a/src/curriculum/recommendations.py +++ b/src/curriculum/recommendations.py @@ -14,6 +14,8 @@ from collections.abc import Iterable, Mapping from typing import Any +from src.skilltree.summary import SkillTreeSummaryError, validate_export_payload + _TOKEN_RE = re.compile(r"[A-Za-z][A-Za-z0-9_-]{3,}") _STOP_WORDS = frozenset( { @@ -164,6 +166,10 @@ def _index_project( dict[str, set[str]], dict[str, tuple[Mapping[str, Any], ...]], ]: + try: + validate_export_payload(project) + except SkillTreeSummaryError: + return {}, {}, {}, {} raw_subjects = project.get("subjects") raw_dependencies = project.get("dependencies") raw_quizzes = project.get("quizzes") @@ -256,9 +262,7 @@ def _depths( if indegree[dependent] == 0: queue.append(dependent) if visited != len(skill_ids): - # The export validator normally prevents this. Keeping the recommendation - # helper fail-closed avoids inventing a path if it is called on raw data. - return {skill_id: 0 for skill_id in skill_ids} + raise SkillTreeSummaryError("SkillTree project dependencies contain a cycle") return depths diff --git a/src/dashboard/artifacts.py b/src/dashboard/artifacts.py index 3cd5631..70e29b8 100644 --- a/src/dashboard/artifacts.py +++ b/src/dashboard/artifacts.py @@ -12,6 +12,7 @@ DashboardArtifact, ) from src.dashboard.figure_metadata import load_figure_metadata as _load_figure_metadata +from src.skilltree.content import export_content_digest from src.skilltree.summary import ( SkillTreeSummaryError, summarize_export_payload, @@ -26,7 +27,7 @@ class DashboardArtifactError(ValueError): # Derived expected artifact count. Roster integrity is guarded by tests for unique ids, # unique paths, and registry/dashboard parity rather than a stale hand-maintained literal. EXPECTED_ARTIFACT_COUNT = len(ARTIFACTS) -_ANSWER_KEY_FIELDS = frozenset({"isCorrect"}) +_ANSWER_KEY_FIELDS = frozenset({"isCorrect", "answers"}) def artifact_inventory(project_root: Path) -> list[dict[str, object]]: @@ -62,7 +63,7 @@ def read_json_file(path: Path) -> tuple[dict[str, Any] | None, str | None]: def redact_skilltree_answer_keys(payload: Any) -> Any: - """Return a browser-safe SkillTree export projection without quiz answer keys.""" + """Remove answer options and correctness keys; official SkillTree owns quizzes.""" if isinstance(payload, dict): return { key: redact_skilltree_answer_keys(value) @@ -92,10 +93,16 @@ def load_redacted_skilltree_export( return redacted, None -def load_skilltree_project(project_root: Path) -> dict[str, Any]: +def load_skilltree_project( + project_root: Path, + *, + export_payload: dict[str, Any] | None = None, +) -> dict[str, Any]: """Load the exported SkillTree project, summarized for dashboard use.""" path = project_root / "output" / "data" / "skilltree_project.json" - payload, error = read_json_file(path) + payload, error = ( + (export_payload, None) if export_payload is not None else read_json_file(path) + ) if payload is None: return { "available": False, @@ -142,6 +149,7 @@ def load_skilltree_project(project_root: Path) -> dict[str, Any]: "id": payload.get("projectId"), "name": payload.get("name"), }, + "exportContentDigest": export_content_digest(payload), "subjects": subjects, "dependencies": dependencies, "quizzes": quizzes, diff --git a/src/dashboard/assets/app.js b/src/dashboard/assets/app.js index 4c2aad9..d619dff 100644 --- a/src/dashboard/assets/app.js +++ b/src/dashboard/assets/app.js @@ -1161,6 +1161,8 @@ function formatBytes(bytes) { function renderOfficialFallback(message, config) { const container = el("skills-client-container"); container.setAttribute("aria-busy", "false"); + container.dataset.mountState = "fallback"; + container.dataset.route = ""; const wrapper = document.createElement("div"); wrapper.className = "embed-state warn"; const title = document.createElement("strong"); @@ -1191,6 +1193,11 @@ async function mountOfficialDisplay(payload) { const config = payload.config; const container = el("skills-client-container"); container.setAttribute("aria-busy", "true"); + container.dataset.mountState = "pending"; + container.dataset.projectId = config.project.id; + container.dataset.route = ""; + container.dataset.instanceId = payload.instanceId || ""; + container.dataset.exportContentDigest = payload.project?.exportContentDigest || ""; if (!payload.skilltree.available) { renderOfficialFallback( "SkillTree is offline from this dashboard session. Start the local service or use the artifact panels.", @@ -1226,6 +1233,22 @@ async function mountOfficialDisplay(payload) { client.SkillsConfiguration.configure(skilltreeConfig); await client.SkillsConfiguration.afterConfigure(); const display = new client.SkillsDisplayJS({ + handleRouteChanged: (route) => { + if (container.dataset.mountState === "fallback") { + return; + } + const nativeLearnerRoute = route === "/" || route === "/rank" + || ["/subjects/", "/quizzes/", "/badges/", "/tags/"].some( + (prefix) => String(route).startsWith(prefix), + ) || route === "/badges" || route === "/tags"; + if (!nativeLearnerRoute) { + renderOfficialFallback("The official client did not reach a learner route.", config); + return; + } + container.dataset.route = route; + container.dataset.mountState = "ready"; + container.setAttribute("aria-busy", "false"); + }, options: { serviceUrl: config.skilltree.url, projectId: config.project.id, @@ -1257,7 +1280,6 @@ async function mountOfficialDisplay(payload) { }, }); container.innerHTML = ""; - container.setAttribute("aria-busy", "false"); display.attachTo(container); } catch (error) { renderOfficialFallback( @@ -1431,6 +1453,7 @@ main().catch((error) => { setText("service-state", "error"); document.body.classList.remove("is-loading"); el("skills-client-container").setAttribute("aria-busy", "false"); + el("skills-client-container").dataset.mountState = "fallback"; el("skills-client-container").innerHTML = `
Dashboard data could not load. diff --git a/src/dashboard/browser_validation.py b/src/dashboard/browser_validation.py new file mode 100644 index 0000000..8fa2eaf --- /dev/null +++ b/src/dashboard/browser_validation.py @@ -0,0 +1,333 @@ +"""Real-browser proof of official SkillTree routes, not merely an empty console.""" + +from __future__ import annotations + +import html +import math +import re +import time +from collections.abc import Mapping +from typing import Any +from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit + +from src.dashboard.live_validation import LiveSkillTreeDashboardError, _fetch_json +from src.skilltree.form_login_core import SkillTreeFormLoginError, parse_axi_result +from src.skilltree.form_login_validation import run_browser_command +from src.skilltree.local_auth_state import LocalSkillTreeAuthError, _skilltree_origin + +_ALLOWED_ERROR_PATTERNS = ( + re.compile(r"projects/Inception/.*token.*401", re.IGNORECASE), + re.compile(r"Unable to authenticate.*Inception", re.IGNORECASE), +) + +DISPLAY_PROBE_JS = r"""(activate = false) => { + const container = document.getElementById('skills-client-container'); + if (!container) return {state: 'pending', url: location.href}; + const panel = container.closest('.panel'); + const tab = panel && [...document.querySelectorAll('.tab')].find(item => item.dataset.panel === panel.id); + if (activate && tab) tab.click(); + const frame = container.querySelector('iframe'); + const bounds = frame && frame.getBoundingClientRect(); + return { + state: container.dataset.mountState || 'pending', + url: location.href, + projectId: container.dataset.projectId || '', + route: container.dataset.route || '', + dashboardInstanceId: container.dataset.instanceId || '', + exportContentDigest: container.dataset.exportContentDigest || '', + frameUrl: frame ? frame.src : '', + frameVisible: !!bounds && bounds.width > 0 && bounds.height > 0, + busy: container.getAttribute('aria-busy') === 'true' + }; +}""" + + +class BrowserDisplayError(RuntimeError): + """The real official frame could not prove the requested learner view.""" + + +def unexpected_errors(errors: list[str]) -> list[str]: + return [ + error + for error in errors + if not any(pattern.search(error) for pattern in _ALLOWED_ERROR_PATTERNS) + ] + + +def browser_output( + arguments: list[str], *, browser_command: str, timeout: float +) -> str: + try: + result = run_browser_command([browser_command, *arguments], timeout=timeout) + except SkillTreeFormLoginError as exc: + raise BrowserDisplayError(str(exc)) from exc + if re.search( + r"(?im)^\s*(?:code:\s*(?:TIMEOUT|ERROR)|isError:\s*true)\s*$", result.stdout + ): + raise BrowserDisplayError("browser command returned an explicit tool failure") + return result.stdout + + +def display_probe( + *, + browser_command: str, + timeout: float, + activate: bool = False, +) -> dict[str, Any]: + script = f"() => ({DISPLAY_PROBE_JS})({'true' if activate else 'false'})" + stdout = browser_output( + ["eval", script], browser_command=browser_command, timeout=timeout + ) + try: + value = parse_axi_result(stdout) + except SkillTreeFormLoginError as exc: + raise BrowserDisplayError(str(exc)) from exc + if not isinstance(value, dict) or value.get("isError") is True: + raise BrowserDisplayError("browser display probe returned no state object") + return value + + +def wait_for_official_display( + *, + browser_command: str, + dashboard_url: str, + skilltree_url: str, + project_id: str, + route: str, + timeout: float = 45.0, + dashboard_instance_id: str | None = None, + export_digest: str | None = None, +) -> dict[str, Any]: + if not math.isfinite(timeout) or timeout <= 0: + raise BrowserDisplayError( + "browser readiness timeout must be finite and positive" + ) + deadline = time.monotonic() + timeout + while True: + remaining = deadline - time.monotonic() + if remaining <= 0: + raise BrowserDisplayError( + f"official SkillTree display did not reach ready route {route!r}" + ) + probe = display_probe( + browser_command=browser_command, + timeout=min(remaining, 30.0), + activate=True, + ) + if _origin(str(probe.get("url", ""))) != _origin(dashboard_url): + raise BrowserDisplayError( + "browser navigated away from the requested dashboard origin" + ) + if probe.get("state") == "fallback": + raise BrowserDisplayError( + "official SkillTree display is in fallback, not mounted" + ) + if probe.get("state") == "ready" and probe.get("route") == route: + if ( + probe.get("projectId") != project_id + or _origin(str(probe.get("frameUrl", ""))) != _origin(skilltree_url) + or ( + dashboard_instance_id is not None + and probe.get("dashboardInstanceId") != dashboard_instance_id + ) + or ( + export_digest is not None + and probe.get("exportContentDigest") != export_digest + ) + ): + raise BrowserDisplayError( + "official frame identity or export association is invalid" + ) + if probe.get("busy") is False and probe.get("frameVisible") is True: + return probe + time.sleep(min(0.25, max(0.0, deadline - time.monotonic()))) + + +def _view_contracts(status: Mapping[str, Any]) -> tuple[str, str, list[dict[str, str]]]: + config = status.get("config", {}) + project = status.get("project", {}) + if ( + not isinstance(config, Mapping) + or not isinstance(project, Mapping) + or not project.get("available") + ): + raise BrowserDisplayError("dashboard has no available exported project") + identity = config.get("project", {}) + if not isinstance(identity, Mapping) or not isinstance(identity.get("id"), str): + raise BrowserDisplayError("dashboard project identity is unavailable") + project_id = identity["id"] + skilltree = config.get("skilltree", {}) + if not isinstance(skilltree, Mapping) or not isinstance(skilltree.get("url"), str): + raise BrowserDisplayError("SkillTree target is unavailable") + quizzes = { + quiz["quizId"]: quiz + for quiz in project.get("quizzes", []) + if isinstance(quiz, Mapping) and isinstance(quiz.get("quizId"), str) + } + for subject in project.get("subjects", []): + if not isinstance(subject, Mapping): + continue + for skill in subject.get("skills", []): + if not isinstance(skill, Mapping) or skill.get("quizId") not in quizzes: + continue + quiz = quizzes[skill["quizId"]] + views = [ + { + "kind": "project", + "route": "/", + "visibleText": str(identity.get("name", "")), + }, + { + "kind": "subject", + "route": f"/subjects/{subject['subjectId']}", + "visibleText": str(subject.get("name", "")), + }, + { + "kind": "quiz", + "route": f"/subjects/{subject['subjectId']}/skills/{skill['skillId']}/quizzes/{quiz['quizId']}", + "visibleText": str(quiz.get("name", "")), + }, + ] + if any(not view["visibleText"].strip() for view in views): + raise BrowserDisplayError("official view contract lacks a display name") + return project_id, skilltree["url"], views + raise BrowserDisplayError( + "export has no quiz-gated skill for the required official quiz proof" + ) + + +def validate_dashboard_browser( + dashboard_url: str, + *, + browser_command: str = "chrome-devtools-axi", + timeout: float = 45.0, + settle_seconds: float = 5.0, +) -> dict[str, Any]: + if not math.isfinite(timeout) or timeout <= 0: + raise BrowserDisplayError("timeout must be finite and positive") + if not math.isfinite(settle_seconds) or settle_seconds < 0: + raise BrowserDisplayError("settle_seconds must be finite and non-negative") + parsed = urlsplit(dashboard_url) + base = urlunsplit((parsed.scheme, parsed.netloc, parsed.path.rstrip("/"), "", "")) + try: + status = _fetch_json(f"{base}/api/status?defer_live=1", min(timeout, 30.0)) + except LiveSkillTreeDashboardError as exc: + raise BrowserDisplayError(str(exc)) from exc + project_id, skilltree_url, views = _view_contracts(status) + instance_id = status.get("instanceId") + digest = status.get("project", {}).get("exportContentDigest") + if ( + not isinstance(instance_id, str) + or not instance_id + or not isinstance(digest, str) + or len(digest) != 64 + ): + raise BrowserDisplayError("dashboard instance/export identity is missing") + allowed_count = 0 + results = [] + for view in views: + query = dict(parse_qsl(parsed.query)) + query["skillsClientDisplayPath"] = view["route"] + url = urlunsplit( + (parsed.scheme, parsed.netloc, parsed.path or "/", urlencode(query), "") + ) + browser_output(["open", url], browser_command=browser_command, timeout=60.0) + probe = wait_for_official_display( + browser_command=browser_command, + dashboard_url=dashboard_url, + skilltree_url=skilltree_url, + project_id=project_id, + route=view["route"], + timeout=timeout, + dashboard_instance_id=instance_id, + export_digest=digest, + ) + if settle_seconds: + time.sleep(settle_seconds) + snapshot = browser_output( + ["snapshot"], browser_command=browser_command, timeout=30.0 + ) + visible = " ".join(html.unescape(snapshot).split()).casefold() + expected = " ".join(html.unescape(view["visibleText"]).split()).casefold() + if expected not in visible: + raise BrowserDisplayError( + f"official {view['kind']} view lacks its expected visible content" + ) + console = browser_output( + ["console", "--type", "error"], + browser_command=browser_command, + timeout=30.0, + ) + errors = [line.strip() for line in console.splitlines() if "[error]" in line] + if unexpected_errors(errors): + raise BrowserDisplayError("dashboard has unexpected browser console errors") + allowed_count += len(errors) + results.append( + { + "kind": view["kind"], + "route": probe["route"], + "state": "ready", + "visibleContentChecked": True, + } + ) + return { + "ok": True, + "checked": True, + "projectId": project_id, + "dashboardInstanceId": instance_id, + "exportContentDigest": digest, + "views": results, + "allowedConsoleErrors": allowed_count, + "evidenceKind": "real official SDK route, visible frame and browser accessibility content", + } + + +def _origin(url: str) -> tuple[str, str, int]: + try: + return _skilltree_origin(url) + except LocalSkillTreeAuthError as exc: + raise BrowserDisplayError("browser/configured origin is invalid") from exc + + +def browser_receipt_complete(payload: object) -> bool: + """Require all actual official view observations; an empty console is insufficient.""" + if ( + not isinstance(payload, Mapping) + or payload.get("ok") is not True + or payload.get("checked") is not True + ): + return False + if ( + not isinstance(payload.get("projectId"), str) + or not payload["projectId"] + or not isinstance(payload.get("dashboardInstanceId"), str) + or not payload["dashboardInstanceId"] + or not isinstance(payload.get("exportContentDigest"), str) + or len(payload["exportContentDigest"]) != 64 + ): + return False + views = payload.get("views") + if not isinstance(views, list) or len(views) != 3: + return False + kinds: set[str] = set() + for view in views: + if not isinstance(view, Mapping): + return False + kind, route = view.get("kind"), view.get("route") + if ( + not isinstance(kind, str) + or kind not in {"project", "subject", "quiz"} + or kind in kinds + or not isinstance(route, str) + or view.get("state") != "ready" + or view.get("visibleContentChecked") is not True + ): + return False + if kind == "project" and route != "/": + return False + if kind == "subject" and not route.startswith("/subjects/"): + return False + if kind == "quiz" and "/quizzes/" not in route: + return False + kinds.add(kind) + return kinds == {"project", "subject", "quiz"} diff --git a/src/dashboard/config.py b/src/dashboard/config.py index cdbdb9e..65243e5 100644 --- a/src/dashboard/config.py +++ b/src/dashboard/config.py @@ -59,6 +59,7 @@ class DashboardConfig: local_auth_enabled: bool = False local_state_dir: Path | None = None learner_id: str = DEFAULT_LOCAL_LEARNER_ID + instance_id: str = "" def __post_init__(self) -> None: if not self.skilltree_url.strip(): @@ -188,6 +189,7 @@ def make_dashboard_config( local_auth_enabled: bool = False, local_state_dir: Path | None = None, learner_id: str = DEFAULT_LOCAL_LEARNER_ID, + instance_id: str = "", ) -> DashboardConfig: """Build the default dashboard config, resolving the project root once.""" root = project_root or resolve_project_root("src") @@ -203,6 +205,7 @@ def make_dashboard_config( local_auth_enabled=local_auth_enabled, local_state_dir=local_state_dir, learner_id=learner_id, + instance_id=instance_id, ) diff --git a/src/dashboard/live_validation.py b/src/dashboard/live_validation.py index c1ef7e3..d73fd70 100644 --- a/src/dashboard/live_validation.py +++ b/src/dashboard/live_validation.py @@ -17,6 +17,7 @@ ) from src.net_guard import NonHttpUrlError, require_http_url from src.project_identity import SKILLTREE_PROJECT_ID, SKILLTREE_PROJECT_NAME +from src.skilltree.content import compare_live_content, export_content_digest from src.skilltree.summary import SkillTreeSummaryError, summarize_export_payload @@ -54,7 +55,7 @@ def _fetch_json(url: str, timeout: float) -> dict[str, Any]: return payload -def _count_exported_project(project_path: Path) -> dict[str, int]: +def _load_exported_project(project_path: Path) -> dict[str, Any]: try: payload = read_json_object(project_path) except FileNotFoundError as exc: @@ -68,11 +69,19 @@ def _count_exported_project(project_path: Path) -> dict[str, int]: except JsonArtifactError as exc: raise LiveSkillTreeDashboardError(str(exc)) from exc try: - return summarize_export_payload(payload).project_counts() + summarize_export_payload(payload) + export_content_digest(payload) + return payload except SkillTreeSummaryError as exc: raise LiveSkillTreeDashboardError(str(exc)) from exc +def _count_exported_project(project_path: Path) -> dict[str, int]: + return summarize_export_payload( + _load_exported_project(project_path) + ).project_counts() + + def _require(condition: bool, message: str) -> None: if not condition: raise LiveSkillTreeDashboardError(message) @@ -284,7 +293,8 @@ def validate_live_skilltree_dashboard( material_index = _fetch_json(f"{dashboard_url}/api/course-materials", timeout) project_path = project_root / "output" / "data" / "skilltree_project.json" - exported_counts = _count_exported_project(project_path) + exported_project = _load_exported_project(project_path) + exported_counts = summarize_export_payload(exported_project).project_counts() project = dashboard_status.get("project", {}) config = dashboard_status.get("config", {}) dashboard_skilltree = dashboard_status.get("skilltree", {}) @@ -309,10 +319,12 @@ def validate_live_skilltree_dashboard( counts == exported_counts, f"dashboard counts {counts!r} != export {exported_counts!r}", ) - public_project = config.get("project", {}) - _require(isinstance(public_project, dict), "public project config is not an object") - project_id = project.get("projectId") or public_project.get("id") - project_name = project.get("name") or public_project.get("name") + public_project = project.get("project", {}) + _require( + isinstance(public_project, dict), "dashboard project identity is not an object" + ) + project_id = public_project.get("id") + project_name = public_project.get("name") _require( project_id == SKILLTREE_PROJECT_ID, "dashboard project id mismatch", @@ -429,6 +441,7 @@ def validate_live_skilltree_dashboard( local_state_dir=local_state_dir, timeout=timeout, required=require_live_admin, + exported_project=exported_project, ) live_dependencies = _validate_live_dependencies( skilltree_url=skilltree_url, @@ -439,6 +452,17 @@ def validate_live_skilltree_dashboard( timeout=timeout, required=require_live_dependencies, ) + live_content = live_admin.get("content", {"checked": False}) + _require( + not require_live_admin + or ( + live_content.get("checked") is True + and live_content.get("complete") is True + and project.get("exportContentDigest") + == export_content_digest(exported_project) + ), + "live SkillTree content or dashboard export digest does not match the current export", + ) figure_contracts = ( _validate_figure_contracts(project_root=project_root, artifacts=artifacts) if require_figure_contracts @@ -449,6 +473,8 @@ def validate_live_skilltree_dashboard( "ok": True, "skilltreeStatus": skilltree_status, "dashboardUrl": dashboard_url, + "dashboardInstanceId": dashboard_status.get("instanceId"), + "exportContentDigest": export_content_digest(exported_project), "skilltreeUrl": skilltree_url, "counts": exported_counts, "artifacts": {"present": len(artifacts), "missing": 0}, @@ -459,6 +485,7 @@ def validate_live_skilltree_dashboard( "figureContracts": figure_contracts, "liveAdminProject": live_admin, "liveDependencies": live_dependencies, + "liveContent": live_content, "tokenKeys": token_keys, } @@ -548,6 +575,7 @@ def _validate_live_admin_count( local_state_dir: Path | None, timeout: float, required: bool, + exported_project: dict[str, Any], ) -> dict[str, Any]: """Optionally verify the live SkillTree admin project count.""" from src.skilltree.client import SkillTreeClientError @@ -585,6 +613,11 @@ def _validate_live_admin_count( try: client.login(state.admin_username, state.admin_password) live_skills = client.live_skill_count() + content = compare_live_content( + get=lambda path: client.request_json("GET", path), + base_url=skilltree_url, + project=exported_project, + ) except (LocalSkillTreeAuthError, SkillTreeClientError) as exc: _require(not required, f"live SkillTree admin count check failed: {exc}") return { @@ -599,4 +632,5 @@ def _validate_live_admin_count( return { "checked": True, "skills": live_skills, + "content": content, } diff --git a/src/dashboard/payload.py b/src/dashboard/payload.py index fd8dd19..6ca76af 100644 --- a/src/dashboard/payload.py +++ b/src/dashboard/payload.py @@ -11,10 +11,12 @@ load_figure_table_companions, load_scholarship_health_network, load_skilltree_project, + read_json_file, ) from src.dashboard.config import DashboardConfig, DashboardPayload from src.dashboard.graph_analysis import analyze_project_graph from src.dashboard.status import check_skilltree_status +from src.skilltree.content import compare_live_content from src.skilltree.deployment_matrix import skilltree_deployment_matrix from src.skilltree.local_auth import ( DEFAULT_LOCAL_STATE_DIR, @@ -22,6 +24,7 @@ SkillTreeSessionClient, load_local_state, ) +from src.skilltree.local_auth_state import _require_same_skilltree_origin DEFERRED_LIVE_PARITY_REASON = ( "live dependency parity was deferred for first paint; " @@ -41,7 +44,10 @@ def dashboard_payload( that otherwise blocks every panel on first paint. The default is unchanged, so validators and proof artifacts still get a real parity check. """ - project = load_skilltree_project(config.project_root) + exported, _ = read_json_file( + config.project_root / "output" / "data" / "skilltree_project.json" + ) + project = load_skilltree_project(config.project_root, export_payload=exported) public_config = config.to_public_dict() skilltree_status = check_skilltree_status(config.skilltree_url) public_skilltree = public_config.get("skilltree", {}) @@ -66,7 +72,7 @@ def dashboard_payload( "project": project, "liveSkilltree": deferred_live_skilltree_summary(project) if defer_live - else live_skilltree_summary(config, project), + else live_skilltree_summary(config, project, export_payload=exported), "graphAnalysis": analyze_project_graph(project), "claims": load_claims_summary(config.project_root), "courseMaterials": load_course_materials_summary(config.project_root), @@ -96,8 +102,10 @@ def deferred_live_skilltree_summary(project: dict[str, Any]) -> dict[str, object def live_skilltree_summary( config: DashboardConfig, project: dict[str, Any], + *, + export_payload: dict[str, Any] | None = None, ) -> dict[str, object]: - """Return browser-safe live SkillTree admin/count parity when local auth is ready.""" + """Return browser-safe dependency and semantic-content parity when auth is ready.""" from src.skilltree.client import SkillTreeClientError from src.skilltree.dependency_reconciler import ( dependency_pairs_from_payload, @@ -133,6 +141,14 @@ def live_skilltree_summary( state = load_local_state(state_dir) if state is None: raise LocalSkillTreeAuthError("local SkillTree auth state is missing") + _require_same_skilltree_origin( + base_url=config.skilltree_url, + state_url=state.skilltree_url, + ) + if state.project_id != config.project_id: + raise LocalSkillTreeAuthError( + "configured project does not match bootstrapped local auth state" + ) expected_pairs = dependency_pairs_from_payload(dependencies, strict=False) client = SkillTreeSessionClient( config.skilltree_url, @@ -140,6 +156,17 @@ def live_skilltree_summary( timeout=15.0, ) client.login(state.admin_username, state.admin_password) + if export_payload is None: + export_payload, _ = read_json_file( + config.project_root / "output" / "data" / "skilltree_project.json" + ) + if export_payload is None: + raise LocalSkillTreeAuthError("private exported project is unavailable") + content = compare_live_content( + get=lambda path: client.request_json("GET", path), + base_url=config.skilltree_url, + project=export_payload, + ) live_pairs = client.live_dependency_pairs() parity = summarize_dependency_parity( expected_pairs=expected_pairs, @@ -147,8 +174,9 @@ def live_skilltree_summary( ) return { **parity, - "skills": client.live_skill_count(), - "subjects": client.live_subject_count(), + "skills": content["skills"], + "subjects": content["subjects"], + "content": content, } except (LocalSkillTreeAuthError, SkillTreeClientError) as exc: return { diff --git a/src/dashboard/server.py b/src/dashboard/server.py index 29da681..2c8125d 100644 --- a/src/dashboard/server.py +++ b/src/dashboard/server.py @@ -4,6 +4,7 @@ import gzip import json +import uuid from http import HTTPStatus from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer from pathlib import Path @@ -32,8 +33,21 @@ def dashboard_recommendations(config: DashboardConfig) -> dict[str, Any]: - """Return the read-only path prototype for the current exported project.""" - return recommend_next_skills(load_skilltree_project(config.project_root)) + """Validate the private export; return only answer-key-free path candidates.""" + from src.dashboard.artifacts import read_json_file + + project, error = read_json_file( + config.project_root / "output" / "data" / "skilltree_project.json" + ) + if project is None: + return { + "available": False, + "error": error, + "recommendations": [], + "personalized": False, + "signalSource": "none", + } + return recommend_next_skills(project) ASSETS_ROOT = Path(__file__).resolve().parent / "assets" @@ -163,6 +177,7 @@ def _host_header_hostname(host_header: str) -> str: def make_dashboard_handler(config: DashboardConfig) -> type[BaseHTTPRequestHandler]: """Create a request handler bound to one immutable dashboard config.""" + instance_id = config.instance_id or uuid.uuid4().hex class DashboardRequestHandler(BaseHTTPRequestHandler): server_version = "ActiveSkillferenceDashboard/1.0" @@ -227,6 +242,7 @@ def do_GET(self) -> None: self._send_json( { "ok": True, + "instanceId": instance_id, "projectId": config.project_id, "skilltreeUrl": config.skilltree_url, } @@ -238,10 +254,13 @@ def do_GET(self) -> None: if path == "/api/status": query = parse_qs(parsed.query) self._send_json( - dashboard_payload( - config, - defer_live=_query_flag(query, "defer_live"), - ) + { + **dashboard_payload( + config, + defer_live=_query_flag(query, "defer_live"), + ), + "instanceId": instance_id, + } ) return if path == "/api/recommendations": diff --git a/src/figures/plots/registry_io.py b/src/figures/plots/registry_io.py index aa93272..f11f47c 100644 --- a/src/figures/plots/registry_io.py +++ b/src/figures/plots/registry_io.py @@ -99,10 +99,12 @@ def render_all( include_scholarship = scholarship_audit is not None include_claim_support = claim_support_audit is not None include_roadmap = roadmap is not None - planned_count = planned_figure_count( - include_scholarship=include_scholarship, - include_claim_support=include_claim_support, - include_roadmap=include_roadmap, + planned_count = len( + planned_figure_registry_entries( + include_scholarship=include_scholarship, + include_claim_support=include_claim_support, + include_roadmap=include_roadmap, + ) ) figures = [ plot_curriculum_dag(graph, out_dir / "curriculum_dag.png"), @@ -233,7 +235,7 @@ def planned_figure_count( ) -> int: """Return the canonical number of registered figures for a render configuration.""" return len( - planned_figure_registry_entries( + figure_registry_payload( include_scholarship=include_scholarship, include_claim_support=include_claim_support, include_roadmap=include_roadmap, diff --git a/src/gnn_conformance.py b/src/gnn_conformance.py index bfcbd0c..5882bbe 100644 --- a/src/gnn_conformance.py +++ b/src/gnn_conformance.py @@ -315,6 +315,18 @@ def require(condition: bool, message: str) -> None: ) passed.append("dimensions match the kernel's canonical model (2/2/2)") + try: + DiscreteActiveInferenceAgent( + A=np.asarray(model["A_matrix"], dtype=float), + B=np.asarray(model["B_matrix"], dtype=float), + C=np.asarray(model["C_vector"], dtype=float), + D=np.asarray(model["D_vector"], dtype=float), + ) + except (TypeError, ValueError) as exc: + raise ConformanceError( + f"declared probability/transition model is invalid: {exc}" + ) from exc + require( np.allclose(np.asarray(model["A_matrix"], dtype=float), agent.A), "A does not match the kernel's informative sensor", diff --git a/src/integration/full_app_stack.py b/src/integration/full_app_stack.py index 985bf42..01ae40f 100644 --- a/src/integration/full_app_stack.py +++ b/src/integration/full_app_stack.py @@ -208,7 +208,7 @@ def bootstrap_command( state_dir: str, learner_id: str, skip_dependencies: bool = False, - skip_seed_if_live_counts_ready: bool = False, + skip_seed_if_live_content_ready: bool = False, ) -> list[str]: command = [ python, @@ -222,8 +222,8 @@ def bootstrap_command( ] if skip_dependencies: command.append("--skip-dependencies") - if skip_seed_if_live_counts_ready: - command.append("--skip-seed-if-live-counts-ready") + if skip_seed_if_live_content_ready: + command.append("--skip-seed-if-live-content-ready") return command @@ -254,8 +254,9 @@ def dashboard_command( port: int, state_dir: str, learner_id: str, + instance_id: str | None = None, ) -> list[str]: - return [ + command = [ python, "scripts/serve_learner_dashboard.py", "--skilltree-url", @@ -270,6 +271,9 @@ def dashboard_command( "--learner-id", learner_id, ] + if instance_id is not None: + command.extend(["--instance-id", instance_id]) + return command def live_validation_command( @@ -313,6 +317,7 @@ def console_validation_command( dashboard_url, "--require-browser", "--require-dashboard", + "--json", ] @@ -441,6 +446,7 @@ def wait_for_dashboard( *, dashboard_url: str, timeout: float, + instance_id: str | None = None, ) -> dict[str, Any]: health_url = f"{dashboard_url.rstrip('/')}/healthz" deadline = time.monotonic() + timeout @@ -452,7 +458,12 @@ def wait_for_dashboard( f"dashboard process exited before /healthz became ready\n{output}" ) try: - return _fetch_json(health_url, timeout=_REQUEST_TIMEOUT_SECONDS) + health = _fetch_json(health_url, timeout=_REQUEST_TIMEOUT_SECONDS) + if instance_id is not None and health.get("instanceId") != instance_id: + raise FullAppStackError( + "health endpoint belongs to another dashboard instance" + ) + return health except ( FullAppStackError, TimeoutError, @@ -482,19 +493,62 @@ def proof_summary( dependency_reconciliation: dict[str, Any] | None = None, app_ready: bool = False, ) -> dict[str, Any]: - checks = [live_validation, dashboard_health] + from src.dashboard.browser_validation import browser_receipt_complete + + browser: dict[str, Any] = {"checked": False, "reason": "skipped by --skip-browser"} + if browser_console_stdout is not None: + try: + candidate = json.loads(browser_console_stdout) + except json.JSONDecodeError as exc: + raise FullAppStackError( + "browser validator did not emit a structured proof" + ) from exc + if not browser_receipt_complete(candidate): + raise FullAppStackError( + "browser validator did not prove all official learner views" + ) + browser = candidate + content = live_validation.get("liveContent", {}) + dependencies = live_validation.get("liveDependencies", {}) + if not isinstance(content, dict) or not isinstance(dependencies, dict): + raise FullAppStackError("live content/dependency proof must contain objects") + instance_id = dashboard_health.get("instanceId") + digest = live_validation.get("exportContentDigest") + ok = ( + live_validation.get("ok") is True + and dashboard_health.get("ok") is True + and isinstance(instance_id, str) + and bool(instance_id) + and live_validation.get("dashboardInstanceId") == instance_id + and isinstance(digest, str) + and len(digest) == 64 + and content.get("checked") is True + and content.get("complete") is True + and content.get("exportContentDigest") == digest + ) + if not app_ready: + ok = ( + ok + and dependencies.get("checked") is True + and dependencies.get("complete") is True + ) if raw_skilltree_login is not None: - checks.append(raw_skilltree_login) + ok = ok and raw_skilltree_login.get("ok") is True if dependency_reconciliation is not None: - checks.append(dependency_reconciliation) - ok = all(check.get("ok") is not False for check in checks) - if ( - dependency_reconciliation is not None - and dependency_reconciliation.get("complete") is False - ): - ok = False + ok = ok and dependency_reconciliation.get("complete") is True + if browser_console_stdout is not None: + ok = ( + ok + and browser.get("dashboardInstanceId") == instance_id + and browser.get("exportContentDigest") == digest + and browser.get("projectId") == content.get("projectId") + ) return { "ok": ok, + "runId": instance_id, + "dashboardInstanceId": instance_id, + "exportContentDigest": digest, + "projectId": content.get("projectId"), "skilltreeUrl": skilltree_url, "dashboardUrl": dashboard_url, "skilltreeStatus": skilltree_status, @@ -504,13 +558,11 @@ def proof_summary( "courseMaterials": live_validation.get("courseMaterials", {}), "figureContracts": live_validation.get("figureContracts", {"checked": False}), "liveAdminProject": live_validation.get("liveAdminProject", {}), - "liveDependencies": live_validation.get("liveDependencies", {}), + "liveDependencies": dependencies, + "liveContent": content, "dependencyReconciliation": dependency_reconciliation or {"checked": False}, "mode": "app-ready" if app_ready else "full-parity", - "browserConsole": { - "checked": browser_console_stdout is not None, - "summary": (browser_console_stdout or "skipped by --skip-browser").strip(), - }, + "browserConsole": browser, "rawSkillTreeLogin": raw_skilltree_login if raw_skilltree_login is not None else {"checked": False, "reason": "skipped by --skip-browser"}, @@ -530,7 +582,7 @@ def write_proof_artifact(proof: dict[str, Any], path: Path) -> Path: safe = _redact_proof_value(proof) if not isinstance(safe, dict): # pragma: no cover - defensive type narrowing raise FullAppStackError("full-app proof must be a JSON object") - safe["proofVersion"] = 1 + safe["proofVersion"] = 2 return write_json_artifact(Path(path), safe, ensure_ascii=False) @@ -601,7 +653,7 @@ def validate_full_app_stack(args: argparse.Namespace) -> dict[str, Any]: state_dir=state_dir, learner_id=args.learner_id, skip_dependencies=True, - skip_seed_if_live_counts_ready=True, + skip_seed_if_live_content_ready=True, ), ], timeout=args.timeout, @@ -626,6 +678,7 @@ def validate_full_app_stack(args: argparse.Namespace) -> dict[str, Any]: ) pid_path, log_path = dashboard_runtime_paths(state_dir) + instance_id = secrets.token_hex(16) dashboard = start_dashboard( [ "uv", @@ -636,15 +689,18 @@ def validate_full_app_stack(args: argparse.Namespace) -> dict[str, Any]: port=args.port, state_dir=state_dir, learner_id=args.learner_id, + instance_id=instance_id, ), ], log_path=log_path if args.keep_running else None, ) + proof_written = False try: dashboard_health = wait_for_dashboard( dashboard, dashboard_url=dashboard_url, timeout=args.timeout, + instance_id=instance_id, ) live_result = run_checked( [ @@ -713,9 +769,34 @@ def validate_full_app_stack(args: argparse.Namespace) -> dict[str, Any]: ) proof["proofPath"] = str(proof_path) write_proof_artifact(proof, proof_path) + proof_written = True return proof finally: - if args.keep_running and dashboard.poll() is None: - record_dashboard_process(state_dir=state_dir, pid=dashboard.pid) - elif not args.keep_running: - stop_dashboard_process(dashboard) + finalize_dashboard_process( + dashboard, + state_dir=state_dir, + keep_running=args.keep_running, + successful=proof_written, + ) + + +def finalize_dashboard_process( + process: subprocess.Popen[str], + *, + state_dir: str, + keep_running: bool, + successful: bool, +) -> None: + """Keep an owned process only after successful proof publication.""" + if successful and keep_running: + if process.poll() is not None: + raise FullAppStackError( + "owned dashboard exited before keep-running finalization" + ) + try: + record_dashboard_process(state_dir=state_dir, pid=process.pid) + except BaseException: + stop_dashboard_process(process) + raise + else: + stop_dashboard_process(process) diff --git a/src/kernels/active_inference.py b/src/kernels/active_inference.py index d69c4e1..03099e1 100644 --- a/src/kernels/active_inference.py +++ b/src/kernels/active_inference.py @@ -10,7 +10,7 @@ from numpy.typing import ArrayLike, NDArray from .distributions import log_stable, normalize, softmax -from .inference import bayes_posterior +from .inference import _as_probability_vector, _validate_model, bayes_posterior from .information import kl_divergence FloatArray: TypeAlias = NDArray[np.float64] @@ -36,9 +36,12 @@ def expected_free_energy( Tuple `(G, epistemic, pragmatic)` in nats. """ - state_probs = np.asarray(qs, dtype=np.float64) + state_probs = _as_probability_vector(qs, name="qs") likelihood = np.asarray(A, dtype=np.float64) + _validate_model(state_probs, likelihood, 0) preferences = normalize(C) + if preferences.shape != (likelihood.shape[0],): + raise ValueError("C length must match the number of observations") qo = likelihood @ state_probs pragmatic = float(np.dot(qo, log_stable(preferences))) @@ -64,7 +67,7 @@ def policy_posterior(G: ArrayLike, gamma: float = 1.0) -> FloatArray: Posterior policy distribution `softmax(-gamma * G)`. """ - return softmax(-gamma * np.asarray(G, dtype=np.float64)) + return softmax(-np.asarray(G, dtype=np.float64), precision=gamma) @dataclass @@ -77,12 +80,27 @@ class DiscreteActiveInferenceAgent: D: FloatArray def __post_init__(self) -> None: - """Coerce model components into float arrays.""" + """Coerce and validate the existing probability-weight model contract.""" self.A = np.asarray(self.A, dtype=np.float64) self.B = np.asarray(self.B, dtype=np.float64) self.C = np.asarray(self.C, dtype=np.float64) - self.D = np.asarray(self.D, dtype=np.float64) + self.D = _as_probability_vector(self.D, name="D") + _validate_model(self.D, self.A, 0) + n_states = self.D.size + if ( + self.B.ndim != 3 + or self.B.shape[:2] != (n_states, n_states) + or self.B.shape[2] == 0 + or not np.all(np.isfinite(self.B)) + or np.any(self.B < 0.0) + or not np.allclose(self.B.sum(axis=0), 1.0, atol=1e-9, rtol=0.0) + ): + raise ValueError( + "B must contain finite non-negative column-stochastic state transitions" + ) + if normalize(self.C).shape != (self.A.shape[0],): + raise ValueError("C length must match the number of observations") def infer_states(self, obs: int) -> FloatArray: """Infer the posterior state distribution for an observation.""" @@ -92,7 +110,15 @@ def infer_states(self, obs: int) -> FloatArray: def predict(self, qs: ArrayLike, action: int) -> FloatArray: """Predict the next-state distribution under an action.""" - state_probs = np.asarray(qs, dtype=np.float64) + state_probs = _as_probability_vector(qs, name="qs") + if state_probs.shape != self.D.shape: + raise ValueError("qs length must match the model state dimension") + if not isinstance(action, (int, np.integer)) or isinstance( + action, (bool, np.bool_) + ): + raise TypeError("action must be an integer index") + if action < 0 or action >= self.B.shape[2]: + raise ValueError("action index out of range") return self.B[:, :, action] @ state_probs def evaluate_policies(self, qs: ArrayLike) -> FloatArray: diff --git a/src/kernels/distributions.py b/src/kernels/distributions.py index d517d28..c87bd48 100644 --- a/src/kernels/distributions.py +++ b/src/kernels/distributions.py @@ -25,14 +25,15 @@ def normalize(x: ArrayLike) -> FloatArray: """ values = np.asarray(x, dtype=np.float64) + if values.ndim != 1 or not values.size or not np.all(np.isfinite(values)): + raise ValueError("values must be a non-empty finite vector") if np.any(values < 0.0): raise ValueError("values must be non-negative") - - total = float(np.sum(values)) - if total <= 0.0: + scale = float(np.max(values)) + if scale <= 0.0: raise ValueError("values must have a positive sum") - - return values / total + scaled = values / scale + return scaled / np.sum(scaled) def softmax(x: ArrayLike, precision: float = 1.0) -> FloatArray: @@ -52,14 +53,22 @@ def softmax(x: ArrayLike, precision: float = 1.0) -> FloatArray: ValueError: If precision is not strictly positive. """ - if precision <= 0.0: - raise ValueError("precision must be > 0") - + if not np.isfinite(precision) or precision <= 0.0: + raise ValueError("precision must be finite and > 0") values = np.asarray(x, dtype=np.float64) - scaled = precision * values - shifted = scaled - np.max(scaled) - exp_shifted = np.exp(shifted) - return exp_shifted / np.sum(exp_shifted) + if values.ndim != 1 or not values.size or not np.all(np.isfinite(values)): + raise ValueError("scores must be a non-empty finite vector") + # Scale small precision first so centering cannot erase finite scaled mass. + # At precision >= 1, center first to avoid overflowing large absolute scores. + with np.errstate(over="ignore", under="ignore"): + if precision < 1.0: + shifted = values * precision + shifted -= np.max(shifted) + else: + shifted = values - np.max(values) + shifted *= precision + np.exp(shifted, out=shifted) + return shifted / np.sum(shifted) def log_stable(x: ArrayLike, eps: float = 1e-16) -> FloatArray: @@ -91,10 +100,8 @@ def dirichlet_expected(alpha: ArrayLike) -> FloatArray: ValueError: If any concentration parameter is non-positive. """ - concentration = np.asarray(alpha, dtype=np.float64) - if np.any(concentration <= 0.0): - raise ValueError("alpha must be strictly positive") - return concentration / np.sum(concentration) + concentration = _concentrations(alpha) + return normalize(concentration) def dirichlet_expected_log(alpha: ArrayLike) -> FloatArray: @@ -110,10 +117,27 @@ def dirichlet_expected_log(alpha: ArrayLike) -> FloatArray: ValueError: If any concentration parameter is non-positive. """ + concentration = _concentrations(alpha) + scale = float(np.max(concentration)) + scaled_total = float(np.sum(concentration / scale)) + if scale > np.finfo(np.float64).max / scaled_total: + # At this magnitude the digamma asymptotic correction is unrepresentable. + total_digamma = np.log(scale) + np.log(scaled_total) + else: + total_digamma = digamma(scale * scaled_total) + return digamma(concentration) - total_digamma + + +def _concentrations(alpha: ArrayLike) -> FloatArray: concentration = np.asarray(alpha, dtype=np.float64) - if np.any(concentration <= 0.0): - raise ValueError("alpha must be strictly positive") - return digamma(concentration) - digamma(np.sum(concentration)) + if ( + concentration.ndim != 1 + or not concentration.size + or not np.all(np.isfinite(concentration)) + or np.any(concentration <= 0.0) + ): + raise ValueError("alpha must be a non-empty vector of finite positive values") + return concentration __all__ = [ diff --git a/src/kernels/inference.py b/src/kernels/inference.py index 6ea7770..ca03fd3 100644 --- a/src/kernels/inference.py +++ b/src/kernels/inference.py @@ -8,7 +8,7 @@ from numpy.typing import ArrayLike, NDArray from scipy.special import logsumexp -from .distributions import log_stable, normalize +from .distributions import normalize FloatArray: TypeAlias = NDArray[np.float64] _PROB_TOL = 1e-9 @@ -20,8 +20,8 @@ def _as_probability_vector(x: ArrayLike, *, name: str) -> FloatArray: values = np.asarray(x, dtype=np.float64) if values.ndim != 1: raise ValueError(f"{name} must be a 1D probability vector") - if np.any(values < 0.0): - raise ValueError(f"{name} must be non-negative and sum to 1") + if not np.all(np.isfinite(values)) or np.any(values < 0.0): + raise ValueError(f"{name} must be finite, non-negative and sum to 1") if not np.isclose(np.sum(values), 1.0, atol=_PROB_TOL, rtol=0.0): raise ValueError(f"{name} must sum to 1") return values @@ -43,8 +43,10 @@ def _validate_model(prior: FloatArray, A: FloatArray, obs: int) -> None: raise ValueError("prior length must match the number of states in A") if obs < 0 or obs >= A.shape[0]: raise ValueError("obs index out of range for A") - if np.any(A < 0.0): - raise ValueError("A must be non-negative (it is a likelihood P(o|s))") + if not np.all(np.isfinite(A)) or np.any(A < 0.0): + raise ValueError( + "A must be finite and non-negative (it is a likelihood P(o|s))" + ) if not np.allclose(np.sum(A, axis=0), 1.0, atol=_PROB_TOL, rtol=0.0): raise ValueError("each column of A (P(o|s) for fixed state s) must sum to 1") @@ -103,11 +105,14 @@ def variational_free_energy( raise ValueError("q length must match prior length") mask = q_probs > 0.0 - energy = np.sum( - q_probs[mask] * (log_stable(q_probs[mask]) - log_stable(prior_probs[mask])) + if np.any(prior_probs[mask] == 0.0) or np.any(likelihood[obs, mask] == 0.0): + return float("inf") + terms = ( + np.log(q_probs[mask]) + - np.log(prior_probs[mask]) + - np.log(likelihood[obs, mask]) ) - accuracy_cost = np.sum(q_probs[mask] * log_stable(likelihood[obs, mask])) - return float(energy - accuracy_cost) + return float(np.dot(q_probs[mask], terms)) def log_evidence(A: ArrayLike, obs: int, prior: ArrayLike) -> float: @@ -128,7 +133,12 @@ def log_evidence(A: ArrayLike, obs: int, prior: ArrayLike) -> float: prior_probs = _as_probability_vector(prior, name="prior") likelihood = np.asarray(A, dtype=np.float64) _validate_model(prior_probs, likelihood, obs) - return float(logsumexp(log_stable(likelihood[obs, :]) + log_stable(prior_probs))) + support = (likelihood[obs, :] > 0.0) & (prior_probs > 0.0) + if not np.any(support): + return float("-inf") + return float( + logsumexp(np.log(likelihood[obs, support]) + np.log(prior_probs[support])) + ) def belief_propagation_chain( diff --git a/src/kernels/predictive_coding.py b/src/kernels/predictive_coding.py index a46d98c..3a45c6a 100644 --- a/src/kernels/predictive_coding.py +++ b/src/kernels/predictive_coding.py @@ -50,9 +50,9 @@ class PredictiveCodingResult: trajectory: FloatArray # estimate at each step (length steps + 1) free_energy: FloatArray # free energy at each step estimate: float # final estimate - sensory_prediction_error: float # y - estimate at convergence - prior_prediction_error: float # estimate - prior_mean at convergence - steps_to_converge: int # first step within tolerance of the fixed point + sensory_prediction_error: float # y - final estimate + prior_prediction_error: float # final estimate - prior_mean + steps_to_converge: int # first update <= tolerance, or steps if none qualifies def gaussian_free_energy( @@ -102,13 +102,15 @@ def predictive_coding_descent( obs_variance: Observation variance (inverse sensory precision), > 0. rate: Positive learning rate for the gradient step. steps: Number of descent steps. - tolerance: Convergence tolerance on the distance to the fixed point. + tolerance: Threshold on absolute update magnitude, not fixed-point distance. start: Initial estimate (defaults to the prior mean). Returns: - A :class:`PredictiveCodingResult` with the trajectory, free-energy curve, - the converged estimate, the two prediction errors, and the step at which the - estimate first came within ``tolerance`` of the fixed point. + A :class:`PredictiveCodingResult` with all ``steps + 1`` trajectory and + free-energy samples, the final estimate and prediction errors, and the + first step whose absolute update is at most ``tolerance``. If no update + qualifies, ``steps_to_converge`` is ``steps``; the trajectory is never + truncated when the threshold is reached. """ if prior_variance <= 0.0 or obs_variance <= 0.0: @@ -126,9 +128,8 @@ def predictive_coding_descent( estimate, observation, prior_mean, prior_variance, obs_variance ) steps_to_converge = steps - # steps_to_converge is measured by the update magnitude (the descent's own stopping - # condition), NOT by distance to the analytic target — so this diagnostic stays - # independent of the closed form and cannot mask a closed-form bug. + # This update-magnitude diagnostic is independent of the analytic target. + # It records the first qualifying step but does not stop the trajectory. converged = False for step in range(1, steps + 1): sensory_error = (observation - estimate) / obs_variance diff --git a/src/live_counts.py b/src/live_counts.py index d993c49..b36cfb0 100644 --- a/src/live_counts.py +++ b/src/live_counts.py @@ -165,7 +165,10 @@ def audit_live_count_literals(root: Path) -> list[LiveCountIssue]: for path in audited_count_paths(root): rel = path.relative_to(root) - text = _CODE_BLOCK_RE.sub("", path.read_text(encoding="utf-8")) + text = _CODE_BLOCK_RE.sub( + lambda match: "\n" * match.group(0).count("\n"), + path.read_text(encoding="utf-8"), + ) for line_number, line in enumerate(text.splitlines(), start=1): for key, pattern in patterns.items(): expected = counts[key].value diff --git a/src/manuscript/pdf_receipt.py b/src/manuscript/pdf_receipt.py new file mode 100644 index 0000000..d24ea4e --- /dev/null +++ b/src/manuscript/pdf_receipt.py @@ -0,0 +1,246 @@ +"""Content-bound PDF receipts for repository inputs, not release authorization. + +Primary inputs are captured before assembly; actual LaTeX image inputs are added +before compilation. Every bound file and the section set must still match before +publication. Historical receipts are never rewritten by validation. +""" + +from __future__ import annotations + +import hashlib +import json +import re +from collections.abc import Iterable, Mapping, Sequence +from pathlib import Path +from typing import Any + +from src.artifacts.json_io import write_json_artifact + +SECTION_EXCLUDE = frozenset( + {"config.yaml", "preamble.md", "README.md", "AGENTS.md", "SYNTAX.md"} +) +PDF_NAME = "Active_Skillference_combined.pdf" +_PRIMARY_INPUTS = { + "manuscript/config.yaml": "config", + "manuscript/preamble.md": "preamble", + "manuscript/references.bib": "bibliography", +} +_IMAGE_RE = re.compile(r"\\includegraphics\*?(?:\[[^\]]*\])?\{([^{}]+)\}") + + +class PdfReceiptError(ValueError): + """A PDF or its actual repository inputs are missing, changed or unverifiable.""" + + +def hydrated_section_files(directory: Path) -> list[Path]: + return [ + path + for path in sorted(directory.glob("*.md")) + if path.name not in SECTION_EXCLUDE + ] + + +def _relative(path: Path, root: Path) -> str: + try: + return path.resolve().relative_to(root.resolve()).as_posix() + except ValueError as exc: + raise PdfReceiptError( + "PDF inputs/output must remain inside the project root" + ) from exc + + +def _record(path: Path, root: Path, kind: str) -> dict[str, Any]: + relative = _relative(path, root) + try: + with path.open("rb") as stream: + digest = hashlib.file_digest(stream, "sha256").hexdigest() + size = path.stat().st_size + except OSError as exc: + raise PdfReceiptError(f"PDF {kind} file is unreadable: {relative}") from exc + return {"path": relative, "kind": kind, "size_bytes": size, "sha256": digest} + + +def capture_pdf_inputs(root: Path, sections: Sequence[Path]) -> dict[str, Any]: + if not sections: + raise PdfReceiptError("PDF input manifest needs hydrated sections") + rows = [_record(root / name, root, kind) for name, kind in _PRIMARY_INPUTS.items()] + rows.extend(_record(path, root, "section") for path in sections) + return { + "sections": [_relative(path, root) for path in sections], + "assets": [], + "files": sorted(rows, key=lambda row: row["path"]), + } + + +def latex_asset_paths(tex: Path, *, build_directory: Path, root: Path) -> list[Path]: + text = tex.read_text(encoding="utf-8") + text = re.sub(r"(? dict[str, Any]: + validate_pdf_inputs(root, manifest) + rows = {row["path"]: dict(row) for row in manifest["files"]} + for path in assets: + row = _record(path, root, "asset") + rows.setdefault(row["path"], row) + result = { + "sections": list(manifest["sections"]), + "assets": sorted(name for name, row in rows.items() if row["kind"] == "asset"), + "files": [rows[name] for name in sorted(rows)], + } + validate_pdf_inputs(root, result) + return result + + +def validate_pdf_inputs(root: Path, manifest: Mapping[str, Any]) -> None: + sections, rows = manifest.get("sections"), manifest.get("files") + assets = manifest.get("assets") + if ( + not isinstance(sections, list) + or not sections + or any(not isinstance(path, str) for path in sections) + or len(set(sections)) != len(sections) + or not isinstance(rows, list) + or not rows + or not isinstance(assets, list) + or any(not isinstance(path, str) for path in assets) + or len(set(assets)) != len(assets) + ): + raise PdfReceiptError( + "PDF input manifest is missing its complete section/file set" + ) + expected_sections = [ + _relative(path, root) + for path in hydrated_section_files(root / "output" / "manuscript") + ] + if sections != expected_sections: + raise PdfReceiptError("hydrated PDF section set/order changed") + by_path: dict[str, Mapping[str, Any]] = {} + for row in rows: + if not isinstance(row, Mapping) or not isinstance(row.get("path"), str): + raise PdfReceiptError("PDF input manifest contains a malformed file row") + name = row["path"] + path = Path(name) + if ( + path.is_absolute() + or ".." in path.parts + or _relative(root / path, root) != name + or name in by_path + ): + raise PdfReceiptError("PDF input manifest has an unsafe or duplicate path") + if ( + not isinstance(row.get("sha256"), str) + or not re.fullmatch(r"[0-9a-f]{64}", row["sha256"]) + or type(row.get("size_bytes")) is not int + or row["size_bytes"] < 0 + or row.get("kind") + not in ("config", "preamble", "bibliography", "section", "asset") + ): + raise PdfReceiptError(f"PDF input digest row is malformed: {name}") + current = _record(root / path, root, row["kind"]) + if ( + current["sha256"] != row["sha256"] + or current["size_bytes"] != row["size_bytes"] + ): + raise PdfReceiptError(f"PDF input changed: {name}") + by_path[name] = row + if any( + name not in by_path or by_path[name]["kind"] != kind + for name, kind in _PRIMARY_INPUTS.items() + ): + raise PdfReceiptError( + "PDF input manifest is missing configuration/preamble/bibliography" + ) + if {name for name, row in by_path.items() if row["kind"] == "section"} != set( + sections + ): + raise PdfReceiptError( + "PDF input manifest does not cover the complete hydrated section set" + ) + if {name for name, row in by_path.items() if row["kind"] == "asset"} != set(assets): + raise PdfReceiptError( + "PDF input manifest does not cover its complete image set" + ) + + +def write_pdf_receipt( + target: Path, + *, + root: Path, + source_revision: str, + inputs: Mapping[str, Any], +) -> Path: + validate_pdf_inputs(root, inputs) + if not re.fullmatch(r"[0-9a-f]{40}", source_revision): + raise PdfReceiptError( + "PDF source revision must be the observed full Git commit" + ) + output = _record(target, root, "pdf") + if output["size_bytes"] <= 0: + raise PdfReceiptError("finalized PDF is empty") + receipt = target.with_suffix(".receipt.json") + return write_json_artifact( + receipt, + { + "schema": "active-skillference-pdf/v2", + "output_path": output["path"], + "size_bytes": output["size_bytes"], + "sha256": output["sha256"], + "source_revision": source_revision, + "inputManifest": dict(inputs), + }, + ) + + +def pdf_receipt_issues(root: Path, target: Path | None = None) -> list[str]: + target = target if target is not None else root / "output" / "pdf" / PDF_NAME + receipt = target.with_suffix(".receipt.json") + try: + payload = json.loads(receipt.read_text(encoding="utf-8")) + if ( + not isinstance(payload, dict) + or payload.get("schema") != "active-skillference-pdf/v2" + ): + raise PdfReceiptError( + "missing/legacy PDF input provenance; a genuine re-render is required" + ) + manifest = payload.get("inputManifest") + if not isinstance(manifest, Mapping): + raise PdfReceiptError("PDF receipt has no complete input manifest") + validate_pdf_inputs(root, manifest) + output = _record(target, root, "pdf") + if ( + payload.get("output_path") != output["path"] + or payload.get("sha256") != output["sha256"] + or payload.get("size_bytes") != output["size_bytes"] + ): + raise PdfReceiptError( + "finalized PDF bytes do not match the recorded render" + ) + revision = payload.get("source_revision") + if not isinstance(revision, str) or not re.fullmatch(r"[0-9a-f]{40}", revision): + raise PdfReceiptError("PDF receipt has no observed source revision") + except (OSError, ValueError) as exc: + return [f"PDF freshness failed: {exc}; re-render before publishing"] + return [] diff --git a/src/pilot/contract.py b/src/pilot/contract.py index 0f9bb10..154dd57 100644 --- a/src/pilot/contract.py +++ b/src/pilot/contract.py @@ -32,8 +32,7 @@ def validate_pilot_contract(payload: Mapping[str, Any]) -> None: raise PilotContractError("contractVersion must be 1") _require_text(payload, "status", expected="protocol-only") _require_text(payload, "purpose") - _require_mapping(payload, "claimsBoundary") - boundary = payload["claimsBoundary"] + boundary = _require_mapping(payload, "claimsBoundary") _require_list(boundary, "allowed") _require_list(boundary, "prohibited") if not boundary["allowed"] or not boundary["prohibited"]: @@ -50,13 +49,14 @@ def validate_pilot_contract(payload: Mapping[str, Any]) -> None: if participants.get("directIdentifiersAllowed") is not False: raise PilotContractError("directIdentifiersAllowed must be false") fields = _require_list(participants, "requiredFields") - _unique_texts(fields, "participants.requiredFields") + _unique_texts(fields, "participants.requiredFields", {"participant_id_hash"}) events = _require_mapping(payload, "events") event_types = _require_list(events, "allowedEventTypes") _unique_texts(event_types, "events.allowedEventTypes") event_fields = _require_list(events, "requiredFields") - _unique_texts(event_fields, "events.requiredFields") + linking_fields = {"participant_id_hash", "event_type", "event_timestamp"} + _unique_texts(event_fields, "events.requiredFields", linking_fields) outcomes = _require_mapping(payload, "outcomes") primary = _require_mapping(outcomes, "primary") @@ -87,7 +87,7 @@ def validate_pilot_contract(payload: Mapping[str, Any]) -> None: export = _require_mapping(payload, "export") _require_text(export, "sourceBoundary") export_fields = _require_list(export, "fields") - _unique_texts(export_fields, "export.fields") + _unique_texts(export_fields, "export.fields", linking_fields) def _require_mapping(payload: Mapping[str, Any], key: str) -> dict[str, Any]: @@ -115,7 +115,11 @@ def _require_text( return value -def _unique_texts(values: list[Any], location: str) -> None: +def _unique_texts( + values: list[Any], location: str, required: set[str] | None = None +) -> None: + if not values: + raise PilotContractError(f"{location} must not be empty") texts = [] for index, value in enumerate(values, start=1): if not isinstance(value, str) or not value.strip(): @@ -123,3 +127,8 @@ def _unique_texts(values: list[Any], location: str) -> None: texts.append(value) if len(texts) != len(set(texts)): raise PilotContractError(f"{location} must not contain duplicates") + missing = (required or set()) - set(texts) + if missing: + raise PilotContractError( + f"{location} is missing linking fields: {', '.join(sorted(missing))}" + ) diff --git a/src/pipeline/generated_outputs.py b/src/pipeline/generated_outputs.py index 539fe6d..87b8ec7 100644 --- a/src/pipeline/generated_outputs.py +++ b/src/pipeline/generated_outputs.py @@ -98,7 +98,7 @@ def build_generated_outputs(project_root: Path | None = None) -> GeneratedOutput banned = audit_manuscript_banned_phrases(manuscript_dir) if banned: detail = "; ".join(f"{name}: {hits}" for name, hits in banned.items()) - raise SystemExit( + raise ValueError( f"manuscript endorsement-overreach phrases found: {detail}" ) scholarship_audit = validate_scholarship( diff --git a/src/provenance/binding.py b/src/provenance/binding.py index 1686a8d..f3aa201 100644 --- a/src/provenance/binding.py +++ b/src/provenance/binding.py @@ -20,12 +20,11 @@ from __future__ import annotations import re -import unicodedata from dataclasses import dataclass, replace from pathlib import Path from src.curriculum.graph import CurriculumGraph -from src.curriculum.model import CLAIM_TOKEN_RE, Skill, Subject +from src.curriculum.model import CLAIM_TOKEN_RE, Skill, Subject, _strip_format_chars from src.provenance.claims import ClaimRegistry # The shape a computed *result* takes: either a decimal with >=2 fractional digits @@ -66,19 +65,6 @@ class UnbackedNumber: context: str -def _strip_format_chars(text: str) -> str: - """Remove Unicode *format* characters (category Cf: zero-width space U+200B, - ZWJ/ZWNJ, BOM, word-joiner, …). - - These are invisible and neither ``\\s`` nor ``\\b`` match them, so a token or number - split by one (``{{claim:e}}``, ``1.6094``) would slip past both the - malformed-token check and the unbacked-number linter. Stripping them before any - integrity scan closes that bypass class; they have no legitimate place in curriculum - prose, so removal is safe and makes the gates see the real glyph sequence. - """ - return "".join(ch for ch in text if unicodedata.category(ch) != "Cf") - - def render(text: str, registry: ClaimRegistry) -> str: """Replace every ``{{claim:NAME}}`` with its rendered value. @@ -104,21 +90,26 @@ def _sub(match: re.Match[str]) -> str: def validate_skill(skill: Skill, registry: ClaimRegistry) -> None: - """Raise if any claim token in the skill is unbacked.""" - missing = sorted(t for t in skill.claim_tokens() if not registry.has(t)) - if missing: - raise ProvenanceError( - f"skill {skill.skill_id!r} references unbacked claims: {', '.join(missing)}" - ) + """Apply canonical fail-closed token admission to every learner-facing skill field.""" + texts = [skill.name, skill.objective, skill.description] + for question in skill.assessments: + texts.append(question.prompt) + texts.extend(question.answers) + try: + for text in texts: + render(text, registry) + except ProvenanceError as exc: + raise ProvenanceError(f"skill {skill.skill_id!r}: {exc}") from exc def validate_curriculum(skills: list[Skill], registry: ClaimRegistry) -> None: - """Fail closed if ANY skill references an unbacked claim. Used by the exporter.""" + """Fail closed on unbacked or malformed learner-field tokens before export.""" problems: list[str] = [] for skill in skills: - missing = sorted(t for t in skill.claim_tokens() if not registry.has(t)) - if missing: - problems.append(f"{skill.skill_id}: {', '.join(missing)}") + try: + validate_skill(skill, registry) + except ProvenanceError as exc: + problems.append(str(exc)) if problems: raise ProvenanceError( "unbacked claim tokens (export refused):\n " + "\n ".join(problems) @@ -127,7 +118,7 @@ def validate_curriculum(skills: list[Skill], registry: ClaimRegistry) -> None: def render_graph(graph: CurriculumGraph, registry: ClaimRegistry) -> CurriculumGraph: """Return a copy of the graph with every claim token resolved in EVERY learner-facing - field: subject description, skill name/objective/description, and each assessment's + field: subject name/description, skill name/objective/description, and each assessment's prompt and answer options. Fails closed: :func:`render` raises :class:`ProvenanceError` on any unbacked OR @@ -160,11 +151,14 @@ def _render(text: str) -> str: new_subjects.append( replace( subject, + name=_render(subject.name), description=_render(subject.description), skills=rendered_skills, ) ) - return CurriculumGraph(subjects=new_subjects) + rendered = CurriculumGraph(subjects=new_subjects) + rendered.validate() + return rendered def _strip_protected(text: str) -> str: @@ -266,10 +260,10 @@ def lint_curriculum( def lint_graph( graph: CurriculumGraph, *, allow: tuple[str, ...] = () ) -> list[UnbackedNumber]: - """Lint every learner-facing free-text field in the graph: subject descriptions, - skill names, and skill descriptions. Empty list == clean.""" + """Lint subject names/descriptions and learner-facing skill prose.""" violations = lint_curriculum(graph.skills, allow=allow) for subject in graph.subjects: + violations.extend(_scan_text(subject.name, subject.subject_id, allow)) violations.extend(_scan_text(subject.description, subject.subject_id, allow)) return violations diff --git a/src/publication/zenodo.py b/src/publication/zenodo.py index 239ff74..26e04c8 100644 --- a/src/publication/zenodo.py +++ b/src/publication/zenodo.py @@ -1,19 +1,15 @@ -"""Reserve a Zenodo concept DOI without publishing anything. +"""Explicit Zenodo reservation, upload and publication operations. -Zenodo allocates a DOI to an unpublished draft deposition. That allocation is -real — the identifier is issued to this account and nobody else can receive it — -but it does not resolve until the draft is published, and a draft can still be -discarded. Reserving is therefore the one DOI action that is not irreversible, -which is why this module can perform it and deliberately cannot publish. - -There is no publish call here on purpose. Minting a live DOI is a permanent, -outward-facing act and belongs to a human running the release tooling, not to a -build step. +A reserved DOI belongs to an unpublished draft and does not resolve until that +draft is published. Publication is a separate irreversible operation guarded by +the expected DOI. No build/import path publishes or contacts Zenodo implicitly; +release authority and credentials remain with the operator. """ from __future__ import annotations import json +import os import urllib.error import urllib.parse import urllib.request @@ -162,37 +158,40 @@ def find_draft(*, token: str, title: str) -> ReservedDoi | None: def upload_deposit_file(*, token: str, deposition_id: int, path: Path) -> str: - """Attach one file to a draft deposition and return its Zenodo filename. - - Zenodo will not publish a deposition with no files, so this runs first. It - uses the modern bucket API, which streams the body rather than building a - multipart form in memory. - """ + """Stream one real file to the draft bucket with an explicit content length.""" deposition = _request(f"/deposit/depositions/{deposition_id}", token=token) - bucket = str(deposition.get("links", {}).get("bucket", "")) - if not bucket: + links = deposition.get("links", {}) if isinstance(deposition, dict) else {} + bucket = links.get("bucket") if isinstance(links, dict) else None + if not isinstance(bucket, str) or not bucket: raise ZenodoError( - f"deposition {deposition_id} exposes no upload bucket; it may already " - "be published" + f"deposition {deposition_id} exposes no upload bucket; it may already be published" ) - request = urllib.request.Request( - f"{bucket}/{urllib.parse.quote(path.name)}", - data=path.read_bytes(), - method="PUT", - headers={ - "Authorization": f"Bearer {token}", - "Content-Type": "application/octet-stream", - }, - ) try: - with urllib.request.urlopen(request, timeout=600) as response: - payload = json.loads(response.read().decode("utf-8")) + with path.open("rb") as stream: + request = urllib.request.Request( + f"{bucket}/{urllib.parse.quote(path.name)}", + data=stream, + method="PUT", + headers={ + "Authorization": f"Bearer {token}", + "Content-Type": "application/octet-stream", + "Content-Length": str(os.fstat(stream.fileno()).st_size), + }, + ) + with urllib.request.urlopen(request, timeout=600) as response: + payload = json.loads(response.read().decode("utf-8")) except urllib.error.HTTPError as exc: - detail = exc.read().decode("utf-8")[:500] + detail = exc.read().decode("utf-8", errors="replace")[:500] raise ZenodoError(f"Zenodo file upload failed ({exc.code}): {detail}") from exc - except (urllib.error.URLError, TimeoutError, ValueError) as exc: + except (OSError, ValueError) as exc: raise ZenodoError(f"Zenodo file upload failed: {exc}") from exc - return str(payload.get("key", path.name)) + if ( + not isinstance(payload, dict) + or not isinstance(payload.get("key"), str) + or not payload["key"].strip() + ): + raise ZenodoError("Zenodo upload returned no confirmed filename") + return payload["key"] def publish_deposit( @@ -261,7 +260,7 @@ def reserve_doi(*, token: str, metadata: dict[str, Any]) -> ReservedDoi: ) if result.is_published: raise ZenodoError( - f"deposition {deposition_id} reports as published; this module only " - "reserves and must never mint a resolving DOI" + f"deposition {deposition_id} reports as published; the reservation " + "operation refuses an already-published record" ) return result diff --git a/src/release_readiness.py b/src/release_readiness.py index a221ee7..7950067 100644 --- a/src/release_readiness.py +++ b/src/release_readiness.py @@ -115,13 +115,21 @@ def build_release_readiness_snapshot(root: Path) -> ReleaseReadinessSnapshot: ) -def collect_blocker_probes(root: Path) -> BlockerProbeSummary: - """Probe volatile external blockers for inventory rendering.""" - public = _run_command( - ["git", "ls-remote", "--heads", PUBLIC_REPOSITORY_URL, "HEAD"], +def probe_public_repository( + root: Path, + repository_url: str = PUBLIC_REPOSITORY_URL, +) -> subprocess.CompletedProcess[str]: + """Observe all advertised refs without fetching or mutating either repository.""" + return _run_command( + ["git", "ls-remote", "--refs", repository_url], cwd=root, timeout=20, ) + + +def collect_blocker_probes(root: Path) -> BlockerProbeSummary: + """Probe volatile external blockers for inventory rendering.""" + public = probe_public_repository(root) docker = _run_command(["docker", "info"], cwd=root, timeout=20) return BlockerProbeSummary( public_repository=_summarize_public_repository_probe(public), @@ -132,12 +140,9 @@ def collect_blocker_probes(root: Path) -> BlockerProbeSummary: def _summarize_public_repository_probe( result: subprocess.CompletedProcess[str], ) -> str: - """Report reachability *and* whether the public target has any content. + """Separate transport failure, an empty target and advertised repository refs. - ``git ls-remote`` exits 0 for a repository that exists but holds no refs, so - a bare exit-code check reads an empty placeholder repository as ready. That - is the difference between "the target exists" and "the release is published", - and DOI/archive metadata must stay blank across that gap. + Neither transport success nor advertised refs proves a release/archive/DOI. """ if result.returncode != 0: return _summarize_probe(result) @@ -158,6 +163,22 @@ def render_release_readiness_inventory( public_repository="not refreshed by this validation run", docker_daemon="not refreshed by this validation run", ) + if probes.public_repository == "reachable (refs present)": + public_observation = "The repository is reachable and advertises refs." + elif probes.public_repository.startswith("reachable but empty"): + public_observation = "The repository is reachable but advertises no refs." + elif probes.public_repository.startswith("not refreshed"): + public_observation = ( + "No fresh public-repository observation was made by this validation run." + ) + else: + public_observation = "The public-repository probe failed or was unavailable." + public_release_line = ( + f"- {public_observation} Advertised-ref probe: " + f"`git ls-remote --refs {PUBLIC_REPOSITORY_URL}`. " + f"Current probe: {probes.public_repository}. " + "Release approval, public archive availability and DOI publication require independent evidence." + ) status_count = len(snapshot.status_lines) tracked_count = len(snapshot.modified_tracked_files) untracked_count = len(snapshot.untracked_files) @@ -269,11 +290,7 @@ def render_release_readiness_inventory( "", "## Remaining Release Blockers", "", - ( - "- Public DOI/repository remains blocked until " - f"`git ls-remote {PUBLIC_REPOSITORY_URL}` succeeds. Current probe: " - f"{probes.public_repository}." - ), + public_release_line, live_stack_line, ( "- Cross-vendor security audit evidence is closed as a point-in-time " diff --git a/src/scholarship/source_health_report.py b/src/scholarship/source_health_report.py index 260738b..1609452 100644 --- a/src/scholarship/source_health_report.py +++ b/src/scholarship/source_health_report.py @@ -35,10 +35,6 @@ def scholarship_source_health_network( bot_blocks = _issue_rows(source_health.issue_list, _BOT_BLOCK_CODES) role_gaps = _issue_rows(source_health.issue_list, _ROLE_GAP_CODES) metadata_issues = _issue_rows(source_health.issue_list, _METADATA_CODES) - bot_blocks.extend(_online_status_rows(source_health, "http_error:403")) - dead_links.extend(_online_status_rows(source_health, "url_error:")) - dead_links.extend(_online_status_rows(source_health, "timeout:")) - dead_links.extend(_online_status_rows(source_health, "tls_error:")) drift = { "singleSectionWithoutClaimSupport": list( advisories.get("singleSectionWithoutClaimSupport", []) @@ -104,17 +100,6 @@ def _issue_rows( ] -def _online_status_rows( - audit: SourceHealthAudit, - prefix: str, -) -> list[dict[str, object]]: - return [ - {"code": "online_status", "key": key, "message": status} - for key, status in audit.online_results.items() - if status.startswith(prefix) - ] - - def _boundary_source_row(row: dict[str, Any]) -> dict[str, object]: return { "area": str(row.get("area", "")), diff --git a/src/screenshots/capture.py b/src/screenshots/capture.py index d4cbdbc..1f5ae79 100644 --- a/src/screenshots/capture.py +++ b/src/screenshots/capture.py @@ -15,6 +15,7 @@ import shutil import subprocess import urllib.error +import urllib.parse import urllib.request from dataclasses import dataclass from datetime import UTC, datetime @@ -22,7 +23,13 @@ from typing import Any from src.artifacts.json_io import write_json_artifact -from src.net_guard import NonHttpUrlError, require_http_url +from src.dashboard.browser_validation import ( + BrowserDisplayError, + browser_receipt_complete, + display_probe, + wait_for_official_display, +) +from src.net_guard import NonHttpUrlError, is_loopback_host, require_http_url from src.screenshots.manifest import ( SCREENSHOT_PROVENANCE_FILENAME, VIEWPORT_HEIGHT, @@ -30,6 +37,7 @@ ScreenshotView, screenshot_views, ) +from src.skilltree.local_auth_state import LocalSkillTreeAuthError, _skilltree_origin DEFAULT_BROWSER_COMMAND = "chrome-devtools-axi" #: The browser automation server writes only inside this root on this platform. @@ -54,6 +62,7 @@ class CaptureSettings: viewport_height: int = VIEWPORT_HEIGHT #: Directory the browser automation server is permitted to write to. staging_root: Path = Path(DEFAULT_STAGING_ROOT) + proof_path: Path | None = None def _run( @@ -110,13 +119,69 @@ def ensure_browser_ready(*, browser_command: str) -> bool: return True -def _fetch_json(url: str, *, timeout: float) -> dict[str, Any]: +def _capture_url(url: str) -> str: try: - require_http_url(url, context="dashboard status read for screenshot capture") - except NonHttpUrlError as exc: - raise ScreenshotCaptureError(str(exc)) from exc + parsed = urllib.parse.urlsplit(url) + except ValueError as exc: + raise ScreenshotCaptureError("capture URL is invalid") from exc + if ( + parsed.username is not None + or parsed.password is not None + or parsed.query + or parsed.fragment + ): + raise ScreenshotCaptureError( + "capture requires a credential-free dashboard base URL without query or fragment" + ) try: - with urllib.request.urlopen(url, timeout=timeout) as response: + require_http_url(url, context="screenshot capture") + port = parsed.port + except (NonHttpUrlError, ValueError) as exc: + raise ScreenshotCaptureError( + "capture requires an absolute loopback HTTP(S) URL" + ) from exc + if not parsed.hostname or not is_loopback_host(parsed.hostname): + raise ScreenshotCaptureError( + "capture requires an absolute loopback HTTP(S) URL" + ) + if port is not None and port <= 0: + raise ScreenshotCaptureError("capture URL port is invalid") + return url.rstrip("/") + + +def _capture_target(url: str) -> tuple[tuple[str, str, int], str]: + normalized = _capture_url(url) + return _skilltree_origin(normalized), urllib.parse.urlsplit(normalized).path.rstrip( + "/" + ) or "/" + + +class _LoopbackCaptureRedirect(urllib.request.HTTPRedirectHandler): + def redirect_request( + self, + req: urllib.request.Request, + fp: Any, + code: int, + msg: str, + headers: Any, + newurl: str, + ) -> urllib.request.Request | None: + _capture_url(newurl) + if _skilltree_origin(req.full_url) != _skilltree_origin(newurl): + raise ScreenshotCaptureError( + "capture redirect changed the requested origin" + ) + return super().redirect_request(req, fp, code, msg, headers, newurl) + + +def _capture_opener() -> urllib.request.OpenerDirector: + return urllib.request.build_opener(_LoopbackCaptureRedirect) + + +def _fetch_json(url: str, *, timeout: float) -> dict[str, Any]: + _capture_url(url) + try: + with _capture_opener().open(url, timeout=timeout) as response: payload = json.loads(response.read().decode("utf-8")) except (urllib.error.URLError, TimeoutError, ValueError) as exc: raise ScreenshotCaptureError(f"could not read {url}: {exc}") from exc @@ -125,53 +190,183 @@ def _fetch_json(url: str, *, timeout: float) -> dict[str, Any]: return payload -def capture_context(settings: CaptureSettings) -> dict[str, Any]: - """Return the system state the capture run is evidence of.""" - status = _fetch_json( - f"{settings.dashboard_url.rstrip('/')}/api/status", - timeout=_STATUS_TIMEOUT_SECONDS, - ) - project_counts = status.get("project", {}).get("counts", {}) - live = status.get("liveSkilltree", {}) - proof_path = ( - settings.project_root - / ".active_skillference_local" - / "full_app_stack_proof.json" +def capture_context( + settings: CaptureSettings, + *, + browser_probe: dict[str, Any] | None = None, +) -> dict[str, Any]: + """Observe current status and associate only an explicit, matching successful proof.""" + dashboard_url = _capture_url(settings.dashboard_url) + status = _fetch_json(f"{dashboard_url}/api/status", timeout=_STATUS_TIMEOUT_SECONDS) + project, config, live = ( + status.get("project"), + status.get("config"), + status.get("liveSkilltree"), ) - proof_mode = "not recorded" - if proof_path.is_file(): + service = status.get("skilltree") + if not ( + isinstance(project, dict) + and isinstance(config, dict) + and isinstance(live, dict) + and isinstance(service, dict) + ): + raise ScreenshotCaptureError( + "dashboard project/config/live state must contain objects" + ) + project_counts = project.get("counts", {}) + public_project, skilltree = config.get("project", {}), config.get("skilltree", {}) + if not all( + isinstance(value, dict) for value in (project_counts, public_project, skilltree) + ): + raise ScreenshotCaptureError( + "dashboard project counts/identity/target must contain objects" + ) + instance_id = status.get("instanceId") + digest = project.get("exportContentDigest") + project_id = public_project.get("id") + skilltree_url = str(skilltree.get("url", "")) + if browser_probe is not None: try: - proof = json.loads(proof_path.read_text(encoding="utf-8")) - proof_mode = str(proof.get("mode", "not recorded")) - except (OSError, ValueError): - proof_mode = "unreadable proof artifact" + browser_matches = _capture_target( + str(browser_probe.get("url", "")) + ) == _capture_target(dashboard_url) + except (ScreenshotCaptureError, LocalSkillTreeAuthError, ValueError): + browser_matches = False + if ( + browser_probe.get("state") != "ready" + or browser_probe.get("dashboardInstanceId") != instance_id + or browser_probe.get("exportContentDigest") != digest + or browser_probe.get("projectId") != project_id + or not browser_matches + ): + raise ScreenshotCaptureError( + "browser capture state does not match current dashboard content" + ) + association = _proof_association( + settings, + instance_id=instance_id, + digest=digest, + project_id=project_id, + skilltree_url=skilltree_url, + browser_probe=browser_probe, + ) + content = live.get("content", {}) return { - "dashboardUrl": settings.dashboard_url, - "skilltreeUrl": str( - status.get("config", {}).get("skilltree", {}).get("url", "") - ), - "skilltreeAvailable": bool(status.get("skilltree", {}).get("available")), + "dashboardUrl": dashboard_url, + "dashboardInstanceId": instance_id, + "exportContentDigest": digest, + "projectId": project_id, + "skilltreeUrl": skilltree_url, + "skilltreeAvailable": bool(service.get("available")), "viewport": { "width": settings.viewport_width, "height": settings.viewport_height, }, "exportedCounts": { - "skills": project_counts.get("skills", 0), - "subjects": project_counts.get("subjects", 0), - "dependencies": project_counts.get("dependencies", 0), - "quizzes": project_counts.get("quizzes", 0), + key: project_counts.get(key, 0) + for key in ("skills", "subjects", "dependencies", "quizzes") }, "liveDependencies": { - "checked": bool(live.get("checked")), + "checked": live.get("checked") is True, "matched": live.get("matchedDependencies"), "expected": live.get("expectedDependencies"), - "complete": bool(live.get("complete")), + "complete": live.get("complete") is True, }, - "fullAppStackProofMode": proof_mode, - "evidenceKind": ( - "captured browser recording of a live system; pixels are not " - "reproducible byte-for-byte and are not a deterministic render" - ), + "liveContent": { + key: content.get(key) + for key in ("checked", "complete", "exportContentDigest") + } + if isinstance(content, dict) + else {"checked": False}, + "officialDisplay": { + key: browser_probe.get(key) + for key in ("state", "route", "frameUrl", "frameVisible") + } + if browser_probe is not None + else {"state": "unobserved"}, + "proofAssociation": association, + "fullAppStackProofMode": association.get("mode", "not associated"), + "evidenceKind": "captured browser recording of a live system; pixels are not reproducible byte-for-byte and are not a deterministic render", + } + + +def _proof_association( + settings: CaptureSettings, + *, + instance_id: Any, + digest: Any, + project_id: Any, + skilltree_url: str, + browser_probe: dict[str, Any] | None, +) -> dict[str, Any]: + if settings.proof_path is None: + return {"checked": False, "reason": "no explicit proof path"} + path = settings.proof_path + if not path.is_absolute(): + path = settings.project_root / path + try: + raw = path.read_bytes() + proof = json.loads(raw) + except (OSError, ValueError) as exc: + return { + "checked": False, + "reason": f"proof is unreadable: {type(exc).__name__}", + } + if not isinstance(proof, dict) or proof.get("proofVersion") != 2: + return { + "checked": False, + "reason": "historical/unsupported proof schema is unassociated", + } + content = proof.get("liveContent", {}) + browser = proof.get("browserConsole", {}) + try: + same_targets = _capture_target( + str(proof.get("dashboardUrl", "")) + ) == _capture_target(settings.dashboard_url) and _skilltree_origin( + str(proof.get("skilltreeUrl", "")) + ) == _skilltree_origin(skilltree_url) + except (ScreenshotCaptureError, LocalSkillTreeAuthError, ValueError): + same_targets = False + complete = ( + proof.get("ok") is True + and same_targets + and proof.get("mode") in ("app-ready", "full-parity") + and isinstance(instance_id, str) + and bool(instance_id) + and isinstance(digest, str) + and len(digest) == 64 + and proof.get("runId") == proof.get("dashboardInstanceId") == instance_id + and proof.get("exportContentDigest") == digest + and proof.get("projectId") == project_id + and isinstance(content, dict) + and content.get("checked") is True + and content.get("complete") is True + and content.get("exportContentDigest") == digest + and browser_probe is not None + and browser_receipt_complete(browser) + and browser.get("dashboardInstanceId") == instance_id + and browser.get("exportContentDigest") == digest + and browser.get("projectId") == project_id + ) + if proof.get("mode") == "full-parity": + dependencies = proof.get("liveDependencies", {}) + complete = ( + complete + and isinstance(dependencies, dict) + and dependencies.get("checked") is True + and dependencies.get("complete") is True + ) + if not complete: + return { + "checked": False, + "reason": "proof does not match this successful current browser/content/instance state", + } + return { + "checked": True, + "mode": proof["mode"], + "runId": proof["runId"], + "proofSha256": hashlib.sha256(raw).hexdigest(), + "proofPath": str(path), } @@ -182,6 +377,11 @@ def _capture_one( ) -> dict[str, Any]: _run([settings.browser_command, "eval", view.setup_js]) _run([settings.browser_command, "wait", str(view.settle_ms)]) + try: + probe = display_probe(browser_command=settings.browser_command, timeout=30.0) + except BrowserDisplayError as exc: + raise ScreenshotCaptureError(str(exc)) from exc + context = capture_context(settings, browser_probe=probe) # The browser automation server runs under its own filesystem sandbox and # silently reports success while writing nothing outside /tmp, so captures # are staged there and moved into the tracked tree by this process. @@ -192,6 +392,23 @@ def _capture_one( f"browser reported success but {staged} was not written; the browser " "automation server may not be permitted to write to that directory" ) + try: + after = display_probe(browser_command=settings.browser_command, timeout=30.0) + except BrowserDisplayError as exc: + raise ScreenshotCaptureError(str(exc)) from exc + if any( + after.get(key) != probe.get(key) + for key in ( + "state", + "route", + "url", + "dashboardInstanceId", + "exportContentDigest", + ) + ): + raise ScreenshotCaptureError( + "browser state changed while the screenshot was being recorded" + ) target = settings.output_dir / view.filename target.write_bytes(staged.read_bytes()) staged.unlink() @@ -203,14 +420,18 @@ def _capture_one( "settleMs": view.settle_ms, "sha256": digest, "bytes": target.stat().st_size, + "context": context, } def capture_screenshots(settings: CaptureSettings) -> dict[str, Any]: """Capture every manifest view and write the registry plus provenance record.""" - settings.output_dir.mkdir(parents=True, exist_ok=True) + dashboard_url = _capture_url(settings.dashboard_url) + with _capture_opener().open( + dashboard_url, timeout=_STATUS_TIMEOUT_SECONDS + ) as response: + _capture_url(response.geturl()) restarted = ensure_browser_ready(browser_command=settings.browser_command) - context = capture_context(settings) _run( [ settings.browser_command, @@ -220,9 +441,26 @@ def capture_screenshots(settings: CaptureSettings) -> dict[str, Any]: ] ) _run([settings.browser_command, "open", settings.dashboard_url]) - # The page defers the live-parity read, so give the swap-in time to land - # before the first capture claims a parity number. - _run([settings.browser_command, "wait", "12000"]) + initial = _fetch_json( + f"{dashboard_url}/api/status", timeout=_STATUS_TIMEOUT_SECONDS + ) + try: + probe = wait_for_official_display( + browser_command=settings.browser_command, + dashboard_url=dashboard_url, + skilltree_url=str(initial["config"]["skilltree"]["url"]), + project_id=str(initial["config"]["project"]["id"]), + route="/", + dashboard_instance_id=str(initial["instanceId"]), + export_digest=str(initial["project"]["exportContentDigest"]), + timeout=_STATUS_TIMEOUT_SECONDS, + ) + except (BrowserDisplayError, KeyError, TypeError) as exc: + raise ScreenshotCaptureError( + "official display did not provide capture-ready identity" + ) from exc + context = capture_context(settings, browser_probe=probe) + settings.output_dir.mkdir(parents=True, exist_ok=True) staging_dir = Path(settings.staging_root) / f"asf-capture-{os.getpid()}" staging_dir.mkdir(parents=True, exist_ok=True) diff --git a/src/skilltree/client.py b/src/skilltree/client.py index a8cccc3..b21b21b 100644 --- a/src/skilltree/client.py +++ b/src/skilltree/client.py @@ -8,6 +8,7 @@ from __future__ import annotations import json +import math import urllib.error import urllib.request from typing import Any @@ -28,6 +29,7 @@ subject_create_path, subject_payload, ) +from src.skilltree.content import missing_quiz_questions from src.skilltree.model import ( Quiz, QuizQuestion, @@ -49,12 +51,19 @@ class SkillTreeClient: """Thin admin/report client for a single SkillTree project.""" def __init__( - self, base_url: str, project_id: str, token: str | None = None + self, + base_url: str, + project_id: str, + token: str | None = None, + *, + timeout: float = 30.0, ) -> None: if not base_url.strip(): raise ValueError("base_url must be non-empty") if not project_id.strip(): raise ValueError("project_id must be non-empty") + if not math.isfinite(timeout) or timeout <= 0: + raise ValueError("timeout must be finite and positive") try: self.base_url = require_http_url( base_url.rstrip("/"), context="SkillTree client" @@ -63,6 +72,7 @@ def __init__( raise ValueError(str(exc)) from exc self.project_id = project_id self.token = token + self.timeout = timeout self._opener = urllib.request.build_opener(NoCrossOriginBearerRedirect) def _get(self, path: str) -> Any: @@ -72,7 +82,7 @@ def _get(self, path: str) -> Any: headers["Authorization"] = f"Bearer {self.token}" request = urllib.request.Request(url, headers=headers, method="GET") try: - with self._opener.open(request) as response: + with self._opener.open(request, timeout=self.timeout) as response: status = response.status raw = response.read().decode("utf-8") except urllib.error.HTTPError as exc: @@ -82,6 +92,8 @@ def _get(self, path: str) -> Any: status=exc.code, body=raw, ) from exc + except TimeoutError as exc: + raise SkillTreeClientError(f"GET {path} timed out") from exc except urllib.error.URLError as exc: # pragma: no cover - network-dependent raise SkillTreeClientError( f"GET {path} transport error: {exc.reason}" @@ -101,7 +113,7 @@ def _post(self, path: str, body: dict[str, Any] | None = None) -> Any: headers["Authorization"] = f"Bearer {self.token}" request = urllib.request.Request(url, data=data, headers=headers, method="POST") try: - with self._opener.open(request) as response: + with self._opener.open(request, timeout=self.timeout) as response: status = response.status raw = response.read().decode("utf-8") except urllib.error.HTTPError as exc: @@ -111,6 +123,8 @@ def _post(self, path: str, body: dict[str, Any] | None = None) -> Any: status=exc.code, body=raw, ) from exc + except TimeoutError as exc: + raise SkillTreeClientError(f"POST {path} timed out") from exc except urllib.error.URLError as exc: # pragma: no cover - network-dependent raise SkillTreeClientError( f"POST {path} transport error: {exc.reason}" @@ -152,13 +166,8 @@ def create_quiz(self, quiz: Quiz) -> Any: return result def _ensure_quiz_questions(self, quiz: Quiz) -> None: - if not quiz.questions: - return existing = self._get(quiz_questions_path(quiz.quiz_id)) - existing_count = min(quiz_question_count(existing), len(quiz.questions)) - if existing_count >= len(quiz.questions): - return - for question in quiz.questions[existing_count:]: + for question in missing_quiz_questions(existing, quiz.questions): self.create_quiz_question(quiz.quiz_id, question) def create_quiz_question(self, quiz_id: str, question: QuizQuestion) -> Any: diff --git a/src/skilltree/content.py b/src/skilltree/content.py new file mode 100644 index 0000000..c7d2e6a --- /dev/null +++ b/src/skilltree/content.py @@ -0,0 +1,299 @@ +"""Semantic, multiplicity-aware comparison of exported and live admin content. + +This module never updates or deletes live resources. Missing resources can be +created by the existing seeder; changed or unverifiable resources fail closed. +Only identifiers, counts and a digest leave the admin boundary, never answer keys. +""" + +from __future__ import annotations + +import hashlib +import json +from collections import Counter +from collections.abc import Callable, Mapping, Sequence +from typing import Any + +from src.skilltree.admin_ops import ( + project_create_path, + quiz_definition_path, + quiz_questions_path, + skill_create_path, + subject_create_path, +) +from src.skilltree.local_auth_state import _skilltree_origin +from src.skilltree.model import QuizQuestion +from src.skilltree.summary import validate_export_payload +from src.skilltree.transport import SkillTreeClientError + + +class SkillTreeContentError(SkillTreeClientError): + """Live content cannot safely be identified with an exported resource.""" + + +def export_content_digest(project: Mapping[str, Any]) -> str: + """Bind a receipt to the complete private export, including answer correctness.""" + validate_export_payload(project) + raw = json.dumps(project, sort_keys=True, separators=(",", ":"), ensure_ascii=False) + return hashlib.sha256(raw.encode("utf-8")).hexdigest() + + +def _object(payload: Any, resource: str) -> Mapping[str, Any]: + if isinstance(payload, Mapping): + data = payload.get("data") + if isinstance(data, Mapping): + return data + return payload + raise SkillTreeContentError(f"{resource} returned no verifiable content object") + + +def _rows(payload: Any, key: str, resource: str) -> list[Mapping[str, Any]]: + if isinstance(payload, Mapping): + if key in payload: + payload = payload[key] + elif "data" in payload: + return _rows(payload["data"], key, resource) + if not isinstance(payload, list) or any( + not isinstance(row, Mapping) for row in payload + ): + raise SkillTreeContentError(f"{resource} returned no verifiable {key} list") + return payload + + +def _question_key(question: Mapping[str, Any]) -> str: + text = question.get("question") + question_type = question.get("questionType") + answers = question.get("answers") + if ( + not isinstance(text, str) + or not text.strip() + or not isinstance(question_type, str) + or question_type not in {"SingleChoice", "MultipleChoice", "TextInput"} + or not isinstance(answers, list) + or not answers + ): + raise SkillTreeContentError( + "quiz question response lacks verifiable semantic content" + ) + normalized_answers = [] + for answer in answers: + if ( + not isinstance(answer, Mapping) + or not isinstance(answer.get("answer"), str) + or not isinstance(answer.get("isCorrect"), bool) + ): + raise SkillTreeContentError( + "quiz answer response lacks verifiable correctness" + ) + normalized_answers.append((answer["answer"], answer["isCorrect"])) + return json.dumps( + [text, question_type, sorted(normalized_answers)], + ensure_ascii=False, + separators=(",", ":"), + ) + + +def missing_quiz_questions( + payload: Any, + expected: Sequence[QuizQuestion], +) -> list[QuizQuestion]: + """Return missing question occurrences; refuse extras or changed answer keys.""" + current = Counter(_question_key(row) for row in _rows(payload, "questions", "quiz")) + wanted = Counter(_question_key(question.to_dict()) for question in expected) + if current - wanted: + raise SkillTreeContentError( + "live quiz contains extra or changed questions; existing content was not overwritten" + ) + missing = [] + for question in expected: + key = _question_key(question.to_dict()) + if current[key]: + current[key] -= 1 + else: + missing.append(question) + return missing + + +def _index( + rows: list[Mapping[str, Any]], field: str, resource: str +) -> dict[str, Mapping[str, Any]]: + result = {} + for row in rows: + identifier = row.get(field) + if not isinstance(identifier, str) or not identifier or identifier in result: + raise SkillTreeContentError(f"{resource} has missing or duplicate {field}") + result[identifier] = row + return result + + +def _quiz_completion_contract(payload: Any, question_count: int) -> tuple[int, int]: + """Resolve native 4.6.0 length/pass-count settings, never an export-only percent.""" + rows = _index(_rows(payload, "settings", "quiz"), "setting", "quiz settings") + + def integer(name: str) -> int: + if name not in rows: + return -1 + value = rows[name].get("value") + if not isinstance(value, str): + raise SkillTreeContentError(f"quiz setting {name} is not a string integer") + try: + return int(value) + except ValueError as exc: + raise SkillTreeContentError( + f"quiz setting {name} is not an integer" + ) from exc + + length = integer("quizLength") + passing = integer("quizPassingReq") + if length == -1: + length = question_count + if ( + length <= 0 + or length > question_count + or passing < -1 + or passing > question_count + ): + raise SkillTreeContentError("quiz length/pass-count settings are unverifiable") + return length, passing if passing > 0 else length + + +def compare_live_content( + *, + get: Callable[[str], Any], + base_url: str, + project: Mapping[str, Any], +) -> dict[str, Any]: + """Read actual admin content and return a browser-safe comparison receipt. + + A count-only response is not evidence of content parity. Server-generated + identifiers and quiz ordering are ignored; duplicate question/answer + occurrences and correctness flags are not. Global unrelated quiz definitions + are outside the project boundary; every exported quiz is checked explicitly. + """ + digest = export_content_digest(project) + project_id = str(project["projectId"]) + root = project_create_path(project_id) + live_project = _object(get(root), "project") + if live_project.get("projectId") != project_id or not isinstance( + live_project.get("name"), str + ): + raise SkillTreeContentError("live project identity/name is unverifiable") + missing: list[str] = [] + extra: list[str] = [] + changed: list[str] = [] + if live_project["name"] != project["name"]: + changed.append(f"project:{project_id}") + live_subjects = _index( + _rows(get(f"{root}/subjects"), "subjects", "project"), "subjectId", "project" + ) + expected_subjects = { + subject["subjectId"]: subject for subject in project["subjects"] + } + missing.extend( + f"subject:{identifier}" + for identifier in expected_subjects.keys() - live_subjects.keys() + ) + extra.extend( + f"subject:{identifier}" + for identifier in live_subjects.keys() - expected_subjects.keys() + ) + live_skill_count = 0 + for subject_id, subject in expected_subjects.items(): + if subject_id not in live_subjects: + continue + subject_path = subject_create_path(project_id, subject_id) + live_subject = _object(get(subject_path), "subject") + fields: tuple[str, ...] = ("subjectId", "name", "description") + if any(field not in live_subject for field in fields): + raise SkillTreeContentError(f"subject:{subject_id} content is unverifiable") + if any(live_subject[field] != subject.get(field, "") for field in fields): + changed.append(f"subject:{subject_id}") + live_skills = _index( + _rows(get(f"{subject_path}/skills"), "skills", "subject"), + "skillId", + "subject", + ) + wanted_skills = {skill["skillId"]: skill for skill in subject["skills"]} + live_skill_count += len(live_skills) + missing.extend( + f"skill:{identifier}" + for identifier in wanted_skills.keys() - live_skills.keys() + ) + extra.extend( + f"skill:{identifier}" + for identifier in live_skills.keys() - wanted_skills.keys() + ) + for skill_id, skill in wanted_skills.items(): + if skill_id not in live_skills: + continue + live_skill = _object( + get(skill_create_path(project_id, subject_id, skill_id)), "skill" + ) + fields = ( + "skillId", + "name", + "description", + "pointIncrement", + "numPerformToCompletion", + "helpUrl", + "selfReportingType", + ) + if any(field not in live_skill for field in fields): + raise SkillTreeContentError(f"skill:{skill_id} content is unverifiable") + if any( + live_skill[field] != skill.get(field) for field in fields + ) or live_skill.get("quizId", "") != skill.get("quizId", ""): + changed.append(f"skill:{skill_id}") + for quiz in project["quizzes"]: + quiz_id = quiz["quizId"] + try: + live_quiz = _object(get(quiz_definition_path(quiz_id)), "quiz") + except SkillTreeClientError as exc: + if exc.status != 404: + raise + missing.append(f"quiz:{quiz_id}") + continue + fields = ("quizId", "name", "type") + if any(field not in live_quiz for field in fields): + raise SkillTreeContentError(f"quiz:{quiz_id} definition is unverifiable") + if any(live_quiz[field] != quiz.get(field) for field in fields): + changed.append(f"quiz:{quiz_id}") + question_count = len(quiz["questions"]) + percent = quiz.get("passingPercent") + if type(percent) is not int or not 1 <= percent <= 100: + raise SkillTreeContentError( + f"quiz:{quiz_id} export pass percentage is invalid" + ) + # Normalize defaults for the intended completed quiz. Missing questions + # still make the receipt incomplete below and can be seeded without + # misclassifying native default settings as an existing-content change. + length, required = _quiz_completion_contract( + get(f"{quiz_definition_path(quiz_id)}/settings"), + question_count, + ) + if (length, required) != ( + question_count, + (question_count * percent + 99) // 100, + ): + changed.append(f"quiz:{quiz_id}") + current = Counter( + _question_key(row) + for row in _rows(get(quiz_questions_path(quiz_id)), "questions", "quiz") + ) + wanted = Counter(_question_key(row) for row in quiz["questions"]) + if current - wanted: + changed.append(f"quiz-questions:{quiz_id}") + if wanted - current: + missing.append(f"quiz-questions:{quiz_id}") + scheme, host, port = _skilltree_origin(base_url) + return { + "checked": True, + "complete": not (missing or extra or changed), + "targetOrigin": {"scheme": scheme, "host": host, "port": port}, + "projectId": project_id, + "exportContentDigest": digest, + "subjects": len(live_subjects), + "skills": live_skill_count, + "missingResources": sorted(missing), + "extraResources": sorted(extra), + "changedResources": sorted(set(changed)), + } diff --git a/src/skilltree/deployment_matrix.py b/src/skilltree/deployment_matrix.py index c3de30d..f4b2e33 100644 --- a/src/skilltree/deployment_matrix.py +++ b/src/skilltree/deployment_matrix.py @@ -44,7 +44,7 @@ def skilltree_deployment_matrix( else "ready", }, { - "browserExposure": "same-origin token endpoint URL, never the token value", + "browserExposure": "same-origin endpoint and a temporary learner bearer token in browser memory", "evidence": ( "trusted-client bridge configured" if local_auth_ready and authenticator_configured @@ -52,7 +52,7 @@ def skilltree_deployment_matrix( ), "id": "token_backed_admin_seeding", "label": "Token-backed admin seeding", - "secretBoundary": "Client secret and admin credentials are exchanged server-side only.", + "secretBoundary": "Client secret and admin credentials stay server-side; the learner token is issued to browser memory.", "status": ( "ready" if local_auth_ready and authenticator_configured else "blocked" ), diff --git a/src/skilltree/local_auth.py b/src/skilltree/local_auth.py index e2e2f54..ad7b640 100644 --- a/src/skilltree/local_auth.py +++ b/src/skilltree/local_auth.py @@ -1,15 +1,15 @@ """Local-only SkillTree FORM auth bootstrap and trusted-client token bridge. -This module intentionally keeps credentials server-side. It creates or reuses a -single local root/admin account, stores local-only secrets in an ignored state -directory, and exchanges the SkillTree project trusted-client secret for learner -tokens. Browser code only ever sees the dashboard token endpoint path. +Root credentials and trusted-client secrets stay server-side in ignored local +state. The dashboard token bridge issues a temporary learner bearer token to +browser memory for the official SkillTree client; that token is not persisted. """ from __future__ import annotations import base64 import json +import math import os import sys import urllib.error @@ -31,6 +31,7 @@ subject_create_path, subject_payload, ) +from src.skilltree.content import compare_live_content, missing_quiz_questions from src.skilltree.local_auth_state import ( DEFAULT_LOCAL_ADMIN_PASSWORD, DEFAULT_LOCAL_ADMIN_USERNAME, @@ -69,7 +70,7 @@ SkillTreeSubject, ) from src.skilltree.seeder import seed -from src.skilltree.transport import SkillTreeClientError, quiz_question_count +from src.skilltree.transport import SkillTreeClientError, parse_json_body # Backwards-compatible re-export surface pulled from local_auth_state so downstream # callers can import these names from src.skilltree.local_auth. Declared explicitly @@ -104,6 +105,8 @@ def __init__( raise ValueError("base_url must be non-empty") if not project_id.strip(): raise ValueError("project_id must be non-empty") + if not math.isfinite(timeout) or timeout <= 0: + raise ValueError("timeout must be finite and positive") self.base_url = base_url.rstrip("/") self.project_id = project_id self.timeout = timeout @@ -133,7 +136,7 @@ def request_json( form_body=form_body, headers=headers, ) - return json.loads(raw) if raw.strip() else {} + return parse_json_body(raw) def request_text( self, @@ -432,16 +435,11 @@ def create_quiz(self, quiz: Quiz) -> Any: return result def _ensure_quiz_questions(self, quiz: Quiz) -> None: - """Create any exported quiz questions missing from the live quiz.""" - if not quiz.questions: - return + """Create missing semantic question occurrences, never infer a prefix by count.""" existing = self.request_json( "GET", f"/admin/quiz-definitions/{quiz.quiz_id}/questions" ) - existing_count = min(quiz_question_count(existing), len(quiz.questions)) - if existing_count >= len(quiz.questions): - return - for question in quiz.questions[existing_count:]: + for question in missing_quiz_questions(existing, quiz.questions): self.create_quiz_question(quiz.quiz_id, question) def create_quiz_question(self, quiz_id: str, question: QuizQuestion) -> Any: @@ -550,17 +548,20 @@ def verify_seed_landed( "server-side rejection swallowed as idempotent). Inspect for duplicate " "names / invalid ids before trusting the live instance." ) + if not live_project_content(client, project)["complete"]: + raise LocalSkillTreeAuthError( + "seed verification failed: live content differs from the exported project" + ) -def live_project_counts_match( +def live_project_content( client: SkillTreeSessionClient, project: SkillTreeProject -) -> bool: - """Return whether live SkillTree already has this export's core resources.""" - expected_skills = sum(len(subject.skills) for subject in project.subjects) - expected_subjects = len(project.subjects) - return ( - client.live_skill_count() == expected_skills - and client.live_subject_count() == expected_subjects +) -> dict[str, Any]: + """Compare identities, learner content and answer keys, not resource counts.""" + return compare_live_content( + get=lambda path: client.request_json("GET", path), + base_url=client.base_url, + project=project.to_export_dict(), ) @@ -651,19 +652,25 @@ def _seed_project_if_requested( project: SkillTreeProject | None, seed_project: bool, seed_dependencies: bool, - skip_seed_if_live_counts_ready: bool, + skip_seed_if_live_content_ready: bool, verify_seed: bool, ) -> tuple[bool, dict[str, int] | None]: if not seed_project or project is None: return False, None - should_seed = True - if skip_seed_if_live_counts_ready: - try: - should_seed = not live_project_counts_match(client, project) - except (LocalSkillTreeAuthError, SkillTreeClientError): - should_seed = True - if not should_seed: - return False, None + try: + content = live_project_content(client, project) + except SkillTreeClientError as exc: + if exc.status != 404: + raise + content = None + if content is not None: + if content["changedResources"] or content["extraResources"]: + raise LocalSkillTreeAuthError( + "live SkillTree content differs from the export; existing content " + "was not overwritten. Review the reported drift before reseeding." + ) + if skip_seed_if_live_content_ready and content["complete"]: + return False, None seed_summary = seed(client, project, include_dependencies=seed_dependencies) if verify_seed: verify_seed_landed(client, project) @@ -723,11 +730,15 @@ def ensure_local_skilltree_auth( project: SkillTreeProject | None = None, seed_project: bool = True, seed_dependencies: bool = True, - skip_seed_if_live_counts_ready: bool = False, + skip_seed_if_live_content_ready: bool = False, verify_seed: bool = True, verify_token: bool = True, ) -> BootstrapResult: """Bootstrap local root auth, seed the project, and store client credentials.""" + if project is not None and project.project_id != project_id: + raise LocalSkillTreeAuthError( + "exported project id does not match the requested bootstrap target" + ) client = SkillTreeSessionClient( base_url, project_id, timeout=default_local_request_timeout() ) @@ -744,7 +755,7 @@ def ensure_local_skilltree_auth( project=project, seed_project=seed_project, seed_dependencies=seed_dependencies, - skip_seed_if_live_counts_ready=skip_seed_if_live_counts_ready, + skip_seed_if_live_content_ready=skip_seed_if_live_content_ready, verify_seed=verify_seed, ) state = _project_client_secret_state( @@ -851,6 +862,8 @@ def exchange_trusted_client_token( status=exc.code, body=raw, ) from exc + except TimeoutError as exc: + raise SkillTreeClientError("POST /oauth/token timed out") from exc except urllib.error.URLError as exc: # pragma: no cover - network dependent raise SkillTreeClientError( f"POST /oauth/token transport error: {exc.reason}" @@ -862,7 +875,12 @@ def exchange_trusted_client_token( raise LocalSkillTreeAuthError( "SkillTree token endpoint returned an empty response" ) - payload = json.loads(raw) + try: + payload = json.loads(raw) + except json.JSONDecodeError as exc: + raise LocalSkillTreeAuthError( + "SkillTree token endpoint returned malformed JSON" + ) from exc if not isinstance(payload, dict): raise LocalSkillTreeAuthError( "SkillTree token endpoint returned non-object JSON" diff --git a/src/skilltree/transport.py b/src/skilltree/transport.py index 6c629c4..84afd8b 100644 --- a/src/skilltree/transport.py +++ b/src/skilltree/transport.py @@ -49,7 +49,10 @@ def redirect_request( # type: ignore[override] def parse_json_body(raw: str) -> Any: - return json.loads(raw) if raw.strip() else {} + try: + return json.loads(raw) if raw.strip() else {} + except json.JSONDecodeError as exc: + raise SkillTreeClientError("SkillTree returned malformed JSON") from exc def quiz_question_count(payload: Any) -> int: diff --git a/tests/README.md b/tests/README.md index fe61174..6c0c64d 100644 --- a/tests/README.md +++ b/tests/README.md @@ -16,6 +16,11 @@ uv run pytest tests/ --cov=src --cov-fail-under=90 Tests import from the `src.*` namespace (`pythonpath = [".", "src"]`); `conftest.py` also puts `src/` on `sys.path` and forces `MPLBACKEND=Agg` so figure tests run headless. +Script smoke tests copy the public source inputs into a module-scoped temporary +project before running the real orchestrators. Generated artifacts and web-math +fixtures stay there; running the suite must not regenerate the audited checkout's +tracked output or alter its source binding. + ## Coverage facts - The authoritative command is `uv run pytest tests/ --cov=src --cov-fail-under=90`. @@ -24,6 +29,14 @@ puts `src/` on `sys.path` and forces `MPLBACKEND=Agg` so figure tests run headle - **Zero mocks.** Real numeric computation everywhere; `pytest-httpserver` provides a real local HTTP server for SkillTree REST boundaries. +Distribution regressions include extreme finite scores at small inverse temperature, +where centering before scaling can overflow. Claim-rendering regressions also +require the rendered graph to retain unique subject and skill names before export. + +The admin-content fixture serves the pinned SkillTree quiz definition and its +separate settings endpoint. Pass requirements are native correct-question counts; +missing/default settings must not be mistaken for a missing export-only percentage. + ## Test Files | File | Covers | diff --git a/tests/conftest.py b/tests/conftest.py index f3f8308..e24f2d5 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -3,6 +3,8 @@ import os import sys +import pytest + # Force headless backend for matplotlib in tests os.environ.setdefault("MPLBACKEND", "Agg") @@ -15,3 +17,265 @@ for _path in (ROOT, SRC): if _path not in sys.path: sys.path.insert(0, _path) + + +@pytest.fixture +def skilltree_admin_content(): + """Serve actual model content over loopback, including private answer keys.""" + + def serve(httpserver, project, *, initially_missing=False, quiz_settings=None): + payload = ( + project.to_export_dict() if hasattr(project, "to_export_dict") else project + ) + root = f"/admin/projects/{payload['projectId']}" + import json + + from werkzeug.wrappers import Response + + root_requests = 0 + + def project_response(request): + nonlocal root_requests + root_requests += 1 + missing = initially_missing and root_requests == 1 + body = ( + {"error": "not found"} + if missing + else { + "projectId": payload["projectId"], + "name": payload["name"], + "numSubjects": len(payload["subjects"]), + "numSkills": sum( + len(subject["skills"]) for subject in payload["subjects"] + ), + } + ) + return Response( + json.dumps(body), + status=404 if missing else 200, + content_type="application/json", + ) + + httpserver.expect_request(root, method="GET").respond_with_handler( + project_response + ) + httpserver.expect_request(f"{root}/subjects", method="GET").respond_with_json( + payload["subjects"] + ) + for subject in payload["subjects"]: + subject_path = f"{root}/subjects/{subject['subjectId']}" + httpserver.expect_request(subject_path, method="GET").respond_with_json( + subject + ) + httpserver.expect_request( + f"{subject_path}/skills", method="GET" + ).respond_with_json(subject["skills"]) + for skill in subject["skills"]: + httpserver.expect_request( + f"{subject_path}/skills/{skill['skillId']}", + method="GET", + ).respond_with_json(skill) + for quiz in payload["quizzes"]: + quiz_path = f"/admin/quiz-definitions/{quiz['quizId']}" + httpserver.expect_request(quiz_path, method="GET").respond_with_json( + {key: quiz[key] for key in ("quizId", "name", "type")} + ) + httpserver.expect_request( + f"{quiz_path}/settings", method="GET" + ).respond_with_json((quiz_settings or {}).get(quiz["quizId"], [])) + questions = [] if initially_missing else list(quiz["questions"]) + + def read_questions(_request, rows=questions): + return Response( + json.dumps({"questions": rows}), + content_type="application/json", + ) + + def create_question(request, rows=questions): + rows.append(request.get_json()) + return Response("{}", content_type="application/json") + + httpserver.expect_request( + f"{quiz_path}/questions", method="GET" + ).respond_with_handler(read_questions) + if initially_missing: + httpserver.expect_request( + f"{quiz_path}/create-question", + method="POST", + ).respond_with_handler(create_question) + + return serve + + +@pytest.fixture +def content_project(): + from src.skilltree.model import ( + Quiz, + QuizQuestion, + SkillTreeProject, + SkillTreeSkill, + SkillTreeSubject, + ) + + quiz = Quiz( + quiz_id="q", + name="Quiz", + questions=[ + QuizQuestion( + question="First?", answers=[{"answer": "one", "isCorrect": True}] + ), + QuizQuestion( + question="Second?", answers=[{"answer": "two", "isCorrect": True}] + ), + ], + ) + return SkillTreeProject( + project_id="p", + name="Project", + subjects=[ + SkillTreeSubject( + subject_id="s", + name="Subject", + description="Subject content", + skills=[ + SkillTreeSkill( + skill_id="k", + name="Skill", + description="Lesson content", + self_reporting_type="Quiz", + quiz_id="q", + ) + ], + ) + ], + quizzes=[quiz], + ) + + +@pytest.fixture +def pdf_bytes(): + """Build a structurally valid one-page PDF, not a copied historical output.""" + + def make(text="Fixture image"): + escaped = text.replace("\\", "\\\\").replace("(", "\\(").replace(")", "\\)") + stream = f"BT /F1 18 Tf 20 120 Td ({escaped}) Tj ET\n".encode("ascii") + objects = [ + b"<< /Type /Catalog /Pages 2 0 R >>", + b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>", + b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 300 200] /Resources << /Font << /F1 4 0 R >> >> /Contents 5 0 R >>", + b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>", + b"<< /Length " + + str(len(stream)).encode("ascii") + + b" >>\nstream\n" + + stream + + b"endstream", + ] + content = bytearray(b"%PDF-1.4\n%\xe2\xe3\xcf\xd3\n") + offsets = [] + for index, body in enumerate(objects, 1): + offsets.append(len(content)) + content.extend(f"{index} 0 obj\n".encode("ascii") + body + b"\nendobj\n") + xref = len(content) + content.extend( + f"xref\n0 {len(objects) + 1}\n0000000000 65535 f \n".encode("ascii") + ) + for offset in offsets: + content.extend(f"{offset:010d} 00000 n \n".encode("ascii")) + content.extend( + f"trailer\n<< /Root 1 0 R /Size {len(objects) + 1} >>\nstartxref\n{xref}\n%%EOF\n".encode( + "ascii" + ) + ) + return bytes(content) + + return make + + +@pytest.fixture +def pdf_project(tmp_path, pdf_bytes): + """Create disposable source inputs and observe their real local Git revision.""" + import os + import subprocess + + from src.manuscript.pdf_receipt import PDF_NAME, add_pdf_assets, capture_pdf_inputs + + root = tmp_path / "publication" + manuscript = root / "manuscript" + hydrated = root / "output" / "manuscript" + manuscript.mkdir(parents=True) + hydrated.mkdir(parents=True) + assets_dir = hydrated / "assets" + assets_dir.mkdir() + assets = [assets_dir / name for name in ("cover.pdf", "atlas.pdf", "body.pdf")] + for path, label in zip( + assets, ("CoverGraphic", "AtlasGraphic", "BodyGraphic"), strict=True + ): + path.write_bytes(pdf_bytes(label)) + (manuscript / "config.yaml").write_text( + "paper:\n title: FixtureTitlePage\n" + "authors:\n - name: Fixture Author\n" + "book:\n cover:\n image: assets/cover.pdf\n" + "front_matter:\n page_two_visual:\n image: assets/atlas.pdf\n" + " title: FixtureAtlasPage\n caption: FixtureAtlasCaption\n", + encoding="utf-8", + ) + (manuscript / "preamble.md").write_text( + "```latex\n\\usepackage{graphicx}\n```\n", encoding="utf-8" + ) + (manuscript / "references.bib").write_text( + "@book{fixture,\n title={Fixture Publication},\n author={Example, Test},\n" + " year={2020},\n publisher={Fixture Press}\n}\n", + encoding="utf-8", + ) + section = hydrated / "01_fixture.md" + section.write_text( + f"# FixtureBodySection\n\nA cited statement [@fixture].\n\n![Body image]({assets[2].as_posix()})\n", + encoding="utf-8", + ) + target = root / "output" / "pdf" / PDF_NAME + target.parent.mkdir() + target.write_bytes(pdf_bytes("Finalized fixture PDF")) + git_env = { + key: value for key, value in os.environ.items() if not key.startswith("GIT_") + } + git_env.update(GIT_CONFIG_NOSYSTEM="1", GIT_CONFIG_GLOBAL=os.devnull) + subprocess.run(["git", "init", "--quiet", str(root)], env=git_env, check=True) + subprocess.run(["git", "add", "."], cwd=root, env=git_env, check=True) + subprocess.run( + [ + "git", + "-c", + "user.name=Receipt Test", + "-c", + "user.email=receipt@example.org", + "-c", + "commit.gpgsign=false", + "-c", + "core.hooksPath=/dev/null", + "commit", + "--quiet", + "-m", + "Disposable publication source", + ], + cwd=root, + env=git_env, + check=True, + ) + revision = subprocess.run( + ["git", "rev-parse", "HEAD"], + cwd=root, + env=git_env, + check=True, + capture_output=True, + text=True, + ).stdout.strip() + inputs = add_pdf_assets(root, capture_pdf_inputs(root, [section]), assets) + return { + "root": root, + "section": section, + "assets": assets, + "target": target, + "revision": revision, + "inputs": inputs, + "git_env": git_env, + } diff --git a/tests/test_atomic_json_publication.py b/tests/test_atomic_json_publication.py new file mode 100644 index 0000000..7003998 --- /dev/null +++ b/tests/test_atomic_json_publication.py @@ -0,0 +1,68 @@ +"""Atomic publication preserves the previous complete artifact after real I/O failure.""" + +from __future__ import annotations + +import json +import os +import subprocess +import sys +from pathlib import Path + +import pytest + +from src.artifacts.json_io import write_json_artifact + + +@pytest.mark.skipif(os.name != "posix", reason="RLIMIT_FSIZE is a POSIX I/O boundary") +def test_failed_write_preserves_existing_bytes_and_cleans_temporary( + tmp_path: Path, +) -> None: + artifact = tmp_path / "artifact.json" + original = b'{"generation": "previous"}\n' + artifact.write_bytes(original) + program = """ +import errno +from pathlib import Path +import resource +import signal +import sys +from src.artifacts.json_io import write_json_artifact +signal.signal(signal.SIGXFSZ, signal.SIG_IGN) +resource.setrlimit(resource.RLIMIT_FSIZE, (1024, 1024)) +try: + write_json_artifact(Path(sys.argv[1]), {"data": "x" * 16384}) +except OSError as error: + if error.errno != errno.EFBIG: + raise +else: + raise AssertionError("expected the actual filesystem write to exceed RLIMIT_FSIZE") +""" + result = subprocess.run( + [sys.executable, "-B", "-c", program, str(artifact)], + cwd=Path(__file__).resolve().parents[1], + capture_output=True, + text=True, + timeout=15, + check=False, + ) + assert result.returncode == 0, result.stdout + result.stderr + assert artifact.read_bytes() == original + assert sorted(path.name for path in tmp_path.iterdir()) == [artifact.name] + + +def test_atomic_replacement_preserves_content_format_and_existing_mode( + tmp_path: Path, +) -> None: + artifact = tmp_path / "artifact.json" + artifact.write_text('{"old": true}\n', encoding="utf-8") + artifact.chmod(0o640) + write_json_artifact(artifact, {"z": "λ", "a": [1, 2]}, ensure_ascii=False) + assert ( + artifact.read_text(encoding="utf-8") + == json.dumps( + {"z": "λ", "a": [1, 2]}, indent=2, sort_keys=True, ensure_ascii=False + ) + + "\n" + ) + assert artifact.stat().st_mode & 0o777 == 0o640 + assert sorted(path.name for path in tmp_path.iterdir()) == [artifact.name] diff --git a/tests/test_course_materials.py b/tests/test_course_materials.py index d3e1728..40b806e 100644 --- a/tests/test_course_materials.py +++ b/tests/test_course_materials.py @@ -4,7 +4,6 @@ from copy import deepcopy from pathlib import Path -from typing import Self import pytest @@ -363,124 +362,143 @@ def test_course_material_normalizer_surfaces_malformed_record_shape() -> None: assert any(issue.code == "record-type" for issue in audit.issues) -def test_github_tree_fetch_retries_transient_5xx(monkeypatch) -> None: - """A transient 5xx is retried with backoff and the fetch then succeeds.""" - import io - import json - import urllib.error - - from src.curriculum.materials import github_sources as gs - - body = json.dumps({"tree": [{"path": "a.md", "type": "blob"}]}).encode() - state = {"calls": 0} - - class FakeResponse: - def __init__(self, data: bytes) -> None: - self._data = data - - def __enter__(self) -> Self: - return self - - def __exit__(self, *exc: object) -> bool: - return False - - def read(self) -> bytes: - return self._data - - def fake_urlopen(request, timeout): - state["calls"] += 1 - if state["calls"] == 1: - raise urllib.error.HTTPError( - request.full_url, - 503, - "Service Unavailable", - {}, - io.BytesIO(b"boom"), - ) - return FakeResponse(body) +def test_github_tree_fetch_retries_transient_5xx(httpserver) -> None: + import urllib.request - monkeypatch.setattr(gs.urllib.request, "urlopen", fake_urlopen) - monkeypatch.setattr(gs.time, "sleep", lambda _seconds: None) + from src.curriculum.materials.github_sources import _github_tree_payload - paths = gs._fetch_github_tree_paths("o", "r", "c" * 40, timeout=2.0) - assert paths == {"a.md"} - assert state["calls"] == 2 + httpserver.expect_ordered_request("/tree").respond_with_json({}, status=503) + httpserver.expect_ordered_request("/tree").respond_with_json( + {"tree": [{"path": "a.md", "type": "blob"}]} + ) + result = _github_tree_payload( + urllib.request.Request(httpserver.url_for("/tree")), + timeout=2, + base_delay=0, + ) + assert result == {"tree": [{"path": "a.md", "type": "blob"}]} + assert len(httpserver.log) == 2 -def test_github_tree_fetch_raises_immediately_on_permanent_4xx(monkeypatch) -> None: - """A permanent 4xx is not retried (it can never succeed).""" - import io +def test_github_tree_fetch_raises_immediately_on_permanent_4xx(httpserver) -> None: import urllib.error import urllib.request - from src.curriculum.materials import github_sources as gs - - def fake_urlopen(request, timeout): - raise urllib.error.HTTPError( - request.full_url, 404, "Not Found", {}, io.BytesIO(b"missing") - ) - - monkeypatch.setattr(gs.time, "sleep", lambda _seconds: None) + from src.curriculum.materials.github_sources import _github_tree_payload - with pytest.raises(urllib.request.HTTPError): - gs._github_tree_payload( - urllib.request.Request("https://api.github.com/x"), timeout=2.0 + httpserver.expect_ordered_request("/tree").respond_with_json({}, status=404) + with pytest.raises(urllib.error.HTTPError) as error: + _github_tree_payload( + urllib.request.Request(httpserver.url_for("/tree")), + timeout=2, + base_delay=0, ) + assert error.value.code == 404 + assert len(httpserver.log) == 1 -def test_github_tree_fetch_retries_transient_network_error(monkeypatch) -> None: - """A transient network/timeout error is retried and the fetch then succeeds.""" - import json - - from src.curriculum.materials import github_sources as gs +def test_github_tree_fetch_raises_connection_error_from_real_loopback() -> None: + import socket + import urllib.error + import urllib.request - body = json.dumps({"tree": [{"path": "a.md", "type": "blob"}]}).encode() - state = {"calls": 0} + from src.curriculum.materials.github_sources import _github_tree_payload + + # Reserve a non-listening endpoint without stopping the shared HTTP fixture. + with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as endpoint: + endpoint.bind(("127.0.0.1", 0)) + url = f"http://127.0.0.1:{endpoint.getsockname()[1]}/tree" + with pytest.raises(urllib.error.URLError): + _github_tree_payload( + urllib.request.Request(url), + timeout=0.2, + attempts=2, + base_delay=0, + ) - class FakeResponse: - def __enter__(self) -> Self: - return self - def __exit__(self, *exc: object) -> bool: - return False +def test_github_tree_fetch_raises_final_error_after_exhausting_attempts( + httpserver, +) -> None: + import urllib.error + import urllib.request - def read(self) -> bytes: - return body + from src.curriculum.materials.github_sources import _github_tree_payload - def fake_urlopen(request, timeout): - state["calls"] += 1 - if state["calls"] == 1: - raise TimeoutError("timed out") - return FakeResponse() + for _ in range(3): + httpserver.expect_ordered_request("/tree").respond_with_json({}, status=503) + with pytest.raises(urllib.error.HTTPError) as error: + _github_tree_payload( + urllib.request.Request(httpserver.url_for("/tree")), + timeout=2, + base_delay=0, + ) + assert error.value.code == 503 + assert len(httpserver.log) == 3 - monkeypatch.setattr(gs.urllib.request, "urlopen", fake_urlopen) - monkeypatch.setattr(gs.time, "sleep", lambda _seconds: None) - paths = gs._fetch_github_tree_paths("o", "r", "c" * 40, timeout=2.0) - assert paths == {"a.md"} - assert state["calls"] == 2 +@pytest.mark.parametrize("malformed", [[], None, {"tree": None}]) +def test_malformed_successful_github_response_becomes_one_source_issue( + httpserver, + monkeypatch, + malformed, +) -> None: + httpserver.expect_request( + "/repos/testorg/testrepo/git/trees/0123456789abcdef0123456789abcdef01234567" + ).respond_with_json(malformed) + monkeypatch.setenv("ASF_GITHUB_API_BASE", httpserver.url_for("")) + issues = validate_material_sources(_online_payload(), online=True, timeout=2) + findings = [issue for issue in issues if issue.code == "github-tree"] + assert len(findings) == 1 + assert len(httpserver.log) == 1 -def test_github_tree_fetch_raises_final_error_after_exhausting_attempts( +@pytest.mark.parametrize("include_second", [False, True]) +def test_offline_and_online_pointer_owners_agree_on_whitespace_and_all_list_entries( + httpserver, monkeypatch, + include_second: bool, ) -> None: - """If every retry fails with 5xx, the final HTTPError is re-raised.""" - import io - import urllib.error - import urllib.request - - from src.curriculum.materials import github_sources as gs + from src.curriculum.model import ALL_STRATA - def fake_urlopen(request, timeout): - raise urllib.error.HTTPError( - request.full_url, 503, "Service Unavailable", {}, io.BytesIO(b"boom") - ) + monkeypatch.setenv("ASF_GITHUB_API_BASE", httpserver.url_for("")) + payload = _online_payload() + source = payload["sources"][0] + source_id = source["source_id"].strip() + for field in ("source_id", "repo_url", "commit"): + source[field] = " " + source[field].strip() + " " + source["path_globs"] = ["*.md"] + payload["stratum_mappings"] = { + stratum.value: [ + { + "source_id": f" {source_id} ", + "path": " first.md ", + "title": " First ", + "role": " Context ", + }, + { + "source_id": f" {source_id} ", + "path": " second.md ", + "title": " Second ", + "role": " Context ", + }, + ] + for stratum in ALL_STRATA + } + payload["topic_mappings"] = [] + assert validate_material_sources(payload) == [] + assert httpserver.log == [] + paths = ["first.md", "second.md"] if include_second else ["first.md"] + httpserver.expect_request( + "/repos/testorg/testrepo/git/trees/0123456789abcdef0123456789abcdef01234567" + ).respond_with_json( + {"tree": [{"path": path, "type": "blob"} for path in paths], "truncated": False} + ) - monkeypatch.setattr(gs.urllib.request, "urlopen", fake_urlopen) - monkeypatch.setattr(gs.time, "sleep", lambda _seconds: None) + issues = validate_material_sources(payload, online=True, timeout=2) - with pytest.raises(urllib.request.HTTPError) as exc_info: - gs._github_tree_payload( - urllib.request.Request("https://api.github.com/x"), timeout=2.0 - ) - assert exc_info.value.code == 503 + if include_second: + assert issues == [] + else: + assert issues and all("second.md" in issue.message for issue in issues) + assert len(httpserver.log) == 1 diff --git a/tests/test_curriculum_graph.py b/tests/test_curriculum_graph.py index 2924a25..10331d8 100644 --- a/tests/test_curriculum_graph.py +++ b/tests/test_curriculum_graph.py @@ -196,3 +196,62 @@ def test_connectivity_fails_closed_when_no_foundational_roots() -> None: ) assert g.reachable_from_foundations() == set() assert not g.is_connected_foundations_to_applications() + + +def test_smallest_newly_ready_id_precedes_a_larger_ready_root() -> None: + graph = _graph(_skill("z", _M, ()), _skill("b", _M, ("a",)), _skill("a", _M, ())) + assert [skill.skill_id for skill in graph.topological_order()] == ["a", "b", "z"] + + +def test_reverse_stored_deep_chain_has_exact_stack_safe_prerequisite_closure() -> None: + from src.curriculum.quality import transitive_prerequisites + + identifiers = [f"k{index:04d}" for index in range(1200)] + skills = [ + _skill(identifier, _M, () if index == 0 else (identifiers[index - 1],)) + for index, identifier in enumerate(identifiers) + ] + graph = _graph(*reversed(skills)) + graph.validate() + closure = transitive_prerequisites(graph) + assert closure[identifiers[0]] == set() + assert closure[identifiers[600]] == set(identifiers[:600]) + assert closure[identifiers[-1]] == set(identifiers[:-1]) + assert graph.levels()[identifiers[-1]] == len(identifiers) - 1 + + +def test_diamond_closure_deduplicates_shared_ancestors_and_refuses_cycles() -> None: + from src.curriculum.quality import transitive_prerequisites + + graph = _graph( + _skill("a", _M, ()), + _skill("b", _M, ("a",)), + _skill("c", _M, ("a",)), + _skill("d", _M, ("b", "c")), + ) + assert transitive_prerequisites(graph) == { + "a": set(), + "b": {"a"}, + "c": {"a"}, + "d": {"a", "b", "c"}, + } + with pytest.raises(CurriculumError): + transitive_prerequisites( + _graph(_skill("a", _M, ("b",)), _skill("b", _M, ("a",))) + ) + with pytest.raises(CurriculumError): + transitive_prerequisites(_graph(_skill("a", _M, ("missing",)))) + + +def test_call_scoped_snapshots_do_not_cache_mutable_subject_membership() -> None: + graph = _graph(_skill("a", _M, ())) + before = graph.skills + graph.subjects.append( + Subject("extra", "Extra", _M, "d", (_skill("b", _M, ("a",)),)) + ) + graph.validate() + assert [skill.skill_id for skill in before] == ["a"] + assert [skill.skill_id for skill in graph.skills] == ["a", "b"] + assert [skill.skill_id for skill in graph.topological_order()] == ["a", "b"] + assert graph.stats()["skills"] == 2 + assert graph.stats()["max_depth"] == 1 diff --git a/tests/test_dashboard_console.py b/tests/test_dashboard_console.py index 43cd07e..ea3ae66 100644 --- a/tests/test_dashboard_console.py +++ b/tests/test_dashboard_console.py @@ -13,6 +13,8 @@ import validate_dashboard_console as gate +from src.dashboard.browser_validation import unexpected_errors + _INCEPTION = ( "msgid=89 [error] Error getting service status Error: SkillTree: Unable to " "authenticate using [/app/projects/Inception/users/local-learner/token] endpoint. " @@ -21,12 +23,12 @@ def test_known_inception_probe_is_allow_listed() -> None: - assert gate.unexpected_errors([_INCEPTION]) == [] + assert unexpected_errors([_INCEPTION]) == [] def test_unexpected_error_is_surfaced() -> None: other = "msgid=12 [error] Uncaught TypeError: cannot read property 'x' of undefined" - unexpected = gate.unexpected_errors([_INCEPTION, other]) + unexpected = unexpected_errors([_INCEPTION, other]) # The cosmetic probe is filtered; a real error must remain so the gate fails on it. assert unexpected == [other] @@ -37,7 +39,7 @@ def test_inception_allow_list_does_not_swallow_other_401s() -> None: "[error] Unable to authenticate using " "[/app/projects/activeInference/users/local-learner/token] Response Code=[401]." ) - assert gate.unexpected_errors([real_401]) == [real_401] + assert unexpected_errors([real_401]) == [real_401] def test_dashboard_reachable_rejects_non_http_scheme() -> None: diff --git a/tests/test_dashboard_live_validation.py b/tests/test_dashboard_live_validation.py index 2316698..62bfcb8 100644 --- a/tests/test_dashboard_live_validation.py +++ b/tests/test_dashboard_live_validation.py @@ -1,17 +1,20 @@ from __future__ import annotations import json +from copy import deepcopy from pathlib import Path import pytest -from src.dashboard.artifacts import EXPECTED_ARTIFACT_COUNT +from src.dashboard.artifacts import EXPECTED_ARTIFACT_COUNT, load_skilltree_project from src.dashboard.live_validation import ( LiveSkillTreeDashboardError, _fetch_json, validate_live_skilltree_dashboard, ) from src.project_identity import SKILLTREE_PROJECT_ID, SKILLTREE_PROJECT_NAME +from src.skilltree.content import export_content_digest +from src.skilltree.local_auth import LocalSkillTreeState, save_local_state def _write_project_export(project_root: Path, payload: object | None = None) -> Path: @@ -43,6 +46,7 @@ def _serve_dashboard_payloads( *, counts: dict[str, int] | None = None, token_payload: dict[str, object] | None = None, + project_summary: dict[str, object] | None = None, ) -> str: base_url = httpserver.url_for("").rstrip("/") project_counts = counts or { @@ -54,9 +58,11 @@ def _serve_dashboard_payloads( httpserver.expect_request("/public/status").respond_with_json({"status": "OK"}) httpserver.expect_request("/api/status").respond_with_json( { - "project": { - "projectId": SKILLTREE_PROJECT_ID, - "name": SKILLTREE_PROJECT_NAME, + "instanceId": "live-validation-fixture", + "project": project_summary + if project_summary is not None + else { + "project": {"id": SKILLTREE_PROJECT_ID, "name": SKILLTREE_PROJECT_NAME}, "dependencies": [], "counts": project_counts, }, @@ -173,3 +179,69 @@ def test_live_validator_rejects_token_secret_leak( def test_fetch_json_rejects_non_http_scheme() -> None: with pytest.raises(LiveSkillTreeDashboardError, match="only http/https URLs"): _fetch_json("file:///etc/passwd", timeout=1.0) + + +@pytest.mark.parametrize("changed", [False, True]) +def test_strict_live_validator_binds_actual_admin_content_not_just_matching_counts( + tmp_path: Path, + httpserver, + content_project, + skilltree_admin_content, + changed: bool, +) -> None: + exported = content_project.to_export_dict() + exported.update(projectId=SKILLTREE_PROJECT_ID, name=SKILLTREE_PROJECT_NAME) + exported["subjects"][0]["skills"][0]["skillId"] = "live_validation_01" + exported["quizzes"][0]["questions"][0]["answers"].append( + {"answer": "other", "isCorrect": False} + ) + _write_project_export(tmp_path, exported) + public = load_skilltree_project(tmp_path, export_payload=exported) + base_url = _serve_dashboard_payloads(httpserver, project_summary=public) + live = deepcopy(exported) + if changed: + answers = live["quizzes"][0]["questions"][0]["answers"] + answers[0]["isCorrect"], answers[1]["isCorrect"] = False, True + skilltree_admin_content(httpserver, live) + httpserver.expect_request("/", method="GET").respond_with_data( + "Session", content_type="text/html" + ) + httpserver.expect_request("/performLogin", method="POST").respond_with_json({}) + httpserver.expect_request( + f"/admin/projects/{SKILLTREE_PROJECT_ID}/dependency/graph", + method="GET", + ).respond_with_json({"nodes": [], "edges": []}) + state_dir = tmp_path / "private-state" + save_local_state( + state_dir, + LocalSkillTreeState( + admin_username="fixture-admin", + admin_password="fixture-password", + learner_id="fixture-learner", + project_id=SKILLTREE_PROJECT_ID, + skilltree_url=base_url, + client_secret="fixture-client-secret", + ), + ) + arguments = { + "dashboard_url": base_url, + "skilltree_url": base_url, + "project_root": tmp_path, + "local_state_dir": state_dir, + "require_token": False, + "require_live_admin": True, + } + if changed: + with pytest.raises(LiveSkillTreeDashboardError, match="content"): + validate_live_skilltree_dashboard(**arguments) + else: + result = validate_live_skilltree_dashboard(**arguments) + assert ( + result["liveContent"]["checked"] is True + and result["liveContent"]["complete"] is True + ) + assert result["exportContentDigest"] == export_content_digest(exported) + assert result["dashboardInstanceId"] == "live-validation-fixture" + assert "isCorrect" not in json.dumps( + result + ) and "fixture-password" not in json.dumps(result) diff --git a/tests/test_dashboard_mount.py b/tests/test_dashboard_mount.py new file mode 100644 index 0000000..dbcc7f5 --- /dev/null +++ b/tests/test_dashboard_mount.py @@ -0,0 +1,99 @@ +"""Opt-in actual-browser acceptance; never substitutes a fake SDK or CLI result.""" + +from __future__ import annotations + +import json +import os +import shutil +import threading +from http.server import ThreadingHTTPServer +from pathlib import Path + +import pytest + +from src.dashboard.browser_validation import ( + BrowserDisplayError, + browser_receipt_complete, + display_probe, + validate_dashboard_browser, +) +from src.dashboard.config import make_dashboard_config +from src.dashboard.server import make_dashboard_handler + + +@pytest.fixture +def authorized_browser(): + url = os.environ.get("ASF_TEST_DASHBOARD_URL", "") + if not url: + pytest.skip( + "set ASF_TEST_DASHBOARD_URL only after live browser acceptance is authorized" + ) + command = os.environ.get("ASF_TEST_BROWSER_COMMAND", "chrome-devtools-axi") + if shutil.which(command) is None: + pytest.skip(f"existing browser CLI unavailable: {command}") + return url, command + + +@pytest.mark.requires_browser +def test_actual_official_project_subject_and_quiz_are_mounted( + authorized_browser, +) -> None: + url, command = authorized_browser + receipt = validate_dashboard_browser( + url, browser_command=command, timeout=60, settle_seconds=1 + ) + assert browser_receipt_complete(receipt) + quiz = next(view for view in receipt["views"] if view["kind"] == "quiz") + observed = display_probe(browser_command=command, timeout=10) + assert observed["state"] == "ready" and observed["route"] == quiz["route"] + assert observed["frameVisible"] is True and observed["busy"] is False + assert observed["projectId"] == receipt["projectId"] + assert observed["dashboardInstanceId"] == receipt["dashboardInstanceId"] + assert observed["exportContentDigest"] == receipt["exportContentDigest"] + + +@pytest.mark.requires_browser +def test_real_dashboard_fallback_cannot_pass_mounted_view_proof( + tmp_path: Path, + httpserver, + content_project, + authorized_browser, +) -> None: + _authorized_url, command = authorized_browser + export = tmp_path / "output" / "data" / "skilltree_project.json" + export.parent.mkdir(parents=True) + export.write_text(json.dumps(content_project.to_export_dict()), encoding="utf-8") + httpserver.expect_request("/public/status").respond_with_json({"status": "OK"}) + config = make_dashboard_config( + project_root=tmp_path, + skilltree_url=httpserver.url_for(""), + project_id="p", + project_name="Project", + host="127.0.0.1", + port=8765, + skills_client_module_url="/assets/missing-official-module.js", + local_auth_enabled=False, + ) + server = ThreadingHTTPServer(("127.0.0.1", 0), make_dashboard_handler(config)) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + url = f"http://127.0.0.1:{server.server_address[1]}" + try: + with pytest.raises(BrowserDisplayError, match="fallback|did not reach ready"): + validate_dashboard_browser( + url, browser_command=command, timeout=15, settle_seconds=0 + ) + observed = display_probe(browser_command=command, timeout=5) + assert observed["state"] != "ready" + finally: + server.shutdown() + server.server_close() + thread.join(timeout=5) + + +@pytest.mark.parametrize("timeout", [0.0, -1.0, float("nan"), float("inf")]) +def test_invalid_browser_timeout_is_rejected_before_network_or_cli( + timeout: float, +) -> None: + with pytest.raises(BrowserDisplayError, match="finite and positive"): + validate_dashboard_browser("not a URL", timeout=timeout) diff --git a/tests/test_dashboard_server.py b/tests/test_dashboard_server.py index 13b4f7c..2f06f76 100644 --- a/tests/test_dashboard_server.py +++ b/tests/test_dashboard_server.py @@ -640,3 +640,82 @@ def test_request_without_host_header_is_allowed(tmp_path: Path) -> None: status_line = raw.split(b"\r\n", 1)[0].decode("ascii") assert "200" in status_line assert b'"ok": true' in raw + + +def test_malformed_provider_token_is_a_secret_safe_service_error( + tmp_path: Path, + httpserver: HTTPServer, +) -> None: + provider = httpserver.url_for("").rstrip("/") + httpserver.expect_request("/oauth/token", method="POST").respond_with_data( + '{"access_token": "private-provider-fragment"', + content_type="application/json", + ) + state_dir = tmp_path / "state" + _write_local_auth_state(state_dir, skilltree_url=provider) + config = make_dashboard_config( + project_root=tmp_path, + skilltree_url=provider, + local_auth_enabled=True, + local_state_dir=state_dir, + ) + with _DashboardServer(config) as base_url: + try: + _request(f"{base_url}/api/skilltree/token") + except urllib.error.HTTPError as error: + status, raw = error.code, error.read() + else: + raise AssertionError("malformed provider token was accepted") + assert status == 503 + assert "private-provider-fragment" not in raw.decode("utf-8") + assert "trusted-secret" not in raw.decode("utf-8") + + +def test_status_requests_see_atomic_export_replacement_without_exposing_answer_keys( + tmp_path: Path, + httpserver, + content_project, +) -> None: + from copy import deepcopy + + from src.artifacts.json_io import write_json_artifact + from src.skilltree.content import export_content_digest + + exported = content_project.to_export_dict() + path = tmp_path / "output" / "data" / "skilltree_project.json" + path.parent.mkdir(parents=True, exist_ok=True) + write_json_artifact(path, exported) + httpserver.expect_request("/public/status").respond_with_json({"status": "OK"}) + config = make_dashboard_config( + project_root=tmp_path, + project_id="p", + project_name="Project", + skilltree_url=httpserver.url_for(""), + ) + with _DashboardServer(config) as url: + code, _headers, body = _request(url + "/api/status?defer_live=1") + first = json.loads(body) + assert code == 200 and first["project"]["available"] is True + assert first["project"]["exportContentDigest"] == export_content_digest( + exported + ) + replacement = deepcopy(exported) + replacement["subjects"][0]["skills"][0]["description"] = "Replacement lesson" + replacement["quizzes"][0]["questions"][0]["answers"][0]["answer"] = ( + "private-replacement-key-7ce8" + ) + write_json_artifact(path, replacement) + code, _headers, body = _request(url + "/api/status?defer_live=1") + second = json.loads(body) + + assert code == 200 + assert first["instanceId"] == second["instanceId"] + assert second["project"]["exportContentDigest"] == export_content_digest( + replacement + ) + assert ( + first["project"]["exportContentDigest"] + != second["project"]["exportContentDigest"] + ) + assert "isCorrect" not in body.decode("utf-8") + assert "private-replacement-key-7ce8" not in body.decode("utf-8") diff --git a/tests/test_figures.py b/tests/test_figures.py index 47bc619..287f206 100644 --- a/tests/test_figures.py +++ b/tests/test_figures.py @@ -605,12 +605,16 @@ def test_validation_surface_matrix_figure_written(tmp_path) -> None: def test_validation_intent_mismatch_matrix_figure_written(tmp_path) -> None: + from src.figures.plots.registry_io import planned_figure_registry_entries + path = plot_validation_intent_mismatch_matrix( tmp_path / "validation-intent.png", - figure_count=planned_figure_count( - include_scholarship=True, - include_claim_support=True, - include_roadmap=True, + figure_count=len( + planned_figure_registry_entries( + include_scholarship=True, + include_claim_support=True, + include_roadmap=True, + ) ), include_scholarship=True, include_claim_support=True, @@ -913,3 +917,25 @@ def test_validation_surface_matrix_is_derived_from_validator_files( # Against an empty root no validator files exist -> every cell is n/a (0). _, _, empty = validation_surface_matrix(tmp_path) assert empty.sum() == 0 + + +def test_registered_total_and_rendered_subset_keep_distinct_contracts() -> None: + from itertools import product + + from src.figures.plots.registry_io import planned_figure_registry_entries + from src.figures.registry_manifest import figure_registry_payload + + for scholarship, claims, roadmap in product((False, True), repeat=3): + flags = { + "include_scholarship": scholarship, + "include_claim_support": claims, + "include_roadmap": roadmap, + } + registered = figure_registry_payload(**flags) + rendered = planned_figure_registry_entries(**flags) + assert planned_figure_count(**flags) == len(registered) + assert {row["filename"] for row in rendered} == { + row["filename"] + for row in registered + if row.get("evidence_kind", "rendered") != "captured" + } diff --git a/tests/test_full_app_stack.py b/tests/test_full_app_stack.py index ae4110f..ee2e710 100644 --- a/tests/test_full_app_stack.py +++ b/tests/test_full_app_stack.py @@ -8,6 +8,8 @@ import sys from pathlib import Path +import pytest + from src.integration import full_app_stack as stack REPO = Path(__file__).resolve().parents[1] @@ -20,7 +22,7 @@ def test_full_app_stack_builds_strict_commands() -> None: state_dir="state", learner_id="learner", skip_dependencies=True, - skip_seed_if_live_counts_ready=True, + skip_seed_if_live_content_ready=True, ) dashboard = stack.dashboard_command( python="python", @@ -57,7 +59,7 @@ def test_full_app_stack_builds_strict_commands() -> None: assert bootstrap[:2] == ["python", "scripts/bootstrap_local_skilltree.py"] assert "--learner-id" in bootstrap assert "--skip-dependencies" in bootstrap - assert "--skip-seed-if-live-counts-ready" in bootstrap + assert "--skip-seed-if-live-content-ready" in bootstrap assert dashboard[:2] == ["python", "scripts/serve_learner_dashboard.py"] assert "--local-auth" in dashboard assert reconcile[:2] == ["python", "scripts/reconcile_live_dependencies.py"] @@ -312,48 +314,86 @@ def test_full_app_stack_starts_keep_running_dashboard_with_log(tmp_path: Path) - stack.stop_dashboard_process(process, timeout=5) +def _browser_receipt() -> dict: + return { + "ok": True, + "checked": True, + "projectId": "p", + "dashboardInstanceId": "instance", + "exportContentDigest": "a" * 64, + "views": [ + { + "kind": "project", + "route": "/", + "state": "ready", + "visibleContentChecked": True, + }, + { + "kind": "subject", + "route": "/subjects/s", + "state": "ready", + "visibleContentChecked": True, + }, + { + "kind": "quiz", + "route": "/subjects/s/skills/k/quizzes/q", + "state": "ready", + "visibleContentChecked": True, + }, + ], + } + + +def _live_receipt() -> dict: + return { + "ok": True, + "dashboardInstanceId": "instance", + "exportContentDigest": "a" * 64, + "counts": {"skills": 630}, + "artifacts": {"present": 30, "missing": 0}, + "courseMaterials": {"records": 630}, + "figureContracts": {"checked": True, "registered": 19}, + "liveAdminProject": {"checked": True, "skills": 630}, + "liveDependencies": { + "checked": True, + "complete": True, + "expectedDependencies": 1199, + }, + "liveContent": { + "checked": True, + "complete": True, + "projectId": "p", + "exportContentDigest": "a" * 64, + }, + } + + def test_full_app_stack_proof_summary_shape() -> None: - pid_path = Path("/tmp/dashboard.pid") - log_path = Path("/tmp/dashboard.log") + pid_path, log_path = Path("/tmp/dashboard.pid"), Path("/tmp/dashboard.log") proof = stack.proof_summary( skilltree_url="http://localhost:8080", dashboard_url="http://127.0.0.1:8765", skilltree_status={"status": "OK"}, - dashboard_health={"ok": True}, - live_validation={ - "counts": {"skills": 630}, - "artifacts": {"present": 30, "missing": 0}, - "courseMaterials": {"records": 630}, - "figureContracts": {"checked": True, "registered": 19}, - "liveAdminProject": {"checked": True, "skills": 630}, - "liveDependencies": {"checked": True, "expectedDependencies": 1199}, - }, - browser_console_stdout="dashboard console gate OK", + dashboard_health={"ok": True, "instanceId": "instance"}, + live_validation=_live_receipt(), + browser_console_stdout=json.dumps(_browser_receipt()), raw_skilltree_login={"ok": True, "adminSurfaceVisible": True}, kept_running=True, dashboard_pid=12345, dashboard_pid_path=pid_path, dashboard_log_path=log_path, - dependency_reconciliation={"ok": True, "missingAfter": 0}, - app_ready=False, + dependency_reconciliation={"ok": True, "complete": True, "missingAfter": 0}, ) - - encoded = json.dumps(proof, sort_keys=True) assert proof["ok"] is True - assert proof["counts"] == {"skills": 630} assert proof["browserConsole"]["checked"] is True - assert proof["rawSkillTreeLogin"] == {"ok": True, "adminSurfaceVisible": True} - assert proof["figureContracts"] == {"checked": True, "registered": 19} - assert proof["liveDependencies"] == { - "checked": True, - "expectedDependencies": 1199, - } - assert proof["dependencyReconciliation"] == {"ok": True, "missingAfter": 0} + assert proof["liveContent"]["complete"] is True + assert proof["liveDependencies"]["complete"] is True + assert proof["runId"] == proof["dashboardInstanceId"] == "instance" + assert proof["exportContentDigest"] == proof["liveContent"]["exportContentDigest"] assert proof["mode"] == "full-parity" assert proof["dashboardPid"] == 12345 assert proof["dashboardPidPath"] == str(pid_path) assert proof["dashboardLogPath"] == str(log_path) - assert "dashboard console gate OK" in encoded def test_full_app_stack_proof_summary_does_not_certify_failed_subproof() -> None: @@ -383,7 +423,81 @@ def test_full_app_stack_writes_redacted_deterministic_proof(tmp_path: Path) -> N path = stack.write_proof_artifact(proof, tmp_path / "proof.json") payload = json.loads(path.read_text(encoding="utf-8")) - assert payload["proofVersion"] == 1 + assert payload["proofVersion"] == 2 assert payload["token"] == "[REDACTED]" assert "super-secret" not in path.read_text(encoding="utf-8") assert payload["counts"] == {"skills": 630} + + +@pytest.mark.parametrize( + "stdout", ["dashboard console gate OK", "{}", '{"ok": true, "checked": false}'] +) +def test_console_exit_text_or_unchecked_payload_cannot_certify_a_display( + stdout: str, +) -> None: + with pytest.raises(stack.FullAppStackError): + stack.proof_summary( + skilltree_url="http://localhost:8080", + dashboard_url="http://127.0.0.1:8765", + skilltree_status={"status": "OK"}, + dashboard_health={"ok": True, "instanceId": "instance"}, + live_validation=_live_receipt(), + browser_console_stdout=stdout, + raw_skilltree_login={"ok": True}, + kept_running=False, + dashboard_pid=None, + ) + + +def test_proof_failure_stops_owned_dashboard_even_with_keep_running( + tmp_path: Path, + httpserver, +) -> None: + import urllib.request + + httpserver.expect_request("/unrelated").respond_with_data("still running") + process = stack.start_dashboard( + [sys.executable, "-c", "import time; time.sleep(30)"] + ) + destination = tmp_path / "proof-is-a-directory" + destination.mkdir() + try: + with pytest.raises(OSError): + try: + stack.write_proof_artifact({"ok": True}, destination) + finally: + stack.finalize_dashboard_process( + process, + state_dir=str(tmp_path), + keep_running=True, + successful=False, + ) + assert process.poll() is not None + assert not (tmp_path / "dashboard.pid").exists() + with urllib.request.urlopen( + httpserver.url_for("/unrelated"), timeout=2 + ) as response: + assert response.read() == b"still running" + finally: + stack.stop_dashboard_process(process) + + +def test_failed_pid_publication_does_not_leave_the_owned_process_alive( + tmp_path: Path, +) -> None: + (tmp_path / "dashboard.pid").mkdir() + process = stack.start_dashboard( + [sys.executable, "-c", "import time; time.sleep(30)"] + ) + try: + with pytest.raises(OSError): + stack.finalize_dashboard_process( + process, + state_dir=str(tmp_path), + keep_running=True, + successful=True, + ) + assert process.poll() is not None + assert (tmp_path / "dashboard.pid").is_dir() + finally: + stack.stop_dashboard_process(process) diff --git a/tests/test_generated_output_cli.py b/tests/test_generated_output_cli.py new file mode 100644 index 0000000..26a5d8a --- /dev/null +++ b/tests/test_generated_output_cli.py @@ -0,0 +1,24 @@ +"""The CLI owns process exits while the importable pipeline raises package errors.""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from scripts import run_kernels + + +def test_cli_reports_real_manuscript_rejection_without_replacing_the_builder( + tmp_path: Path, monkeypatch +) -> None: + manuscript = tmp_path / "manuscript" + manuscript.mkdir() + (manuscript / "00_fixture.md").write_text( + "The platform is NSA approved for all learners.\n", encoding="utf-8" + ) + monkeypatch.setattr(run_kernels, "_ROOT", tmp_path) + with pytest.raises(SystemExit) as error: + run_kernels.main() + assert isinstance(error.value.code, str) + assert "endorsement-overreach" in error.value.code diff --git a/tests/test_gnn_conformance.py b/tests/test_gnn_conformance.py index 55ff181..d7d2ca2 100644 --- a/tests/test_gnn_conformance.py +++ b/tests/test_gnn_conformance.py @@ -114,11 +114,6 @@ def test_declared_dimensions_match_the_kernel() -> None: assert dimensions["o"] == (agent["num_observations"], 1) -def test_kernel_canonical_model_conforms() -> None: - checks = check_conformance(canonical_model()) - assert len(checks) == 6 - - def test_mutated_model_is_rejected() -> None: model = copy.deepcopy(canonical_model()) model["B_matrix"][0][0][1] = 0.0 # break the commit transition @@ -126,6 +121,24 @@ def test_mutated_model_is_rejected() -> None: check_conformance(model) +@pytest.mark.parametrize( + ("field", "value"), + [ + ("A_matrix", [[1.0, 0.0]]), + ("B_matrix", [[1.0, 0.0], [0.0, 1.0]]), + ("C_vector", [1.0]), + ("D_vector", [1.0]), + ], +) +def test_invalid_declared_component_shapes_raise_conformance_error( + field, value +) -> None: + model = copy.deepcopy(canonical_model()) + model[field] = value + with pytest.raises(ConformanceError, match="declared probability/transition model"): + check_conformance(model) + + @pytest.mark.parametrize("schema", [None, "", 1, {}, "gnn-importer-pin/v2"]) def test_unsupported_pin_schema_is_rejected(tmp_path, schema) -> None: data = {**CANONICAL_PIN, "schema": schema} diff --git a/tests/test_kernels_distributions.py b/tests/test_kernels_distributions.py index 5ca67c5..96922f8 100644 --- a/tests/test_kernels_distributions.py +++ b/tests/test_kernels_distributions.py @@ -61,9 +61,14 @@ def test_higher_precision_sharpens_the_distribution(self) -> None: def test_rejects_non_positive_precision(self) -> None: """softmax() requires a positive inverse temperature.""" - with pytest.raises(ValueError, match="precision must be > 0"): + with pytest.raises(ValueError): softmax(np.array([1.0, 2.0], dtype=np.float64), precision=0.0) + def test_small_precision_preserves_mass_across_extreme_scores(self) -> None: + expected = np.exp([-2.0, 0.0]) + expected /= expected.sum() + assert_allclose(softmax([-1e308, 1e308], precision=1e-308), expected) + class TestLogStable: """Tests for log_stable().""" @@ -94,7 +99,7 @@ def test_dirichlet_expected_log_matches_digamma_identity(self) -> None: def test_dirichlet_helpers_reject_non_positive_concentrations(self) -> None: """Dirichlet expectations require strictly positive concentration parameters.""" alpha = np.array([1.0, 0.0, 2.0], dtype=np.float64) - with pytest.raises(ValueError, match="strictly positive"): + with pytest.raises(ValueError): dirichlet_expected(alpha) - with pytest.raises(ValueError, match="strictly positive"): + with pytest.raises(ValueError): dirichlet_expected_log(alpha) diff --git a/tests/test_manuscript_hydration.py b/tests/test_manuscript_hydration.py new file mode 100644 index 0000000..70e7e1f --- /dev/null +++ b/tests/test_manuscript_hydration.py @@ -0,0 +1,44 @@ +"""Unknown prose placeholders cannot silently become plausible manuscript text.""" + +from __future__ import annotations + +import pytest + +from scripts.z_generate_manuscript_variables import _render_markdown + + +def test_known_prose_variables_and_claims_resolve_to_the_supplied_value() -> None: + assert ( + _render_markdown("Result {{value}} / {{claim:value}}.", {"value": "computed"}) + == "Result computed / computed." + ) + + +@pytest.mark.parametrize( + "text", ["Result {{misspelled}}.", "Result {{claim:missing}}."] +) +def test_unknown_prose_placeholder_fails_instead_of_stripping_its_delimiters( + text: str, +) -> None: + with pytest.raises(KeyError): + _render_markdown(text, {"value": "computed"}) + + +def test_code_examples_normalize_locally_without_admitting_unknown_adjacent_prose() -> ( + None +): + example = "Use `{{unknown}}` and:\n```text\n{{claim:missing}}\n```\n" + assert ( + _render_markdown(example, {}) + == "Use `unknown` and:\n```text\nclaim:missing\n```\n" + ) + with pytest.raises(KeyError): + _render_markdown(example + "Prose {{unknown}}.", {}) + + +@pytest.mark.parametrize( + "text", ["Result {{missing token}}.", "Result {{ claim:value }}."] +) +def test_unrecognized_prose_token_shape_remains_an_error(text: str) -> None: + with pytest.raises(ValueError, match="unresolved manuscript tokens"): + _render_markdown(text, {"value": "computed"}) diff --git a/tests/test_material_claim_fields.py b/tests/test_material_claim_fields.py new file mode 100644 index 0000000..2a08db6 --- /dev/null +++ b/tests/test_material_claim_fields.py @@ -0,0 +1,234 @@ +"""Learner fields share canonical claim admission and retain original claim IDs.""" + +from __future__ import annotations + +from copy import deepcopy +from dataclasses import replace + +import pytest + +from src.curriculum.graph import CurriculumGraph +from src.curriculum.materials import ( + audit_course_material_index, + build_course_material_index, +) +from src.curriculum.model import AssessmentQuestion, Skill, Stratum, Subject +from src.provenance.binding import ( + ProvenanceError, + lint_graph, + render_graph, + validate_curriculum, + validate_skill, +) +from src.provenance.claims import ClaimRegistry + +TOKEN = "{{cl\u200baim:bo\u200bund}}" + + +def _registry() -> ClaimRegistry: + registry = ClaimRegistry() + registry.register("bound", "bound-value") + return registry + + +def _skill() -> Skill: + return Skill( + "a", + "A", + Stratum.MATH_FOUNDATIONS, + "Objective", + "Description", + (), + (AssessmentQuestion("Question", ("one", "two"), (0,)),), + ) + + +def _field(skill: Skill, name: str, value: str) -> Skill: + if name == "assessment_prompt": + return replace( + skill, assessments=(replace(skill.assessments[0], prompt=value),) + ) + if name == "assessment_answer": + return replace( + skill, assessments=(replace(skill.assessments[0], answers=(value, "two")),) + ) + return replace(skill, **{name: value}) + + +@pytest.mark.parametrize( + "field", + ["name", "objective", "description", "assessment_prompt", "assessment_answer"], +) +@pytest.mark.parametrize( + "token", + [ + "{{claim:missing}}", + "{{ claim:bound }}", + "{{claim:bad-id}}", + "{{cl\u200baim:missing}}", + ], +) +def test_standalone_skill_and_curriculum_validation_reject_every_bad_field( + field: str, token: str +) -> None: + skill = _field(_skill(), field, token) + with pytest.raises(ProvenanceError): + validate_skill(skill, _registry()) + with pytest.raises(ProvenanceError): + validate_curriculum([skill], _registry()) + + +@pytest.mark.parametrize( + "field", + ["name", "objective", "description", "assessment_prompt", "assessment_answer"], +) +def test_format_character_normalization_preserves_original_claim_identity( + field: str, +) -> None: + skill = _field(_skill(), field, TOKEN) + assert skill.claim_tokens() == {"bound"} + validate_skill(skill, _registry()) + + +def _graph() -> CurriculumGraph: + first = replace( + _skill(), + name=f"A {TOKEN}", + objective=f"Objective {TOKEN}", + description=f"Description {TOKEN}", + ) + second = replace(_skill(), skill_id="b", name=f"B {TOKEN}", prerequisites=("a",)) + return CurriculumGraph( + [ + Subject( + "sub_m", + f"Subject {TOKEN}", + Stratum.MATH_FOUNDATIONS, + "Subject description", + (first, second), + ) + ] + ) + + +def _sources(*, claims: bool = True) -> dict: + value = TOKEN if claims else "bound-value" + pointer = { + "source_id": "fixture", + "path": "lessons/intro.md", + "title": f"Pointer {value}", + "role": f"Role {value}", + } + return { + "version": 1, + "sources": [ + { + "source_id": "fixture", + "repo_url": "https://github.com/fixture/materials", + "commit": "a" * 40, + "title": f"Source {value}", + "license_status": f"License {value}", + "status_note": f"Status {value}", + "path_globs": ["lessons/*.md"], + } + ], + "stratum_mappings": {stratum.value: [dict(pointer)] for stratum in Stratum}, + "topic_mappings": [], + } + + +def test_material_names_references_context_and_source_labels_render_without_losing_claim_ids() -> ( + None +): + graph = _graph() + index = build_course_material_index(graph, _sources(), registry=_registry()) + first, second = index["skills"]["a"], index["skills"]["b"] + + assert ( + first["name"] == "A bound-value" + and first["subjectName"] == "Subject bound-value" + ) + assert second["prerequisites"][0]["name"] == "A bound-value" + assert second["relatedSkills"][0]["subjectName"] == "Subject bound-value" + assert first["downstreamSkills"][0]["name"] == "B bound-value" + assert "A bound-value (a)" in "\n".join( + section["body"] for section in second["sections"] + ) + assert "B bound-value (b)" in "\n".join( + section["body"] for section in first["sections"] + ) + assert first["provenance"]["claimTokens"] == ["bound"] + assert first["externalPointers"][0]["title"] == "Pointer bound-value" + assert first["externalPointers"][0]["role"] == "Role bound-value" + assert index["sources"][0]["title"] == "Source bound-value" + assert index["sources"][0]["statusNote"] == "Status bound-value" + assert audit_course_material_index(index, graph).issues == () + + +def test_material_builder_without_registry_accepts_claim_free_input_but_refuses_raw_claims() -> ( + None +): + graph = _graph() + with pytest.raises(ProvenanceError): + build_course_material_index(graph, _sources(claims=False)) + rendered = render_graph(graph, _registry()) + index = build_course_material_index(rendered, _sources(claims=False)) + assert index["skills"]["a"]["name"] == "A bound-value" + + +@pytest.mark.parametrize( + "field", ["name", "subjectName", "prerequisite", "section", "source", "pointer"] +) +@pytest.mark.parametrize( + "token", ["{{claim:missing}}", "{{ claim:bound }}", "{{cl\u200baim:missing}}"] +) +def test_material_audit_catches_unrendered_tokens_outside_section_bodies( + field: str, token: str +) -> None: + graph = _graph() + index = deepcopy( + build_course_material_index(graph, _sources(), registry=_registry()) + ) + record = index["skills"]["b"] + if field in ("name", "subjectName"): + record[field] = token + elif field == "prerequisite": + record["prerequisites"][0]["name"] = token + elif field == "section": + record["sections"][0]["body"] = token + elif field == "source": + index["sources"][0]["title"] = token + else: + record["externalPointers"][0]["title"] = token + issues = audit_course_material_index(index, graph).issues + assert any(issue.code == "unresolved-claim-token" for issue in issues) + + +def test_subject_name_is_rendered_and_is_part_of_numeric_lint() -> None: + graph = _graph() + assert render_graph(graph, _registry()).subjects[0].name == "Subject bound-value" + malformed = CurriculumGraph([replace(graph.subjects[0], name="{{ claim:bound }}")]) + with pytest.raises(ProvenanceError): + render_graph(malformed, _registry()) + unbacked = CurriculumGraph([replace(graph.subjects[0], name="Result 2.75 nats")]) + violations = lint_graph(unbacked) + assert len(violations) == 1 and violations[0].value == "2.75" + + +def test_subject_names_that_collide_after_rendering_are_refused() -> None: + from src.curriculum.model import CurriculumError + + registry = ClaimRegistry() + registry.register("first", "same") + registry.register("second", "same") + first = _skill() + second = replace(first, skill_id="b", name="B", prerequisites=("a",)) + graph = CurriculumGraph( + [ + Subject("sub_a", "Course {{claim:first}}", first.stratum, "", (first,)), + Subject("sub_b", "Course {{claim:second}}", second.stratum, "", (second,)), + ] + ) + graph.validate() + with pytest.raises(CurriculumError, match="duplicate subject name"): + render_graph(graph, registry) diff --git a/tests/test_pdf_provenance.py b/tests/test_pdf_provenance.py new file mode 100644 index 0000000..b3b0da4 --- /dev/null +++ b/tests/test_pdf_provenance.py @@ -0,0 +1,177 @@ +"""Receipt/freshness contracts use disposable real files, never historical outputs.""" + +from __future__ import annotations + +from copy import deepcopy + +import pytest + +from src.manuscript.pdf_receipt import ( + PdfReceiptError, + add_pdf_assets, + capture_pdf_inputs, + latex_asset_paths, + pdf_receipt_issues, + validate_pdf_inputs, + write_pdf_receipt, +) + + +def _receipt(project): + return write_pdf_receipt( + project["target"], + root=project["root"], + source_revision=project["revision"], + inputs=project["inputs"], + ) + + +@pytest.mark.parametrize( + "relative", + [ + "manuscript/config.yaml", + "manuscript/preamble.md", + "manuscript/references.bib", + "output/manuscript/01_fixture.md", + "output/manuscript/assets/cover.pdf", + "output/manuscript/assets/atlas.pdf", + "output/manuscript/assets/body.pdf", + ], +) +def test_changed_actual_input_invalidates_unchanged_pdf_and_cannot_refresh_receipt( + pdf_project, relative: str +) -> None: + receipt = _receipt(pdf_project) + old_receipt, old_pdf = receipt.read_bytes(), pdf_project["target"].read_bytes() + path = pdf_project["root"] / relative + path.write_bytes(path.read_bytes() + b"\nchanged input\n") + + assert pdf_receipt_issues(pdf_project["root"]) + with pytest.raises(PdfReceiptError, match="changed"): + _receipt(pdf_project) + assert receipt.read_bytes() == old_receipt + assert pdf_project["target"].read_bytes() == old_pdf + + +@pytest.mark.parametrize( + "change", + ["add-section", "delete-section", "delete-image", "replace-pdf", "delete-pdf"], +) +def test_changed_input_set_or_final_pdf_is_not_fresh( + pdf_project, pdf_bytes, change: str +) -> None: + receipt = _receipt(pdf_project) + old_receipt = receipt.read_bytes() + if change == "add-section": + (pdf_project["section"].parent / "02_added.md").write_text("# Added section\n") + elif change == "delete-section": + pdf_project["section"].unlink() + elif change == "delete-image": + pdf_project["assets"][1].unlink() + elif change == "replace-pdf": + pdf_project["target"].write_bytes(pdf_bytes("Different finalized PDF")) + else: + pdf_project["target"].unlink() + assert pdf_receipt_issues(pdf_project["root"]) + assert receipt.read_bytes() == old_receipt + + +@pytest.mark.parametrize( + "raw", + [ + b"not JSON", + b"[]", + b'{"schema":"active-skillference-pdf/v1"}', + b'{"schema":"active-skillference-pdf/v2"}', + ], +) +def test_legacy_or_incomplete_receipt_requires_real_rerender_without_relabeling( + pdf_project, raw: bytes +) -> None: + receipt = pdf_project["target"].with_suffix(".receipt.json") + receipt.write_bytes(raw) + assert pdf_receipt_issues(pdf_project["root"]) + assert receipt.read_bytes() == raw + + +def test_missing_receipt_is_not_created_by_read_only_freshness_audit( + pdf_project, +) -> None: + receipt = pdf_project["target"].with_suffix(".receipt.json") + assert pdf_receipt_issues(pdf_project["root"]) + assert not receipt.exists() + + +@pytest.mark.parametrize( + "damage", + [ + "missing-primary", + "missing-section", + "missing-image", + "duplicate-path", + "unsafe-path", + "bad-digest", + "missing-assets-list", + ], +) +def test_incomplete_or_unsafe_input_manifest_is_rejected( + pdf_project, damage: str +) -> None: + manifest = deepcopy(pdf_project["inputs"]) + rows = manifest["files"] + if damage.startswith("missing-") and damage != "missing-assets-list": + kind = { + "missing-primary": "config", + "missing-section": "section", + "missing-image": "asset", + }[damage] + rows.remove(next(row for row in rows if row["kind"] == kind)) + elif damage == "duplicate-path": + rows.append(dict(rows[0])) + elif damage == "unsafe-path": + rows[0]["path"] = "../outside" + elif damage == "bad-digest": + rows[0]["sha256"] = "not-a-content-digest" + else: + manifest.pop("assets") + with pytest.raises(PdfReceiptError): + validate_pdf_inputs(pdf_project["root"], manifest) + + +def test_actual_latex_image_inputs_are_deduplicated_and_comments_do_not_create_inputs( + pdf_project, +) -> None: + root = pdf_project["root"] + tex = root / "build.tex" + cover, _atlas, body = pdf_project["assets"] + tex.write_text( + "% \\includegraphics{missing-commented.pdf}\n" + f"\\includegraphics*[width=.5\\linewidth]{{{cover.as_posix()}}}\n" + f"\\includegraphics{{{cover.as_posix()}}}\n\\includegraphics{{{body.as_posix()}}}\n" + ) + paths = latex_asset_paths(tex, build_directory=root, root=root) + assert paths == sorted([cover.resolve(), body.resolve()]) + manifest = add_pdf_assets( + root, capture_pdf_inputs(root, [pdf_project["section"]]), paths + ) + assert set(manifest["assets"]) == { + path.relative_to(root).as_posix() for path in (cover, body) + } + + +def test_nonliteral_latex_image_input_cannot_create_incomplete_provenance( + pdf_project, +) -> None: + root = pdf_project["root"] + tex = root / "build.tex" + tex.write_text(r"\includegraphics{\chosenImage{}}") + with pytest.raises(PdfReceiptError, match="non-literal"): + latex_asset_paths(tex, build_directory=root, root=root) + + +def test_unrelated_repository_file_does_not_invalidate_actual_input_snapshot( + pdf_project, +) -> None: + _receipt(pdf_project) + (pdf_project["root"] / "unrelated.txt").write_text("Not a PDF compiler input.") + assert pdf_receipt_issues(pdf_project["root"]) == [] diff --git a/tests/test_pilot_schema_boundaries.py b/tests/test_pilot_schema_boundaries.py new file mode 100644 index 0000000..c33c04d --- /dev/null +++ b/tests/test_pilot_schema_boundaries.py @@ -0,0 +1,58 @@ +"""A protocol needs linkable event schemas, not merely list-shaped fields.""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from src.pilot.contract import PilotContractError, validate_pilot_contract + + +@pytest.fixture +def pilot_payload() -> dict: + path = Path(__file__).resolve().parents[1] / "data" / "learner_pilot_contract.json" + return json.loads(path.read_text(encoding="utf-8")) + + +@pytest.mark.parametrize( + ("section", "field"), + [ + ("participants", "requiredFields"), + ("events", "allowedEventTypes"), + ("events", "requiredFields"), + ("export", "fields"), + ], +) +def test_schema_lists_cannot_be_empty( + pilot_payload: dict, section: str, field: str +) -> None: + pilot_payload[section][field] = [] + with pytest.raises(PilotContractError, match="must not be empty"): + validate_pilot_contract(pilot_payload) + + +@pytest.mark.parametrize( + ("section", "field", "missing"), + [ + ("participants", "requiredFields", "participant_id_hash"), + ("events", "requiredFields", "event_timestamp"), + ("export", "fields", "participant_id_hash"), + ("export", "fields", "event_type"), + ], +) +def test_schema_must_retain_linking_fields( + pilot_payload: dict, section: str, field: str, missing: str +) -> None: + pilot_payload[section][field].remove(missing) + with pytest.raises(PilotContractError, match=missing): + validate_pilot_contract(pilot_payload) + + +def test_linkage_does_not_require_demographics_or_secondary_outcomes( + pilot_payload: dict, +) -> None: + pilot_payload["participants"]["requiredFields"] = ["participant_id_hash"] + pilot_payload["outcomes"]["secondary"] = [] + validate_pilot_contract(pilot_payload) diff --git a/tests/test_pipeline_generated_outputs.py b/tests/test_pipeline_generated_outputs.py index afb1df9..ad1e8c7 100644 --- a/tests/test_pipeline_generated_outputs.py +++ b/tests/test_pipeline_generated_outputs.py @@ -1,5 +1,7 @@ from __future__ import annotations +import json +import shutil from pathlib import Path import pytest @@ -11,12 +13,27 @@ REPO = Path(__file__).resolve().parents[1] -# Rebuilds the whole generated artifact bundle; `./gates.sh fast` deselects it. -pytestmark = pytest.mark.slow - -def test_build_generated_outputs_returns_existing_artifact_bundle() -> None: - bundle = build_generated_outputs(REPO) +@pytest.mark.slow +def test_build_generated_outputs_returns_existing_artifact_bundle( + tmp_path: Path, +) -> None: + root = tmp_path / "project" + shutil.copytree( + REPO, + root, + ignore=shutil.ignore_patterns( + ".git", + ".venv", + ".local", + "__pycache__", + ".pytest_cache", + ".mypy_cache", + ".ruff_cache", + "node_modules", + ), + ) + bundle = build_generated_outputs(root) assert isinstance(bundle, GeneratedOutputBundle) assert bundle.provenance_claims == 79 @@ -25,20 +42,42 @@ def test_build_generated_outputs_returns_existing_artifact_bundle() -> None: assert bundle.claim_rows == 74 assert bundle.frontier_sources == 45 assert bundle.material_records == 630 - assert (REPO / "output" / "data" / "claims.json") in bundle.paths + assert (root / "output" / "data" / "claims.json") in bundle.paths assert ( - REPO / "output" / "data" / "curriculum_diagnostic_gap_map.json" + root / "output" / "data" / "curriculum_diagnostic_gap_map.json" ) in bundle.paths - assert (REPO / "output" / "data" / "manuscript_variables.json") in bundle.paths + assert (root / "output" / "data" / "manuscript_variables.json") in bundle.paths assert ( - REPO / "output" / "data" / "scholarship_source_health_network.json" + root / "output" / "data" / "scholarship_source_health_network.json" ) in bundle.paths - assert (REPO / "output" / "data" / "figure_table_companions.json") in bundle.paths - assert (REPO / "output" / "data" / "course_material_index.json") in bundle.paths + assert (root / "output" / "data" / "figure_table_companions.json") in bundle.paths + assert (root / "output" / "data" / "course_material_index.json") in bundle.paths assert ( - REPO / "output" / "figures" / "active_skillference_cover.png" + root / "output" / "figures" / "active_skillference_cover.png" ) in bundle.paths assert ( - REPO / "output" / "figures" / "active_skillference_skill_atlas.png" + root / "output" / "figures" / "active_skillference_skill_atlas.png" ) in bundle.paths assert all(path.exists() for path in bundle.paths) + variables = json.loads( + (root / "output" / "data" / "manuscript_variables.json").read_text() + ) + registered = json.loads( + (root / "output" / "figures" / "figure_registry.json").read_text() + ) + assert ( + int(variables["figure_count"]) == len(registered) == bundle.registered_figures + ) + + +def test_importable_builder_raises_package_error_for_endorsement_overreach( + tmp_path: Path, +) -> None: + manuscript = tmp_path / "manuscript" + manuscript.mkdir() + (manuscript / "00_fixture.md").write_text( + "The platform is NSA approved for all learners.", + encoding="utf-8", + ) + with pytest.raises(ValueError, match="endorsement-overreach"): + build_generated_outputs(tmp_path) diff --git a/tests/test_predictive_coding_diagnostics.py b/tests/test_predictive_coding_diagnostics.py new file mode 100644 index 0000000..87cf1c4 --- /dev/null +++ b/tests/test_predictive_coding_diagnostics.py @@ -0,0 +1,43 @@ +"""The update-threshold diagnostic does not certify distance to the fixed point.""" + +import numpy as np +import pytest + +from src.kernels.predictive_coding import ( + gaussian_posterior_mean, + predictive_coding_descent, +) + + +def test_small_update_can_qualify_far_from_the_analytic_target() -> None: + result = predictive_coding_descent( + observation=10.0, + prior_mean=0.0, + prior_variance=1.0, + obs_variance=1.0, + rate=1e-9, + steps=3, + tolerance=1e-6, + ) + target = gaussian_posterior_mean(10.0, 0.0, 1.0, 1.0) + assert target == pytest.approx(5.0) + assert result.steps_to_converge == 1 + assert abs(result.estimate - target) > 4.0 + assert result.trajectory.shape == (4,) + assert result.estimate == result.trajectory[-1] + assert np.all(np.diff(result.trajectory) > 0.0) + + +def test_no_qualifying_update_retains_steps_sentinel_and_full_trajectory() -> None: + result = predictive_coding_descent( + observation=10.0, + prior_mean=0.0, + prior_variance=1.0, + obs_variance=1.0, + rate=0.1, + steps=2, + tolerance=1e-12, + ) + np.testing.assert_allclose(result.trajectory, [0.0, 1.0, 1.8]) + assert result.steps_to_converge == 2 + assert result.estimate == pytest.approx(1.8) diff --git a/tests/test_probability_support.py b/tests/test_probability_support.py new file mode 100644 index 0000000..fda0c51 --- /dev/null +++ b/tests/test_probability_support.py @@ -0,0 +1,109 @@ +"""Exact support and magnitude-independent probability boundary contracts.""" + +from __future__ import annotations + +import numpy as np +import pytest + +from src.kernels.active_inference import ( + DiscreteActiveInferenceAgent, + expected_free_energy, + policy_posterior, +) +from src.kernels.distributions import ( + dirichlet_expected, + dirichlet_expected_log, + normalize, + softmax, +) +from src.kernels.inference import elbo, log_evidence, variational_free_energy + + +def test_impossible_observation_retains_exact_zero_support() -> None: + likelihood = np.eye(2) + assert log_evidence(likelihood, 1, [1.0, 0.0]) == -np.inf + assert variational_free_energy([0.0, 1.0], likelihood, 1, [1.0, 0.0]) == np.inf + assert elbo([0.0, 1.0], likelihood, 1, [1.0, 0.0]) == -np.inf + assert variational_free_energy([1.0, 0.0], likelihood, 0, [1.0, 0.0]) == 0.0 + + +def test_small_nonzero_evidence_is_not_epsilon_regularized() -> None: + mass = 1e-30 + expected = np.log(mass) + assert log_evidence(np.eye(2), 0, [mass, 1.0 - mass]) == pytest.approx(expected) + assert variational_free_energy( + [1.0, 0.0], np.eye(2), 0, [mass, 1.0 - mass] + ) == pytest.approx(-expected) + + +def test_extreme_masses_and_policy_scores_preserve_normalization() -> None: + np.testing.assert_allclose(normalize([1e308, 1e308]), [0.5, 0.5]) + np.testing.assert_allclose(dirichlet_expected([1e308, 1e308]), [0.5, 0.5]) + np.testing.assert_allclose( + dirichlet_expected_log([1e308, 1e308]), + [-np.log(2.0)] * 2, + atol=1e-12, + rtol=0.0, + ) + np.testing.assert_allclose(softmax([1e308, 1e308], precision=2.0), [0.5, 0.5]) + np.testing.assert_allclose( + policy_posterior([-1e308, -1e308], gamma=2.0), [0.5, 0.5] + ) + np.testing.assert_allclose(softmax([-1e308, 1e308], precision=2.0), [0.0, 1.0]) + + +@pytest.mark.parametrize("bad", [np.nan, np.inf, -np.inf]) +@pytest.mark.parametrize( + "function", [normalize, dirichlet_expected, dirichlet_expected_log] +) +def test_nonfinite_masses_are_rejected(function, bad: float) -> None: + with pytest.raises(ValueError): + function([bad, 1.0]) + + +@pytest.mark.parametrize("gamma", [-1.0, 0.0, np.inf, np.nan]) +def test_invalid_policy_precision_is_rejected(gamma: float) -> None: + with pytest.raises(ValueError): + policy_posterior([0.0, 1.0], gamma=gamma) + + +@pytest.mark.parametrize("belief", [[-1.0, 2.0], [0.2, 0.2], [np.nan, 1.0]]) +def test_planning_rejects_invalid_state_beliefs(belief: list[float]) -> None: + with pytest.raises(ValueError): + expected_free_energy(belief, np.full((2, 2), 0.5), [0.5, 0.5]) + + +def test_agent_rejects_invalid_transition_at_its_owning_boundary() -> None: + with pytest.raises(ValueError, match="B"): + DiscreteActiveInferenceAgent( + A=np.eye(2), + B=np.array([[[-1.0], [-1.0]], [[2.0], [2.0]]]), + C=np.array([0.5, 0.5]), + D=np.array([0.5, 0.5]), + ) + + +def test_prediction_rejects_out_of_range_actions_and_wrong_belief_dimension() -> None: + agent = DiscreteActiveInferenceAgent( + A=np.eye(2), + B=np.eye(2)[:, :, None], + C=np.array([0.5, 0.5]), + D=np.array([0.5, 0.5]), + ) + with pytest.raises(ValueError, match="action"): + agent.predict([0.5, 0.5], -1) + with pytest.raises(ValueError, match="dimension"): + agent.predict([1.0], 0) + np.testing.assert_allclose(agent.predict([0.25, 0.75], 0), [0.25, 0.75]) + + +@pytest.mark.parametrize("action", [True, np.bool_(False), 0.5, "0", None]) +def test_prediction_rejects_noninteger_action_types(action) -> None: + agent = DiscreteActiveInferenceAgent( + A=np.eye(2), + B=np.eye(2)[:, :, None], + C=np.array([0.5, 0.5]), + D=np.array([0.5, 0.5]), + ) + with pytest.raises(TypeError, match="action"): + agent.predict([0.5, 0.5], action) diff --git a/tests/test_recommendations.py b/tests/test_recommendations.py index 23ca073..043252e 100644 --- a/tests/test_recommendations.py +++ b/tests/test_recommendations.py @@ -9,6 +9,8 @@ def _project() -> dict[str, object]: return { + "projectId": "recommendations", + "name": "Recommendations", "subjects": [ { "subjectId": "sub_foundations", @@ -48,15 +50,32 @@ def _project() -> dict[str, object]: "quizzes": [ { "quizId": "math_sets_quiz", - "questions": [{"question": "What is a set?"}], + "questions": [ + { + "question": "What is a set?", + "answers": [{"answer": "A collection", "isCorrect": True}], + } + ], }, { "quizId": "prob_events_quiz", - "questions": [{"question": "How do events relate to sets?"}], + "questions": [ + { + "question": "How do events relate to sets?", + "answers": [{"answer": "Events are sets", "isCorrect": True}], + } + ], }, { "quizId": "bayes_update_quiz", - "questions": [{"question": "How does evidence update belief?"}], + "questions": [ + { + "question": "How does evidence update belief?", + "answers": [ + {"answer": "Bayesian conditioning", "isCorrect": True} + ], + } + ], }, ], } @@ -134,25 +153,23 @@ def test_frontier_skill_whose_prerequisite_failed_is_remediation() -> None: assert "retake the official quiz" not in prob["reason"] -def test_cyclic_exports_fail_closed_to_zero_depths() -> None: - """A dependency cycle in a malformed export makes _depths fail closed (depth 0).""" - project = { - "subjects": [ - { - "subjectId": "s", - "name": "S", - "skills": [ - {"skillId": "a", "name": "A", "description": "a"}, - {"skillId": "b", "name": "B", "description": "b"}, - ], - } - ], - "dependencies": [ - {"skillId": "a", "dependsOnSkillId": "b"}, - {"skillId": "b", "dependsOnSkillId": "a"}, - ], - "quizzes": [], - } - result = recommend_next_skills(project) - assert result["available"] is True - assert all(row["depth"] == 0 for row in result["recommendations"]) +@pytest.mark.parametrize( + "dependency", + [ + {"skillId": "math_sets", "dependsOnSkillId": "prob_events"}, + {"skillId": "prob_events", "dependsOnSkillId": "missing"}, + {"skillId": "prob_events", "dependsOnSkillId": "prob_events"}, + {"skillId": "prob_events", "dependsOnSkillId": "math_sets"}, + "not an edge", + ], +) +def test_invalid_export_graph_cannot_produce_an_unsuccessful_retake(dependency) -> None: + project = _project() + project["dependencies"].append(dependency) + result = recommend_next_skills( + project, + unsuccessful_skill_ids=["bayes_update"], + ) + assert result["available"] is False + assert result["recommendations"] == [] + assert result["error"] diff --git a/tests/test_release_readiness_inventory.py b/tests/test_release_readiness_inventory.py index 4bf8145..193d9c4 100644 --- a/tests/test_release_readiness_inventory.py +++ b/tests/test_release_readiness_inventory.py @@ -280,10 +280,44 @@ def test_summarize_probe_distinguishes_reachable_and_blocked() -> None: assert _summarize_probe(silent) == "blocked (exit 3)" -def test_collect_blocker_probes_returns_string_summary(tmp_path: Path) -> None: - from src.release_readiness import BlockerProbeSummary, collect_blocker_probes +def test_public_probe_distinguishes_empty_populated_and_tags_only_local_repositories( + tmp_path: Path, +) -> None: + from src.release_readiness import probe_public_repository + + empty = tmp_path / "empty.git" + subprocess.run(["git", "init", "--bare", "--quiet", str(empty)], check=True) + result = probe_public_repository(tmp_path, str(empty)) + assert result.returncode == 0 + assert result.stdout.strip() == "" + + root = _repo_with_dirty_release_state(tmp_path) + populated = probe_public_repository(tmp_path, str(root)) + assert populated.returncode == 0 + assert any( + line.split()[1].startswith("refs/heads/") + for line in populated.stdout.splitlines() + ) - summary = collect_blocker_probes(tmp_path) - assert isinstance(summary, BlockerProbeSummary) - assert isinstance(summary.public_repository, str) and summary.public_repository - assert isinstance(summary.docker_daemon, str) and summary.docker_daemon + subprocess.run(["git", "-C", str(root), "tag", "fixture_tag"], check=True) + branch = subprocess.run( + ["git", "-C", str(root), "symbolic-ref", "HEAD"], + check=True, + capture_output=True, + text=True, + ).stdout.strip() + subprocess.run(["git", "-C", str(root), "update-ref", "-d", branch], check=True) + tags_only = probe_public_repository(tmp_path, str(root)) + assert tags_only.returncode == 0 + assert {line.split()[1] for line in tags_only.stdout.splitlines()} == { + "refs/tags/fixture_tag" + } + + +def test_public_probe_missing_local_target_is_not_an_empty_success( + tmp_path: Path, +) -> None: + from src.release_readiness import probe_public_repository + + result = probe_public_repository(tmp_path, str(tmp_path / "missing.git")) + assert result.returncode != 0 diff --git a/tests/test_render_manuscript.py b/tests/test_render_manuscript.py index 5f6e105..7addfdc 100644 --- a/tests/test_render_manuscript.py +++ b/tests/test_render_manuscript.py @@ -6,49 +6,83 @@ import json import shutil import subprocess -from pathlib import Path - -from scripts.render_manuscript import PDF_NAME, ROOT, _write_pdf_receipt - - -def test_pdf_receipt_records_finalized_output(tmp_path: Path) -> None: - subprocess.run(["git", "init", "--quiet", str(tmp_path)], check=True) - subprocess.run( - [ - "git", - "-c", - "user.name=Receipt Test", - "-c", - "user.email=receipt@example.org", - "commit", - "--quiet", - "--allow-empty", - "-m", - "Source revision", - ], - cwd=tmp_path, - check=True, - ) - revision = subprocess.run( - ["git", "rev-parse", "HEAD"], - cwd=tmp_path, - capture_output=True, - text=True, - check=True, - ).stdout.strip() - target = tmp_path / "output" / "pdf" / PDF_NAME - target.parent.mkdir(parents=True) - shutil.copyfile(ROOT / "output" / "pdf" / PDF_NAME, target) + +import pytest + +from scripts import render_manuscript as renderer +from src.manuscript.pdf_receipt import capture_pdf_inputs, pdf_receipt_issues + + +def test_pdf_receipt_records_finalized_output_and_actual_inputs(pdf_project) -> None: + target, root = pdf_project["target"], pdf_project["root"] receipt = target.with_suffix(".receipt.json") receipt.write_text("outdated receipt", encoding="utf-8") - assert _write_pdf_receipt(target, root=tmp_path) == receipt + assert ( + renderer._write_pdf_receipt(target, root=root, inputs=pdf_project["inputs"]) + == receipt + ) content = target.read_bytes() - assert json.loads(receipt.read_text(encoding="utf-8")) == { - "schema": "active-skillference-pdf/v1", - "output_path": f"output/pdf/{PDF_NAME}", - "size_bytes": len(content), - "sha256": hashlib.sha256(content).hexdigest(), - "source_revision": revision, + payload = json.loads(receipt.read_text(encoding="utf-8")) + assert payload["schema"] == "active-skillference-pdf/v2" + assert payload["output_path"] == f"output/pdf/{renderer.PDF_NAME}" + assert payload["size_bytes"] == len(content) + assert payload["sha256"] == hashlib.sha256(content).hexdigest() + assert payload["source_revision"] == pdf_project["revision"] + expected = { + "manuscript/config.yaml", + "manuscript/preamble.md", + "manuscript/references.bib", + "output/manuscript/01_fixture.md", + "output/manuscript/assets/cover.pdf", + "output/manuscript/assets/atlas.pdf", + "output/manuscript/assets/body.pdf", + } + rows = payload["inputManifest"]["files"] + assert {row["path"] for row in rows} == expected + for row in rows: + actual = (root / row["path"]).read_bytes() + assert row["size_bytes"] == len(actual) + assert row["sha256"] == hashlib.sha256(actual).hexdigest() + assert pdf_receipt_issues(root) == [] + + +@pytest.mark.slow +def test_real_render_places_configured_atlas_on_physical_page_two( + pdf_project, monkeypatch +) -> None: + required = ("pandoc", "pandoc-crossref", "xelatex", "bibtex", "pdftotext") + missing = [tool for tool in required if shutil.which(tool) is None] + if missing: + pytest.skip("existing render tools unavailable: " + ", ".join(missing)) + root = pdf_project["root"] + monkeypatch.setattr(renderer, "ROOT", root) + monkeypatch.setattr(renderer, "MANUSCRIPT", root / "manuscript") + monkeypatch.setattr(renderer, "OUTPUT_MANUSCRIPT", root / "output" / "manuscript") + monkeypatch.setattr(renderer, "OUTPUT_PDF", root / "output" / "pdf") + scratch = root / "build" + scratch.mkdir() + inputs = capture_pdf_inputs(root, [pdf_project["section"]]) + + target = renderer._render_pdf(scratch, renderer._config(), inputs=inputs) + + extracted = subprocess.run( + ["pdftotext", "-layout", str(target), "-"], + check=True, + capture_output=True, + text=True, + ).stdout.split("\f") + assert "FixtureTitlePage" in extracted[0] + assert "FixtureAtlasPage" in extracted[1] + assert "FixtureAtlasCaption" in extracted[1] + assert "AtlasGraphic" in extracted[1] + assert "FixtureBodySection" not in extracted[1] + assert "FixtureBodySection" in "\f".join(extracted[2:]) + receipt = json.loads( + target.with_suffix(".receipt.json").read_text(encoding="utf-8") + ) + assert set(receipt["inputManifest"]["assets"]) == { + path.relative_to(root).as_posix() for path in pdf_project["assets"] } + assert pdf_receipt_issues(root) == [] diff --git a/tests/test_roadmap.py b/tests/test_roadmap.py index 870e0f5..165ca22 100644 --- a/tests/test_roadmap.py +++ b/tests/test_roadmap.py @@ -671,3 +671,55 @@ def _copy_minimal_source_surfaces(tmp_path: Path) -> None: (manuscript_dir / src.name).write_text( src.read_text(encoding="utf-8"), encoding="utf-8" ) + + +def test_blocked_task_retains_named_blocker_without_completion_evidence( + tmp_path: Path, +) -> None: + payload = _valid_roadmap_payload() + task = payload["tasks"][1] + task.update( + status="blocked", + evidence=[], + notes="Blocked until docker info reported: daemon unavailable.", + ) + _write_roadmap(tmp_path, payload) + + assert audit_roadmap(tmp_path) == [] + todo = (tmp_path / "TODO.md").read_text(encoding="utf-8") + details = "\n".join(render_todo_detail_pages(payload).values()) + assert "[!]" in todo and task["id"] in todo + assert "daemon unavailable." in details + + task.update(status="in_progress", notes="Implementation admitted.") + _write_roadmap(tmp_path, payload) + assert audit_roadmap(tmp_path) == [] + assert "[~]" in (tmp_path / "TODO.md").read_text(encoding="utf-8") + assert "daemon unavailable." not in "\n".join( + render_todo_detail_pages(payload).values() + ) + + +def test_live_count_diagnostic_keeps_real_line_number_after_fenced_example( + tmp_path: Path, +) -> None: + _copy_minimal_source_surfaces(tmp_path) + current = build_live_counts(REPO)["skill_count"].value + stale = current + 7 + (tmp_path / "README.md").write_text( + f"# Counts\n```text\nExample: {stale} skills.\n```\n\nCurrent: {stale} skills.\n", + encoding="utf-8", + ) + + issues = [ + issue + for issue in audit_live_count_literals(tmp_path) + if issue.key == "skill_count" + ] + + assert len(issues) == 1 + assert ( + issues[0].line == 6 + and issues[0].found == stale + and issues[0].expected == current + ) diff --git a/tests/test_run_entrypoint.py b/tests/test_run_entrypoint.py new file mode 100644 index 0000000..35b747a --- /dev/null +++ b/tests/test_run_entrypoint.py @@ -0,0 +1,80 @@ +"""Run the real entrypoint without fake uv/docker executables or repository writes.""" + +from __future__ import annotations + +import os +import shutil +import stat +import subprocess +from pathlib import Path + +import pytest + +REPO = Path(__file__).resolve().parents[1] + + +@pytest.mark.parametrize( + "argument,code", [("--help", 0), ("--unknown", 2), ("--check", 1), ("--offline", 1)] +) +@pytest.mark.parametrize("existing", [False, True]) +def test_argument_and_prerequisite_exits_do_not_initialize_postgres_state( + tmp_path: Path, + argument: str, + code: int, + existing: bool, +) -> None: + bash = shutil.which("bash", path=os.defpath) + if bash is None: + pytest.skip("existing bash runtime unavailable") + script = tmp_path / "run.sh" + shutil.copyfile(REPO / "run.sh", script) + programs = tmp_path / "programs" + programs.mkdir() + # Real system utilities make the old entropy path observable, but no uv or + # Docker binary is admitted: --check/--offline must stop at that prerequisite. + for name in ("dirname", "sed", "mkdir", "cat", "tr", "head", "chmod"): + source = shutil.which(name, path=os.defpath) + if source is None: + pytest.skip(f"existing system utility unavailable: {name}") + (programs / name).symlink_to(source) + state = tmp_path / "private-state" + password = state / "postgres_password" + before = None + if existing: + state.mkdir() + password.write_text("kept-synthetic-password\n", encoding="utf-8") + password.chmod(0o600) + before = ( + password.read_bytes(), + password.stat().st_mtime_ns, + stat.S_IMODE(password.stat().st_mode), + ) + env = { + key: value + for key, value in os.environ.items() + if key not in {"ACTIVE_SKILLFERENCE_POSTGRES_PASSWORD", "BASH_ENV", "ENV"} + and not key.startswith("BASH_FUNC_") + } + env.update(PATH=str(programs), ACTIVE_SKILLFERENCE_LOCAL_STATE_DIR=str(state)) + + result = subprocess.run( + [bash, str(script), argument], + cwd=tmp_path, + env=env, + capture_output=True, + text=True, + timeout=5, + check=False, + ) + + assert result.returncode == code, result.stdout + result.stderr + if argument in ("--check", "--offline"): + assert "uv is not installed" in result.stderr + if existing: + assert ( + password.read_bytes(), + password.stat().st_mtime_ns, + stat.S_IMODE(password.stat().st_mode), + ) == before + else: + assert not state.exists() diff --git a/tests/test_scholarship.py b/tests/test_scholarship.py index 60c3f08..94c7c77 100644 --- a/tests/test_scholarship.py +++ b/tests/test_scholarship.py @@ -1386,3 +1386,38 @@ def test_role_gate_rejects_lane_name_padding() -> None: ) is True ) + + +@pytest.mark.parametrize("status,bucket", [(403, "botBlocks"), (404, "deadLinks")]) +def test_online_source_failure_appears_once_in_network_report( + tmp_path: Path, httpserver, status: int, bucket: str +) -> None: + httpserver.expect_request("/failed-source", method="HEAD").respond_with_data( + "", status=status + ) + _manuscript, matrix_path, bib_path = _write_fixture( + tmp_path, + manuscript="Claim [@known].", + bib="@article{known,\n}\n", + matrix=_health_matrix(httpserver.url_for("/failed-source")), + ) + health = audit_source_health( + matrix_path, bib_path, online=True, timeout=5.0, allow_private=True + ) + scholarship = validate_scholarship( + REPO / "manuscript", + REPO / "data" / "scholarship_sources.yaml", + REPO / "manuscript" / "references.bib", + ) + gaps = validate_scholarship_gaps( + scholarship, + _real_claim_support(), + REPO / "manuscript" / "references.bib", + ) + + report = scholarship_source_health_network( + scholarship=scholarship, source_health=health, gap_report=gaps + ) + + assert report["summary"][bucket] == 1 + assert [row["key"] for row in report[bucket]] == ["known"] diff --git a/tests/test_screenshot_context.py b/tests/test_screenshot_context.py new file mode 100644 index 0000000..30591c9 --- /dev/null +++ b/tests/test_screenshot_context.py @@ -0,0 +1,257 @@ +"""Association-level fixtures are not browser proof; HTTP and proof files are real.""" + +from __future__ import annotations + +import hashlib +import json +from dataclasses import replace +from pathlib import Path + +import pytest +from pytest_httpserver import HTTPServer + +from src.screenshots.capture import ( + CaptureSettings, + ScreenshotCaptureError, + _capture_target, + _capture_url, + capture_context, + capture_screenshots, +) + + +def _case(httpserver, tmp_path: Path): + url = httpserver.url_for("").rstrip("/") + digest = hashlib.sha256(b"private-export-fixture").hexdigest() + instance = "capture-run-fixture" + status = { + "instanceId": instance, + "project": { + "exportContentDigest": digest, + "counts": {"skills": 1, "subjects": 1, "dependencies": 0, "quizzes": 1}, + }, + "config": {"project": {"id": "p"}, "skilltree": {"url": url}}, + "skilltree": {"available": True}, + "liveSkilltree": { + "checked": True, + "complete": True, + "expectedDependencies": 0, + "matchedDependencies": 0, + "content": { + "checked": True, + "complete": True, + "exportContentDigest": digest, + }, + }, + } + httpserver.expect_request("/api/status", method="GET").respond_with_json(status) + browser = { + "state": "ready", + "url": url, + "projectId": "p", + "route": "/", + "dashboardInstanceId": instance, + "exportContentDigest": digest, + "frameUrl": url + "/skills-display/p", + "frameVisible": True, + "busy": False, + } + receipt = { + "checked": True, + "ok": True, + "projectId": "p", + "dashboardInstanceId": instance, + "exportContentDigest": digest, + "views": [ + { + "kind": kind, + "route": route, + "state": "ready", + "visibleContentChecked": True, + } + for kind, route in ( + ("project", "/"), + ("subject", "/subjects/s"), + ("quiz", "/subjects/s/skills/k/quizzes/q"), + ) + ], + } + proof = { + "proofVersion": 2, + "ok": True, + "mode": "full-parity", + "dashboardUrl": url, + "skilltreeUrl": url, + "projectId": "p", + "runId": instance, + "dashboardInstanceId": instance, + "exportContentDigest": digest, + "liveContent": { + "checked": True, + "complete": True, + "exportContentDigest": digest, + }, + "liveDependencies": {"checked": True, "complete": True}, + "browserConsole": receipt, + } + path = tmp_path / "explicit-proof.json" + settings = CaptureSettings(url, tmp_path / "captures", tmp_path, proof_path=path) + return settings, status, browser, proof + + +def test_explicit_matching_v2_proof_is_bound_to_real_capture_time_status( + httpserver, tmp_path: Path +) -> None: + settings, _status, browser, proof = _case(httpserver, tmp_path) + settings.proof_path.write_text(json.dumps(proof), encoding="utf-8") + raw = settings.proof_path.read_bytes() + context = capture_context(settings, browser_probe=browser) + association = context["proofAssociation"] + assert association["checked"] is True and association["mode"] == "full-parity" + assert association["runId"] == context["dashboardInstanceId"] + assert association["proofSha256"] == hashlib.sha256(raw).hexdigest() + assert context["exportContentDigest"] == proof["exportContentDigest"] + assert settings.proof_path.read_bytes() == raw + + +@pytest.mark.parametrize( + "damage", + [ + "failed", + "legacy", + "url", + "skilltree", + "project", + "digest", + "instance", + "content", + "browser", + "dependencies", + ], +) +def test_unrelated_or_incomplete_proof_cannot_certify_current_capture( + httpserver, tmp_path: Path, damage: str +) -> None: + settings, _status, browser, proof = _case(httpserver, tmp_path) + if damage == "failed": + proof["ok"] = False + elif damage == "legacy": + proof["proofVersion"] = 1 + elif damage == "url": + proof["dashboardUrl"] += "/another-dashboard" + elif damage == "skilltree": + proof["skilltreeUrl"] = "http://127.0.0.1:1" + elif damage == "project": + proof["projectId"] = "other" + elif damage == "digest": + proof["exportContentDigest"] = "b" * 64 + elif damage == "instance": + proof["dashboardInstanceId"] = "different-run" + elif damage == "content": + proof["liveContent"]["complete"] = False + elif damage == "browser": + proof["browserConsole"]["checked"] = False + else: + proof["liveDependencies"]["complete"] = False + settings.proof_path.write_text(json.dumps(proof), encoding="utf-8") + before = settings.proof_path.read_bytes() + context = capture_context(settings, browser_probe=browser) + assert context["proofAssociation"]["checked"] is False + assert context["fullAppStackProofMode"] == "not associated" + assert settings.proof_path.read_bytes() == before + + +def test_missing_malformed_or_implicit_proof_is_unassociated_without_rewriting_it( + httpserver, tmp_path: Path +) -> None: + settings, _status, browser, proof = _case(httpserver, tmp_path) + assert ( + capture_context(settings, browser_probe=browser)["proofAssociation"]["checked"] + is False + ) + settings.proof_path.write_text("not JSON", encoding="utf-8") + assert ( + capture_context(settings, browser_probe=browser)["proofAssociation"]["checked"] + is False + ) + assert settings.proof_path.read_text(encoding="utf-8") == "not JSON" + settings.proof_path.write_text(json.dumps(proof), encoding="utf-8") + assert ( + capture_context(replace(settings, proof_path=None), browser_probe=browser)[ + "proofAssociation" + ]["checked"] + is False + ) + assert capture_context(settings)["proofAssociation"]["checked"] is False + + +@pytest.mark.parametrize( + "field,value", + [ + ("state", "fallback"), + ("dashboardInstanceId", "old"), + ("exportContentDigest", "c" * 64), + ("projectId", "other"), + ("url", "not a URL"), + ], +) +def test_browser_snapshot_must_match_current_status( + httpserver, tmp_path: Path, field: str, value: str +) -> None: + settings, _status, browser, _proof = _case(httpserver, tmp_path) + browser[field] = value + with pytest.raises(ScreenshotCaptureError, match="does not match"): + capture_context(settings, browser_probe=browser) + assert not settings.output_dir.exists() + + +@pytest.mark.parametrize( + "url", + [ + "file:///tmp/not-a-dashboard", + "https://example.invalid", + "http://user:password@127.0.0.1", + "http://127.0.0.1/?query=yes", + "http://127.0.0.1/#fragment", + ], +) +def test_capture_url_admission_is_pure_and_rejects_nonlocal_or_credential_bearing_targets( + url: str, +) -> None: + with pytest.raises(ScreenshotCaptureError): + _capture_url(url) + + +def test_capture_target_comparison_normalizes_origin_and_default_port_without_merging_paths() -> ( + None +): + assert _capture_target("HTTP://LOCALHOST:80/") == _capture_target( + "http://localhost" + ) + assert _capture_target("http://localhost/a") != _capture_target( + "http://localhost/b" + ) + + +def test_off_origin_loopback_redirect_is_refused_before_browser_or_output_creation( + httpserver, tmp_path: Path +) -> None: + unrelated = HTTPServer(host="127.0.0.1", port=0) + unrelated.start() + try: + httpserver.expect_request("/", method="GET").respond_with_data( + "", + status=302, + headers={"Location": unrelated.url_for("/")}, + ) + settings = CaptureSettings( + httpserver.url_for(""), tmp_path / "captures", tmp_path + ) + with pytest.raises( + ScreenshotCaptureError, match="changed the requested origin" + ): + capture_screenshots(settings) + assert unrelated.log == [] + assert not settings.output_dir.exists() + finally: + unrelated.stop() diff --git a/tests/test_screenshots.py b/tests/test_screenshots.py index 74fcdf2..726673d 100644 --- a/tests/test_screenshots.py +++ b/tests/test_screenshots.py @@ -9,6 +9,7 @@ from __future__ import annotations +import hashlib import json import re import subprocess @@ -173,7 +174,10 @@ def test_captured_images_are_tracked_and_carry_provenance() -> None: image = SCREENSHOT_DIR / view.filename assert image.is_file(), f"{view.filename} is not tracked" assert image.stat().st_size > 0 - assert recorded[view.filename]["sha256"] + assert ( + recorded[view.filename]["sha256"] + == hashlib.sha256(image.read_bytes()).hexdigest() + ) def test_captured_images_are_readable_and_not_blank() -> None: diff --git a/tests/test_scripts_smoke.py b/tests/test_scripts_smoke.py index bbf90c9..50c0567 100644 --- a/tests/test_scripts_smoke.py +++ b/tests/test_scripts_smoke.py @@ -5,6 +5,7 @@ import importlib.util import json import os +import shutil import subprocess import sys import urllib.error @@ -28,6 +29,27 @@ _SUBPROCESS_TIMEOUT = 300 +@pytest.fixture(scope="module", autouse=True) +def isolated_script_project(tmp_path_factory): + """Run real generators in disposable source, never in the audited checkout.""" + global REPO + source = REPO + project = tmp_path_factory.mktemp("script-project") + for name in ("data", "manuscript", "scripts", "src", "gnn", "tests"): + shutil.copytree( + source / name, + project / name, + ignore=shutil.ignore_patterns("__pycache__", "*.pyc"), + ) + for name in ("docker-compose.yml", "tasks.yaml", "pyproject.toml"): + shutil.copyfile(source / name, project / name) + REPO = project + try: + yield project + finally: + REPO = source + + def _run(script: str) -> subprocess.CompletedProcess[str]: return subprocess.run( [sys.executable, str(REPO / "scripts" / script)], @@ -70,22 +92,6 @@ def test_validate_docker_digests_helper_flags_tag_only_lines() -> None: ) -def test_local_skilltree_runtime_security_defaults() -> None: - compose = (REPO / "docker-compose.yml").read_text(encoding="utf-8") - run_skilltree = (REPO / "scripts" / "run_skilltree.sh").read_text(encoding="utf-8") - run_dashboard = (REPO / "scripts" / "run_learning_dashboard.sh").read_text( - encoding="utf-8" - ) - - assert "127.0.0.1:5432:5432" in compose - assert "127.0.0.1:8080:8080" in compose - assert "ACTIVE_SKILLFERENCE_POSTGRES_PASSWORD:?" in compose - assert '--token "$SKILLTREE_TOKEN"' not in run_skilltree - assert '--token "$SKILLTREE_TOKEN"' not in run_dashboard - assert "POSTGRES_PASSWORD_FILE" in run_skilltree - assert "POSTGRES_PASSWORD_FILE" in run_dashboard - - def test_form_login_password_handoff_avoids_eval_argv_secret() -> None: module = _load_script_module("validate_skilltree_form_login.py") @@ -282,7 +288,7 @@ def test_validate_american_english_script_runs() -> None: def test_validate_web_math_script_runs_with_rendered_html() -> None: web_dir = REPO / "output" / "web" web_dir.mkdir(parents=True, exist_ok=True) - (web_dir / "math_smoke.html").write_text( + (web_dir / "index.html").write_text( """ @@ -477,8 +483,7 @@ def test_validate_live_skilltree_dashboard_script_accepts_healthy_payload( httpserver.expect_request("/api/status").respond_with_json( { "project": { - "projectId": "activeInference", - "name": "Active Skillference", + "project": {"id": "activeInference", "name": "Active Skillference"}, "counts": counts, }, "config": { @@ -635,8 +640,7 @@ def _serve_strict_live_dashboard_payload( httpserver.expect_request("/api/status").respond_with_json( { "project": { - "projectId": "activeInference", - "name": "Active Skillference", + "project": {"id": "activeInference", "name": "Active Skillference"}, "counts": counts, }, "config": { @@ -897,8 +901,7 @@ def test_validate_live_skilltree_dashboard_rejects_missing_live_dependency( httpserver.expect_request("/api/status").respond_with_json( { "project": { - "projectId": "activeInference", - "name": "Active Skillference", + "project": {"id": "activeInference", "name": "Active Skillference"}, "dependencies": [{"skillId": "leaf", "dependsOnSkillId": "root"}], "counts": { "subjects": 1, diff --git a/tests/test_skilltree_client.py b/tests/test_skilltree_client.py index 197016c..0545c0f 100644 --- a/tests/test_skilltree_client.py +++ b/tests/test_skilltree_client.py @@ -133,7 +133,9 @@ def test_create_quiz_does_not_duplicate_existing_questions( ).respond_with_json({"quizId": "bayes_quiz"}) httpserver.expect_request( "/admin/quiz-definitions/bayes_quiz/questions", method="GET" - ).respond_with_json({"quizType": "Quiz", "questions": [{"id": 1}]}) + ).respond_with_json( + {"quizType": "Quiz", "questions": [quiz.questions[0].to_dict()]} + ) assert client.create_quiz(quiz) == {"quizId": "bayes_quiz"} @@ -154,7 +156,9 @@ def test_create_quiz_appends_missing_question_suffix( ).respond_with_json({"quizId": "bayes_quiz"}) httpserver.expect_request( "/admin/quiz-definitions/bayes_quiz/questions", method="GET" - ).respond_with_json({"quizType": "Quiz", "questions": [{"id": 1}]}) + ).respond_with_json( + {"quizType": "Quiz", "questions": [quiz.questions[0].to_dict()]} + ) httpserver.expect_request( "/admin/quiz-definitions/bayes_quiz/create-question", method="POST", diff --git a/tests/test_skilltree_content.py b/tests/test_skilltree_content.py new file mode 100644 index 0000000..7bebf6d --- /dev/null +++ b/tests/test_skilltree_content.py @@ -0,0 +1,330 @@ +"""Real loopback regressions for identity/content parity and idempotent quiz seeding.""" + +from __future__ import annotations + +import copy +import json +from pathlib import Path + +import pytest +from pytest_httpserver import HTTPServer + +from src.dashboard.artifacts import load_skilltree_project +from src.dashboard.config import DashboardConfig +from src.dashboard.payload import live_skilltree_summary +from src.skilltree.client import SkillTreeClient +from src.skilltree.content import ( + SkillTreeContentError, + compare_live_content, + export_content_digest, +) +from src.skilltree.local_auth import ( + LocalSkillTreeState, + SkillTreeSessionClient, + ensure_local_skilltree_auth, + save_local_state, +) +from src.skilltree.local_auth_state import LocalSkillTreeAuthError +from src.skilltree.model import SkillTreeProject + + +def _compare(httpserver: HTTPServer, project: SkillTreeProject) -> dict: + client = SkillTreeClient( + httpserver.url_for(""), project.project_id, token="synthetic-test-token" + ) + return compare_live_content( + get=client._get, + base_url=client.base_url, + project=project.to_export_dict(), + ) + + +def test_matching_actual_content_is_digest_bound_and_answer_key_free( + httpserver: HTTPServer, + content_project, + skilltree_admin_content, +) -> None: + skilltree_admin_content(httpserver, content_project) + result = _compare(httpserver, content_project) + assert result["checked"] is True and result["complete"] is True + assert result["projectId"] == "p" + assert result["exportContentDigest"] == export_content_digest( + content_project.to_export_dict() + ) + assert "isCorrect" not in json.dumps(result) + assert "synthetic-test-token" not in json.dumps(result) + + +@pytest.mark.parametrize("change", ["answer", "lesson", "extra_question"]) +def test_same_counts_cannot_hide_changed_content( + httpserver: HTTPServer, + content_project, + skilltree_admin_content, + change: str, +) -> None: + live = copy.deepcopy(content_project.to_export_dict()) + if change == "answer": + live["quizzes"][0]["questions"][0]["answers"][0]["isCorrect"] = False + elif change == "lesson": + live["subjects"][0]["skills"][0]["description"] = "Different lesson" + else: + live["quizzes"][0]["questions"].append(live["quizzes"][0]["questions"][0]) + skilltree_admin_content(httpserver, live) + result = _compare(httpserver, content_project) + assert result["checked"] is True and result["complete"] is False + assert result["changedResources"] + assert result["subjects"] == 1 and result["skills"] == 1 + + +def test_count_only_admin_response_is_unverifiable( + httpserver: HTTPServer, content_project +) -> None: + httpserver.expect_request("/admin/projects/p", method="GET").respond_with_json( + {"numSkills": 1, "numSubjects": 1} + ) + with pytest.raises(SkillTreeContentError, match="unverifiable"): + _compare(httpserver, content_project) + + +@pytest.mark.parametrize("client_type", [SkillTreeClient, SkillTreeSessionClient]) +def test_existing_second_question_does_not_suppress_missing_first( + httpserver: HTTPServer, + content_project, + client_type, +) -> None: + quiz = content_project.quizzes[0] + httpserver.expect_request( + "/admin/quiz-definitions/q", method="POST" + ).respond_with_json({}) + httpserver.expect_request( + "/admin/quiz-definitions/q/questions", method="GET" + ).respond_with_json({"questions": [quiz.questions[1].to_dict()]}) + httpserver.expect_request( + "/admin/quiz-definitions/q/create-question", + method="POST", + json=quiz.questions[0].to_admin_create_payload(), + ).respond_with_json({"id": 1}) + client = client_type(httpserver.url_for(""), "p") + client.create_quiz(quiz) + posted = [ + request.get_json() + for request, _ in httpserver.log + if request.path.endswith("/create-question") + ] + assert posted == [quiz.questions[0].to_admin_create_payload()] + httpserver.check_assertions() + + +@pytest.mark.parametrize("client_type", [SkillTreeClient, SkillTreeSessionClient]) +def test_extra_question_occurrence_is_refused_without_any_question_write( + httpserver: HTTPServer, + content_project, + client_type, +) -> None: + quiz = content_project.quizzes[0] + questions = [question.to_dict() for question in quiz.questions] + httpserver.expect_request( + "/admin/quiz-definitions/q", method="POST" + ).respond_with_json({}) + httpserver.expect_request( + "/admin/quiz-definitions/q/questions", method="GET" + ).respond_with_json({"questions": questions + [questions[0]]}) + with pytest.raises(SkillTreeContentError, match="extra or changed"): + client_type(httpserver.url_for(""), "p").create_quiz(quiz) + assert not any( + request.path.endswith("/create-question") for request, _ in httpserver.log + ) + + +def test_live_summary_never_sends_stored_credentials_to_second_origin( + tmp_path: Path, + httpserver: HTTPServer, + content_project, +) -> None: + original = httpserver.url_for("").rstrip("/") + second = HTTPServer(host="127.0.0.1", port=0) + second.start() + try: + state_dir = tmp_path / "private" + save_local_state( + state_dir, + LocalSkillTreeState( + admin_username="admin@localhost.invalid", + admin_password="stored-secret", + learner_id="learner", + project_id="p", + skilltree_url=original, + client_secret="trusted-secret", + ), + ) + config = DashboardConfig( + project_root=tmp_path, + project_id="p", + project_name="Project", + skilltree_url=second.url_for(""), + local_auth_enabled=True, + local_state_dir=state_dir, + ) + exported = content_project.to_export_dict() + public = load_skilltree_project(tmp_path, export_payload=exported) + result = live_skilltree_summary(config, public, export_payload=exported) + assert result["checked"] is False + assert "does not match" in result["reason"] + assert second.log == [] + assert httpserver.log == [] + assert "stored-secret" not in json.dumps(result) + finally: + second.stop() + + +def test_bootstrap_target_mismatch_is_refused_before_any_network( + tmp_path: Path, + httpserver: HTTPServer, + content_project, +) -> None: + with pytest.raises(LocalSkillTreeAuthError, match="bootstrap target"): + ensure_local_skilltree_auth( + base_url=httpserver.url_for(""), + project_id="different", + project=content_project, + state_dir=tmp_path, + ) + assert httpserver.log == [] + + +@pytest.mark.parametrize("client_type", [SkillTreeClient, SkillTreeSessionClient]) +def test_missing_duplicate_question_occurrence_is_created_once( + httpserver, content_project, client_type +) -> None: + quiz = content_project.quizzes[0] + first = copy.deepcopy(quiz.questions[0]) + quiz.questions[:] = [first, copy.deepcopy(first)] + httpserver.expect_request( + "/admin/quiz-definitions/q", method="POST" + ).respond_with_json({}) + httpserver.expect_request( + "/admin/quiz-definitions/q/questions", method="GET" + ).respond_with_json({"questions": [first.to_dict()]}) + httpserver.expect_request( + "/admin/quiz-definitions/q/create-question", + method="POST", + json=first.to_admin_create_payload(), + ).respond_with_json({"id": 2}) + + client_type(httpserver.url_for(""), "p").create_quiz(quiz) + + posted = [ + request.get_json() + for request, _response in httpserver.log + if request.path.endswith("/create-question") + ] + assert posted == [first.to_admin_create_payload()] + httpserver.check_assertions() + + +@pytest.mark.parametrize("changed", ["lesson", "answer"]) +def test_existing_changed_content_is_refused_before_any_seed_write( + httpserver, + content_project, + skilltree_admin_content, + changed: str, +) -> None: + from src.skilltree.local_auth import _seed_project_if_requested + + live = copy.deepcopy(content_project.to_export_dict()) + if changed == "lesson": + live["subjects"][0]["skills"][0]["description"] = "Different lesson" + else: + live["quizzes"][0]["questions"][0]["answers"][0]["isCorrect"] = False + skilltree_admin_content(httpserver, live) + client = SkillTreeSessionClient(httpserver.url_for(""), "p") + with pytest.raises(LocalSkillTreeAuthError, match="not overwritten"): + _seed_project_if_requested( + client=client, + project=content_project, + seed_project=True, + seed_dependencies=False, + skip_seed_if_live_content_ready=True, + verify_seed=True, + ) + assert httpserver.log and all( + request.method == "GET" for request, _response in httpserver.log + ) + + +@pytest.mark.parametrize( + ("settings", "complete"), + [ + ([], True), + ([{"setting": "quizUserRole", "value": "ROLE_QUIZ_ADMIN"}], True), + ([{"setting": "quizLength", "value": "-1"}], True), + ([{"setting": "quizPassingReq", "value": "-1"}], True), + ([{"setting": "quizPassingReq", "value": "0"}], True), + ([{"setting": "quizPassingReq", "value": "2"}], True), + ([{"setting": "quizPassingReq", "value": "1"}], False), + ([{"setting": "quizLength", "value": "1"}], False), + ], +) +def test_native_quiz_settings_preserve_full_question_completion( + httpserver, + content_project, + skilltree_admin_content, + settings, + complete, +) -> None: + skilltree_admin_content( + httpserver, + content_project, + quiz_settings={"q": settings}, + ) + result = _compare(httpserver, content_project) + assert result["complete"] is complete + assert result["changedResources"] == ([] if complete else ["quiz:q"]) + assert result["missingResources"] == [] + httpserver.check_assertions() + + +def test_export_percentage_is_compared_with_native_correct_question_count( + httpserver, + content_project, + skilltree_admin_content, +) -> None: + from dataclasses import replace + + content_project.quizzes[0] = replace(content_project.quizzes[0], passing_percent=50) + skilltree_admin_content( + httpserver, + content_project, + quiz_settings={"q": [{"setting": "quizPassingReq", "value": "1"}]}, + ) + assert _compare(httpserver, content_project)["complete"] is True + + +@pytest.mark.parametrize( + "settings", + [ + [{"setting": "quizPassingReq", "value": True}], + [{"setting": "quizPassingReq", "value": "NaN"}], + [{"setting": "quizPassingReq", "value": "3"}], + [{"setting": "quizLength", "value": "0"}], + [{"setting": "quizLength", "value": "-2"}], + [ + {"setting": "quizPassingReq", "value": "2"}, + {"setting": "quizPassingReq", "value": "1"}, + ], + ], +) +def test_unverifiable_native_quiz_settings_fail_closed( + httpserver, + content_project, + skilltree_admin_content, + settings, +) -> None: + skilltree_admin_content( + httpserver, + content_project, + quiz_settings={"q": settings}, + ) + with pytest.raises(SkillTreeContentError): + _compare(httpserver, content_project) + assert all(request.method == "GET" for request, _response in httpserver.log) diff --git a/tests/test_skilltree_local_auth.py b/tests/test_skilltree_local_auth.py index 6ccaf05..1a33c5a 100644 --- a/tests/test_skilltree_local_auth.py +++ b/tests/test_skilltree_local_auth.py @@ -686,6 +686,9 @@ def test_admin_create_methods_hit_expected_paths(httpserver: HTTPServer) -> None httpserver.expect_request( "/admin/quiz-definitions/m_sets_quiz", method="POST" ).respond_with_json({}) + httpserver.expect_request( + "/admin/quiz-definitions/m_sets_quiz/questions", method="GET" + ).respond_with_json({"questions": []}) client.create_project("Active Inference") client.create_subject(subject) @@ -745,7 +748,9 @@ def test_admin_create_quiz_appends_missing_question_suffix( ).respond_with_json({}) httpserver.expect_request( "/admin/quiz-definitions/m_sets_quiz/questions", method="GET" - ).respond_with_json({"quizType": "Quiz", "questions": [{"id": 1}]}) + ).respond_with_json( + {"quizType": "Quiz", "questions": [quiz.questions[0].to_dict()]} + ) httpserver.expect_request( "/admin/quiz-definitions/m_sets_quiz/create-question", method="POST", @@ -763,7 +768,7 @@ def test_admin_create_quiz_appends_missing_question_suffix( def _bootstrap_httpserver_expectations( - httpserver: HTTPServer, *, needs_bootstrap: bool + httpserver: HTTPServer, skilltree_admin_content, *, needs_bootstrap: bool ) -> None: httpserver.expect_request("/public/config", method="GET").respond_with_json( {"needToBootstrap": needs_bootstrap} @@ -788,11 +793,7 @@ def _bootstrap_httpserver_expectations( httpserver.expect_request( "/admin/quiz-definitions/m_sets_quiz", method="POST" ).respond_with_json({}) - # Post-seed verification (verify_seed_landed) reads the live skill count; the - # tiny project defines exactly 1 skill, so report 1 to confirm the seed landed. - httpserver.expect_request( - "/admin/projects/activeInference", method="GET" - ).respond_with_json({"numSkills": 1, "numSubjects": 1}) + skilltree_admin_content(httpserver, _tiny_project(), initially_missing=True) httpserver.expect_request( "/admin/projects/activeInference/clientSecret", method="GET" ).respond_with_data("bootstrap-secret", content_type="text/plain") @@ -823,14 +824,40 @@ def _tiny_project() -> SkillTreeProject: ) ], dependencies=[], - quizzes=[Quiz(quiz_id="m_sets_quiz", name="Sets Quiz")], + quizzes=[ + Quiz( + quiz_id="m_sets_quiz", + name="Sets Quiz", + questions=[ + QuizQuestion( + question="Which collection is a set?", + answers=[ + { + "answer": "A collection of distinct elements", + "isCorrect": True, + }, + { + "answer": "An ordered list requiring repetition", + "isCorrect": False, + }, + ], + ) + ], + ) + ], ) def test_ensure_local_skilltree_auth_full_first_bootstrap( - tmp_path: Path, httpserver: HTTPServer + tmp_path: Path, + httpserver: HTTPServer, + skilltree_admin_content, ) -> None: - _bootstrap_httpserver_expectations(httpserver, needs_bootstrap=True) + _bootstrap_httpserver_expectations( + httpserver, + skilltree_admin_content, + needs_bootstrap=True, + ) base = httpserver.url_for("").rstrip("/") result = ensure_local_skilltree_auth( @@ -854,6 +881,14 @@ def test_ensure_local_skilltree_auth_full_first_bootstrap( assert stored is not None assert stored.client_secret == "bootstrap-secret" assert stored.skilltree_url == base + question_writes = [ + request.get_json() + for request, _response in httpserver.log + if request.method == "POST" and request.path.endswith("/create-question") + ] + assert question_writes == [ + _tiny_project().quizzes[0].questions[0].to_admin_create_payload() + ] def test_ensure_local_skilltree_auth_reuses_existing_state( @@ -929,8 +964,10 @@ def test_ensure_local_skilltree_auth_rejects_state_url_mismatch_before_login( httpserver.check_assertions() -def test_ensure_local_skilltree_auth_skips_seed_when_live_counts_ready( - tmp_path: Path, httpserver: HTTPServer +def test_ensure_local_skilltree_auth_skips_seed_when_live_content_matches( + tmp_path: Path, + httpserver: HTTPServer, + skilltree_admin_content, ) -> None: base = httpserver.url_for("").rstrip("/") save_local_state( @@ -953,9 +990,7 @@ def test_ensure_local_skilltree_auth_skips_seed_when_live_counts_ready( content_type="text/html", ) httpserver.expect_request("/performLogin", method="POST").respond_with_json({}) - httpserver.expect_request( - "/admin/projects/activeInference", method="GET" - ).respond_with_json({"numSkills": 1, "numSubjects": 1}) + skilltree_admin_content(httpserver, _tiny_project()) httpserver.expect_request( "/admin/projects/activeInference/clientSecret", method="GET" ).respond_with_data("trusted-secret", content_type="text/plain") @@ -966,7 +1001,7 @@ def test_ensure_local_skilltree_auth_skips_seed_when_live_counts_ready( state_dir=tmp_path, project=_tiny_project(), seed_project=True, - skip_seed_if_live_counts_ready=True, + skip_seed_if_live_content_ready=True, verify_token=False, ) @@ -1283,10 +1318,11 @@ def test_live_skill_count_rejects_missing_num_skills(httpserver: HTTPServer) -> raise AssertionError("missing numSkills did not raise") -def test_verify_seed_landed_passes_when_live_matches(httpserver: HTTPServer) -> None: - httpserver.expect_request("/admin/projects/p", method="GET").respond_with_json( - {"numSkills": 2, "numSubjects": 1} - ) +def test_verify_seed_landed_passes_when_live_matches( + httpserver: HTTPServer, + skilltree_admin_content, +) -> None: + skilltree_admin_content(httpserver, _two_skill_project()) client = SkillTreeSessionClient(httpserver.url_for("/"), "p") # Exactly-equal live count must not raise. verify_seed_landed(client, _two_skill_project()) diff --git a/tests/test_skilltree_seeder.py b/tests/test_skilltree_seeder.py index 2e8fb99..e8572cd 100644 --- a/tests/test_skilltree_seeder.py +++ b/tests/test_skilltree_seeder.py @@ -68,7 +68,8 @@ def test_seed_returns_correct_summary(httpserver: HTTPServer) -> None: def handler(request): # type: ignore[no-untyped-def] order.append(request.path) - return Response("{}", content_type="application/json", status=200) + body = '{"questions": []}' if request.path.endswith("/questions") else "{}" + return Response(body, content_type="application/json", status=200) httpserver.expect_request(re.compile(r"^/.*$")).respond_with_handler(handler) @@ -91,7 +92,8 @@ def test_seed_issues_dependencies_after_all_skills(httpserver: HTTPServer) -> No def handler(request): # type: ignore[no-untyped-def] order.append(request.path) - return Response("{}", content_type="application/json", status=200) + body = '{"questions": []}' if request.path.endswith("/questions") else "{}" + return Response(body, content_type="application/json", status=200) httpserver.expect_request(re.compile(r"^/.*$")).respond_with_handler(handler) @@ -128,7 +130,8 @@ def test_seed_full_path_ordering_exact(httpserver: HTTPServer) -> None: def handler(request): # type: ignore[no-untyped-def] order.append(f"{request.method} {request.path}") - return Response("{}", content_type="application/json", status=200) + body = '{"questions": []}' if request.path.endswith("/questions") else "{}" + return Response(body, content_type="application/json", status=200) httpserver.expect_request(re.compile(r"^/.*$")).respond_with_handler(handler) @@ -159,7 +162,8 @@ def test_seed_can_defer_dependencies_for_dedicated_reconciler( def handler(request): # type: ignore[no-untyped-def] order.append(f"{request.method} {request.path}") - return Response("{}", content_type="application/json", status=200) + body = '{"questions": []}' if request.path.endswith("/questions") else "{}" + return Response(body, content_type="application/json", status=200) httpserver.expect_request(re.compile(r"^/.*$")).respond_with_handler(handler) client = SkillTreeClient( diff --git a/tests/test_skilltree_transport_boundaries.py b/tests/test_skilltree_transport_boundaries.py new file mode 100644 index 0000000..daa21f5 --- /dev/null +++ b/tests/test_skilltree_transport_boundaries.py @@ -0,0 +1,107 @@ +"""Malformed JSON and real socket stalls stay inside the SkillTree error contract.""" + +from __future__ import annotations + +import threading +from concurrent.futures import ThreadPoolExecutor + +import pytest +from pytest_httpserver import HTTPServer +from werkzeug.wrappers import Response + +from src.skilltree.client import SkillTreeClient, SkillTreeClientError +from src.skilltree.local_auth import ( + LocalSkillTreeAuthError, + SkillTreeSessionClient, + exchange_trusted_client_token, +) + + +@pytest.mark.parametrize("method", ["GET", "POST"]) +def test_bearer_read_timeout_is_bounded_and_domain_typed( + httpserver: HTTPServer, method: str +) -> None: + release = threading.Event() + + def stalled_body(): + yield b"{" + release.wait() + yield b"}" + + httpserver.expect_request("/slow", method=method).respond_with_handler( + lambda request: Response(stalled_body(), content_type="application/json") + ) + client = SkillTreeClient( + httpserver.url_for(""), "p", token="private-token", timeout=0.05 + ) + operation = client._get if method == "GET" else client._post + with ThreadPoolExecutor(max_workers=1) as pool: + future = pool.submit(operation, "/slow") + try: + with pytest.raises(SkillTreeClientError, match="timed out") as error: + future.result(timeout=5) + assert "private-token" not in str(error.value) + finally: + release.set() + + +@pytest.mark.parametrize("timeout", [0.0, -1.0, float("inf"), float("nan")]) +def test_bearer_timeout_must_be_finite_and_positive(timeout: float) -> None: + with pytest.raises(ValueError, match="finite and positive"): + SkillTreeClient("http://127.0.0.1:1", "p", timeout=timeout) + + +def test_session_malformed_json_is_domain_typed_and_secret_safe( + httpserver: HTTPServer, +) -> None: + httpserver.expect_request("/broken", method="GET").respond_with_data( + '{"secret": "private-response-fragment"', + content_type="application/json", + ) + client = SkillTreeSessionClient(httpserver.url_for(""), "p") + with pytest.raises(SkillTreeClientError, match="malformed JSON") as error: + client.request_json("GET", "/broken") + assert "private-response-fragment" not in str(error.value) + + +def test_token_malformed_json_is_auth_domain_error(httpserver: HTTPServer) -> None: + httpserver.expect_request("/oauth/token", method="POST").respond_with_data( + '{"access_token": "private-fragment"', + content_type="application/json", + ) + with pytest.raises(LocalSkillTreeAuthError, match="malformed JSON") as error: + exchange_trusted_client_token( + base_url=httpserver.url_for(""), + project_id="p", + client_secret="private-secret", + learner_id="learner", + ) + assert "private-fragment" not in str(error.value) + assert "private-secret" not in str(error.value) + + +def test_token_socket_timeout_is_domain_typed(httpserver: HTTPServer) -> None: + release = threading.Event() + + def stalled_body(): + yield b"{" + release.wait() + yield b"}" + + httpserver.expect_request("/oauth/token", method="POST").respond_with_handler( + lambda request: Response(stalled_body(), content_type="application/json") + ) + with ThreadPoolExecutor(max_workers=1) as pool: + future = pool.submit( + exchange_trusted_client_token, + base_url=httpserver.url_for(""), + project_id="p", + client_secret="synthetic-secret", + learner_id="learner", + timeout=0.05, + ) + try: + with pytest.raises(SkillTreeClientError, match="timed out"): + future.result(timeout=5) + finally: + release.set() diff --git a/tests/test_zenodo.py b/tests/test_zenodo.py new file mode 100644 index 0000000..611efe4 --- /dev/null +++ b/tests/test_zenodo.py @@ -0,0 +1,193 @@ +"""Publication API contracts over loopback, with synthetic credentials only.""" + +from __future__ import annotations + +import json +import os +import urllib.request +from pathlib import Path + +import pytest +from werkzeug.wrappers import Response + +from src.publication import zenodo + +TOKEN = "loopback-test-token" +DOI = "10.5072/fixture.7" + + +@pytest.fixture +def api(httpserver, monkeypatch): + monkeypatch.setattr(zenodo, "ZENODO_API", httpserver.url_for("/").rstrip("/")) + return httpserver + + +def _draft(identifier: int, title: str = "Wanted") -> dict: + return { + "id": identifier, + "title": title, + "state": "unsubmitted", + "submitted": False, + "metadata": {"prereserve_doi": {"doi": DOI}}, + "links": {"html": "http://localhost/draft"}, + } + + +@pytest.fixture +def observed_upload(monkeypatch): + streams = [] + + class ObserveRequest(urllib.request.BaseHandler): + def http_request(self, request): + if request.get_method() == "PUT": + stream = request.data + assert hasattr(stream, "read") and not stream.closed + assert not isinstance(stream, (bytes, bytearray)) + assert stream.tell() == 0 + assert request.get_header("Content-length") == str( + os.fstat(stream.fileno()).st_size + ) + streams.append(stream) + return request + + https_request = http_request + + # A request processor observes the real urllib transfer; no response is faked. + monkeypatch.setattr( + urllib.request, "_opener", urllib.request.build_opener(ObserveRequest()) + ) + return streams + + +def _upload_bucket(api) -> None: + api.expect_request("/deposit/depositions/7", method="GET").respond_with_json( + {"links": {"bucket": api.url_for("/bucket/fixture")}} + ) + + +def test_upload_streams_open_file_with_exact_length_and_closes_it( + api, tmp_path: Path, observed_upload +) -> None: + content = b"actual streamed archive bytes\n" * 32768 + path = tmp_path / "artifact.zip" + path.write_bytes(content) + _upload_bucket(api) + + def receive(request): + assert request.headers["Authorization"] == f"Bearer {TOKEN}" + assert request.headers["Content-Length"] == str(len(content)) + assert request.get_data() == content + return Response(json.dumps({"key": path.name}), content_type="application/json") + + api.expect_request( + "/bucket/fixture/artifact.zip", method="PUT" + ).respond_with_handler(receive) + assert ( + zenodo.upload_deposit_file(token=TOKEN, deposition_id=7, path=path) == path.name + ) + api.check_assertions() + assert len(observed_upload) == 1 and observed_upload[0].closed + + +@pytest.mark.parametrize( + "status,body", [(500, "failed"), (200, "not JSON"), (200, "{}")] +) +def test_upload_failure_closes_stream_and_never_invents_a_confirmed_filename( + api, + tmp_path: Path, + observed_upload, + status: int, + body: str, +) -> None: + path = tmp_path / "artifact.zip" + path.write_bytes(b"archive") + _upload_bucket(api) + api.expect_request("/bucket/fixture/artifact.zip", method="PUT").respond_with_data( + body, status=status + ) + with pytest.raises(zenodo.ZenodoError): + zenodo.upload_deposit_file(token=TOKEN, deposition_id=7, path=path) + assert len(observed_upload) == 1 and observed_upload[0].closed + + +def test_missing_upload_bucket_refuses_before_opening_or_sending_a_file( + api, tmp_path: Path +) -> None: + api.expect_request("/deposit/depositions/7", method="GET").respond_with_json( + {"links": {}} + ) + with pytest.raises(zenodo.ZenodoError, match="no upload bucket"): + zenodo.upload_deposit_file( + token=TOKEN, deposition_id=7, path=tmp_path / "absent.zip" + ) + assert [request.method for request, _response in api.log] == ["GET"] + + +def test_metadata_preserves_declared_identity_but_excludes_non_api_creator_fields( + tmp_path: Path, +) -> None: + path = tmp_path / ".zenodo.json" + path.write_text( + json.dumps( + { + "title": "Fixture", + "description": "Declared description", + "license": "MIT", + "version": "", + "creators": [ + { + "name": "Fixture Author", + "affiliation": "Lab", + "email": "private@example.org", + } + ], + } + ) + ) + metadata = zenodo.zenodo_metadata(path) + assert metadata["title"] == "Fixture" and metadata["license"] == "mit" + assert metadata["creators"] == [{"name": "Fixture Author", "affiliation": "Lab"}] + assert "version" not in metadata and "private@example.org" not in json.dumps( + metadata + ) + + +@pytest.mark.parametrize("payload", [{}, {"id": 7}, {**_draft(7), "submitted": True}]) +def test_reservation_rejects_missing_identity_or_doi_and_already_published_result( + api, payload +) -> None: + api.expect_request("/deposit/depositions", method="POST").respond_with_json(payload) + with pytest.raises(zenodo.ZenodoError): + zenodo.reserve_doi(token=TOKEN, metadata={"title": "Fixture"}) + + +def test_doi_mismatch_sends_no_publication_post(api) -> None: + api.expect_request("/deposit/depositions/7", method="GET").respond_with_json( + _draft(7) + ) + with pytest.raises(zenodo.ZenodoError, match="refusing to publish"): + zenodo.publish_deposit( + token=TOKEN, deposition_id=7, expected_doi="10.5072/other" + ) + assert [request.method for request, _response in api.log] == ["GET"] + + +def test_explicit_matching_doi_allows_only_the_requested_loopback_publication( + api, +) -> None: + api.expect_request("/deposit/depositions/7", method="GET").respond_with_json( + _draft(7) + ) + api.expect_request( + "/deposit/depositions/7/actions/publish", method="POST" + ).respond_with_json( + { + "doi": DOI, + "state": "done", + "submitted": True, + "links": {"record_html": api.url_for("/record/7")}, + } + ) + result = zenodo.publish_deposit(token=TOKEN, deposition_id=7, expected_doi=DOI) + assert result["doi"] == DOI and result["submitted"] and result["state"] == "done" + assert [request.method for request, _response in api.log] == ["GET", "POST"]