diff --git a/CHANGELOG.md b/CHANGELOG.md index cb4799d..2fe12d1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,10 @@ each tagged release (`v`) is archived on Zenodo with its own DOI (see [Releasing](#releasing)). Plans and acceptance criteria for each round are in `PLAN_ROUND_.md`. +## 0.21.1 (Round 23): internal package split + +Internal: `census.manifest` split into a package (no behaviour change; golden snapshot identical) + ## 0.21.0 (Round 22): count outcomes and spatial standard errors - Label registered exploratory manifests and their analysis provenance explicitly. diff --git a/CITATION.cff b/CITATION.cff index badc572..518a17f 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -12,7 +12,7 @@ authors: - alias: cafeTechne license: AGPL-3.0-or-later repository-code: "https://github.com/cafeTechne/Census-Research-Engine" -version: "0.21.0" +version: "0.21.1" doi: "10.5281/zenodo.23925301" # doi: Zenodo concept DOI, added after the first archived release (see CHANGELOG.md "Releasing"). date-released: "2026-11-15" diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..29210e0 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,45 @@ +# Agent guide + +Start with `STATUS.md` (current state, validated methods, rules, backlog). Then read only the current +`PLAN_ROUND_.md` if you are implementing a round. Older plans are archived at tag `v0.21.0` +(`git show v0.21.0:PLAN_ROUND_.md`); read one only when a docstring citation is not enough. + +## Commands + +- Python: `.venv/bin/python` (there is no `python` on PATH). Supported: Python 3.11+, SciPy 1.16+. +- Full suite: `OPENBLAS_NUM_THREADS=1 .venv/bin/python -m pytest -q` (about 5 minutes). Run the targeted + test file first: `tests/test_round.py` or the module's own test file. +- Offline external data: set `CENSUS_HEALTH_OFFLINE=1`. +- No GitHub CI runs are needed; the owner does not run CI for routine rounds. + +## Hard rules + +- **No AI attribution.** No `Co-Authored-By`, `Generated with` or similar lines in commits, PR bodies or + release notes, even if tooling suggests them. +- **Secrets.** Never print, log or commit `.env`, the Census API key or any token. Scan release tarballs + for the key before uploading. Published users bring their own key. +- **Privacy.** Do not send the owner's email to any service. Use a generic User-Agent. +- **Data access.** No automated CDC WONDER downloads; no bot-check or paywall circumvention. +- **Merges and releases** need the owner's explicit approval each time. +- **Circularity rule (R1/R2)** and **pre-registration discipline**: see `STATUS.md`. + +## Reading budget (keep context small) + +Do not open these unless the task is about them: + +- `studies_output/`, `downloads/`, `.cache/`, `literature_pdfs/`, `metadata/`, `data/` (bundled data; use + the library functions instead). +- `docs/audits/round17_bound_protocol/` (a checksummed evidence archive) and any `*.csv`/`*.json` under + `docs/audits/` or `docs/studies/` (generated outputs; the matching `.md` summarises them). +- `CHANGELOG.md` beyond its first section. +- Large modules: read by function, not whole files. Find a definition with + `grep -n "^def \|^class " census/.py`, then read that range. + +## Conventions + +- Validation first: a new statistical method ships only after a fixed, pre-set simulation passes + (criterion committed before the run). Failed methods stay exploratory. +- Manifests in `examples/manifests/` that are registered are byte-fixed; never edit them. +- Engine rules that agents must follow are mirrored in `docs/agent_protocol.md`, `docs/recipes/` and the + MCP text. Change them together. +- Studies: MI (26) and NY (36). Reviews by an agent are labelled "author self-review, not independent". diff --git a/PLAN_ROUND_1.md b/PLAN_ROUND_1.md deleted file mode 100644 index 883a362..0000000 --- a/PLAN_ROUND_1.md +++ /dev/null @@ -1,172 +0,0 @@ -# PLAN_ROUND_1.md — US Census ACS Data Batch Downloader - -**Planning Round:** 1 -**Date:** 2026-11-01 -**Status:** Not yet started — awaiting implementation -**Next Round:** Create `PLAN_ROUND_2.md` when Phase 1 is complete, updating status and leaving notes for the next agent. - ---- - -## Project Overview - -A Python desktop tool (Tkinter GUI) that batch-downloads **ACS 5-Year estimates** from the [US Census Bureau API](https://api.census.gov/data.html) for a selected U.S. state, place, and table group across multiple years (originally 2010–2019). Outputs are saved as CSVs in both wide and long/melted formats. - -**Owner:** cafeTechne -**Repo (original):** https://github.com/cafeTechne/acs5year_place_downloader -**API Docs:** https://www.census.gov/data/developers/data-sets/acs-5year.html -**Census API Key:** Hardcoded in source files — see Phase 1 task to fix this. - ---- - -## Current File Inventory - -| File | Purpose | Keep? | -|---|---|---| -| `acs_downloader.py` | **Primary working Tkinter GUI** — the canonical app. Fix and extend this file. | Yes | -| `dickbird.py` | Byte-for-byte duplicate of `acs_downloader.py`. No meaningful changes. | Delete | -| `app.py` | Incomplete Flask web app skeleton. Not pursuing this direction. | Delete | -| `tempCodeRunnerFile.py` | VS Code Code Runner auto-generated scratch file. | Delete | -| `variables.json` | ~10MB local cache of all ACS variable definitions. | Keep | -| `groups.json` | Local cache of all ACS table group names and descriptions. | Keep | -| `state_places/` | State place files (pipe-delimited .txt) for FIPS resolution. | Keep | -| `fips_data.xlsx` | FIPS reference data. | Keep | -| `aggregated_data.xlsx` | Output artifact from a prior run. | Keep | -| `zip2fips.json` | ZIP-to-FIPS mapping (useful for future geography expansion). | Keep | -| `kitty.ico` | App icon for the Tkinter window. | Keep | -| `README.md` | Outdated. Update after work is complete. | Keep | - ---- - -## Known Bugs (Must Fix in Phase 1) - -### Bug 1 — format_dataframe TypeError (Critical) -**File:** `acs_downloader.py` — `format_dataframe()` (~line 145) - -VARIABLE_DEFINITIONS stores flat strings (labels) but the code indexes them as dicts: - attributes = VARIABLE_DEFINITIONS[var]["attributes"].split(",") ... - -Fix: simplify to a direct rename since the value is already the label string: - def format_dataframe(df, table_code): - variable_descriptions = get_variable_descriptions(table_code) - df.rename(columns=variable_descriptions, inplace=True) - return df - -### Bug 2 — process_data_and_save never writes data (Critical) -**File:** `acs_downloader.py` — `process_data_and_save()` (~line 191) - -The guard checks for "GEO_ID" which the Census ACS5 API does NOT return by default. -The API actually returns: NAME, state, place, and the variable columns. -This condition is always False, so no data is ever saved. - -Fix: change the guard to: - if "NAME" in formatted_df.columns and "state" in formatted_df.columns: - -And update enhanced_melt id_vars to ["NAME", "state", "place"]. - -### Bug 3 — API Key Hardcoded in Source -All Python files contain the raw API key committed to git. - -Fix: Use python-dotenv. - - Create .env with: CENSUS_API_KEY= - - Add .env to .gitignore - - Provide .env.example - - In acs_downloader.py: API_KEY = os.getenv("CENSUS_API_KEY", "") - -### Bug 4 — Year Range Hardcoded -`on_submit()` has: for year in range(2010, 2020) -ACS 5-Year data is available through at least 2022. - -Fix: Add start/end year spinboxes to the UI, defaulting to 2010 and the current available max. - ---- - -## Architecture Decision - -**Keep Tkinter. Do not migrate to Flask.** - -The Flask migration in app.py / tempCodeRunnerFile.py was an incomplete detour. -The Tkinter app was working before and is the correct foundation. - -Plan: - 1. Fix the Tkinter app (Phase 1) - 2. Improve it as a research tool (Phase 2) - 3. Extract a headless library layer (Phase 3) - 4. Build an MCP server on top of that layer (Phase 4) - ---- - -## Roadmap - -### Phase 1 — Cleanup and Bug Fixes -Goal: Get acs_downloader.py to a correct, runnable baseline. - -- Delete dickbird.py, app.py, tempCodeRunnerFile.py -- Fix Bug 1: format_dataframe TypeError -- Fix Bug 2: process_data_and_save GEO_ID guard -- Fix Bug 3: Move API key to .env / python-dotenv -- Fix Bug 4: Make year range configurable (start/end year inputs in UI) -- Add requirements.txt (requests, pandas, openpyxl, python-dotenv) -- Verify end-to-end: pick a state/place/table, confirm CSVs are actually written -- Create PLAN_ROUND_2.md when done - -### Phase 2 — Research Tool UX Improvements -Goal: Make the tool genuinely useful for research workflows. - -- Multi-place selection (same table, multiple places at once) -- Multi-table selection -- Year range UI inputs (start/end spinboxes) -- Output format options: wide CSV, long/melted CSV, Excel, JSON -- Data preview pane (show sample before saving) -- Lazy variable browser (variables.json is 10MB — do not load at startup) -- Progress bar / status log for batch operations (use threading to prevent UI freeze) -- Geography level selector: place (current), county, census tract, state, national -- Friendly error handling for 404s, rate limits, network failures - -### Phase 3 — Headless Library Layer -Goal: Extract all data logic into a clean importable Python module with no GUI dependency. -This is a prerequisite for the MCP server. - -Create census/ package: - census/__init__.py - census/api.py — Census API calls (fetch_census_data, fetch_variables, etc.) - census/geo.py — state/place file loading, FIPS lookups - census/tables.py — groups/table search - census/export.py — DataFrame to CSV/Excel/JSON - -- acs_downloader.py imports from census/ (GUI becomes a thin wrapper) -- All functions fully type-annotated (Python type hints) -- Basic unit tests with pytest for pure data functions - -### Phase 4 — MCP Server -Goal: Expose the tool to AI agents via the Model Context Protocol (https://modelcontextprotocol.io/). - -Install: pip install mcp -Create: mcp_server.py - -Tools to implement: - list_states — Return all U.S. states/territories with FIPS codes - list_places — Return places in a given state (by FIPS or name) - search_tables — Search ACS table groups by keyword - list_variables — Return variable descriptions for a given table code - fetch_year — Fetch ACS data for one year/state/place/table - batch_fetch — Fetch across a year range, return aggregated result - export_csv — Save current data to CSV, return file path - -Document MCP server config for Claude Desktop and Cursor in README. - ---- - -## Notes for Future Agents - -- The canonical file to work on is acs_downloader.py. - Do NOT work on dickbird.py, app.py, or tempCodeRunnerFile.py — delete them. -- Before non-trivial changes, create a git branch: git checkout -b phase1/cleanup -- variables.json is 10MB. It loads lazily (only on submit). Keep it that way. -- groups.json is loaded at startup to populate the table listbox — this is fine. -- State place files are in state_places/, pipe-delimited, with a header row to skip. - Each line has 9 fields: state, state_code, place_code, _, place_name, place_type, _, _, counties -- Census API base URL: https://api.census.gov/data/{year}/acs/acs5 - Params: get=NAME,group({table_code}), for=place:{place_code}, in=state:{state_code}, key={API_KEY} -- ACS5 data availability: generally 2009-2022 (some tables start later). - Check https://api.census.gov/data.html for year availability per table. -- The Dracula color scheme (BACKGROUND_COLOR="#282A36" etc.) is intentional — preserve it. diff --git a/PLAN_ROUND_10.md b/PLAN_ROUND_10.md deleted file mode 100644 index 92291c3..0000000 --- a/PLAN_ROUND_10.md +++ /dev/null @@ -1,136 +0,0 @@ -# PLAN_ROUND_10.md: Microdata, Timelier Estimates, and Research Designs - -**Planning Round:** 10 -**Date:** 2026-11-03 -**Status:** Implemented (section 13); audit fixes from PLAN_ROUND_11.md C1-C7 included -**Branch:** `phase10/microdata-and-designs` (from `defaultBranch` after the Round 9 merge) - ---- - -## 1. Summary - -Rounds 5–9 built a verified path from published ACS tables to maps and models. Three limits remain for research: - -1. **Published tables fix the question.** If no table crosses the variables a hypothesis needs (for example rent burden by disability status by age), the toolkit cannot answer it. The Public Use Microdata Sample (PUMS) can, with its own 80 replicate weights. -2. **5-year estimates are slow and blur change.** ACS 1-year estimates (areas of 65,000+ people) are timelier and do not overlap; they suit trend and event analyses for large places. -3. **Designs, not just models.** Common causal and comparative designs on ACS data (difference-in-differences across non-overlapping periods, multilevel models of tracts within counties) need sampling error handled correctly and are easy to get wrong. - -Round 10 adds PUMS with replicate-weight inference, ACS 1-year support, cost-of-living adjustment between places, multilevel and difference-in-differences designs, and the Round 9 carryovers. - -| Priority | Workstream | Section | -| :--- | :--- | :--- | -| P0 | A. PUMS microdata with replicate-weight variance | 3 | -| P0 | B. Validation of PUMS tabulations against published tables | 4 | -| P1 | C. ACS 1-year estimates | 5 | -| P1 | D. Research designs: multilevel models, difference-in-differences | 6 | -| P1 | E. Cost-of-living adjustment between places (BEA RPP) | 7 | -| P1 | F. Round 9 carryovers | 8 | -| P2 | G. Release, DOI, and documentation site | 9 | - ---- - -## 2. Principles carried forward - -Census documentation is the specification; every result records its method and the uncertainty sources in its SEs; nothing is accepted without validation against published Census values or simulation with known truth; tools work for local models through the core profile. - ---- - -## 3. Workstream A (P0): PUMS microdata - -- **Source:** ACS PUMS person and housing files (1-year and 5-year) from `www2.census.gov/programs-surveys/acs/data/pums/`, and the PUMS API (`api.census.gov/data//acs/acs5/pums`) for targeted pulls. Data dictionaries per release (variable codes and value labels change across releases, as table codes do; the Round 6 concordance approach applies). -- **Weights:** person weight `PWGTP` with replicates `PWGTP1`–`PWGTP80`; housing weight `WGTP` with `WGTP1`–`WGTP80`. Variance by successive difference replication, `Var = 4/80 Σ (θ_r − θ)²`, exactly as for table replicates (`census.variance`). -- **Tabulation engine:** weighted counts, means, proportions, medians (weighted quantiles with the Census PUMS median method), ratios, and crosstabs, each with replicate SEs and the `DerivedEstimate` fields. -- **Geography:** Public Use Microdata Areas (PUMAs, 100,000+ people); PUMA boundaries per vintage (2010 PUMAs through the 2021 releases, 2020 PUMAs from 2022), PUMA-to-county/tract relationship files for context; maps through Round 9. -- **Models on microdata:** survey-weighted regression with replicate-weight SEs (refit on each replicate weight), reusing the Round 8 `ModelResult` contract. -- **Disclosure caution:** PUMS is public but individual-level; outputs are aggregates only, and small cells are flagged (unweighted n below a threshold). - -## 4. Workstream B (P0): PUMS validation - -Census publishes verification files for every PUMS release: selected estimates and their replicate SEs computed from the same PUMS. Our tabulations must reproduce them. - -- Reproduce the Census **PUMS verification estimates** (estimate and SE) for the latest 1-year and 5-year releases. -- Reproduce the worked examples in the **PUMS Accuracy of the Data** statements (used in Round 7 for the median method). -- Compare PUMS-derived totals with the corresponding published tables for large geographies, with tolerances reflecting the PUMS sampling of the full ACS. - -## 5. Workstream C (P1): ACS 1-year estimates - -- Datasets `acs1`, `acs1/profile`, `acs1/subject`, `acs1/cprofile`; per-vintage metadata archive extended to 1-year releases (2005 onward), with the Round 6 concordance. -- **Geography rules:** published only for areas of 65,000+; the planner reports which requested areas exist. -- **2020 experimental estimates:** the 2020 1-year release was replaced by experimental estimates with different methods; treat 2020 as unavailable by default with an advisory and an explicit opt-in. -- **Comparisons:** 1-year periods do not overlap, so the overlap adjustment is 0; comparing 1-year with 5-year estimates is refused (Census advises against it). -- **Replicates:** 1-year variance replicate tables are used where published. - -## 6. Workstream D (P1): research designs - -- **Multilevel models** (tracts within counties, people within PUMAs): `statsmodels` MixedLM with known sampling variance at level 1 (a Fay–Herriot-style likelihood), random intercepts and slopes, and replicate refits for sampling SEs. -- **Difference-in-differences** on non-overlapping periods: two-group and event-style designs using area-level estimates with known variances; the overlap check from Round 8 enforced; parallel-trends diagnostics across pre-periods where at least three non-overlapping periods exist; replicate-based SEs where available. -- **Validation:** simulation coverage (88–92% for nominal 90%) and reproduction of a published reference example for each design. - -## 7. Workstream E (P1): cost-of-living adjustment between places - -- **BEA Regional Price Parities** (RPP) by state and metropolitan area, file-backed with checksums like the price indexes. -- `adjust_for_place(value, geography, year)` converts nominal dollars to national-average purchasing power; combined with R-CPI-U-RS for time, giving dollars comparable across places and years. -- Advisory when an area falls outside RPP coverage (nonmetropolitan portions use state nonmetro RPPs) and a note that RPPs carry their own estimation error that is not in ACS MOEs. - -## 8. Workstream F (P1): Round 9 carryovers - -1. Local-model evaluation tasks for maps and spatial tools (`make_map`, `spatial_autocorrelation`); decide whether `make_map` stays in the core profile from the results. -2. ZCTA and place crosswalks from block assignment files (block-to-ZCTA and block-to-place), using the Round 9 block weights. -3. Interactive HTML map export, self-contained with no external tile servers (P2 in Round 9). -4. A second local model in the evaluation (another open-weights family at a similar size) to separate model limits from tool-design problems, especially for the recipe task. - -## 9. Workstream G (P2): release and documentation - -- First tagged release (v0.9.0 or v0.10.0) once the Zenodo integration is enabled, with the concept DOI added to `CITATION.cff` and the README. -- Optional documentation site built from the Markdown docs and recipes. - ---- - -## 10. Out of scope (later) - -- Decennial census tables beyond the block counts used for crosswalks. -- Restricted-use microdata (Federal Statistical Research Data Centers). -- Bayesian hierarchical models beyond Fay–Herriot and multilevel likelihoods. - ---- - -## 11. Decisions - -| # | Question | Decision | -| :--- | :--- | :--- | -| 1 | PUMS access | **Decided:** download full state files on demand (cached, checksummed) for tabulation; use the PUMS API only for small targeted pulls | -| 2 | Small-cell flag | **Decided:** flag estimates from fewer than 30 unweighted records and any with CV > 30%; do not suppress, since PUMS is already public | -| 3 | 2020 1-year experimental estimates | **Decided:** unavailable by default; opt-in with an advisory | -| 4 | Multilevel and DiD SEs | **Decided:** same interval targets as Round 8. Exception by design: `pums_regression` defaults to `target='these_areas'` (replicate-weight SEs), because microdata regression estimates the coefficient in the surveyed population (the state or nation), which is the design-based inference Census recommends for PUMS; area-level models keep `population` as the default (population by default, these-areas with replicates), validated by simulation before becoming defaults | -| 5 | Cost-of-living adjustment | **Decided:** optional and never applied by default; results state when RPPs were used | - ---- - -## 12. Acceptance criteria - -1. PUMS tabulations reproduce Census verification estimates and SEs for at least one 1-year and one 5-year release. -2. PUMS replicate-weight regression SEs match a hand-computed replicate formula and pass simulation coverage. -3. 1-year extractions report unavailable geographies and refuse 1-year vs 5-year comparisons. -4. Multilevel and DiD designs meet coverage targets in simulation and reproduce a published reference example. -5. RPP adjustment reproduces BEA published real personal income for at least two areas. -6. The Round 7 and Round 8 validation batteries still pass; CI green on Linux and Windows, Python 3.10–3.12. - ---- - -## 13. Implementation status - -| Workstream | Result | -| :--- | :--- | -| A PUMS | `census/pums.py`: download, dictionaries, tabulations, crosstabs, weighted medians, replicate-weight regression; PUMA boundaries | -| B PUMS validation | `census/pums_validation.py`: all 32 Census verification estimates and SEs reproduced for Wyoming (2024 1-year, 2020-2024 5-year), max SE error 0.498 (published values are whole numbers) | -| C ACS 1-year | acs1 datasets 2005-2025 (no standard 2020 release; `/2020/acs/acs1` returns 404); 26 MB metadata archive; 65,000+ availability probe; mixed-product comparisons refused; no 1-year replicate tables exist | -| D Designs | `census/multilevel.py` (Kenward-Roger/Satterthwaite intervals, coverage 0.891/0.890), `census/did.py` (coverage 0.909 two-group, 0.902 event and these-areas after full replicate refits); references verified against the statsmodels documentation page and Card and Krueger (1994) | -| E Regional price parities | `census/rpp.py`: BEA SARPP/PARPP/MARPP 2008-2024; BEA real personal income reproduced for all states in 2008 within 2e-5 (requires a common per-year scaling BEA does not document; recorded in the module) | -| F Carryovers | ZCTA and place crosswalks; `census/web_maps.py` self-contained HTML maps; evaluation harness tasks for `make_map` and `spatial_autocorrelation` | -| Interfaces | `census/workflows_microdata.py`: six tools (57 total); `product` and `allow_experimental_2020` on extraction tools | -| Audit | `PLAN_ROUND_11.md` C1-C7 fixed with 28 tests (`tests/test_audit_r11.py`) | - -**PUMS geography note:** per the PUMS data dictionaries, the 2018-2022 5-year PUMS codes PUMAs in two columns (`PUMA10` for data years 2018-2021, `PUMA20` for 2022); the 2019-2023 and 2020-2024 5-year files use 2020 PUMAs throughout. `puma_vintage(2022, "5-year")` refuses the mixed release, as for 2012-2015. - -**Not done and carried forward:** the two-model local evaluation (the run was interrupted by a usage limit, and the Linux workstation has no Ollama; the harness and tasks are ready: `python scripts/local_model_eval.py --model qwen3:8b llama3.1:8b`). `make_map` remains in the core profile untested with local models until that run. Round 11 owner decisions R1, R2, Q1, Q2 (D6-D8) remain open. - diff --git a/PLAN_ROUND_11.md b/PLAN_ROUND_11.md deleted file mode 100644 index 9893d18..0000000 --- a/PLAN_ROUND_11.md +++ /dev/null @@ -1,279 +0,0 @@ -# Round 11: Statistical Correctness and Audit Follow-up - -**Date:** 2026-11-05 -**Status:** Plan drafted; first systematic evidence pass complete; integration reproductions and policy decisions pending -**Context:** Audit of the existing Round 9 implementation and intentional Round 10 working-tree changes - -## 1. Objective and boundaries - -Resolve concrete scientific correctness defects and define the remaining reliability decisions without superseding Round 9 or Round 10. Findings describe the working tree inspected during the audit, not a released version. Revalidate each finding after Claude Code resumes: code and line numbers may change. - -The user authorized creation and iterative refinement of this plan and the next-round draft, PLAN_ROUND_12.md, including parallel read-only reviews. This does not authorize source fixes, new test files, dependency installation, live calls, data regeneration, commits, or pushes. Only those two planning files may be edited during this follow-up. PLAN_ROUND_9.md and PLAN_ROUND_10.md remain untouched. Treat all other modified and untracked files as intentional work in progress. - -Future implementation must be coordinated with the owner of Round 10, use the current intended baseline, and preserve existing edits. Do not implement a competing model, feature, or migration. No task below is complete merely because a proposed fix appears straightforward. - -## 2. Evidence and limitations - -The original audit used source and test inspection plus isolated in-memory arithmetic. Selected functions were extracted through Python AST rather than importing the package. No application or live endpoint was run. Python 3.14 and NumPy are available locally; pandas, SciPy, and statsmodels were unavailable. Full pytest and integration results are therefore **not established**. - -Confirmed defects are distinguished from risks and policy questions. The ACS successive-difference factor 4/80 is consistent in the reviewed shared variance, PUMS regression, and Moran calculations. No confirmed error was identified in the reviewed RPP scaling or inflation-factor direction. Documented conservative Moran intervals and incomplete Round 10 integration are not defects by themselves. No critical security exploit was established; this is not a security certification. - -## 3. Prioritized work inventory - -| ID | Severity/type | Work | Relationship | Implementation status | -| :--- | :--- | :--- | :--- | :--- | -| C1 | High / defect | Undefined PUMS replicates must not imply zero uncertainty | Round 10 A/B | Fixed; tests/test_audit_r11.py::test_c1_* | -| C2 | High / defect | Weighted medians must ignore zero-weight insertions | Round 10 A/B | Fixed; tests/test_audit_r11.py::test_c2_* | -| C3 | High / defect | Reject ambiguous spatial replicate release alignment | Round 9 spatial inference | Fixed; tests/test_audit_r11.py::test_c3_* | -| C4 | Medium / defect | Align scalar/MCP Census sentinel handling with pipeline | Round 9 controlled MOEs | Fixed; tests/test_audit_r11.py::test_c4_* | -| C5 | Medium / defect | Handle missing PUMS grouping keys as documented | Round 10 A | Fixed; tests/test_audit_r11.py::test_c5_* | -| C6 | Medium / defect | Validate or correctly difference post-treatment DiD references | Round 10 D | Fixed (D5: pre-treatment reference required); tests/test_audit_r11.py::test_c6_* | -| C7 | Medium / guidance defect | Report the uncertainty basis actually used; qualify SIMEX guidance | Existing inference recipes and wrappers | Fixed; tests/test_audit_r11.py::test_c7_* | -| R1 | Medium / risk | Clarify and enforce intended runtime price-data integrity | Existing inflation; Round 10 E | Decided 2026-11-07: kept as a documented risk; integrity verified at generation and by tests, not at each load (docs/agent_protocol.md section 6) | -| R2 | Medium / risk | Characterize PUMS peak-memory amplification | Round 10 A | Measured 2026-11-07: ~3.6 GB peak at California scale with needed columns; warning added for all-column reads of files over 200 MB | -| Q1 | Question | Define DiD serial-correlation inference scope | Round 10 D | Measured 2026-11-07: coverage 0.896-0.905 under AR(1) rho 0.5 and 0.8 (study_did_serial); no estimator added; CR1 cluster sandwich measured and not better; independence assumption documented. 2-period rho=0 cell rechecked at 3,000 reps: 0.900 | -| Q2 | Question | Define policy for incomplete model replicates | Existing inference; Round 10 D | Decided 2026-11-07: replicate_policy strict for manifests (confirmatory), flag otherwise with an understated-SE note (tests/test_audit_decisions.py) | - -## 4. Confirmed defects and acceptance criteria - -### C1. Undefined PUMS replicate statistics - -**Evidence:** `census/pums.py:579-585`, `_finish`, uses `np.nansum` for squared replicate deviations. `_safe_div` at line 525 returns NaN for a zero denominator. A finite full estimate of 10 with 80 undefined replicate estimates produces SE 0 and MOE 0. One undefined replicate is silently omitted while retaining the factor 4/80. The shared `census/variance.py:77` instead propagates NaN through `np.sum`. - -**Why wrong:** Undefined replicate statistics are not evidence of no sampling error. This can silently produce false precision. - -**Expected behavior:** Reject incomplete replicate statistics or return explicitly undefined uncertainty. Any alternative handling requires a documented statistical justification; do not renormalize or impute silently. - -**Likely future targets:** `census/pums.py`, `tests/test_pums.py`; inspect the shared variance contract before choosing a local change. - -**Regression criteria:** Cover all-undefined and partially undefined replicates; zero replicate denominator with a valid full denominator; ordinary complete replicates; grouped and ungrouped statistics. Undefined uncertainty must not become zero MOE or a reassuring reliability classification. Construct expected results independently. - -**Confidence:** Very high for the arithmetic defect. Full public-API reproduction remains F1. - -### C2. Zero-weight records change a weighted median - -**Evidence:** `census/pums.py:530-555`, especially lines 553-554. At an exact half-total boundary, the implementation averages the current value with the next record, irrespective of the next record's weight. - -**Reproduced example:** Values [0, 100, 200] with weights [1, 0, 1] return 50. Removing the zero-weight record gives [0, 200] with weights [1, 1] and returns 100 under the documented midpoint convention. - -**Expected behavior:** Zero-weight insertion must not change the weighted distribution. At a halfway boundary, use the next positive-weight observation under the selected median convention. Assess negative replicate weights separately rather than silently extending this rule to them. - -**Likely future targets:** `census/pums.py`, `tests/test_pums.py`. - -**Regression criteria:** Leading, internal, trailing, and repeated zero weights; exact-half boundaries; tied values; full and replicate columns; all-zero total remains undefined. Existing `test_weighted_median_exact_half_and_bins` does not cover zero-weight invariance. - -**Confidence:** Very high; the extracted function was executed in memory. - -### C3. Ambiguous spatial replicate alignment - -**Evidence:** `census/spatial.py:207-217`, `_replicate_matrix`, filters by variable and then `drop_duplicates("_gid")`, retaining the first area row without selecting a release. `morans_i` consumes the resulting matrix at lines 305-313. The file workflow in `census/workflows_geo.py:182-185` supplies a requested year to boundary handling but passes the replicate frame through without selecting its year. - -**Example:** A 2023 estimate frame and a replicate frame containing 2018 and 2023 rows for the same areas/variable can use 2018 replicates if those appear first. Reordering rows changes inference. - -**Expected behavior:** Require unique, release-consistent area/variable replicate rows or explicitly select and validate the intended release. Do not resolve conflicting duplicates by file order. Preserve valid single-release inputs without a Year column where that contract is supported. - -**Likely future targets:** `census/spatial.py`, affected geography workflow, `tests/test_spatial.py`, and workflow tests if the contract changes. - -**Regression criteria:** Conflicting duplicate GEOIDs; two releases in reversed order; missing areas; shuffled valid rows; Moran SE, LISA stability, and spatial regression paths that share this helper. Decide whether identical duplicates also fail. Existing tests compare complete single-release replicates against PySAL but do not test ambiguous replicates. - -**Confidence:** Very high from source tracing; no PySAL integration execution. - -### C4. Scalar/MCP sentinel semantics diverge - -**Evidence:** `census/stats.py:136-175`; `mcp_server.py:867-877` delegates to the scalar `calculate_cv`. `standard_error` takes the absolute MOE without sentinel decoding. The array helper `stats.moe_to_se` and cleaned pipeline recognize controlled and missing codes. - -**Reproduced example:** `calculate_cv(1000, -555555555)` gives SE approximately 337723741.64, CV approximately 33772374.16%, reliability Low, and interval [-555554555, 555556555]. The pipeline convention for that controlled MOE is zero uncertainty. The missing code -222222222 also gives a huge finite SE. - -**Expected behavior:** Controlled MOE becomes zero; missing/suppressed MOE yields undefined uncertainty. Preserve deliberate compatibility for ordinary negative MOEs only after distinguishing them from sentinel codes. Check sentinel estimate inputs too. - -**Likely future targets:** `census/stats.py`, `tests/test_stats.py`, `tests/test_controlled_moe.py`, relevant MCP tests. Do not change shared signatures unnecessarily. - -**Regression criteria:** Table-driven scalar/array/cleaned-frame/MCP parity for controlled, suppressed, missing, zero, normal positive and ordinary negative MOEs. Existing scalar tests explicitly accept -1645; retain or deliberately document that behavior. No fabricated finite interval from a missing code. - -**Confidence:** Very high; selected scalar functions were executed in memory. - -### C5. Missing PUMS grouping keys produce float indices - -**Evidence:** `census/pums.py:501-522`, `_groups`, `_group_sum`, `_group_count`. `ngroup` with excluded missing keys produces float codes containing NaN. Filtering NaNs does not convert remaining labels to integer indices. NumPy array indexing and `bincount` require integers. - -**Example:** Group labels [0.0, NaN] leave [0.0] after filtering; `np.bincount` raises TypeError. This downstream failure was reproduced with NumPy. The full pandas path has not been executed locally. - -**Expected behavior:** Drop missing-key rows as `tabulate` documents, then aggregate valid integer group codes with accurate counts. - -**Likely future targets:** `census/pums.py`, `tests/test_pums.py`. - -**Regression criteria:** One missing key, multiple grouping keys, all missing keys, no observed groups, and crosstab paths. Verify actual retained estimates, counts, and replicate SEs rather than merely absence of an exception. Existing missing-value test addresses the outcome, not grouping keys. - -**Confidence:** High; source-established pandas path and reproduced NumPy failure. Full public-API reproduction remains F1. - -### C6. Post-treatment reference in two-group DiD - -**Evidence:** `census/did.py:109-111` accepts any observed reference. Lines 128-138 difference outcomes against it but construct the effect column as treated * post rather than treated * (post - post_reference). - -**Example:** With one pre and one post period and a post-period reference, retained rows are pre-period rows, making the treatment column identically zero. The design is singular. The design arithmetic was reproduced in memory; no full model fit was run. - -**Expected behavior:** Either reject post-treatment references clearly or difference the regressor consistently. Choose the documented contract before implementing. Recheck event-style interpretation separately; do not assume the two-group fix applies unchanged. - -**Likely future targets:** `census/did.py`, `tests/test_did.py`. - -**Regression criteria:** Explicit pre reference, default reference, post reference, nonexistent reference, and independent fixed-effects/design-matrix comparison. Existing tests use the default pre-treatment reference. - -**Confidence:** Very high for the design defect. - -### C7. Regression guidance contradicts the chosen uncertainty target - -**Evidence added during parallel planning review:** `census/models.py:178-206` maps population to se_model, these_areas to se_sampling, and conservative to se_total. `census/report_tables.py:176-185` already uses `interval_se_column` for the reported SE. However, `census/workflows_inference.py:194` and `docs/recipes/regression_sampling_error.md:10` unconditionally tell users to report se_total. Recipe line 9 also offers SIMEX as a fallback without the confirmatory-use qualification stated in README's Inference section. - -**Why wrong:** A reader can report an SE that does not support the displayed CI/p-value, or infer that a method with documented undercoverage is an interchangeable confirmatory fallback. This is a defect in instructions, not proof that the underlying result table is calculated incorrectly. - -**Expected behavior:** Guidance identifies the result's actual SE basis and target, preserves method-specific uncertainty caveats, and distinguishes exploratory SIMEX use from validated confirmatory alternatives. Reuse `interval_se_column`; do not hard-code a second target mapping. Do not claim that replicate refits eliminate bias from noisy covariates. - -**Likely future targets:** Existing regression recipe, `census/workflows_inference.py`, focused workflow/recipe tests. Broader output interpretation belongs to Round 12 B. - -**Regression criteria:** Population/these_areas/conservative examples with deliberately different SE columns yield guidance consistent with the serialized result, table and figure basis. Method-specific corrected models retain their actual basis. Recipe guidance does not recommend SIMEX for confirmatory use. Assert scientific meaning, not the entire wording of prose. - -**Confidence:** Very high from direct source comparison; no wrapper or export execution performed. - -## 5. Risks and design questions - -### R1. Runtime integrity of price data - -`census/inflation.py:55-62` and `census/rpp.py:66-78` parse numerical CSVs without validating sidecar hashes. Valid-looking local corruption can change scientific results while original source metadata remains available. This is a risk, not evidence of corrupted data. - -`tests/test_inflation.py` and `tests/test_rpp.py` do verify shipped checksums. Resolve whether integrity is guaranteed only at generation/test time or also at runtime. If runtime verification is adopted, define custom-data behavior, cache invalidation, error reporting, and verification cost before editing loaders. Use temporary synthetic data for tampering tests; never modify real price files. Follow-up F2. - -### R2. PUMS memory amplification - -`census/pums.py:213-228` materializes response content; lines 311-322 retain parsing chunks before concatenation and conversion; `_weight_matrix` creates an 81-column float64 array. Chunk size does not bound total memory. At 1.9 million records the weight matrix alone occupies 1231200000 bytes (about 1.23 GB / 1.15 GiB), excluding frames, strings, and temporaries. - -No peak-memory benchmark has been run. Preserve the Round 10 decision to download full state files on demand. Characterize realistic resource costs before proposing streaming or architectural changes. Follow-up F3. - -### Q1. DiD serial correlation - -`census/did.py:5-20` explicitly assumes independent area-by-period model errors. Non-overlapping ACS periods remove shared sample years, not temporal correlation in economic shocks. This is a scope/design question, not a demonstrated algebra error. Decide whether to document the restricted model, add sensitivity diagnostics, or later support serial covariance. Coverage claims must state the generating process. Follow-up F4. - -### Q2. Incomplete model replicates - -`census/models.py:258-273` holds incomplete replicate rows at full-sample values; the result reports understatement warnings. Similar behavior exists in multilevel and DiD refits. It is deliberate and disclosed, unlike C1's silent NaN summation. Decide whether confirmatory/these-areas inference should fail, require explicit opt-in, or retain the existing warning policy. Quantify the impact before changing defaults. Follow-up F5. - -## 6. Systematic follow-up register - -Each follow-up records evidence, outcome, remaining dependency, and next action. A planning follow-up can be resolved while implementation remains not started. Do not equate a proposed validation with an executed test. - -| ID | Question / evidence needed | Status | Next action | -| :--- | :--- | :--- | :--- | -| F1 | Reproduce C1-C6 through public APIs; distinguish arithmetic from integration evidence | Partial: arithmetic rechecked; integration pending environment | Use an existing supported environment with dependencies, once its use is authorized | -| F2 | Is runtime integrity promised, and what does existing checksum coverage establish? | Evidence pass complete; runtime policy unresolved | Owner decides runtime verification contract; no current checksum mismatch found | -| F3 | What memory bounds can be established without large allocations or downloads? | Static bound complete; measurement pending | Benchmark only after scope/resource approval; no large allocation performed | -| F4 | Do DiD coverage studies test serially correlated model errors? | Evidence pass complete: reviewed generators use independent shocks | Decide documented model scope versus a future serial-correlation sensitivity study | -| F5 | Where are missing replicate rows held fixed, and how is this disclosed? | Evidence pass complete; policy unresolved | Decide strictness for confirmatory inference before changing defaults | -| F6 | Do guidance and outputs name the same uncertainty basis? | Source pass complete: C7 added | Run target-specific wrapper/table/figure checks in an authorized environment | - -### First evidence pass: 2026-11-05 - -**F1 — Reproduction readiness.** Rechecked module availability without package imports: NumPy available; pandas, SciPy, statsmodels, and esda unavailable. Re-executed the AST-extracted weighted median: [0,100,200]/[1,0,1] gives 50; [0,200]/[1,1] gives 100. Rechecked the C1 expression with 80 NaNs: SE is 0. These confirm arithmetic/function behavior, not end-to-end tabulation or model results. C4 scalar behavior, C5 downstream NumPy type failure, and C6 differenced-design arithmetic were reproduced in the original audit; they were not rerun as integration tests in this pass. C3 remains source-established. No dependencies were installed and no pytest invocation was made. Remaining public-API cases are specified in section 4; run them only in a suitable authorized environment. - -**F2 — Integrity evidence and scope.** Read README section Methodology, `docs/agent_protocol.md:112-124`, both loaders, and checksum tests. The reviewed documentation describes source checksums but does not explicitly promise hash verification at each load. Do not upgrade R1 to a confirmed broken runtime promise. Independently hashed existing CSV bytes with the standard library: `data/price_indexes/cpi_u.csv`, `c_cpi_u.csv`, and `r_cpi_u_rs.csv` match each adjacent JSON's `csv_sha256`; `data/price_parities/rpp.csv` and `pce_price_index.csv` match `rpp.json`'s `csv_sha256` mapping. This checks five current CSVs, not every source archive, provenance field, or future load. No corruption was found in those checks. Still unresolved: runtime verification policy and treatment of intentionally supplied custom tables/directories. - -**F3 — Memory bound and lifetimes.** Recomputed 1900000 * 81 * 8 = 1231200000 bytes = 1.1466443539 GiB for one tabulation weight matrix. The download path materializes the compressed response before returning it; the reader then retains parsed chunks and creates concatenation/conversion output. These are different phases: do not add every allocation as if all were guaranteed to coexist. Within parsing, retained chunks and concatenation overlap; within tabulation the input frame and converted matrix coexist. The static lower bound does not establish peak RSS. A future benchmark should record compressed bytes, row/column counts, selected columns, chunk size, baseline/peak RSS for each phase, platform and dependency versions, and verify equal estimates across chunk sizes. Prefer bounded synthetic inputs first; no download or large benchmark was performed. - -**F4 — DiD coverage scope.** `census/simulation.py:663-666` generates a fresh independent model-error vector for each period, in addition to a shared area intercept. The shared intercept cancels under differencing; this is not a test of serially correlated residual shocks. The two-group and event studies at lines 674-713 add independent sampling noise by period. `tests/test_did.py:155-157` is only a fast smoke gate (coverage > 0.75), not evidence for the full 0.88-0.92 target or serial-correlation robustness. Its Card-Krueger checks validate the separate `did_from_changes` helper, not the complete GLS/REML panel estimator. If a serial-correlation sensitivity study is approved later, specify the residual process (e.g. several positive AR(1) correlations), independent versus correlated sampling error, nominal level, Monte Carlo SE, and estimand before judging coverage. No new coverage results are claimed. - -**F5 — Missing-replicate policy traced.** `census/models.py:258-273` replaces an entire area/variable replicate row with its full-sample value when any of its 80 entries is nonfinite; warnings are attached at lines 350-355. `census/multilevel.py:506-557` reuses that helper and reports the fixed-row counts. `census/did.py:294-327` implements its own unit-period fallback and adds an understatement note. Default zero tolerated failed refits does not prevent these fallbacks: missing input rows are treated as fixed before refitting rather than counted as failed fits. Thus a these-areas target plus a nonempty replicate frame does not establish complete replicate coverage. This remains a disclosed policy concern, not a claim that a new feature is missing. A future policy decision should distinguish truly fixed predictors, unavailable replicates for stochastic variables, and partially missing replicate rows. Recommended future tests: complete versus removed/partially missing rows, warning/count assertions, all-fallback inputs, and any explicit strict-mode contract. No strict mode or new defaults were implemented. - -**F6 — Guidance follow-up: 2026-11-05.** Three parallel read-only reviews examined scientific evidence, researcher workflows, and maintainability. The parent independently checked the C7 source locations. The existing reporting helper already selects the interval SE; the correction should be local to contradictory guidance rather than another result schema. Recipe executability and complete methods text remain Round 12 work. Recommendations from the reviews are proposals, not external statistical verification or executed tests. - -### Remaining dependencies and next pass - -- **Environment dependency:** Public-API reproduction and test execution require an existing supported environment or separately authorized dependency setup. Do not mark F1 complete on arithmetic alone. -- **Owner decisions:** R1 runtime verification, Q1 DiD scope, Q2 incomplete-replicate strictness, and C6 reference contract require explicit recorded choices. The proposed tests do not silently choose those policies. -- **Measurement dependency:** R2 peak RSS requires an approved bounded benchmark; static arithmetic alone cannot resolve it. -- **Coordination dependency:** Revalidate source locations and outcomes against Claude Code's resumed Round 10 baseline before implementation. This plan does not reserve or take over those files. -- **Next iteration order:** Complete available public-API reproductions; record policy decisions; implement only a specifically approved defect scope; run focused regressions and applicable offline validation; update status with actual results. - -## 7. Future implementation sequence and verification - -1. Reconcile this inventory with the resumed Round 10 baseline and obtain approval for a specific fix scope. -2. Reproduce C1 and C2, then C3. Add independent mathematical or invariance checks before minimal fixes. -3. Address C4 sentinel parity, C5 grouping, C6 reference handling, and C7 target-sensitive guidance with focused regressions. -4. Resolve R1/Q1/Q2 policy questions explicitly. Measure R2 before selecting an architectural remedy. -5. Run relevant tests in an existing supported Python 3.10-3.12 environment; do not infer success from isolated arithmetic. With implementation authorization, broaden to the offline suite and concordance validation after focused checks pass. Keep live tests disabled unless separately authorized. -6. Use existing frozen Census/reference fixtures and applicable simulation studies to verify scientific behavior. Fast smoke thresholds alone do not establish nominal coverage. Do not regenerate fixtures or weaken acceptance thresholds to make checks pass. -7. Record changed files, exact checks/results, baseline failures versus regressions, remaining limitations, and Round 10 coordination needs here or in the approved handoff. No commits or pushes without permission. - -## 8. Completion criteria - -- Each defect is revalidated against the intended baseline and has a recorded disposition: an independent regression test plus a minimal reviewed fix; evidence explaining why it no longer applies; or an owner-accepted release limitation with its consequences recorded. An accepted limitation remains unresolved behavior, not a fixed defect or proof of a clean baseline. -- Risks are measured or have explicit, documented policy decisions; questions are not silently treated as defects or implemented defaults. -- CI-relevant offline checks pass on the supported matrix, or failures are explicitly documented and attributed. -- All scientific claims distinguish exact formula checks, fixture reproduction, and simulation coverage. -- Round 9/10 plans and unrelated working-tree content remain preserved. - -## 9. Execution packages and coordination - -This is an execution design for a later authorized implementation, not permission to start it. Preserve C1-C7 identifiers in review and handoff notes. Relative size means review complexity, not a promised schedule: small = localized behavior/doc change; medium = several consuming paths or a policy boundary. - -| Package | Scope and output | Dependencies | Likely targets | Size / owner boundary | -| :--- | :--- | :--- | :--- | :--- | -| S0 Baseline | Record intended revision, existing edits, supported environment and relevant preexisting failures; reproduce findings before fixing | Round 10 handoff | Read-only inspection; authorized temporary test outputs | Small; one coordinator | -| S1 PUMS | C1, C2, C5 as individually reviewable fixes with independent edge cases | S0; median weight-domain clarification | pums.py and its current tests | Medium; one owner because fixes share the file | -| S2 Spatial alignment | C3 and direct tests for all consumers of the helper | S0; release-key contract | spatial.py, affected geo wrapper/tests only if needed | Medium; coordinate with Round 10 geography work | -| S3 Sentinel parity | C4 and raw-versus-cleaned input contract | S0 | stats.py and scalar/MCP tests; trace variance helper before touching it | Small to medium; no broad numeric cleanup | -| S4 DiD reference | C6, a recorded reference contract, independently constructed design check | S0; reference decision | did.py and its tests | Small to medium; coordinate with Round 10 designs | -| S5 Guidance | C7, actual target/basis in existing next steps and regression recipe | S0; preserve current model contracts | inference workflow, recipe, focused tests | Small; no new tools or model options | -| S6 Evidence closeout | Resolve or explicitly defer R1/R2/Q1/Q2 with reasons; focused checks, applicable offline validation, handoff | S1-S5 and supported environment | Existing audit/validation paths only when approved | Medium; do not turn risks into mandatory new subsystems | - -Default implementation order is S0, S1, S2, S3, S4, S5, S6. S5 may move earlier for immediate documentation value. Parallel review of independent packages is useful; concurrent edits to the same source/test file are not. Implementation branches are a future decision based on the then-current checkout, not a reason to switch away from the active Round 10 work now. - -For each package retain: reproduction input, independent expectation, before/after behavior, affected callers, test results, compatibility impact, remaining limitation, and rollback scope. A documentation-only fix need not acquire a broad test framework. If a fix needs a new dependency, new public tool, or substantial model refactor, stop that package at an evidenced proposal and reconsider its scope. - -## 10. Proposed decisions, not adopted defaults - -| Decision | Recommended starting point | Compatibility and evidence needed | -| :--- | :--- | :--- | -| D1 Undefined tabulation replicates | Preserve a finite point estimate with undefined SE/MOE and an explicit reason, or reject where the existing API cannot represent this safely | Never silently substitute zero or rescale 4/80; validate flags and JSON behavior | -| D2 Median weight domain | Ignore zero-weight records for midpoint selection; preserve the documented median convention | Establish release-specific handling of negative replicate weights before extending nonnegative-distribution reasoning; do not infer validity from synthetic tests or clip weights | -| D3 Spatial duplicates | Reject duplicate area/variable rows by default; where a year is explicitly available, select and validate it before uniqueness checks | Decide identical-duplicate behavior; no silent first-row selection or heuristic year guessing | -| D4 Scalar MOEs | Decode Census sentinels before scalar conversion; preserve explicitly supported ordinary negative-MOE behavior unless intentionally changed | Trace callers of stats.moe_to_se versus variance.moe_to_se; raw input decoding and cleaned mathematical primitives need not be identical APIs | -| D5 DiD reference | Initially require a pre-treatment reference for the two-group API unless arbitrary references are an intended supported requirement | Explicit error before fitting; event-style reference semantics assessed independently; no new estimator | -| D6 Runtime price integrity | Keep R1 as a documented risk until the runtime/custom-data guarantee is chosen | Existing shipped CSV hashes match; do not imply current corruption or force a new verification service | -| D7 Incomplete inference replicates | Prefer an explicit strict contract for confirmatory inference, with documented exploratory fallback if retained | Distinguish fixed covariates, missing stochastic variables, missing rows, and failed fits; defaults require a compatibility decision, not a silent switch | -| D8 DiD serial shocks | State current independent-shock assumptions and limits; consider a bounded sensitivity study before another model | No claim that non-overlap alone justifies serial independence; avoid adding cluster/serial estimators without a separate scientific design | - -The final choice can differ if Census specifications or the intended API require it. Record the rationale and supporting evidence; this table is not a statistical source. External verification remains a later separately authorized activity. - -## 11. Scientific contracts to preserve - -- **Raw versus cleaned values:** Recognize sentinels at ingestion boundaries. Do not globally reinterpret mathematical primitives that deliberately accept already-cleaned numbers without tracing callers. -- **Identity:** Match on GEOID, variable, and release/product where present; validate geography vintage at the boundary that has that context. Do not repair lost leading zeros by guessing or silently pick among conflicting identities. -- **Missingness:** Distinguish a true zero, a controlled estimate, unavailable uncertainty, excluded records, and unavailable replicates. Row counts and uncertainty counts answer different questions. -- **Inference:** The estimand, selected SE, confidence level, and interval distribution/df must agree. Do not reconstruct every CI as normal when a fitted method reports t-based inference. -- **Oracles:** Hand calculations, exact reference matrices, and invariances must not call the production helper being checked to generate their expected values. Published count verification alone does not validate means, medians, or regressions. -- **Interfaces:** Compare shared Python workflow, CLI and MCP paths for operations they actually share. The maintenance-mode GUI does not need all new inference features. -- **Compatibility:** Preserve valid supported results within declared numerical tolerances. Deliberate changes in invalid-input behavior must have clear errors/notes and targeted regression evidence. - -## 12. Verification tiers and release decision - -| Tier | Purpose | Evidence required | Limits | -| :--- | :--- | :--- | :--- | -| T0 Source/arithmetic | Localize failure and construct an expected answer | Explicit assumptions and function/expression executed | Not public-API or integration success | -| T1 Focused offline regression | Verify approved fixes through consuming APIs | Deterministic cases, invariances, warnings/errors, result serialization | Does not establish broad coverage | -| T2 Offline references/integration | Detect scientific drift and adapter divergence | Existing frozen Census/reference checks and applicable workflow tests | Synthetic and published evidence must be labeled separately | -| T3 Full seeded studies | Support specified nominal coverage claims | Estimand, generating process, seed, independent repetitions, MC SE, fixed acceptance rule, environment/revision | Smoke thresholds and repeated reruns are not substitute evidence; correlated terms/areas are not independent repetitions | -| T4 Optional live drift checks | Verify endpoint/schema/source changes | Separate authorization and clear source/date | Network failures are not numerical regressions; not needed to implement these fixes | - -Run the smallest meaningful T1 checks first, then applicable T2 checks. Run T3 only for a changed estimator or unresolved scientific concern. Keep current 0.88-0.92 coverage rules unless a scientific decision changes them before results are seen. Do not tune seeds, broaden bands, or relabel failed gates after inspecting outcomes. Keep informational/sensitivity rows distinguishable from claims being gated. - -Before treating a command as a release gate, verify that it signals gated failures to its caller. `scripts/inference_validation.py:38-49` distinguishes FAIL from info in its report, while `main` at line 121 currently returns zero. This source fact does not prove that the report command was intended as a CI gate; Round 12 A must establish report-versus-gate semantics rather than silently changing compatibility. No report command was executed here because its default full run writes repository artifacts. - -Round 11 is ready to close when confirmed findings are fixed and tested, shown inapplicable on the intended baseline, or explicitly accepted as release limitations by the owner with their consequences recorded. Do not declare a clean scientific baseline while a high-severity defect remains unexplained. Risks can remain deferred with documented boundaries; optional feature work is not required for closure. Round 12 implementation is a separate scope decision. - -## 13. Scope limits and handoff to Round 12 - -Round 11 adds no estimator families, providers, public MCP tools, mandatory dependencies, storage format, or replacement orchestration layer. Prefer existing test modules and one shared helper only when at least two reviewed consumers require identical behavior. Do not standardize every result/error across the repository merely because a touched path needs a fix. - -The next-round draft, [PLAN_ROUND_12.md](PLAN_ROUND_12.md), proposes three bounded improvements: independent validation evidence, consistent interpretation of existing results, and one executable research workflow. These are improvements to existing research capabilities, not a continuation of every possible feature in earlier plans. Incomplete Round 10 PUMS, 1-year, RPP, map, or design integration stays with Round 10. - -Planning review used three parallel read-only reviewers. No review produced code changes, dependency installations, integration results, or external source verification. Remaining decisions are visible above; no empty follow-up is marked complete merely to finish the plan. - -**Current handoff:** Planning only. No source fixes, test additions, dataset updates, integration tests, commits, or pushes have been performed for Round 11. diff --git a/PLAN_ROUND_12.md b/PLAN_ROUND_12.md deleted file mode 100644 index d58684d..0000000 --- a/PLAN_ROUND_12.md +++ /dev/null @@ -1,204 +0,0 @@ -# Round 12: Verifiable and Reproducible Research Workflows - -**Date:** 2026-11-05 -**Status:** Implementation authorized by the owner (2026-11-07); Round 11 closed (C1-C7 fixed; R1/R2/Q1/Q2 decided, see PLAN_ROUND_11.md) -**Prerequisite:** Reconcile the final Round 10 baseline and close or explicitly disposition Round 11's relevant correctness findings -**Companion:** [PLAN_ROUND_11.md](PLAN_ROUND_11.md) - -## 1. The improvement this round should deliver - -A researcher should be able to use an existing supported analysis, understand exactly what its interval means, and hand its outputs to a second researcher who can check the result. A maintainer should be able to verify that behavior without maintaining another analysis engine or an expanding catalog of examples. - -This round improves the usefulness of the capabilities already present. It does not add estimator families or finish features reserved to Round 10. The recommended core is three workstreams: independent scientific evidence, consistent interpretation, and one executable reference workflow. Work on one implementation slice at a time, each ending in a usable reviewable result. - -The user authorized planning and parallel read-only review. During this planning task only PLAN_ROUND_11.md and this file may change. Do not install dependencies, access external services, run live endpoints, change datasets, generate reports into the checkout, or change source/tests. Implementation, environment setup, external verification and release actions remain future scope decisions. Earlier planning files and all unrelated working-tree edits remain protected. - -## 2. Research outcomes and evidence of need - -| Desired outcome | Existing foundation | Why improvement is worthwhile | -| :--- | :--- | :--- | -| A reported number can be checked independently | Frozen Census/reference fixtures, simulation harness, ordinary pytest tests | Some tests compare implementations with themselves or cover simpler special cases than the public claim | -| A reader knows which uncertainty supports an interval | ModelResult, diagnostics/notes/provenance, interval_se_column, report tables | Round 11 C7 demonstrates that correct calculations can be paired with contradictory instructions | -| Another researcher can reproduce the substantive analysis | Manifests, hashes, bundles, extraction replay, recipes, fake clients | Individual capabilities need one verified path with exact inputs, transformations and supported artifacts | -| Maintenance grows slower than scientific value | Shared Python workflows, CLI/MCP adapters, plain result objects | Extending those contracts is cheaper than another interface, workflow runner, or metadata framework | - -Success is measured in correct interpretation, independent evidence, and a complete workflow, not a higher tool count. Do not claim a measured usability improvement until someone has actually followed the example; this plan defines acceptance exercises rather than reporting their outcomes. - -## 3. Admission rule and scope budget - -An item enters the round only if it has a concrete researcher problem, an existing extension point, an independent validation strategy, a bounded maintenance cost, and an explicit stop condition. Record an existing alternative before adding capability. Prefer deleting contradictory instructions or duplicated mappings over adding options. - -**Proposed core budget:** Three workstreams, one canonical workflow, no new required dependencies, no new public MCP tools, no new estimator families or data providers, no new bundle/manifest format, and no new persistence or orchestration layer. Implement the contracts needed by selected paths, not a universal framework. Existing commands may need a narrowly reviewed correction; do not expand their parameter surfaces by default. - -A second workflow is conditional, not a promised deliverable. It is admitted only if the first demonstrably cannot exercise an important existing boundary, can reuse existing fixtures/adapters, and replaces or completes an existing recipe rather than starting an example gallery. If new infrastructure becomes necessary, reduce scope or defer with evidence. - -## 4. Baseline and overlap boundaries - -- **Round 10 owns:** PUMS and verification, ACS 1-year support, multilevel/DiD model completion, RPP integration, crosswalk/map carryovers, local-model evaluation, and its release/documentation work. Do not duplicate or redesign them here. -- **Round 11 owns:** C1-C7 fixes and their direct regressions, scientific input contracts at affected boundaries, and disposition of R1/R2/Q1/Q2. Round 12 consumes those decisions; it does not silently pick missing-replicate, median-weight, DiD-reference, or runtime-integrity policy. -- **Round 12 owns:** Additional independent evidence beyond direct bug tests, coherent interpretation across existing selected artifacts, and one existing workflow made executable and reviewable end to end. -- **Current architecture is intentional:** `pyproject.toml:55-61` locates resources in the clone/editable install. Standalone-wheel resource packaging and dependency splitting are not assumed defects. The GUI remains in maintenance mode; do not require new-model feature parity. - -When resumed code already satisfies an item, record evidence and retain only useful regression/documentation work. Never implement a second equivalent path merely to claim plan completion. - -## 5. Workstream A (P0): Independent scientific evidence - -### A1. Two targeted reference checks - -**Need:** Expected values should not inherit the algebra error they are intended to detect. - -**Evidence boundaries observed:** - -- `tests/test_did.py:34-46` checks the separate published-change helper, not the complete GLS/REML panel estimator. `census/simulation.py:733-738` constructs one simulation target with production `_prepare`, `_differences`, and `BlockGLS`. This does not invalidate the study; it limits its independence as an algebra check. -- `tests/test_multilevel.py:59-66` reproduces a documented example with zero known sampling variance. That particular test does not establish the nonzero-known-variance extension. -- `census/pums_validation.py:163-167` documents its compressed cell fixtures as count verification. Those fixtures and the current verification path are designed and tested for published counts; they do not establish general mean, median, or regression correctness. Do not broaden published-verification claims beyond that evidence. - -**Proposed scope:** Add or strengthen at most two independent reference cases in existing tests: (1) a tiny known-variance GLS/block problem with nonzero heterogeneous sampling variances and nonzero within-block covariance, checked against directly assembled dense matrices; (2) a two-group DiD case with an independently constructed fixed-effects/differenced design, including a reference-period edge case if Round 11 has not already supplied the full oracle. Reuse direct Round 11 PUMS hand/invariance cases instead of starting another validation battery. - -The independent reference may use basic matrix algebra, but must not import the production design builder, covariance builder, optimizer helper or result table constructor to generate its answer. Fixed-variance algebra checks validate that layer only; do not describe them as validation of estimated variance components or small-sample corrections. Match the expected estimand and method assumptions explicitly. - -**Acceptance:** A reader can derive the expected coefficient/covariance from the supplied small matrices and see why the test would fail under an incorrect reference transformation, omitted covariance term, or misuse of known variance. Tolerances have a numerical justification. Current published-reference tests remain intact. No large new data archive or parameter grid. - -**Likely targets:** Existing `tests/test_did.py`, `tests/test_multilevel.py`, applicable existing simulation/reference notes. Source changes only if these cases establish a separately scoped defect. - -### A2. Make validation evidence usable as a release decision - -Use Round 11's T0-T4 tiers. Keep fast deterministic regressions on ordinary changes; use applicable full seeded studies for release evidence or changed scientific behavior. Record method, estimand, generating process, independent repetitions, nominal level, acceptance rule, Monte Carlo precision, revision, configuration and environment. Count independent simulation repetitions, not correlated areas or coefficients, when quantifying Monte Carlo uncertainty. - -`scripts/inference_validation.py:38-49` already distinguishes gated FAIL rows from info rows; line 121 currently returns zero. Decide whether the existing command is a report generator, a gate, or both before changing exit behavior. A future gate must fail when any required study fails or is missing, while informational rows do not fail it. Preserve report-only compatibility if that is the intended contract; use existing script/CI machinery rather than another runner. - -Do not infer nominal coverage from `tests/test_did.py:157`'s >0.75 smoke threshold. Do not convert that smoke check into a slow, noisy full-coverage assertion on every PR. Reconcile with any completed Round 10 runner integration before editing. - -**Acceptance:** One documented check map identifies the focused offline command for a touched method and the applicable release studies. A controlled synthetic report-result case establishes success/failure/info/missing-study gate semantics without running long simulations in unit tests. Full studies, when actually run, retain failed results and fixed rules; no seed searching or post hoc threshold adjustment. - -**Maintenance ceiling:** Existing scripts/test modules and existing CI jobs only. No benchmark service, new CI provider, multi-version dependency resolver, or automatic fixture refresh. Runtime budgets are measured and recorded before they become gates. - -## 6. Workstream B (P0): Results that explain their scientific meaning - -### B1. One interpretation contract using existing result fields - -Reuse `census/results.py:60-89` (ModelResult), `census/models.py:175-206` (target/SE mapping and interval construction), `census/report_tables.py:176-187` (actual SE basis), and existing figure/serialization paths. The aim is consistent use of existing fields, not another result class or metadata language. - -For the selected workflow and touched inference paths, a reader should be able to answer: - -| Question | Existing location to inspect/reuse | Required treatment | -| :--- | :--- | :--- | -| What quantity is estimated? | Formula, method, diagnostics interval_target, methods text | Name the estimand; an association is not a causal effect | -| Which uncertainty makes the interval? | interval_se_column, coefficient SE fields, CI/p-value | Use the actual selected basis, not always se_total | -| What confidence level/distribution was used? | Result confidence_level, coefficient df and method notes where applicable | Preserve normal versus t/df distinctions; do not reconstruct stored intervals with the wrong distribution | -| What sample contributed? | n_obs, exclusions, replicate coverage and fallback diagnostics | Distinguish retained units from complete replicate coverage; disclose dropped/fixed rows | -| What error remains outside the result? | Notes, uncertainty_sources and provenance | Explain missing predictor uncertainty, design assumptions, RPP error if relevant, and documented approximation limits | -| Where did the data come from? | Existing extraction/manifest/bundle provenance | Preserve release/product/geography and transformation identity where supplied; do not invent unavailable metadata | - -Fields already present should be reused. Add a small missing field only after proving it is needed by the selected workflow and cannot be represented honestly in existing notes/diagnostics/provenance. Treat any schema compatibility impact explicitly; do not migrate unrelated result types. - -### B2. Agreement across actual consumers - -**Acceptance:** For population, these_areas and conservative targets with deliberately unequal SE columns, serialized results, exported tables, figure intervals, captions/methods and next-step guidance agree on basis, confidence level and interpretation. Include an existing t/df result as a reporting check if that path is supported at the resumed baseline. Test content and numerical meaning, not byte equality of every string or image. - -Undefined uncertainty remains visibly undefined, not zero-width or an apparently significant result. Missingness and incomplete-replicate decisions come from Round 11, not ad hoc formatting. Broad raw/sentinel normalization remains out of scope beyond those decisions. - -Check shared Python workflow, CLI and MCP adapter behavior for the operations exercised here. Normalize timestamps and output paths only for comparison; compare estimates, chosen uncertainty, warnings, and meaningful error codes. Do not force unsupported parameters into the core MCP profile or require GUI modernization. Preserve method-specific uncertainty limitations rather than flattening every method to one claim. - -**Likely targets:** Existing result/interval helper only as necessary, `workflows_inference.py`, `report_tables.py`, `figures.py`, and corresponding existing tests/docs. Round 11 C7 should already remove the immediate contradictory instruction; avoid reopening it as a second implementation. - -**Stop condition:** One interpretation table and selected consumer tests suffice. If the proposal requires a universal result schema, extensible plugin layer or repository-wide error rewrite, defer that redesign. - -## 7. Workstream C (P0): One executable reference workflow - -### C1. Finish a path the project already supports - -**Recommended first path:** A single-release area-level regression with an explicit estimand, known inputs, an uncertainty/readiness check, and an existing results bundle. Start from `examples/manifests/income_population_tracts.json`, existing manifest tests/fake clients, and the regression/preregistered recipes. The existing example is an analysis-mechanics demonstration, not evidence that population causes household income. Label synthetic data as synthetic and keep conclusions descriptive. - -Choose exact variables, units, geography, sample and target before execution. Prefer current frozen inputs and test clients. If those do not provide a defensible real-data example, keep the deterministic example honestly synthetic rather than downloading data or inventing an empirical result. Production data acquisition is a separate use of the existing extraction interface. - -The workflow must make all necessary steps explicit: input identity, variable selection, planning, validated local input, any table join and one-row-per-unit check, derivation and units, exclusions/readiness, model/target, table or figure, methods/limitations, and provenance. Joined inputs must be explicit in the example; do not build a generalized join engine to support one recipe. Do not advertise recipe placeholders such as poverty_rate or median_rent unless their derivation and source columns are actually supplied. - -### C2. Reuse current artifacts and replay honestly - -Existing support includes `census/manifest.py:529-543` (execution and manifest tagging), `:720-729` (exclusions/deviations/sample), `census/report_tables.py:336-343` (bundle artifacts and hashes), and `census/workflows.py:544-559` (extraction replay checks). The deliverable is a verified path through these, not a new bundle format or a second workflow engine. - -From existing artifacts, another researcher must identify release/product where available, geography/vintage, exact variables/derivations/units, retained sample, exclusions, estimand, uncertainty basis, corrections/families if used, manifest hash, deviations and limitations. If identity was never provided, say unavailable rather than inferring it from a filename. - -Separate three claims: extraction replay checks acquisition outputs; manifest execution reruns an analysis; a bundle preserves results and provenance. None alone guarantees a historical external API will return unchanged bytes. An offline acceptance path must inject the existing frozen/fake source and refuse all network access. Use fresh temporary output directories, preserve original inputs, and compare substantive numerical results. Byte equality is required only where the existing format promises determinism; environment-dependent figure bytes are not the general success criterion. - -### C3. Researcher and maintainer acceptance exercises - -1. Follow exact documented commands in a supported existing environment using bounded local inputs. No placeholder column names or undocumented editing is needed; expected result and its independent check are supplied. -2. Produce the existing result/table/methods/provenance artifacts in a temporary directory. The meaning of each interval and the limits of the conclusion are stated beside the result. -3. Re-execute from the same frozen inputs into a second directory. Check estimates, uncertainty basis, exclusions, warnings and manifest identity; distinguish stable numerical results from incidental path/time differences. -4. Exercise one meaningful blocked/degraded case, such as incomplete required replicates, using Round 11's adopted policy. It must not be presented as a clean confirmatory result. Existing concordance guards remain intact; never bypass them to make the example run. -5. Compare the shared workflow and the supported CLI/MCP adapters for semantic agreement without live services. Where an adapter does not expose a step, document the supported route instead of adding a tool solely for symmetry. -6. Have a second reviewer follow the instructions and identify estimand, SE basis, exclusions and limitations from the artifacts. Until this exercise is done, report workflow readiness as unverified rather than claiming usability success. - -**Likely targets:** Existing regression/preregistered recipes, existing example manifest if needed, `tests/test_manifest.py`, `tests/test_replay.py`, workflow/adapter tests and narrow bundle/report corrections. Reuse fixtures; add at most one small synthetic fixture only if existing ones cannot express the case, with generator/expected values documented. Do not change existing scientific fixture values to fit outputs. - -**Optional second path:** Complete the existing county poverty-rate comparison recipe only if a separate scalar-to-derived-rate boundary remains untested. It must reuse existing APIs/fixtures and contribute a distinct independent check. Default scope remains one workflow; maps, PUMS and causal designs do not become extra examples this round. - -## 8. Execution order, review and stop points - -| Slice | Dependency | Reviewable result | Completion evidence | -| :--- | :--- | :--- | :--- | -| 12.0 Baseline | Final Round 10 state; Round 11 disposition | Confirm which proposals remain necessary, exact supported environment, existing artifacts | Read-only inventory and recorded scope | -| 12.A Independent evidence | Baseline and relevant Round 11 fixes | Two targeted independent cases plus check-tier/release-gate contract | Focused tests; full studies only when scientifically required | -| 12.B Interpretation | Adopted Round 11 input/inference policies | Consistent meaning in selected serialized and human outputs | Unequal-SE and applicable df examples; adapter semantic checks | -| 12.C Executable workflow | A/B for selected path | One complete existing recipe/manifest path and artifact handoff | Frozen-input rerun, blocked case, second-reader review | -| 12.Close | A-C | Short evidence/limitations handoff and deferred decisions | No unsupported validation or reproducibility claims | - -Each slice can ship independently after approval and checks. An owner can stop after A or B with useful improvements; do not call the entire round complete unless its agreed core is done. Coordinate shared-file edits serially. Parallel read-only scientific, workflow and maintenance review is useful; parallel editing of the same result helpers is not. - -Maintain a short evidence ledger in existing audit documentation when implementation is approved: claim, scope, oracle, configuration/input identity, outcome, uncertainty/limitation, revision, and responsible reviewer. Extend an existing report or table rather than creating a registry, database, or standalone evidence platform. Clear source-backed documentation can be the correct outcome when an optional feature would add more maintenance than research value. - -## 9. Decisions and follow-up register - -These are explicit planning stubs with current evidence and a bounded next action. Recommendations are not adopted defaults or permission to implement. - -| ID | Status / known evidence | Recommended next action | Stop or escalation condition | -| :--- | :--- | :--- | :--- | -| P12-1 Baseline readiness | Pending; active Round 10 changes remain protected | Revalidate Round 11 findings and final contracts at handoff | No implementation against an assumed historical baseline | -| P12-2 Independent references | Source pass complete; A1 identifies limited-independence examples, not invalid results | Select two smallest nonredundant reference cases; derive expected values independently | If Round 11 supplied an equivalent oracle, reuse it and reduce scope | -| P12-3 Report versus gate | **Decided 2026-11-07: both.** The command stays a report by default (exit 0, compatible); a `--gate` flag exits nonzero when any required study fails or is missing; info rows never fail the gate | Implement in scripts/inference_validation.py with synthetic report tests | No silent exit-code change or second validation runner | -| P12-4 Interpretation fields | Existing ModelResult, interval helper and table basis already present | Inventory selected artifacts against B1 before adding fields | No universal schema or duplicated target mapping | -| P12-5 Example choice | Recommend existing income/population manifest and regression/preregistered recipes | Confirm a complete deterministic input/derivation/target path; label synthetic evidence | If completion requires new data acquisition/engine, simplify the example | -| P12-6 Reproducibility limits | Existing bundle, manifest and extraction replay have different roles | Verify exact supported promises using current tests and frozen adapters | No claim of full analysis replay from extraction metadata alone | -| P12-7 Resource/runtime budget | No benchmark or full-suite runtime measured in planning | Measure the selected checks before setting PR/release budgets | Do not require long stochastic suites on every documentation change | -| P12-8 Human readability | Not evaluated; no user study claimed. **Owner decision: an independent agent session may act as the second reader** (agents may review, as for overrides) | Second reader follows the canonical workflow and interprets artifacts | Correct confusing instructions before adding another example | - -## 10. Explicit deferrals and reopening criteria - -| Deferred work | Reason | Reopen only when | -| :--- | :--- | :--- | -| New Bayesian/causal/spatial/survey estimator families | Scientific and maintenance burden; existing methods need firmer evidence | A concrete research question cannot be answered responsibly by supported methods and has an independent validation design | -| More Census products/providers or external datasets | Coverage expansion is not the current bottleneck | An evidenced workflow need survives review of existing data and Round 10 scope | -| Dashboard, hosted service, GUI redesign, notebook gallery | Additional interface/documentation surface | A supported user workflow demonstrably requires it and has a maintenance owner | -| General plugin/schema/workflow/provenance frameworks | Duplicates existing plain objects, manifests and bundles | Multiple current consumers prove a small shared contract insufficient | -| Broad dependency splitting, wheel resource migration, whole-repo lint/type cleanup | Intentional clone architecture and large protected diff | A reproducible supported-installation or changed-code problem justifies a separate bounded proposal | -| Distributed execution and full PUMS streaming rewrite | Requires measured resource need and larger design | Round 11 R2 measurements demonstrate the bottleneck and a bounded remedy is insufficient | - -Deferral is not a future promise. Keep this list short; do not convert every idea into another round. A new item should replace lower-value scope or have separate authorization, a maintenance owner and a clear completion criterion. - -## 11. Definition of done - -- The agreed independent cases use genuinely independent expectations and state exactly which scientific layer they validate. -- Selected user-visible artifacts and supported adapters agree on estimand, interval basis, confidence level/distribution, sample handling and limitations. -- One existing workflow runs offline with exact inputs and instructions, produces existing reviewable artifacts, reruns substantively, and handles a meaningful blocked case honestly. -- Applicable focused and offline reference checks pass in the supported environment; relevant full studies, if run, are reported with fixed rules and Monte Carlo precision. Skipped, unrun and informational checks are labeled. -- No new estimator families, providers, required dependencies, public tools, persistence systems or bundle/manifest formats were introduced; any scope exception is justified separately. -- A maintainer can identify the smallest relevant check and the release evidence for the touched path, and a second reader can understand the result without reading code. -- Earlier plans, unrelated edits, datasets and credentials remain preserved. Implementation handoff lists actual changes, results, compatibility impact, remaining decisions and deferred work; no commits or pushes without permission. - -**Current handoff:** Draft informed by three parallel read-only reviews and parent source verification. No implementation, dependency setup, new runtime results, external verification or release action is claimed. - ---- - -## 12. Implementation status (2026-11-07) - -| Slice | Result | -| :--- | :--- | -| 12.A Independent evidence | Dense hand-assembled references for the known-variance multilevel GLS layer (fixed components, heterogeneous D, within-block covariance) and the two-group DiD (explicit contrasts, reference-period edge case); no production defect found. `scripts/inference_validation.py --gate` (P12-3: report by default, gate on request); `docs/audits/CHECK_MAP.md` maps each method to its focused test and release study | -| 12.B Interpretation | `census.results.interval_description` is the single source for estimand, interval basis, level, normal vs t (df), undefined terms, and replicate completeness; six disagreements fixed across serialization, tables, figures, methods text, workflow guidance, and CLI/MCP adapters | -| 12.C Executable workflow | `docs/reference_workflow.md` with offline frozen synthetic inputs; `tests/test_reference_workflow.py` runs the documented commands twice with network refused and compares substantive results; blocked case under strict replicate policy. Fixed: results bundles lost hypothesis results (truncated repr) | -| Second reader (P12-8) | Independent agent followed the doc with no prior context; all expected values reproduced; verdict "usable with fixes". Fixed: blocked runs labeled `confirmatory` (now `confirmatory_incomplete`); synthetic data cited as Census API data (new manifest `data_source`, SYNTHETIC citations, caller-supplied replicates described and hashed); doc setup, cleanup, fixture hash, method naming | - -**Deferred:** multilevel and DiD design studies are not yet in the `--gate` required set (run on demand per CHECK_MAP.md); CLI/MCP `run_manifest` cannot take replicates (Python route documented); no CLI command loads frozen responses into the cache (Python step documented). - diff --git a/PLAN_ROUND_13.md b/PLAN_ROUND_13.md deleted file mode 100644 index a0484d5..0000000 --- a/PLAN_ROUND_13.md +++ /dev/null @@ -1,162 +0,0 @@ -# Round 13: Real Data, End to End - -**Date:** 2026-11-07 -**Status:** Implemented (section 11); released as v0.12.0 -**Baseline:** v0.11.0 (Round 12 merged; default multilevel/DiD design studies added to the release gate) -**Companions:** [PLAN_ROUND_11.md](PLAN_ROUND_11.md) (R1/R2/F4 risks), [PLAN_ROUND_12.md](PLAN_ROUND_12.md) (deferrals) - -## 0. Front-loaded findings (fill before implementation starts) - -Items in this section come first in the work order. Each one is either fixed in Round 13 or recorded as a limitation with evidence. - -| ID | Source | Finding | Disposition | -| :--- | :--- | :--- | :--- | -| G0 | v0.11.0 full `--gate` run, 2026-11-07 | **PASS.** All 10 required studies are present and every gated row is inside the band. Wall time is **106 min**: `spatial_fay_herriot` takes 56 min (53%) and `regression_calibration` 18 min; the four default design studies add about 11 min combined | No defects. Runtime goes to C3, which targets `spatial_fay_herriot` first | -| G1 | GitHub Actions API, 2026-11-07 | The weekly `live-validation` job has zero scheduled runs. **Cause verified: it is not yet due, not a defect.** The cron was pushed on Monday 2026-11-02 after that day's 06:00 UTC slot, so the first scheduled run is Monday 2026-11-09. The `CENSUS_API_KEY` secret exists and the workflow is active. A first run was started via `workflow_dispatch` on 2026-11-08 UTC | Workstream A1: triage that run; confirm the 2026-11-09 scheduled run | -| G2 | `.github/workflows/*.yml` | CI never runs `scripts/inference_validation.py`, not even `--fast`. A coverage regression would show up only at release time | Workstream C1 | -| G3 | PLAN_ROUND_12 sec. 12 | Deferred: CLI and MCP `run_manifest` cannot take replicates. There is no CLI command to load frozen responses into the cache | Workstream B1, B2 | -| G4 | PLAN_ROUND_11 F4 | The DiD coverage studies use independent shocks only. Serial correlation is untested | Workstream C2 | -| G5 | PLAN_ROUND_11 R1/F3 | R1 (runtime checksum verification) is kept as a documented risk. R2 (PUMS memory) has a static bound only and no measurement | Workstream D | - -Rule: do not tune thresholds to make a G0 row pass. Before calling a failing row a fluke, rerun it on a new seed with more replications (as was done for the DiD 0.872 cell in Round 12). - -## 1. The improvement this round delivers - -Everything shipped so far has been verified offline, against frozen fixtures and synthetic data. Round 13 runs the toolkit against **live Census data**: - -- A scheduled and on-demand comparison with Census-published values that actually runs. -- One real-data research workflow, from extraction through a journal-quality map and model to a manifest, that a second reader can rerun. -- The few interface gaps that kept the Round 12 reference workflow Python-only. - -No new estimator families, providers, or interfaces. The PLAN_ROUND_12 sec. 10 deferrals still apply. - -## 2. Workstreams - -| Priority | Workstream | Slices | -| :--- | :--- | :--- | -| P0 | A. Live verification | A1, A2 | -| P0 | B. Close the Round 12 interface gaps | B1, B2 | -| P1 | C. Validation in CI and the DiD sensitivity study | C1, C2 | -| P1 | E. Real-data research workflow | E1, E2 | -| P2 | D. Measured risks (R1/R2) | D1, D2 | - -The work order is section 0 items, then A, B, C, E, D. A and B are independent and can run as parallel agents in **separate git worktrees**. C1 touches only CI. E depends on A1, because it needs a working live path. - -## 3. Workstream A: live verification (P0) - -**A1. Make the live job run.** -- Confirm that the `CENSUS_API_KEY` repository secret exists. Check that the API reports the secret by name only; never read or print its value. -- Trigger `live-validation` through `workflow_dispatch` and find out why the schedule has never fired. Possible causes: the cron is on `defaultBranch`, the workflow file was added recently, or Actions schedules are disabled. -- Triage every failing comparison into one of four buckets: - - tool defect (fix it); - - Census revision or errata (record it with its errata ID); - - tolerance or rounding (document it; don't widen silently); - - network or flaky (retry policy). -- Done when one green dispatched run exists, its `validation-report` artifact is linked from `docs/audits/validation_report.md`, and the schedule has fired at least once or its blocker is documented. - -**A2. Run locally with the owner's key.** -- Run `scripts/validate_against_census.py` from this machine with the key in the environment only. -- Commit the regenerated `docs/audits/validation_report.{md,json}` with its date and vintages. -- Done when the committed report reflects a live run, not a frozen one, and says which. - -## 4. Workstream B: close the Round 12 interface gaps (P0) - -**B1. Replicates in the CLI and MCP `run_manifest`.** -- Add `replicates_path` to the CLI (`--arg replicates_path=...`) and the MCP tool, reusing the existing `read_replicates`/`replicates_sha256` adapter. -- Remove the "Python route" caveat from `docs/reference_workflow.md`. -- Tests: CLI and MCP parity with the Python route, including the same manifest hash and the strict-policy blocked case. - -**B2. Cache import CLI.** -- Add `python -m census cache import ` (extending the existing `cache` command, not a new runner) that loads frozen responses and marks the cache synthetic when the source is synthetic. -- The reference workflow should then run entirely from the CLI. -- Test: the documented commands run twice with the network refused and produce identical results. Extend `tests/test_reference_workflow.py`; don't add a new harness. - -## 5. Workstream C: validation in CI and the DiD sensitivity study (P1) - -**C1. Fast gate in CI.** -- Add one CI step, `python scripts/inference_validation.py --fast --gate`, on one OS and one Python version only. -- Measure its runtime first (PLAN_ROUND_12 P12-7). If it costs more than about 5 minutes, run it on `pull_request` to `defaultBranch` only. -- The full gate stays a release-time step, as in the v0.11.0 process. - -**C2. DiD serial-correlation sensitivity (F4).** -- Add an `info`-status study with AR(1) unit errors at rho of 0, 0.3, 0.6 and 0.9, for the default DiD intervals. -- It does not gate. It reports where coverage degrades, and documentation states the model scope (independent errors, or clustered SE where supported). -- Promote it to gated only if the owner decides that serially correlated panels are an in-scope claim (D13-3). - -**C3. Full-gate runtime.** -- The v0.11.0 full gate took 106 minutes; the script header claimed about 40. `spatial_fay_herriot` alone took 56 minutes (G0), so profile its per-replication fit before anything else. -- Profile the gate per study, then: - - run independent studies in separate processes; - - pin BLAS threads per worker so the processes don't oversubscribe the CPU; - - make sure each study's seed does not depend on scheduling order, so results stay identical. -- Done when a parallel run reproduces the serial CSV exactly at the same seed, the wall time is recorded, and the header gives the measured time. - -## 6. Workstream E: real-data research workflow (P1) - -**E1. One worked study on live ACS 5-year data.** -- County median household income, adjusted with RPP, for Michigan and New York (D13-2): - - a choropleth map with an uncertainty layer (journal style); - - a regression with replicate SEs; - - a run manifest showing `data_source=census_api`; - - a results bundle. -- The scope is a single workflow, not a gallery. -- Spot-check its published numbers against data.census.gov through A2. - -**E2. Second reader.** -- An independent agent session reruns E1 from the documentation alone, as in Round 12 P12-8. Its verdict and fixes are recorded. -- Done when the rerun reproduces the substantive results and the manifest hash differs only in the documented volatile fields. - -## 7. Workstream D: measured risks (P2, cut first if time runs short) - -**D1. R1: runtime integrity for bundled price tables.** -- Verify `csv_sha256` when bundled tables load (the files are small, so the cost is negligible). -- Raise an error on a mismatch for bundled data. Supply custom directories with a recorded hash and no verification. -- If the owner keeps R1 as a documented risk (D13-4), skip this slice. - -**D2. R2: PUMS memory benchmark.** -- Run a bounded synthetic benchmark that records peak RSS per phase (parse, concatenate, weight matrix) and checks that estimates match across chunk sizes. -- Size: up to an `81 x 1.9M` weight matrix, about 1.15 GiB static, well inside 128 GB. -- Outcome: a documented limit, with no streaming rewrite unless the measurement shows one is needed. - -## 8. Release - -The round ends as **v0.12.0** with a Zenodo DOI, following the standing release process: version bump, CHANGELOG, full `--gate`, PR, CI, merge, tag, and Zenodo confirmation. - -## 9. Owner decisions - -| ID | Decision (owner, 2026-11-07) | -| :--- | :--- | -| D13-1 | **Yes.** For our own use, agents use the owner's key (environment only, never printed or committed). When the toolkit is published for others, each user supplies their own `CENSUS_API_KEY`; docs and errors must say so, and nothing ships with a key | -| D13-2 | **Michigan (26) and New York (36)** are the test targets for E1 and the live spot checks; all states remain supported (no state-specific code) | -| D13-3 | **Report only.** The serial-correlation DiD study is `info` status and does not gate | -| D13-4 | **Implement D1** (load-time checksum verification for bundled tables) | -| D13-5 | **PRs only.** The fast gate runs on `pull_request` to `defaultBranch`; the full gate stays release-time only | - -## 10. Definition of done - -- Section 0 items are resolved or recorded as limitations with evidence. -- At least one green live validation run, with the report committed and labeled live. -- The reference workflow runs from the CLI with no Python steps. -- The fast gate runs in CI. The DiD sensitivity results are documented. -- One real-data workflow is reproduced by a second reader. -- D1 and D2 are done or explicitly deferred. -- v0.12.0 is released with a DOI. - ---- - -## 11. Implementation status (2026-11-07) - -| Slice | Result | -| :--- | :--- | -| G0 / gate | v0.11.0 gate PASS (106 min). After C3 the full gate PASSES in 17.2 min (`--jobs 0`), with a coverage table identical to v0.11.0 | -| A1 live CI | First dispatched run: 941 live tests passed; one failure was a stale `expectedFailure` on F5 (fixed 2026-11-02, marker never removed). Fixed, along with the F5 criterion text. Scheduled run due 2026-11-09 | -| A2 live battery | 10/10 checks pass live (`docs/audits/validation_report.md`, 2026-11-08) | -| B interfaces | `cache import` (atomic; synthetic into the default cache requires `--allow-default-cache`; `cache clear` removes the marker). Python, CLI and MCP `run_manifest` parity with `replicates_path`. The reference workflow is CLI only | -| C validation | Fast smoke gate on PRs (band widened by 3 MC SE). `did_ar1_sensitivity` (info): coverage 0.88-0.91 for rho 0 to 0.9 in 4-5 period panels. Pinning BLAS to one thread plus `--jobs` cut spatial FH from 3361 s to about 200 s, and its results no longer depend on the machine's thread count | -| D1 integrity | Bundled price tables are verified against their SHA-256 at load; R1 closed | -| D2 PUMS memory | Not done; deferred (R2 stays documented with its static bound) | -| E1 case study | Live MI/NY study (`docs/case_study_mi_ny.md`): pct_bachelors slope 811 RPP-adjusted dollars per point, 90% CI [684, 938] (population target); 10/10 spot-check counties identical to the API. New tools: `place_adjust_frame`, `make_results_bundle` | -| Review | `/code-review high`: 10 findings, all fixed with regression tests (`tests/test_review_fixes_r13.py`) | -| E2 second reader | Verdict "usable as-is": every command ran and every number matched. Clarity edits applied | - -**Deferred:** D2 (PUMS memory benchmark); multi-state `run_manifest` with a place-adjustment step (so the case study is exploratory, not pre-registered); no MCP route for `cache import`; adding Michigan to the validation battery's states (needs fixtures re-recorded). diff --git a/PLAN_ROUND_14.md b/PLAN_ROUND_14.md deleted file mode 100644 index c19e622..0000000 --- a/PLAN_ROUND_14.md +++ /dev/null @@ -1,92 +0,0 @@ -# Round 14: Confirmatory Multi-State Analyses - -**Date:** 2026-11-08 -**Status:** Implemented (section 9); released as v0.13.0 -**Baseline:** v0.12.0 (Round 13 merged) -**Companion:** [PLAN_ROUND_13.md](PLAN_ROUND_13.md) (section 11 lists the deferrals this round takes up) - -## 0. Front-loaded findings - -| ID | Source | Finding | Disposition | -| :--- | :--- | :--- | :--- | -| G1 | Round 13 E1 | The Michigan/New York case study is labelled *exploratory*. A `run_manifest` geography holds a single state, and there is no place-adjustment step, so the study cannot be pre-registered | Workstream A | -| G2 | Round 13 deferrals | The validation battery covers states 06, 36, 48, 17 and 56. Michigan, one of the owner's preferred test states (D13-2), is not included | Workstream B | -| G3 | PLAN_ROUND_11 R2/F3 | PUMS memory has only a static bound (one 81 x 1.9M float64 weight matrix, about 1.15 GiB). Nothing has been measured | Workstream C | -| G4 | Round 13 CI logs | Pandas 4 deprecation (`inplace` in `DataFrame.rename`, `census/export.py:40`). `cache import` has no MCP route | Workstream D | -| G5 | Scheduled CI | The first weekly live-validation run is due 2026-11-09 | Confirm it after the round lands; triage it as in Round 13 A1 | - -Same rules as before: tolerances, bands and thresholds are not tuned to make a check pass. Every failure is classified as a defect, a Census revision, rounding, or flakiness. - -## 1. The improvement this round delivers - -A researcher can pre-register a multi-state, price-adjusted analysis as a manifest and run it confirmatorily from the CLI or MCP, with the same blocking deviations, replicate policy and provenance as the single-state route. The Michigan/New York study then becomes the worked confirmatory example. The validation battery covers Michigan, and PUMS memory use is measured rather than estimated. - -No new estimator families or data providers. PLAN_ROUND_12 sec. 10 deferrals still apply. - -## 2. Workstreams - -| Priority | Workstream | Owner | -| :--- | :--- | :--- | -| P0 | A. Multi-state manifests with a place-adjustment step | agent A | -| P1 | B. Michigan in the validation battery | agent B | -| P1 | C. PUMS memory benchmark (R2) | agent C | -| P2 | D. Small items: pandas 4 deprecations, MCP `cache_import` | agent C (after C) | - -A, B and C touch disjoint modules (`manifest`/workflows, `validation`/fixtures, `pums`/scripts) and run in parallel in separate git worktrees. - -## 3. Workstream A: confirmatory multi-state analyses (P0) - -- **A1. Several states in `geography`.** Accept `"states": ["26", "36"]` as an alternative to `"state"`. A manifest gives one of the two, never both. The extraction runs per state and stacks the results. The manifest hash, validation errors and run status behave as they do for one state. Single-state manifests and their hashes are unchanged; the hash of the existing example manifest must not change. -- **A2. Place adjustment as a declared step.** For example `"adjustments": [{"type": "rpp", "variables": [...], "rpp_year": 2023}]`. It reuses the Round 13 `place_adjust_frame` core (CBSA membership, nonmetro fallback, replicate scaling), not a second implementation. RPP provenance is recorded in the manifest output, including the custom-table hash when a custom table is used. -- **A3. Derived variables** if the manifest cannot already express `pct_bachelors` from B15002 cells: reuse the existing `derive` machinery and its replicate evaluation. -- **A4. Example and test.** - - Add `examples/manifests/mi_ny_income_education.json` (pre-registered hypothesis: positive pct_bachelors slope on RPP-adjusted income). - - Add a frozen-fixture offline test of the whole route, and a `CENSUS_LIVE_TESTS=1` live test. - - Update `docs/case_study_mi_ny.md` so the confirmatory manifest route comes first, with the exploratory route kept as the sensitivity analysis. - - Done when the live confirmatory run reproduces the Round 13 coefficient (811.0, 90% CI [684.3, 937.7]) or explains any difference. - -## 4. Workstream B: Michigan in the validation battery (P1) - -Add `"26"` to the battery's states, re-record the frozen fixtures live (`--record`), and report how much the fixture set grows. Rerun the battery live and offline. A new failure is triaged before anything else; no tolerance changes. Regenerate `docs/audits/validation_report.md`. - -## 5. Workstream C: PUMS memory benchmark (P1) - -Add `scripts/pums_memory_benchmark.py`, using bounded synthetic PUMS-shaped CSVs only (no downloads): -- sizes 100k, 500k and 1.9M person rows; -- peak RSS per phase (download or read, parse, concatenate, weight matrix, tabulation); -- several chunk sizes, with identical estimates across chunk sizes verified; -- platform and dependency versions recorded. - -Write the results to `docs/audits/pums_memory.md`, with a documented limit and a recommendation. Change code only if the measurement shows a defect, such as a needless copy. No streaming rewrite unless the owner approves one. - -## 6. Workstream D: small items (P2) - -- Fix pandas 4 deprecations found by running the suite with `-W error::FutureWarning` and the Pandas4Warning class. -- Add a `cache_import` MCP tool (full profile) wrapping `census.cache.import_frozen`, with the same safety rules as the CLI: atomic, and a synthetic import into the default cache needs an explicit flag. - -## 7. Release - -v0.13.0 with a Zenodo DOI: code review (`/code-review high`), fixes, full gate (`--jobs 0`), PR, CI, merge, tag, Zenodo confirmation. - -## 8. Defaults taken without asking (owner may override) - -| ID | Default | -| :--- | :--- | -| D14-1 | Multi-state manifests stack states; cross-state models need an explicit state term in the formula (no automatic fixed effects) | -| D14-2 | RPP year defaults to the latest bundled year, and the manifest records it. A mismatch with the ACS period is a note, not a blocking deviation | -| D14-3 | The Michigan fixtures are committed if the fixture set stays under about 5 MB in total; otherwise the agent reports the size first | - ---- - -## 9. Implementation status (2026-11-08) - -| Slice | Result | -| :--- | :--- | -| A multi-state manifests | `states`, derivations on replicates, declared RPP `adjustments` (one shared implementation with `place_adjust_frame`). Live MI/NY confirmatory run: no deviations, identical to Round 13 (811.0 [684.3, 937.7]). Existing manifest hashes unchanged | -| B Michigan | Added to the battery; fixtures 2.0 to 2.2 MB; 10/10 checks pass live and offline; no tolerance changes | -| C PUMS memory (R2) | Measured: about 275 MiB + 2,829 B/row; 1.9M rows peaks at 5.4 GiB after removing avoidable copies (was 6.4). 128 GiB holds every state's records at once. R2 closed | -| D small items | MCP `cache_import`; the only pandas 4 deprecation in census/ fixed | -| Review | `/code-review high`: 10 findings fixed with tests (`tests/test_review_fixes_r14.py`). Parent change: `create_manifest` never overwrites its input | -| Gate | Full `--gate --jobs 0`: PASS, 17 min; coverage table unchanged | - -**Deferred:** adjustments limited to one release and not combinable with aggregation; `run_manifest` does not fetch VRE replicates itself; the offline multi-state test uses frozen CBSA membership (the overlay is covered by the live test). diff --git a/PLAN_ROUND_15.md b/PLAN_ROUND_15.md deleted file mode 100644 index c1c229a..0000000 --- a/PLAN_ROUND_15.md +++ /dev/null @@ -1,89 +0,0 @@ -# Round 15: Studies Across Releases, Spatial Coverage, 2020 Census - -**Date:** 2026-11-08 -**Status:** Implemented (section 6); released as v0.14.0 -**Baseline:** v0.13.0 -**Companions:** [PLAN_ROUND_14.md](PLAN_ROUND_14.md), [docs/remote_access_decisions.md](docs/remote_access_decisions.md) - -## 0. Front-loaded findings and boundaries - -| ID | Source | Finding | Disposition | -| :--- | :--- | :--- | :--- | -| G1 | Round 14 deferrals | A manifest's RPP adjustment covers one release only, and `run_manifest` cannot fetch Census VRE replicates itself | Workstream A | -| G2 | `docs/audits/CHECK_MAP.md` | `spatial_regression` (lag/error) and `local_morans` have no coverage study; global Moran's I has only an info study | Workstream B | -| G3 | Owner gap analysis | ACS only. 2020 Decennial Census access exists only internally, as block counts for crosswalks (`census/block_weights.py`) | Workstream C | -| G4 | Scheduled CI | First weekly live run was due 2026-11-09 | Check it and triage as in Round 13 A1 | - -**Boundary (planner):** no networking changes in this round, and no HTTP, persistent service, job queue, custom transfer API or Data Commons adapter. Remote access is the next round (SSH, per `docs/remote_access_decisions.md`). - -As before: no tuning of tolerances, bands or thresholds to make something pass. - -## 1. Workstream A: confirmatory studies across releases (P0) - -- **A1. Several releases with price adjustment.** A manifest with several releases (for example non-overlapping 5-year periods, 2015-2019 vs 2020-2024, or 1-year releases) can declare RPP adjustment for each release's own year. Dollars are converted to one declared dollar year with the existing CPI machinery (R-CPI-U-RS default), *then* place-adjusted. The order is stated in the methods text. - - The declared `dollar_year` and per-release `rpp_year` are required, so the hash fixes them; `create_manifest` fills them as in Round 14. - - Existing overlap rules for comparisons of overlapping 5-year periods stay in force. -- **A2. Automatic replicate retrieval.** `"replicates": {"source": "census_vre"}` in a manifest makes `run_manifest` fetch the VRE tables for the manifest's variables, geography and releases, using the existing `census.replicate` code and its cache. - - The VRE file hashes and URLs go into the run record. - - Variables with no published replicates (for example B19013) are reported. Under the strict confirmatory policy they block, as today, unless the manifest says which analyses use only covariate replicates. The rule is explicit, never silent. -- **A3. Example study and tests.** - - An example manifest for a change study in Michigan and New York counties across two non-overlapping 5-year periods: change in RPP- and CPI-adjusted median household income, or another pre-registered contrast the existing methods support. - - Frozen-fixture offline test, plus a live test gated by `CENSUS_LIVE_TESTS=1`. - - Docs: the preregistration recipe and a short section in the case study. - -## 2. Workstream B: coverage for spatial regression and local Moran's I (P1) - -- **B1. `spatial_regression_coverage`.** A spatial lag and an error model on a lattice with ACS-like sampling error, for each estimator the tool offers (GMM/ML). Report coverage of beta, and of rho/lambda, for the tool's default intervals. Also report bias and the LM diagnostics' size under the null. -- **B2. `local_morans_size`.** Local Moran's I under spatial independence plus sampling noise. Report: - - per-area false-positive rate and family-wise or FDR control, for the tool's default inference and correction; - - power under one clustered alternative. -- **B3. Status.** Both run first as `info` studies. - - A study whose default intervals pass the band can be promoted to gated by owner decision (D15-1). - - If a default fails, the tool's outputs get a visible caveat, or the default changes (for example to a corrected method), with evidence. Never tune the band. -- Update `docs/audits/CHECK_MAP.md`. Keep each study's runtime reasonable under `--jobs`. - -## 3. Workstream C: 2020 Decennial Census (P1, provisional) - -- **C1. Retrieval.** P.L. 94-171 (`dec/pl`) and the Demographic and Housing Characteristics file (`dec/dhc`) through the existing API client, cache and output formats, with variable metadata and codebook. Reuse the block-count code where possible. -- **C2. Correct uncertainty semantics.** Decennial values are **published counts that are not sample-based but not error-free**: - - no sampling MOE; - - coverage error, processing error, and disclosure avoidance through the TopDown differential-privacy algorithm. - - Concretely: - - Outputs never carry a zero MOE or a CV of 0 presented as precision. The uncertainty fields are explicitly "not sample-based (MOE undefined)", with a disclosure-avoidance advisory. - - The advisory notes that small geographies and detailed cells are noisier, that only certain counts are invariant (state total population, block-level housing units and occupied group-quarters facility counts), and that block-level characteristics should be aggregated before use. - - Significance tools refuse decennial-only "sampling" tests, or say exactly what they test. - - Mixing ACS and decennial values in one comparison is allowed only with a stated note. -- **C3. Validation.** A new battery check against the Census Bureau's own invariants and hierarchy: - - state total population matches the published state totals; - - block-level housing-unit counts sum exactly to tract and county totals; - - tract and county totals from DHC agree with P.L. 94-171 where both publish the same item. - - Record live, freeze fixtures, keep fixtures small (Michigan, New York, Wyoming). -- **C4. Scope.** Retrieval, semantics and validation only. No new estimators. DHC tables used in the examples: P1, P12, H1, H3. The 2010 Census is out of scope unless needed. - -## 4. Release - -v0.14.0 with a Zenodo DOI: code review (`/code-review high`), fixes, full gate (`--jobs 0`), PR, CI, live dispatch, merge, tag, Zenodo confirmation. - -## 5. Defaults taken (owner may override) - -| ID | Default | -| :--- | :--- | -| D15-1 | New spatial studies start as `info`. Promotion to gated is an owner decision after the evidence is in | -| D15-2 | Multi-release dollar conversion: CPI to the declared dollar year first, then the RPP of each release's own year | -| D15-3 | Decennial geography levels: nation, state, county, tract, block group, block (block only for P.L. 94-171 and invariant counts), place | - ---- - -## 6. Implementation status (2026-11-08) - -| Slice | Result | -| :--- | :--- | -| A releases | Multi-release adjustments (CPI to dollar year, then per-release RPP) and `census_vre` replicate retrieval. Live MI/NY change study: +3,110.80 (90% CI [2,708.93, 3,512.67]), no deviations; hand recomputation from raw VRE files agrees to about 1e-12 | -| B spatial coverage | Info studies added to the full report. Lag ML rho coverage collapses under outcome sampling error (0.48, 0.003); GMM 0.89-0.90. LISA folded p-values doubled the false-positive rate. **Owner decisions:** lag default GMM; LISA two-sided, 9999 permutations. Error-model lambda attenuation and patterned-noise false clusters remain caveats | -| C 2020 Census | `dec/pl`, `dec/dhc` through the existing pipeline; MOE undefined, never 0; disclosure-avoidance advisory; sampling tests refuse; F11 passes live and offline | -| Review | `/code-review high`: 10 findings fixed with tests (`tests/test_review_fixes_r15.py`), including overlap refusal for every replicate source and explicit-evidence decennial detection | -| Gate | Full `--gate --jobs 0`: PASS, 19 min; existing coverage rows unchanged; new spatial rows are info | - -**Deferred:** 1-year releases and decennial data in manifests; aggregation with adjustment or `census_vre`; `covariates_only` beyond `fit_replicate`; a fix for error-model lambda attenuation and for patterned sampling error in LISA (no tested method yet). **Next:** minimum SSH remote access per `docs/remote_access_decisions.md`. diff --git a/PLAN_ROUND_16.md b/PLAN_ROUND_16.md deleted file mode 100644 index 3cbfa3c..0000000 --- a/PLAN_ROUND_16.md +++ /dev/null @@ -1,173 +0,0 @@ -# Round 16: Validating Spatial Methods Under ACS Sampling Error - -**Date:** 2026-11-08 -**Status:** Implemented (section 7); released as v0.15.0 -**Baseline:** v0.14.0 (spreg 1.9.1, esda 2.10.0, libpysal 4.15.1) -**Companions:** [PLAN_ROUND_15.md](PLAN_ROUND_15.md) (spatial coverage studies), [docs/remote_access_decisions.md](docs/remote_access_decisions.md) (remote access follows this round) - -## 0. Principle and findings carried in - -**Validation first:** understand each failure, test competing explanations, and only then change an estimator. New estimators or substantial methods come in only when the experiments **and** accessible methodological sources support them. Otherwise findings are documented and an implementation round is proposed. - -| ID | Finding (Round 15) | Correct reading | -| :--- | :--- | :--- | -| F1 | Error-model λ coverage 0.16-0.58 under outcome sampling error | λ is spatial autocorrelation of the **disturbances**, not a substantive covariate spillover. The failure is **attenuation** of λ by measurement error in y (no-sampling-error reference covers 0.88-0.90). Earlier summaries calling λ a "spillover term" were imprecise | -| F2 | Lag model reports β and ρ only | β is not the marginal effect in a lag model. There is no direct/indirect/total effects output, so users can misread β | -| F3 | LISA false clusters in 38% of maps ("patterned" scenario) | That scenario has **spatially heterogeneous but independent** sampling errors (CVs sorted along a gradient). It is **not** evidence about spatially **correlated** sampling errors, which are untested | - -**Two different false-cluster situations**, kept distinct throughout: - -| Scenario | Definition | -| :--- | :--- | -| Heterogeneous variances | Sampling variances vary across space, errors independent | -| Correlated errors | Sampling errors covary between areas (survey design, weighting controls) | - -A method that handles one may fail the other. FDR control does not repair p-values computed under a wrong null. - -## 1. Workstream A: spatial regression (P0) - -**A1. λ attenuation (error model).** -- Establish, from the original source, the assumptions under which a measurement-error correction for λ is valid (candidate: Lorincz-Comi et al. 2021, *Spatial Statistics* 45:100523; others if accessible). In particular, does it need only the error variances, or their covariance? -- Compare uncorrected and corrected estimators under known DGPs: bias, RMSE, interval coverage, numerical stability (non-convergence, boundary estimates). -- Sensitivity to the weights matrix (rook, queen, kNN) and to measurement-error magnitude (CV profiles). -- Run under independent and under spatially correlated errors separately. A correction validated under independence is not assumed valid under correlation. -- Implement the correction in the tool only if it validates and its assumptions are verified from an accessible source; otherwise document and propose. - -**A2. Effects for the lag model (approved).** -- First inspect spreg 1.9.1 for established impacts functionality (spatial multiplier, direct/indirect/total effects, inference) and use it if available. -- Verify the reported effects against independently computed impacts from S(W) = (I - ρW)^-1 β: the direct effect is the mean of the diagonal, the total effect is the mean row sum, and indirect = total - direct. -- Report the effects, with uncertainty, in the tool output and results. Make it impossible to mistake β for the marginal effect: the label and notes say β is not the marginal effect, and the effects table comes first. -- Measure effects coverage in the simulation. - -**A3. DGP set (spatial Durbin model approved as a DGP only).** -- Simulate: - - SEM (no substantive spillovers); - - SAR lag with known effects; - - SDM, y = ρWy + Xβ + WXθ + ε, with known effects; - - an omitted spatially correlated covariate. -- Fit the tool's models to each. Report parameter bias and coverage, and recovery of direct/indirect/total effects where they are defined. -- Separate the recovery of disturbance parameters from the recovery of substantive effects. -- **No production SDM estimator in this round.** - -**A4. Common-factor test (SEM as SDM with θ = -ρβ).** -- Evaluate a likelihood-ratio test only where both models can be fitted reliably. Confirm: - - nesting and the restriction; - - comparable likelihoods (same data, same Jacobian treatment); - - finite-sample size and power. -- Asymptotic Wald tests are known to be parameterization-dependent (Political Analysis 2020), so use LR, with simulation or bootstrap critical values if the LR size is off. -- Add the test to the tool only if it earns its place; no checklist additions. - -## 2. Workstream B: LISA under sampling error (P0) - -**B1. Five scenarios** (approved): no true clustering with -1. independent homogeneous errors; -2. heterogeneous independent errors; -3. spatially correlated errors; - -plus true clusters with -4. independent errors; -5. correlated errors. - -Correlation structures in scenarios 3 and 5 are parameterized, with ranges informed by C1 and stated as assumptions. - -**B2. Four candidate methods**, all on the same simulated maps: -1. conventional LISA (current default); -2. `replicate_quadrant_share` (a stability diagnostic, not a calibrated probability); -3. LISA on spatial Fay-Herriot EBLUPs. Explicitly test whether the smoothing **induces or alters** spatial patterns (false clusters created by smoothing; true clusters attenuated); -4. a Wong (2021) threshold-first, uncertainty-aware approach, implemented only as far as the open-access paper specifies it. - -**B3. Metrics, defined in the plan and the report:** -- **false-positive rate:** per-area share of null areas flagged; -- **FDR:** the expected share of flagged areas that are null; -- **false-cluster rate:** the share of maps with at least one flagged contiguous cluster, of size ≥ k, made only of null areas; k is stated; -- **power:** the share of true-cluster areas flagged; -- **cluster-location accuracy:** for example the Jaccard overlap of the flagged cluster with the true cluster; -- **sensitivity to W.** - -Every metric carries a Monte Carlo SE. - -Decompose errors attributable to sampling uncertainty from errors attributable to misspecified spatial dependence (for example by comparing the same maps with and without sampling error, and under the correct vs a misspecified W). - -A preferred method is never chosen by minimizing false positives alone: power and smoothing bias are weighed. - -**B4. Three questions kept separate:** -- stability of a classification across replicate maps; -- statistical significance; -- evidence of a genuine underlying pattern. - -## 3. Workstream C: ACS sampling-error covariance from replicates (P0, central question) - -**C1. Measure it, and measure how reliably it can be measured.** -- Use the ACS VRE replicates for Michigan and New York tracts (and counties), for several variables of different types (a count, a proportion, a median if replicated). -- Estimate the sampling covariance between estimates with the ACS replicate formula, Cov = (4/80) Σ_r (X_r - X)(Y_r - Y), consistent with the variance scaling. -- Analyse: - - neighbouring-pair correlations against non-neighbour pairs, by distance or contiguity order, and inside vs across county lines (weighting controls); - - structured summaries instead of unrestricted matrices, since 80 replicates give a rank-limited covariance across hundreds of tracts: mean or median neighbour correlation, a fitted structured model (for example one correlation parameter per contiguity order), shrinkage or regularization where justified; - - the stability of those summaries: bootstrap or jackknife over replicates, and split-half agreement; - - the noise floor: what pairwise correlation magnitude 80 replicates can distinguish from zero. -- Pairwise correlations are never treated as established individually. - -**C2. Report** whether the covariance structure is estimable well enough to use in a correction (A1) or a null model (B). If so, that is a methodological contribution to the engine; if not, the limit is documented with evidence. Results feed the B1 correlation parameters as stated assumptions. - -## 4. Workstream D: literature notes (P1) - -`docs/literature/spatial_methods.md` lists the references the owner's planner provided. Each entry records: -- full citation and DOI/URL; -- **verification level:** - - (1) verified from full text; - - (2) supported by abstract or accessible excerpt only; - - (3) a hypothesis or implementation suggestion still to verify; -- what it is used for, and which experiment tests it. - -The planner's summaries are not substitutes for the sources. Where access is restricted, document it, and don't implement methods whose essential assumptions cannot be verified. - -References: -- Lacombe & LeSage 2015 -- Koley & Bera 2024 -- Political Analysis 2020 (common-factor Wald test) -- IJGIS 2018 (ACS spatial autocorrelation critique) -- Wong 2021 -- SBC 2019 -- Bayesian areal ACS 2021 -- Sun et al. 2015 -- Arthur 2024 -- LeSage & Pace 2014 -- Lorincz-Comi et al. 2021 -- Rogerson 2024 - -## 5. Scope - -- No unrelated architectural changes; preserve the existing safeguards, reproducibility contracts and working functionality. -- Studies are `info`. Gating is an owner decision after the evidence is in. -- Out of scope this round: - - hierarchical Bayesian measurement-error models (a later research extension); - - Arthur-style constrained resampling and spatial FDR procedures (Sun et al.), unless B shows the basic null-model problem is solved first; - - a production SDM estimator. -- Remote laptop access follows this round. - -## 6. Release - -v0.15.0 with a Zenodo DOI, after code review, fixes, the full gate, and a test run on both Python 3.10 and 3.14 before the push. - ---- - -## 7. Implementation status (2026-11-09) - -| Workstream | Result | -| :--- | :--- | -| A spatial regression | **λ attenuation is real and correctable:** spatial Fay-Herriot recovers λ (coverage 0.86-0.93, uncorrected 0.003-0.85) under independent errors, and fails under positively correlated errors. **Lag effects** were added to the tool. **SDM data** break default GMM lag (documented). **LR common-factor test:** size fine for n ≥ 100, power weak, so not added. **Koley & Bera tests** don't identify SEM vs SDM | -| B LISA | No default change. "38%" corrected. Fay-Herriot smoothing induces clusters only under positively correlated errors. Stability ≠ evidence. Wong 2021 targets a different estimand | -| C covariance | Rates/shares/medians: neighbour error r ≈ 0 (±0.003 tracts). Controlled totals are correlated within county, with a release-dependent sign (2020-2024 negative; 2015-2019 households positive). The single 2019 Michigan population file is rejected by the census.gov firewall (URL-specific, not rate limiting) | -| D literature | 12 sources read in full (owner-supplied PDFs for the blocked ones); verification levels recorded | -| Owner decisions | Include `correct_outcome_error` in v0.15.0 (approved 2026-11-09) | -| Review | `/code-review high`: 10 findings fixed. The shipped λ estimator is now the one the studies run, with coverage unchanged within 0.007 | - -**Proposals for a later round:** -- the spatial Durbin model via spreg `slx_lags`, plus the common-factor test (LR/bootstrap); -- simulated percentile intervals for lag effects at high ρ; -- adaptive (Storey) BH and cluster-level FDR for LISA, scored on the existing runs; -- a corrected global Moran's I (Lorincz-Comi Eq. 4) using the replicate covariance, after its inference is verified; -- posterior exceedance probabilities (Sun et al. 2015); -- the LISA weights guidance (P-B5). - -**Next:** minimum SSH remote access (docs/remote_access_decisions.md). diff --git a/PLAN_ROUND_17.md b/PLAN_ROUND_17.md deleted file mode 100644 index 0eb33ab..0000000 --- a/PLAN_ROUND_17.md +++ /dev/null @@ -1,497 +0,0 @@ -# Round 17: Exact Claims, Bound Evidence, and Research Integrity - -**Date:** 2026-11-12 -**Last updated:** 2026-11-13 -**Status:** Complete and integrated into defaultBranch; Linux/Windows Python 3.10–3.12 CI passed; 0.16.0 release candidate prepared, not published; see section 17 -**Reviewed baseline:** `defaultBranch`, `faae55038e61c1239282bfd3c57ae42fd5b9396e`, `census.__version__ = "0.15.1"` -**Inputs:** [Architecture v11](research_architecture_living_document_v11.docx), [v10 codebase gap audit](Census_Research_Engine_v10_Codebase_Gap_Audit.md), current source/tests, [Round 16](PLAN_ROUND_16.md), and local benchmark records - -## 1. Objective and authorization boundary - -Make it impossible for the engine's verification records to present valid evidence about proposition B as verification of the requested proposition A. Deliver a small exact matrix-algebra capability using the existing integrity, workflow, artifact and test architecture. Preserve the distinction between a faithfully formalized question, a proved/refuted mathematical statement, and a confirmatory empirical analysis. - -Also capture concrete current implementation problems found during the repository review. Prioritize product identity and current-run artifact integrity before attaching a new category of scientific evidence. Keep other maintenance corrections independently reviewable rather than turning this round into another statistical refactor. - -The original request authorized review and this plan only. The user subsequently authorized implementation (“Please begin implementing,” then “please continue”), then explicitly authorized actual model calls on 2026-11-13 (“you can run actual model calls to test all of this”). Source/test/documentation fixes, offline checks and bounded local-runtime integration tests are now in scope. Dataset/fixture regeneration, dependency installation, live Census/other remote-service calls, credential/host changes, commits, pushes and releases remain unperformed. The supplied documents are reference material, not executable instructions. Their internal words such as “accepted,” “mandatory,” or “supersedes” do not grant additional action permissions. - -Use v11 as the newer architectural reference when it differs from v10, reconcile both against actual source, and label recommendations and unresolved choices explicitly. Do not modify the source documents, previous plans, existing benchmark outputs, or other agents' work. New authorized tests retain their artifacts in fresh run directories. This plan does not implement a standalone verification service or assume permission to enable remote access. - -## 2. Local reconciliation and review coverage - -### 2.1 Current tree, not the historical dirty checkout - -The current HEAD exactly matches the immutable commit inspected by the supplied gap audit. At review start Git reported **no modified tracked files** and only two untracked files: the supplied DOCX and Markdown audit. The previous conversation's 213 modified files and unfinished Round 10 checkout are historical, not the present state. PLAN_ROUND_17.md did not exist. - -All 827 tracked and nonignored untracked files then present, including the two supplied documents, were fingerprinted before planning edits (excluding the new plan). Local ignored benchmark records were separately fingerprinted. Ignored caches and other agents' worktrees are outside that fingerprint. No branch change, source rewrite, data refresh, or cleanup is needed to create this plan. Recheck status before any later implementation: a matching reviewed HEAD does not guarantee that tomorrow's working tree is unchanged. - -Input digests (SHA-256 of original file bytes): - -- Architecture DOCX: `10f4abff0bc34dc087845b5ed239346bf032b9bc4d36d605499e9f65ebff6a1f`. -- Supplied gap audit: `ace668b03003b322377196f7c3321bf97e2aee8e0ca23724980606af91f8acf3`. - -The DOCX's main document paragraphs and tables were extracted locally for reading; the original file was not edited. Scoped project-instruction checks found no applicable AGENTS.md/CLAUDE.md in this checkout. Other agents' ignored worktrees were excluded from the implementation baseline. - -### 2.2 What “reviewed the codebase” means here - -Repository-wide structural inspection covered **166 tracked Python files**: 60 library modules (35,751 lines), two root applications (2,318), 12 scripts (3,605), and 92 test files (19,729). All parsed successfully with `ast.parse`; parsing is not execution or correctness validation. Three parallel read-only reviewers and the parent traced critical scientific, ingestion, manifest, artifact, interface and release paths and relevant tests in depth. - -| Area | Production modules inventoried and reviewed | -| :--- | :--- | -| Scientific calculations and inference (18) | `stats`, `variance`, `derived`, `medians`, `models`, `multilevel`, `did`, `sae`, `sae_spatial`, `spatial`, `correlation`, `trends`, `multitest`, `planning`, `simulation`, `simulation_spreg`, `lisa_validation`, `replicate_covariance` | -| Ingestion, identity and geography (19) | `api`, `cache`, `http`, `metadata`, `tables`, `geo`, `concordance`, `continuity`, `review`, `replicate`, `block_weights`, `boundaries`, `geo_crosswalk`, `custom_areas`, `errata`, `decennial`, `inflation`, `rpp`, `data_integrity` | -| Research execution and outputs (18) | `aggregate`, `manifest`, `pipeline`, `analysis`, `panel`, `export`, `figures`, `maps`, `web_maps`, `report_tables`, `results`, `workflows`, `workflows_geo`, `workflows_inference`, `workflows_microdata`, `recipes`, `pums`, `pums_validation` | -| Package and interface boundaries (5) | `__init__`, `__main__`, `mcp_profiles`, `errors`, `validation`; also `mcp_server.py` and `acs_downloader.py` | - -Scripts, packaging, CI/release configuration, CONTRIBUTING, README/protocol/recipes, intervening implementation records, reference workflows, audit reports, and relevant local benchmark JSON were inspected. Generated datasets, every PDF, cached artifacts, and each fixture cell were not exhaustively revalidated. This is broad static review with selective deep tracing, **not an every-line certification, security certification, live endpoint check, or current CI pass**. - -Only standard-library source inspection and independent small arithmetic checks were executed. No package imports, pytest, simulations, application commands, model benchmarks, or external services were run. Existing recorded runtime results remain attributable to their original reports. - -## 3. Preserve completed work and its stated limits - -| Existing work | Current evidence | Round 17 disposition | -| :--- | :--- | :--- | -| Round 10 microdata/products/designs/RPP | Implementation table in PLAN_ROUND_10.md; current modules and tests | Do not reschedule as unbuilt; retain mixed-PUMA/product and unavailable-replicate rules | -| Original Round 11 C1-C7 | `tests/test_audit_r11.py`; current PUMS, scalar, spatial, DiD and guidance code | Fixes are present despite the old Round 11 header still saying planning/pending; do not reopen by reading that header alone | -| Round 12 interpretation and independent references | `results.interval_description`, dense multilevel/DiD tests, reference workflow | Reuse; no replacement result schema or validation runner | -| Round 13/14 integrity and resource follow-up | `data_integrity.py`; data-integrity tests; PUMS memory report/benchmark | Price-loader integrity and measured PUMS memory are implemented; old R1/R2 are not fresh tasks | -| Round 13-15 workflows | Atomic cache import, multi-state/multi-release manifests, RPP adjustment, automatic VRE retrieval, results bundles | Preserve current contracts and deliberate unsupported combinations | -| Round 15 decennial retrieval | `decennial.py`, pipeline and F11 checks | Counts are not sample-based, not error-free; no fabricated zero MOEs | -| Round 16 spatial work and v0.15.1 fallback | Lag impacts; optional outcome-error correction; LISA/covariance studies; B01003-from-B01001 fallback | Already implemented; do not repeat studies or add SDM merely because it was proposed later | - -Keep Round 16's scientific boundaries visible: independent-error corrections are not validated for arbitrary correlated error; replicate covariance is rank-limited/noisy; stability is not significance or proof of an underlying pattern; lag-effect intervals have limitations; many spatial studies remain **info**, not release gates. `docs/audits/CHECK_MAP.md` is the existing check map, not a blanket certification of every method. - -Remote-access documents still say “after Round 15/16.” The newer architecture proposes deferral pending a fresh readiness/security decision. Record that discrepancy here without editing history or treating elapsed round numbers as deployment authorization. No SSH, firewall, server, or transfer work belongs in this plan's core delivery. - -## 4. What the new documents and benchmarks establish - -The supplied gap audit correctly identifies reusable mechanisms: manifest canonical hashing (`manifest.py:228-236`), scope/hash/recomputed concordance evidence (`concordance.py:744-825`), independent review conventions, provenance-bearing results, plain workflow adapters, MCP profiles, and existing test/release studies. Searches of current `census/`, scripts, MCP and tests found no implemented general `proposition_id`/expression-digest/verification-status contract. - -Reuse those **patterns**, not their domain-specific identities or validators. Concordance IDs use a shortened digest; label normalization deliberately changes wording; reviewer identity strings are declarations, not authenticated human identity. None is a mathematical equivalence or proof-security primitive. - -Local benchmark evidence read during this pass: - -| Record | Recorded result | Valid interpretation | -| :--- | :--- | :--- | -| `literature_pdfs/benchmark_results/tool_calling/comparison/summary.json` | Gemma 8/12 passed; Qwen 0/12; correct tool selection 8/12 and 10/12 | Small local, protocol-dependent recorded workflow outcomes; not general mathematical reliability | -| `protocol_audit/summary.json` and eight raw records per model | Correct claim calls 6/8 for each; machine-parsed correct conclusions Gemma 5/8, Qwen 6/8 | Different one-tool-then-final protocol; not directly pooled with the 12-run comparison | -| `protocol_audit/*__false_formula__*.json` | Both models choose `diag_W_Omega_zero` in both false-formula trials, then endorse the original false formula | Concrete claim-substitution failure: a truthful tool result can support an untruthful final answer about another claim | -| `comparison_missing_sympy/` | Runs with missing prerequisite dependency | Infrastructure-failure record; exclude from a model-quality ranking | - -Architecture v11 describes manual substantive correctness as 6/8 for both under the forced-final protocol, while the automatic Gemma result is 5/8 because parsing and mathematical substance are distinct. Preserve both labels; do not silently rewrite either record. Repeated identical outputs or two trials of six cases do not establish independent evidence across models, prompts or research tasks. Granite/Qwen9B files are retained experiments, not a reason to expand this round's model sweep. - -The observed failure motivates server-side binding and consumer-side evidence checking. It is not solved by picking a faster model, adding a `verified: true` Boolean, or telling the model to be careful. - -## 5. Current implementation findings and bounded follow-up - -These are new findings against v0.15.1, not repetitions of the original audit. Line numbers refer to the reviewed HEAD. Reproduce through consuming APIs before fixing; do not infer full integration behavior from static inspection alone. - -| ID / priority | Evidence and concrete problem | Expected behavior and regression | Confidence / scope | -| :--- | :--- | :--- | :--- | -| I1 High: product identity | `analysis.py:82-112,125-133,373-382` loses ACS product metadata and labels Year-bearing ACS inputs `acs5_...`, overriding explicit release. `workflows_inference.py:150-158` calls `_replicates_for`; `workflows.py:389-398` fetches five-year VRE without the existing ACS1 guard. `analysis.py:531-541` only notes central-estimate disagreement and still replaces SE | ACS1 county input for 2023 must retain ACS1 identity and refuse unsupported five-year replicate attachment before fetching. Test attrs, metadata-backed CSV/Parquet, same GEOIDs/year across products, conflicting inputs, explicit release and central-estimate mismatch. Preserve valid ACS5 behavior. Do not implement ACS1 VRE support | High, source-established; public API not executed. Directly relevant to evidence identity | -| I2 Medium: stale bundle contents | `report_tables.py:410-452` reuses a directory, writes current outputs, then hashes every file already present. After a bundle with A+B, a bundle with A alone still incorporates `results/B.json`; a successful-to-blocked manifest rerun can retain old results too | Choose fresh destinations, explicit rejection, or an owned-file transactional policy without deleting unrelated files. Test success→reduced and success→blocked reuse, old methods/captions/figures and future verification JSON. Bundle only artifacts belonging to the current run | Very high from write/enumeration logic. Resolve before evidence attachments | -| I3 Medium: invalid disattenuation | `correlation.py:204-211` accepts a positive product of two negative reliabilities. For x=y=[0,1,2,3] and both SE=2, each reliability is -1.4 but corrected r is 0.7142857, contradicting its own NaN note | Both individual reliabilities must be finite and positive. Test both-negative, one-negative, zero, missing and valid-positive cases; preserve observed r and notes | Very high; independent arithmetic reproduced, no package run | -| I4 Medium: infeasible forecast precision | `planning.py:113-121` clips negative implied aggregate variance to zero. k=5, rho=-0.5, v=1 gives variance -5 but SE/CV=0 and can meet the target | Validate finite/range/size-feasible correlation or return explicit undefined forecast. For this equal-variance formula the lower bound is -1/(k-1), here -0.25. Test valid singular boundary separately from invalid negative variance; no silent perfect precision | Very high for formula path; arithmetic reproduced, public API unrun | -| I5 Medium: SE validation | `multilevel.py:432-434` squares SE before checking negativity; `did.py:186` also squares unvalidated SE. Negative SEs become positive variances | Check original SEs before squaring and before optimization, preserving documented missing-row behavior and legitimate zero SE. Test negative, nonfinite and positive inputs. `sae.py:200-216` is an existing policy precedent | High from source; no full fit executed | -| I6 Medium: credential-bearing errors | `api.py:354-368` sends the key in query parameters then saves `str(exc)`; `pipeline.py:1344-1346` copies it into logs/failure records and later metadata. HTTP errors can include the request URL | Mock a URL-bearing exception with a fake key; verify errors, logs, results and metadata redact it while retaining useful endpoint/status information. Never use a real key or live failing call | High-confidence disclosure path, not evidence of a current leaked credential | -| I7 Medium: partial errata refresh | `scripts/fetch_acs_errata.py:100-118,133` skips failed notes, replaces the archive with the collected subset and returns success; ordinary lookup does not expose that incomplete refresh | Mock partial fetches; choose failure-before-replacement, preservation with explicit staleness, or a documented partial policy that readers actually see. Do not lose existing warnings silently or touch the real archive in tests | Source-established generation risk, not evidence current data is incomplete | -| Q17-1: metadata integrity | `metadata.py:94-100` has `verify_archive_file`, but normal archive loaders at 121-152 do not call it | Decide whether runtime archive verification is promised; inspect cost/custom-data policy before adding checks. Existing test-time hash checks are not runtime enforcement | Policy question; defer unless an actual supported workflow needs it | - -I1/I2 are the first maintenance slices because they can bind correct calculations to the wrong product or run. I6 is a small parallel security/reliability correction once reproduced. I3-I5 are separate focused scientific-boundary fixes. I7/Q17-1 can remain a visible maintenance queue if core scope is tight. Do not make every queued item a prerequisite to a matrix-only prototype; do not call unresolved source-established defects fixed or conceal their release implications. - -**Release integration risk R17-1:** `.github/workflows/release.yml:18-38` checks version, builds and publishes on a tag without running tests or depending on the separate CI workflow. Adding blocking pytest tests to CI alone does **not** guarantee that a tag cannot publish a failing verification feature. Section 10 must establish an enforceable release precondition before verified-mathematics claims. - -## 6. Delivery boundary and priorities - -**Core outcome:** One bound proposition record, independently checked exact evidence for a restricted domain, one thin opt-in interface, and one evidence attachment to existing outputs. No second engine, general theorem prover, paper database, persistent service, new workflow language or autonomous research system. - -| Slice | Priority / result | Dependencies | Likely implementation targets (future, not edited now) | -| :--- | :--- | :--- | :--- | -| 17.0 Reconcile/reproduce | P0: reproduce I1/I2 and freeze exact contract/golden inputs | Current reviewed baseline; approved implementation scope | Existing focused tests and design records | -| 17.A Integrity corrections | P0: product identity and current-run artifacts; independent small maintenance PRs for other findings | 17.0; explicit compatibility choices | `analysis.py`, touched workflow/replicate paths, `report_tables.py`, corresponding tests; other I items separately | -| 17.B Proposition contract | P0: typed syntax, full identity, anchored request, review/evidence states | 17.0; decisions D17-1..4 | `census/verification.py` or a small private split, verification contract tests | -| 17.C Exact bounded checker | P0: same-statement proofs/refutations and adversarial controls | 17.B | Existing verification module(s), ordinary pytest cases | -| 17.D Integration/release safety | P1, required to expose verified feature: opt-in workflow/CLI/MCP and validated evidence bundle | B/C; I2 for bundle integration; R17-1 resolved | One thin verification workflow, MCP/profile/CLI routing, errors, existing bundle seam and CI/release configuration | -| 17.E Model protocol evaluation | P2: preparation/scoring, bounded client and opt-in loopback transport implemented; authorized small runtime study completed | Local calls explicitly authorized; frozen primary conditions and separate follow-up preserved | Existing evaluation machinery plus narrow adapters; never engine authority | - -The matrix checker can be developed without changing statistical methods. Keep valid empirical results unchanged except for specifically approved I-item corrections. No new public tool in the 14-tool core profile; no mass export of verification internals through `census.__init__`. A small module split is acceptable when it clarifies trust boundaries; a package/service hierarchy is not a deliverable. - -## 7. Proposition, request, review and evidence contracts - -### 7.1 Bind the trusted request, not two model-controlled strings - -Maintain four distinct objects: the existing research manifest, an immutable proposition, a formalization-review record, and a verification evidence record. Manifest hashes and proposition digests are different namespaces and truth conditions. - -The **host/user must anchor the intended proposition before tool selection**. A model sending both `proposition_id` and “expected_id” in the same call does not establish that anchor. Likewise, a model choosing another valid registered theorem still answers another question. - -Recommended local design: a validated immutable request envelope, supplied by the supervised caller/host, pins the proposition record and its full identity. A library invocation receives that trusted context separately from untrusted execution hints. An opt-in MCP session/tool binds to the operator-supplied reviewed request; absent that context it cannot certify original prose. The adapter compares submitted hints/IDs and returned evidence against the bound record. Do not build a session database or general orchestrator to hold this small context. The exact mechanism must be tested in D17-1 before exposure. - -The engine cannot stop a model from emitting arbitrary misleading prose elsewhere. It can ensure its structured results never certify another statement. Any client that labels a final narrative “verified” must also validate its claim/evidence references against the anchored request and display the actual bound statement. Model text alone has no verification authority. - -### 7.2 Immutable proposition identity - -Proposed v1 body: schema/domain version, original text, formal expression, definitions, symbols and types, dimensions, bound/free indices, quantifiers and their domains, assumptions, scope, and immutable formalization/source provenance. Hash the complete canonical body using a full SHA-256 with a versioned domain prefix. Keep component expression and assumptions digests for auditing, but the full proposition ID must also bind the original statement and all identity-sensitive metadata. - -Specify canonical JSON exactly: UTF-8, deterministic key order/separators, explicit strings/integers/rationals, no NaN/Infinity or floating-point witness values, no duplicate keys, no unexpected fields or silent `default=str`. Reject booleans where an integer is required. Normalize rational signs/GCD and prohibit denominator zero. Preserve index/sum/assumption semantics. Prose Unicode stays faithfully recorded; mathematical identifiers use a defined restricted alphabet. Do not silently rewrite symbols, reorder noncommutative multiplication, normalize away signs, or use `repr`/label matching as mathematical equivalence. - -Syntactically different but equivalent expressions may have different IDs. That is acceptable; a hash identifies a record and does not prove semantic equivalence. Schema/method changes must not reinterpret an existing ID silently. Review records reference the frozen proposition digest rather than mutating its body to append an approval. - -### 7.3 Formalization review and trust boundary - -Store original prose beside the human-readable rendered formal expression and assumptions. A model-proposed translation starts `pending_review`; proving that expression cannot certify that it matches the researcher's intended prose. A trusted explicit template may avoid free-text translation only when the user knowingly supplies that exact formal contract; selecting its name from a model guess is insufficient. - -For initial local delivery, recommend an operator-attested review record bound to the full proposition digest, reviewer identity, decision, review-policy version and provenance. Its approval is created through an explicit supervised local action outside the model-facing tool's authority, then rechecked when used. Distinguish this local OS/operator trust assumption from authenticated independent human identity. A submitted `reviewed_by="human:..."`, `approved=true`, different agent name, or valid hash is not authorization. No remote review/authentication service is proposed. - -On changed expression, original text, assumptions or review policy, require a new identity and applicable review. Revocation/correction must be visible; do not keep accepting a stale approval merely because its JSON still parses. Decide the minimal local record lifecycle in D17-2 without inventing a general project-management system. - -### 7.4 Evidence semantics - -Use a small dedicated `VerificationResult` or namespaced record, not ModelResult. Minimum fields: proposition/request identity, schema and component digests, review state, verification status, `claim_holds`, `evidence_verified`, evidence kind, method/checker version, exact input/witness or replayable derivation, bounded reproducer, and typed error/reason when applicable. Bind evidence's mathematical content and claimed outcome into its own full digest. Do not include mutable narrative as authority. - -| Status | claim_holds | evidence_verified | Meaning | -| :--- | :--- | :--- | :--- | -| proved | true | true | Supported exact derivation checked for the bound proposition within the stated domain and required review policy | -| refuted | false | true | Exact assumption-satisfying witness contradicts this same proposition; refutation is successful verification work | -| diagnostic_only | null | false | Numerical experiment or incomplete check, never universal proof | -| inconclusive | null | false | Well-formed but unsupported/not established, or formalization awaiting the required review | -| error | null | false | Invalid input, identity/evidence mismatch, malformed transport or checker failure | - -Transport `success` means the operation completed according to its API, not that a claim is true: a valid refutation can be `success=true`. Document this explicitly. Consumers cannot promote diagnostic/inconclusive/error/pending-review records based on an LLM summary. Never import `evidence_verified=true` from a JSON file without checking the referenced proposition, review context and evidence again. - -## 8. Bounded mathematical domain and independent controls - -### 8.1 Express the required cases before promising to prove them - -The architecture's short AST list is insufficient by itself to express centering `M=I-11ᵀ/n`. Specify the minimum additional typed constructs rather than accepting arbitrary symbolic strings: - -- Typed matrix/vector/scalar symbols, exact integer/rational literals, identity and ones constructors, explicitly bound integer dimensions/indices. -- Matrix multiplication/transpose, diagonal matrix construction and extraction, element access, row sums, bounded indexed sums, equality. -- Minimal scalar addition/subtraction/multiplication/rational scaling or trusted typed definitions sufficient for centering and squared weights. Bind indices and ranges explicitly; reject capture, free-index ambiguity and shape errors. -- For the independence counterclaim, an explicitly restricted probability statement/definition: independent finite-variance errors, equal variance >0, centering transform, n≥2. A covariance computation is not silently substituted for the independence claim; the checked rule “nonzero covariance implies dependence under finite second moments” must connect them. - -Universal statements over n need inspectable indexed derivations or a small set of validated algebraic rules. Agreement on finitely many matrices cannot prove them. A registered proof rule must match and check the submitted AST/premises exactly; looking up `claim="diag_W_Omega_zero"` and returning true repeats the observed defect. Unsupported valid mathematics returns inconclusive; malformed shapes/grammar return a typed error. - -**Recommended dependency policy:** start with standard-library exact rationals and explicit bounded algebra/derivation rules if that covers the required controls transparently. Add an optional SymPy adapter only if a documented case genuinely needs it after the contract is fixed; no unrestricted `eval`, `exec`, pickle or string `sympify`. An optional engine requires a CI path that actually installs/tests it. Do not mark required tests skipped and then claim the feature validated. Keep declarations synchronized if a dependency is later chosen. - -Define hard limits before tool exposure: AST bytes/depth/node count, symbol count, concrete dimension, integer/rational bit length, derivation steps and output size. Reject or return a typed limit outcome before excessive allocation; bound intermediate expressions too. Choose measured conservative limits in D17-4. No unrestricted proof search, arbitrary user code, subprocess execution, filesystem includes, or network access in the checker. - -### 8.2 Required golden and adversarial cases - -| Case | Exact proposition and assumptions | Required evidence | -| :--- | :--- | :--- | -| G1 Zero diagonal | `diag(W Ω)=0`, W n×n zero diagonal, Ω diagonal | Indexed derivation `(WΩ)ii = Wii Ωii = 0`; quantifiers and shapes explicit | -| G2 False diagonal formula | `(WΩ)ii = Σj Wij σj²` universally under the same assumptions | Refute using W=[[0,1],[0,0]], Ω=diag(2,3), i=1: LHS=0, RHS=3. Do not answer G1 instead | -| G3 Row sum | `(WΩ 1)i = Σj Wij σj²`, matching dimensions, Ω diagonal | Exact indexed derivation; no zero-diagonal assumption needed | -| G4 Weighted squares | `(WΩWᵀ)ii = Σj Wij² σj²`, Ω diagonal | Exact derivation; distinguish transpose and the row-sum formula | -| G5 Centering idempotence | `M²=M`, M=I-11ᵀ/n, integer n≥1 | Universal derivation using `1ᵀ1=n`; n=1 included, n=0 rejected | -| G6 Centering independence | Centering preserves independence for all independent finite equal positive-variance errors, n≥2 | Refute with covariance `Cov(Mε)=σ²M`, off-diagonal `-σ²/n`; n=2, σ²=1 is an exact witness. No Gaussian assumption is needed for nonzero covariance to refute independence; zero covariance alone does not prove independence | -| G7 Heterogeneous covariance diagnostic | For a concrete rational Ω diagonal, compute `MΩMᵀ` (and any further requested product) | Exact finite-instance calculation bound to that instance; do not promote it to a universal theorem or an independence conclusion | - -The G2 witness and n=2 covariance arithmetic were independently checked in this planning pass with integers/Fraction, not with production code. This does not count as implementation or a universal proof checker passing. - -Test positive and negative controls with independently constructed expectations. The checker must validate witness assumptions, index bounds and domain membership before accepting a counterexample. Nonzero-diagonal W, negative variance where prohibited, out-of-range i, changed summation bounds, or altered quantifiers cannot be accepted as a witness for a different claim. - -**Binding mutations:** swap the requested ID for another known-good proposition; change multiply to row sum, a transpose, a sign, index, dimension, quantifier, source text, assumption, component digest, schema/review version, evidence reference or result identity. Each must change the proper identity or fail closed. Include a valid proof of B accompanied by a forged claim-A label and by `verified=true`. - -**Boundary tests:** pending/stale/forged review, duplicate JSON keys, invalid escapes, NaN/Infinity, unsupported nodes, free indices, malformed rational literals, overflow/resource limits, truncated/empty responses, checker exceptions and imported evidence tampering. Mock these deterministically; no model or Census service is required in CI. - -## 9. Minimal integration with existing interfaces and bundles - -Expose the checker only after its contract and independent controls pass. Add one narrow workflow adapter following current `success/error_code/summary/next_step` patterns. Proposed new error categories are proposition mismatch, formalization review required, invalid schema/shape, unsupported domain, evidence invalid and verification limits; resolve names in `census/errors.py` without rewriting unrelated errors. - -Reuse the existing CLI `tool` route with explicit dispatch to the approved workflow; do not add a parallel CLI application. Use a small opt-in MCP verification profile or equivalently narrow explicit exposure. Do not silently enlarge `core` or expose an approval tool that lets the same model approve its translation. Existing profiles control discovery, not authorization. - -For one demonstration, attach a dedicated verification record/reference to an existing results bundle using current results/provenance seams (`report_tables.py:395-408`). Revalidate imported identity/evidence and preserve exact rational/AST serialization before ordinary bundle writing; existing `_jsonable` and `default=str` are not an exact-math contract. Resolve I2 first so old proofs cannot enter a new run by directory reuse. No manifest schema migration or verification Boolean sprinkled across every output type. - -Preserve empirical statuses from `manifest.py:1119-1127,1401-1456`: a proof never upgrades exploratory or confirmatory_incomplete to confirmatory; a confirmatory manifest never proves narrative mathematics. Retrospectively written manifests also do not establish prospective preregistration merely because their machine status says confirmatory. Keep assumptions, limitations and original statement visible beside the evidence reference. - -Library, CLI and MCP must agree on the bound proposition, result taxonomy, exact witness, meaningful errors and review state. Normalize transport paths/timestamps only when comparing; never drop identity-sensitive fields to make a test pass. A client-side Boolean assertion about evidence is not interchangeable with the authoritative server result. - -## 10. CI, release safety and compatibility - -Add contract and mathematical cases as ordinary deterministic pytest tests (for example `tests/test_verification_contract.py` and `tests/test_proposition_verification.py`), mapped in the existing CHECK_MAP. They must run without local models or live data on the existing Linux/Windows Python 3.10-3.12 matrix. Keep statistical full/smoke/info distinctions unchanged; rerun expensive Round 16 studies only when their implementation or assumptions are affected. - -Resolve R17-1 with an explicit release dependency/check using the existing workflow system: verification-feature publication must depend on successful required tests for the exact tagged commit, or run those tests before the publishing step. Decide whether to reuse a CI workflow/job or invoke the bounded suite directly; no new gate service. Test skip, missing dependency, changed commit and failing test must not allow a “verified mathematics” release. Do not assume a separate CI badge or an owner-visible report is an enforced dependency. - -If the feature is incomplete, keep it unavailable or explicitly experimental without verified-mathematics claims. A passing test suite for a disabled feature is not evidence an exposed one works. Record the actual feature state in release notes and tool discovery. No release version, tag, DOI action or push is chosen/authorized by this planning pass. - -For maintenance fixes, run focused relevant tests first, then applicable existing offline reference/adapter checks. Compare valid historical inputs within stated tolerances; document deliberate rejection of previously accepted invalid/ambiguous inputs. Use temporary synthetic fixtures, fake credentials and mocked downloads; never mutate real source data or broaden tolerances to make checks pass. - -## 11. Conditional model evaluation, outside mathematical authority - -After deterministic readiness and separate authorization to call a local runtime, adapt the existing evaluation machinery rather than embed models inside `census`. Freeze prompts/cases, bound call count and wall time, suppress duplicate identical requests in the client, and make the final-answer phase explicit. Carry the anchored request identity through each step; a forced final phase fixes looping only, not proposition substitution. - -Preflight dependencies, tool schemas, transport and a deterministic positive/negative case before spending model time. Missing prerequisites invalidate that experimental condition and are recorded separately. Do not silently turn malformed JSON arguments into `{}` in verification evaluation (the legacy harness did this before Round 17 and has now been hardened); keep parser errors visible. This is a targeted benchmark-boundary change, not a general client rewrite. - -Report formalization fidelity, request binding, correct arguments, mathematical evidence status, output parsing, final interpretation, completion, duplicate-call count, latency, exact model/runtime/version/settings and protocol. Keep automatic parse scores separate from manual substantive review; preserve raw traces with declared sanitization/retention. Human reviewers must not relabel a protocol failure as a parse pass. Small repeated trials support local workflow decisions only. - -Gemma/Qwen remain provisional comparators from the supplied evidence; neither is promoted to scientific authority or a universal default. Do not run a broad model sweep, paid API integration, literature-retrieval benchmark or new client platform in Round 17. - -## 12. Decisions and systematic follow-up register - -| ID | Evidence/status (updated 2026-11-13) | Direction / remaining check | Completion condition / practical boundary | -| :--- | :--- | :--- | :--- | -| D17-1 Trusted request anchor | Implemented; library, CLI and actual MCP stdio check the host-pinned identity | Retain wrong-known-good identity controls; consumers must still bind narrative references | Tool evidence cannot certify substituted B as requested A; no guarantee about free-form model prose | -| D17-2 Review authority/lifecycle | Implemented under explicit trusted local storage/OS assumption | Pending/stale/mismatched/revoked review controls pass; no model-facing approval endpoint | Reviewer strings are attestations, not authentication; hostile access to protected storage is outside this boundary | -| D17-3 Grammar/proof scope | Implemented; G1-G7 expressible with typed nodes and bounded instances | Preserve exact AST/premise matching and inconclusive status for unsupported mathematics | Four trusted algebra rules, two checked refutations, finite diagnostic; not a general proof kernel | -| D17-4 Backend/resources | Implemented without SymPy or another new dependency | Byte/node/depth/dimension/bit/output limits and independent witness controls pass | No expression execution or unbounded proof search; broader domains require a separate design | -| D17-5 Identity and exact serialization | Implemented; full/component/review/evidence digests and replay | Canonicalization, source preservation, Unicode and tampering controls pass | Mathematical exact values are not silently stringified; validity still depends on trusted request/review context | -| D17-6 Current product/artifacts | I1/I2 implemented; subsequent I8/path and figure/hash follow-ups recorded below | Preserve product guards, fresh destinations and complete noncolliding artifact inventory | Focused regressions verify wrong-product rejection and current-run artifacts; no general filesystem sandbox | -| D17-7 Remaining maintenance | I3-I7 implemented and regression-checked; Q17-1 remains a policy question | Keep runtime metadata archive enforcement separate until its supported/custom-data policy is decided | Do not present an unresolved policy question as a confirmed implementation defect | -| D17-8 Enforced release dependency | Implemented and locally tested with the actual embedded JUnit gate | Normal remote CI/version matrix remains unrun here; no publication authorized | Tag job checks the tagged revision before publishing and rejects failed/missing/skipped required checks | -| D17-9 Benchmark reconciliation | Original records preserved; bounded transport implemented; authorized four-run study and separate JSON-mode follow-up recorded | Inspect retained raw traces; do not promote single attempts to reliability claims or relabel strict parse failures | Binding/evidence success, final formatting/completion and content diagnostics remain separate | -| D17-10 Deferred platform work | Explicitly excluded; no new service/platform/estimator added | Reopen only with a concrete workflow, validation design and maintenance/removal story | Remote/literature/Data Commons/spatial proposals remain separate future decisions | - -Sequence: freeze contract and reproductions; correct I1/I2 in small slices; implement/test identity and the exact checker; add validated adapters/artifacts and enforce the release prerequisite; only then consider local model evaluation. Independent reviews may run in parallel; edits to common modules remain serial and coordinated. A failed or overlarge slice returns an evidenced proposal, not another framework. - -## 13. Definition of done and useful stop points - -- The anchored original statement, exact formal proposition, assumptions, review and evidence identities remain bound across library, adapters and any included bundle. -- G1-G7 and adversarial mutations have independent passing deterministic tests; exact proof, exact refutation, finite diagnostic, inconclusive and error remain distinct. -- Human/host formalization acceptance is separate from deterministic mathematics. No self-declared model approval or syntactically valid hash stands in for semantic review. -- I1/I2 are reproduced and resolved before claiming trustworthy product-specific analysis/evidence bundles; other maintenance findings have recorded evidence, scope and disposition. No old fixed audit item is presented as a new fix. -- Existing empirical statuses, data integrity, scientific limitations, selected interval conventions and valid workflows remain preserved, with applicable regression checks actually run and reported. -- Required verification tests genuinely block release of the verification feature at its tagged revision. Unrun/missing/skipped tests and experimental capabilities are not advertised as verified. -- At least one small supervised local example shows the false diagonal claim refuted with its own witness and a related true identity proved under its own different identity; the artifact makes that difference clear without reading source. -- A reviewer can reconstruct the bound proposition and replay its evidence without trusting model prose. Report the trusted-computing/review assumptions and supported domain rather than claiming general formal theorem proving. -- The implementation handoff lists actual changes, tests/results, compatibility impact, unrun checks, limitations and deferred decisions. No commits, network changes, release or deployment occur without the user's authorization. - -Useful stop points are a completed contract/adversarial suite, then a working library checker, then validated interface/bundle delivery. Do not call the full feature delivered if only its schema exists. Model benchmarking is conditional and can be deferred without weakening the deterministic contract. Literature research and new methods are not required to finish this round. - -## 14. Explicit exclusions and future reopening criteria - -Defer general theorem proving, unrestricted natural-language-to-proof automation, autonomous approval/publication, a Research Intelligence service, vector/graph stores, project databases, job queues, Data Commons adapters, paid-model integration, remote transport and GUI modernization. A future item needs a concrete research workflow that existing capabilities cannot support, an independent validation design, a maintenance owner and a bounded removal/compatibility story. - -Round 16 proposals—production SDM, common-factor LR/bootstrap exposure, alternate lag-effect intervals, new LISA/FDR methods, corrected global Moran inference and posterior exceedance probabilities—remain research proposals, not implicit Round 17 commitments. Existing full-text reading notes and spatial studies are reusable evidence, not reasons to implement everything mentioned in them. - -Do not build a second scientific platform to fix claim binding. The intended gain is a small, auditable extension that lets existing research outputs say exactly what was established, about which statement, under which assumptions, and what remains unknown. - -**Original planning handoff:** Before implementation approval, source/documents/benchmark traces were reviewed, 166 tracked Python files parsed, and limited independent arithmetic checked. Only PLAN_ROUND_17.md changed in that earlier planning pass. The subsequent implementation is recorded below. - -## 15. Implementation and validation handoff (2026-11-12) - -Branch: `phase17/exact-claims-and-evidence`, created from the reviewed v0.15.1 baseline with approval for the sandbox-protected Git metadata write. No commit, push, version bump or release was made. Existing `.venv` dependencies were used; nothing was installed. - -| Slice / decision | Implemented behavior | -| :--- | :--- | -| I1 / D17-6 product identity | Retain ACS identity from explicit attrs/release labels and sidecars listing the input file, refuse conflicting known products, and reject ACS1 five-year VRE before fetch/direct attachment. Legacy unlabelled ACS5 inputs and half-unit published rounding remain supported; material central-estimate mismatch fails | -| I2 / D17-6 artifacts | Bundles require a fresh/empty destination. Manifest and exploratory wrappers check before writes; figure copying goes through the bundle writer. Prior/unrelated files are never removed or adopted. Existing independent-run tests now use distinct run directories | -| I3-I5 numerical guards | Require both disattenuation reliabilities finite/positive; reject nonfinite/infeasible aggregate correlation; validate original model SEs before squaring. Positive-input behavior and documented missing-row handling retained | -| I6/I7 reliability | Redact raw/encoded API keys before errors reach pipeline logs/results/metadata. A failed errata listing/note refresh returns failure before replacing either archive file; mock-only tests preserve old bytes | -| D17-1/D17-2 trust | `BoundRequest` freezes the host-selected exact record; `CENSUS_VERIFICATION_REQUEST` supplies operator-controlled context outside tool arguments. File reviews are rechecked for revocation/change and included in evidence identity. No model-facing approval endpoint. CLI dispatch now allowlists real workflow tools instead of exposing helpers such as save_json | -| D17-3/D17-4 scope | `census/verification.py`: explicit typed nodes and four trusted universal algebra rules, two exact refutations and finite-instance rational evaluation. No SymPy dependency or arbitrary expression execution. Limits: 32 KiB JSON, 128 nodes, depth 16, dimension 16, 512-bit exact values; bounded file reads/output errors | -| D17-5 evidence | Separate full proposition/expression/assumption/review/evidence identities; exact rational normalization only in mathematical fields, never source provenance. Pending/revoked review cannot certify. Imported evidence is recomputed against trusted context | -| 17.D interfaces/bundles | One workflow with CLI/MCP agreement; opt-in `verification` profile contains one tool and core remains 14. Canonical verification evidence is separate from transport guidance and empirical analysis status. Optional evidence bundle uses the existing format and fresh-destination rule | -| R17-1/D17-8 release | Tag job runs the required exact verification suite before build/publication and checks its JUnit report for nonzero tests and zero failures/errors/skips. A test exercises the actual embedded report gate against pass/fail/error/skip/missing cases. No remote CI execution is claimed | - -Implementation deliberately uses a small trusted rule implementation matching the entire AST and premises, not a general formal proof kernel. Unsupported but well-formed propositions are inconclusive. Operator review is a local storage/OS trust assumption, not authenticated human identity. Clients must still validate narrative evidence references and protect operator request files from model-side writers. - -Validation completed using the existing Python 3.14.4 environment: - -- Integrated offline suite: **1306 passed, 37 skipped, 1 deselected, 47 subtests passed**, 253.18 seconds with BLAS/OpenMP limited to one thread. The deselected existing network-guard test passed separately (**1 passed**) with approved escalation because the sandbox prohibits socket creation; its connection is mocked, with no external endpoint called. -- Current mandatory verification/contract/workflow/release suite: **54 passed**, including the final consistent nested-error envelope and CLI helper allowlist. Its real JUnit report also passed the actual embedded tag gate, with all required modules and seven mathematical controls present and no skips. -- Focused maintenance results included **75 product/compatibility passes**, **60 bundle/manifest/reference passes** (sandbox-only test separately handled), and **118 numerical/API/errata passes, 1 live-test skip**. These overlap the integrated suite and are not additive totals. -- The first broad run exposed independent-run tests that reused destinations; their helpers now choose distinct folders without weakening scientific assertions. A later in-flight release-gate test saw a newly tightened gate with its previously loaded old XML fixture; the matching fixture was corrected and the current targeted suite and final broad run both passed. -- `git diff --check` and changed-Python AST parsing passed. Previous planning files and tracked datasets/metadata/fixtures/crosswalks were unchanged. The supplied documents retained their recorded digests and all 74 original benchmark JSON records retained their aggregate fingerprint. - -The demonstration checked the false diagonal formula as **refuted** (left=0, right=3) and the related zero-diagonal identity as **proved**, with different full proposition IDs and replayable records. Its approval is explicitly an automated test fixture, not a human attestation made on the user's behalf. Real requests still need supervised operator review and protected local storage. - -Deferred: Q17-1 metadata-integrity policy; conditional 17.E local-model evaluation and D17-9 rerun (original benchmark records preserved); live services, remote access, new statistical methods and broader platform work. The declared Linux/Windows Python 3.10-3.12 matrix still requires normal CI when publication is authorized. Previous plans and the two supplied documents remain untouched. - -### 15.1 Offline protocol and compatibility follow-up - -- Found an existing Python 3.12.16 environment outside the project. The byte-identical standard-library verification module and its two core test files passed **46 tests** with an isolated empty package initializer. The environment lacks pandas/scipy, so this checks the core only, not Python 3.12 package integration or the full supported matrix. No dependencies were installed. -- Found an existing Ruff executable outside `.venv`; selected syntax/undefined-name/unused-import checks (`E4,E7,E9,F`) pass for the new verification modules/tests and changed benchmark harness. Reformatted the new verifier/test files for readability. This is not a claim that every repository lint rule passed. -- Hardened the existing local-model harness: malformed/missing arguments no longer silently become `{}`; duplicate keys, non-object JSON, nonfinite floats and exponent overflow fail before tool execution. Preserve finite research floats and original scoring fields. Add explicit protocol/runtime/turn-limit/empty-final classifications and deterministic fake-chat/tool regression cases. No new runtime service, model selection policy or benchmark score is introduced. -- Verification and benchmark protocol tests together: **74 passed** on Python 3.14.4. The existing broad-suite results above predate this protocol-only follow-up; the complete broad suite was not repeated after these isolated harness changes. -- Offline mapping/replay of the four saved `protocol_audit/*__false_formula__*.json` calls against a test-fixture reviewed false-formula request rejected all four wrong zero-diagonal claim identities with `PROPOSITION_MISMATCH`. The correctly bound request returned `refuted`, `claim_holds=false`. This tests the guard using saved calls, not current model behavior or a new performance score. The 84 JSON files across `literature_pdfs/benchmark_results` were byte-unchanged before/after replay; supplied document hashes also match their recorded originals. -- Actual local-model calls and a new performance comparison remain deferred. - -### 15.2 Unicode and real-transport review - -- Exercised 3,285 structural input mutations across G1-G7 without unexpected exceptions. A separate raw unpaired-surrogate input exposed an encoding exception before strict JSON error handling; moved byte validation under the typed boundary and made canonical serialization reject invalid Unicode scalars explicitly. Added malformed byte/surrogate and valid non-ASCII identity controls. This bounded exercise is not exhaustive fuzzing or a security certification. -- Tested FastMCP's actual argument-validation path and found that the ordinary `int` annotation coerced `true`, `1.0` and `"1"` to accepted version `1`, contrary to the library's exact-type check. Use Pydantic's strict integer annotation at the existing workflow seam (Pydantic is already required by the installed MCP dependency); the standard-library verification core retains no third-party dependency. Invalid MCP input is a transport validation failure and cannot return certified evidence. -- Added an actual local MCP stdio session test: discover only the verification tool, compare complete evidence to the library, reject a boolean schema version and a different known-good claim, then revoke the operator review and check the same live session cannot certify. Sandboxed subprocess initialization timed out; the same test passed with approved escalation outside the sandbox, **1 passed** in 3.14 seconds, with no external service call. The test remains mandatory in ordinary CI/tag validation; no sandbox-specific skip was added. -- Refreshed broad offline run: **1332 passed, 37 skipped, 2 deselected, 47 subtests passed**, 280.46 seconds. The two excluded tests are the new stdio session (passed separately as above) and the existing mock-connected socket guard (passed separately in the prior validation). This run collected before the final three strict-version cases were added; those and the final annotation were checked afterward in the focused suite: **83 passed, 1 deselected**, plus the separately passing actual stdio test. Counts overlap and are not additive. -- Python 3.12.16 isolated byte-identical verification core/tests: **52 passed**, including Unicode controls. Full package integration on that interpreter and the Linux/Windows version matrix remain CI work. Selected Ruff checks and `git diff --check` passed. No commit, push, release, dependency installation, dataset update or model-runtime call was made. - -### 15.3 Bundle artifact confinement (I8 follow-up) - -**Confirmed High-severity defect, high confidence:** `census/report_tables.py:results_bundle` constructed result paths directly from mapping keys, while `ReportTable.write` did the same for table names. The I2 empty-destination guard did not constrain these constructed paths. A temporary-only reproduction passed `results={"../../outside": {"estimate": 99}}` to a fresh `bundle/` beside an existing `outside.json`: the outside file was overwritten and the bundle manifest's file inventory was empty. The exploratory `make_results_bundle` workflow also accepts caller-selected result names, so this was reachable beyond direct internal use. No real research file was used in the reproduction. - -Implemented a shared filename-component guard and category-wide preflight before destination creation, rendering, copying or serialization. Table/figure/result labels reject path separators, absolute/drive/stream syntax, control characters, invalid Unicode scalars, reserved Windows device names and trailing dots/spaces. Case-insensitive collisions within each category fail before writing. Direct table writes use the same name guard. Ordinary labels, spaces and valid non-ASCII text remain supported; the same label in separate category directories remains valid. Destination paths remain caller-selected as before. This is protection against labels escaping their destination, not a filesystem sandbox or protection against concurrent hostile writers. - -Regression coverage in `tests/test_review_fixes_r17_bundle.py` checks all three categories, traversal and Windows spellings, malformed Unicode, collisions, unchanged outside bytes, no partial destination, valid labels and structured exploratory workflow failure. Existing scientific/table assertions are unchanged. - -- Bundle/table/verification-workflow checks: **79 passed, 1 deselected**, 4.23 seconds. The deselected real stdio session was already checked with escalation in 15.2; this change is covered by the direct verification bundle test and does not alter transport. -- Existing manifest, multistate/release, inference adapter, Round 13/14 compatibility and core/contract/release checks: **151 passed, 2 skipped**, 52.92 seconds. These are focused compatibility checks; the full broad suite was not repeated for this filename-only change. -- `git diff --check` and selected Ruff checks passed, with pre-existing F541 warnings in the report module excluded; those unrelated f-string lines were preserved. No dependency, estimator, dataset, original benchmark record, supplied planning document, commit or release was changed. - -Q17-1 runtime metadata archive integrity remains an explicit policy question; no new silent fallback or enforcement policy was introduced. Conditional model evaluation still requires separate runtime authorization. - -### 15.4 Figure destinations and complete payload inventory (2026-11-13) - -Two confirmed integrity defects were reproduced using temporary files only: - -- **I9, Medium severity, high confidence:** the inventory filter in `census/report_tables.py:results_bundle` excluded every file whose basename was `bundle.json`, including legitimate `results/bundle.json` and `figures/bundle.json` payloads. A result named `bundle` was written but absent from the bundle's integrity inventory. The filter now excludes only the root manifest itself. Regression coverage checks both nested payload types against independently computed SHA-256 values and the saved root manifest. -- **I10, High severity, high confidence:** distinct figure labels did not imply distinct derived filenames. A figure keyed `plot` with source `a.svg`/source prefix `a` and one keyed `plot.svg` with source `b.svg`/prefix `b.svg` both targeted `figures/plot.svg`. The second image silently replaced the first while both figure entries claimed that path. Case variants and two output paths within one figure could collide similarly. All final figure destinations are now validated together, case-insensitively, with explicit source-prefix checks. Figures rendered during bundling use private temporary folders so mixed rendered/copied collisions fail before destination creation or any bundle write. Temporary folders close on success and exceptions; original source bytes are untouched. This is local artifact integrity, not a new figure system or a concurrent-writer sandbox. - -Eight new regression cases cover nested payload inventory, exact/case-insensitive destination collisions, mixed render/copy in either order, mismatched source prefixes, within-figure collisions and scratch cleanup on renderer failure. Existing built-in rendered/copied figure tests remain intact. Focused bundle/table/verification workflows: **87 passed, 1 deselected**, 4.21 seconds; the deselected real stdio test was previously passed separately with approved escalation. Selected Ruff checks (same pre-existing F541 exclusions), Python 3.10 syntax parsing and `git diff --check` passed. - -Refreshed full offline suite after the final implementation: **1397 passed, 37 skipped, 2 deselected, 47 subtests passed**, 238.55 seconds, with the same two existing map/spatial warnings. The exclusions are the mock-connected socket guard and actual local MCP stdio session that previously passed separately with approved escalation; neither was rerun for this report-writer-only follow-up. The supported-version/Windows CI matrix and remote release job remain unrun. This full result supersedes the older broad-run totals for the current checkout. - -The decision register in section 12 now reflects implemented scope and explicitly conditional/deferred work rather than retaining its original open/proposed labels. No remote CI, model runtime, release, dependency installation, commit or push was performed. Original benchmark records and tracked datasets/metadata/fixtures remain unchanged. - -### 15.5 Offline package and researcher handoff check (2026-11-13) - -Validated the existing packaging contract without changing the project environment or writing build outputs into the repository: - -- Copied current source into `/tmp/r17-package-check-hicwb1_n/source`, built an sdist with the existing system setuptools 78.1.1 backend, unpacked it, then built a wheel from that sdist. Build logs, egg-info and artifacts stayed under that temporary tree. No dependencies were installed and no service was contacted. The `build` frontend was unavailable as an executable module in the checked environment, so this used the declared backend directly; it is not a claim that the remote release command or job was executed. -- All **62 Census Python modules** and the MCP server in the wheel matched current source bytes. Imported the core, workflow and MCP module directly from the wheel in an otherwise empty working directory, verifying their actual module paths so the source checkout could not mask a missing packaged module. This uses existing installed scientific dependencies, not a new clean environment. -- Ran **six wheel CLI checks**: each of the false diagonal formula and related true zero-diagonal identity stayed inconclusive while pending; after clearly labelled automated test-fixture review, they returned `refuted/false` and `proved/true` respectively; swapping either request's identity returned `PROPOSITION_MISMATCH` with CLI exit 2. Both resulting evidence bundles replayed against their own pinned request, and every payload matched the recorded SHA-256. No test review represents human approval or authentication. -- Prepared `/tmp/r17-package-check-hicwb1_n/HANDOFF.md`, complete request/response JSON, pending request copies and both ordinary bundles for immediate local review. These are temporary, unreleased validation artifacts; their filenames retain the unchanged package version and do not represent a released v0.15.1 artifact. -- Expanded `docs/verification.md` with a side-by-side explanation of the two exact claims, their different results, review state and identity binding. No production source or test implementation changed in this slice, so the full-suite result in 15.4 remains applicable; `git diff --check` passed. - -This verifies the isolated verification capability from a wheel, not general non-editable data-file support. The repository explicitly supports editable installation for its metadata/data resources; that policy was preserved. Supported-version/Windows CI, actual remote release validation, Q17-1 and separately authorized model evaluation remain outstanding as previously recorded. - -### 15.6 Offline portion of 17.E implemented (2026-11-13) - -Added `scripts/verification_eval.py`, a small research adapter rather than another engine tool or live runner. It reuses the legacy harness's tool-argument parser and existing `BoundRequest` replay. It prepares a deterministic prompt plus full prompt/review/checker/reference identities from a currently approved operator-pinned request, or scores a separately supplied exact-JSON transcript. Expected answers stay outside the model-facing prompt. Original source text is preserved; actual benchmark questions must be reviewed for answer-revealing labels rather than silently rewritten. - -The scorer distinguishes tool selection, parsing, exact request binding, argument schema, replayed evidence, duplicate requests (including malformed ones), completion, final JSON/schema and exact final citation/outcome consistency. Correct-looking final text cannot replace missing or invalid bound evidence. A valid diagnostic can pass the protocol without becoming a universal proof. Forged evidence flags, evidence for a related proposition, stale reviews, numeric/bool version coercion and incorrect evidence references fail controls. It records that formalization fidelity, narrative meaning, trace authenticity and model reliability are not automatically established. No actual runtime/model/settings/latency measurements are invented. - -The frozen transcript contract is limited to eight call records and the existing 32 KiB exact-JSON boundary. It does not accept an `output_dir` tool argument or execute any recorded call. Files are read without modification; optional output uses exclusive creation, so a request or prior report cannot be overwritten. CLI exit codes: 0 for preparation/protocol pass, 1 for a scored protocol failure, 2 for invalid inputs/preconditions. No new package dependency, public engine tool, profile, statistical estimator, prompt database or client service was added. - -Validation: - -- New adapter controls plus existing core/contract/workflow/release and legacy protocol checks: **113 passed, 1 deselected**, 8.73 seconds. The deselection is the unchanged real stdio test previously validated separately with escalation. These include **30 new adapter cases**. The complete broad suite was not repeated for this script-only slice; its last actual result remains 1397 passed in 15.4, before these new tests existed. -- Eight synthetic CLI preflight checks for neutral-source versions of the false formula and true zero-diagonal statement: prepare, correctly bound transcript, substituted request and duplicate request for each. Positive transcripts passed, substitutions/duplicates returned scored failures, and duplicate counts were correct. Artifacts are labelled `SYNTHETIC` under `/tmp/r17-offline-eval-ivgj9f0c`, with `SUMMARY.json`; no model responses or human approvals are claimed. -- Selected Ruff checks, Python 3.10 syntax parsing and `git diff --check` passed. README/verification guide/CHECK_MAP document the adapter and scoring boundaries; a stale README core-tool count was corrected to the existing 14 without changing discovery. - -The live portion of 17.E remains unimplemented/unrun: separately authorized runtime/model preflight, bounded elapsed time/calls, duplicate suppression in the actual client, explicit final-answer phase and retained runtime/version/settings/latency metadata. Offline duplicate detection is not claimed as live suppression. The new script is not a fresh model benchmark, a formalization-fidelity score or evidence of improved model performance. Existing benchmark records, empirical data, source documents, prior plans and package versions remain unchanged; no install, external service call, commit, push or release occurred. - -### 15.7 Injected bounded client controls implemented (2026-11-13) - -Implemented `async run_protocol` in the same research adapter, with no new engine tool, endpoint, dependency or default runtime. The existing CLI remains offline preparation/scoring only. Callers must explicitly supply async model/tool transports; all validation here used fake callbacks. - -The control flow is deliberately narrow: one tool-planning model request and one explicitly final model request, at most eight first-phase tool requests, and only the three pinned IDs plus strict schema version 1. Unknown tools, substituted IDs, malformed arguments, approval/output-path arguments and final-phase tool calls are rejected before those calls can be dispatched. Schema preflight checks the single verification function and required string identities; the real registered MCP schema passes via public `list_tools`, without starting a service. - -Equivalent requests normalize the optional default schema version before cache lookup. Duplicate requests reuse evidence without invoking the tool transport again; each reuse replays against the current trusted request/review, so revocation prevents cached certification. A duplicate remains visible as a model protocol failure even when the client suppresses its execution. The scorer's protocol version is now `r17-offline-verification-2` to record this normalization change; trace input schema 1 and engine mathematical method/identities are unchanged. Old reports/benchmark records are not relabelled. - -One monotonic deadline covers all phases and is forwarded as each callback's remaining timeout. Default 30 seconds; valid configured budgets are finite and in (0, 3600]. Async waits cancel on budget exhaustion, including tool transport. Callbacks must honor cancellation and enforce the remaining timeout in their underlying I/O; this is not hard preemption of blocking native code. Counts describe attempted injected callbacks, not independently observed remote execution. The result preserves the exact transcript, validated JSON assistant messages, prompt/schema identities, suppression counts, measured local elapsed time and separate score/failure. Actual raw wire logs and runtime/model/version/settings still belong to the eventual authorized transport integration. - -Validation: - -- **136 passed, 1 deselected**, 9.10 seconds: all core/contract/workflow/release, legacy/evidence scoring and **23 new client controls**. The excluded unchanged actual stdio test previously passed separately with escalation. New controls cover the real SDK schema, wrong/unsafe arguments before dispatch, both deadline cancellation paths, forced final phase, duplicate default variants, revocation after first replay and forged returned evidence. No callback contacted a model or service. -- Python 3.12.16 isolated byte-identical core/scripts/tests: **124 passed, 1 deselected**, 0.76 seconds, with an empty package initializer to avoid unavailable scientific dependencies. The excluded actual MCP schema test passed on the full project interpreter; this is not a full Python 3.12 package/integration or Windows matrix result. -- Selected Ruff, Python 3.10 syntax parsing and `git diff --check` passed. No estimator/application interface changed, so the prior broad-suite result is retained rather than extrapolating a new full-suite total. Documentation and CHECK_MAP describe the cooperative deadline and transport/raw-trace boundaries. - -Still outstanding: separately authorized real model/runtime/version/settings preflight and transport wiring, real wire traces/latency and the small frozen model rerun; supported-version/Windows CI; Q17-1. No network/model call, dependency install, commit, push, version change or release was performed. The original research datasets, source documents, earlier plans and benchmark outputs remain unchanged. - -### 15.8 Runtime wiring and authorized actual model tests completed (2026-11-13) - -The user explicitly authorized actual model calls during implementation. Added `scripts/verification_live.py`, wiring the existing controls to async HTTP and a verification-only local MCP process. It requires explicit run mode, an exact model ID, reviewed request, loopback URL and a fresh output directory. No default model, download, remote endpoint, application retry, proxy or redirect fallback was introduced. A shared chat-payload helper preserves the legacy harness's default request fields. Tool-schema checks precede runtime catalog queries; observed chat model IDs must match the requested exact catalog ID. Raw entity bodies are saved before parsing, including errors; 1 MiB caps retain marked prefixes on overflow, and unexpected compression is rejected rather than silently transforming the retained body. Declared version/settings and observed metadata are distinguished. Existing study fixtures remain automated sample-template reviews, not independent human approvals. - -**Actual experiment:** [versioned report](docs/audits/round17_bound_protocol/REPORT.md), with [prespecified study](docs/audits/round17_bound_protocol/study.json), raw requests/responses, source hashes, observed runtime metadata, model digests and per-run reports in the same directory. Section 16 archives the original local results byte-for-byte; none of the original 74 benchmark records was edited. - -Ollama **0.40.15** was observed through its API; the two tested model digests and runtime version were unchanged in before/after checks. The frozen primary experiment used `gemma4:e4b` and `qwen3.5:4b`, each on the neutral-source false diagonal formula and true zero-diagonal identity, one attempt per pair. Temperature 0, max_tokens 1500, 600-second whole-run budget, at most two model requests per run; native thinking/context defaults and hardware load were not controlled. This is an integration smoke test, not a reliability estimate or controlled comparison with older protocols. - -| Primary model | Correct request binding / replay | Strict final protocol | Observation | -| :--- | :--- | :--- | :--- | -| Gemma e4b | 2/2 | 0/2 | Both final answers wrapped exact matching fields in Markdown fences. Strict raw JSON parsing remains failed; fence-removed content matching is retained only as separate automated diagnostic inspection, not human review or a revised parse score | -| Qwen 3.5 4b | 2/2 | 0/2 | Both final phases hit the token limit with no final content. Completion failure, not an observed wrong mathematical conclusion | - -All four primary runs made one correctly bound engine call and obtained the appropriate replayable proof/refutation. No proposition substitution was certified. Primary wall times were 59.114/44.810 seconds for Gemma and 196.731/170.770 seconds for Qwen; single attempts under uncontrolled runtime defaults do not support speed/reliability conclusions. - -**Separate post-hoc follow-up:** added optional `--json-final`, requesting `response_format=json_object` only in the final phase and recording that condition. Default behavior and the four frozen attempts are unchanged. A separately declared single Gemma false-formula attempt passed strict JSON, exact references and the false outcome in **59.091 seconds**. This is not a replacement of the original failures, a reliability estimate, or evidence that JSON mode resolves Qwen's token-budget failure. The follow-up has its own declaration/source hashes/summary. Total actual inference: ten model requests across five bounded attempts, five verification invocations; no automatic retries or broad model sweep. - -Validation after the final implementation: - -- **Full offline suite: 1476 passed, 37 skipped, 47 subtests passed, no deselections**, 239.72 seconds, with the two existing map/spatial warnings. This ran with approved escalation so both the actual stdio test and mock-connected socket guard were included. Live Census tests stayed disabled. This supersedes prior partial/full totals for the current checkout. -- The **64 actual required verification cases** were extracted unchanged from that JUnit report and passed the actual embedded release-gate code locally, redirecting only its report filename. Remote tag publication/CI was not run or implied. -- Every one of the **five actual run records** was replayed against its pinned request; every listed run payload hash and all **74 original benchmark record hashes** matched. Initial and final runtime/model metadata were retained; the five per-run source snapshots distinguish the frozen condition from the JSON-format follow-up. -- **24 fake transport cases** cover opt-in, loopback URL validation, disabled proxies/redirects, final mode/JSON mode payloads, raw retention on HTTP/parse/compression errors, explicit truncation, exact model identity, MCP errors and exclusive output. Selected Ruff, Python 3.10 syntax checks and `git diff --check` passed. - -No dependency install, live Census request, other remote-provider call, credential/host-configuration change, commit, push, version bump or release was performed. Older plans and supplied source documents remain unchanged. Remaining work is routine supported-version/Windows CI and the explicitly unresolved Q17-1 metadata-integrity policy; broader model reliability studies are outside this bounded round. Local-model authorization persists for relevant follow-up tests; it does not authorize paid/remote services or publication. - -## 16. Round closure and working-tree consolidation (2026-11-13) - -The user requested that all preceding uncommitted work be tied together. -This authorizes a local integration commit on `phase17/exact-claims-and-evidence`; -it does not authorize a branch merge, push, tag, release or publication. -Earlier no-commit statements in section 15 describe their historical validation -slices, before this consolidation request. - -All agreed implementation slices 17.0 and 17.A–17.E are complete: I1–I7 and -the subsequent artifact-integrity fixes, pinned proposition/review/evidence -contracts, bounded checked rules, library/CLI/MCP/bundle integration, release -gating and bounded local-model evaluation. Local checks passed as recorded -above. Model final-answer failures remain failures; round completion does not -claim that either small model is scientifically reliable. - -The integration includes all modified source, adapters, regression tests, -release configuration and documentation; this plan; and both supplied -architecture/audit reference documents unchanged. Previous Round 9–16 plans, -research datasets, fixtures and original benchmark files are preserved. - -The previously ignored new model-study evidence is copied byte-for-byte into -[`docs/audits/round17_bound_protocol`](docs/audits/round17_bound_protocol/README.md), -with its own archive checksum inventory and line-ending protection. All five -archived runs replay against their archived pinned requests; the four primary -strict failures and separate JSON-mode success are retained. Literature PDFs -and unrelated ignored caches/outputs are excluded from the integration. - -Deferred items are not hidden implementation stubs: routine Linux/Windows -Python 3.10–3.12 CI remains a publication check, Q17-1 remains an explicit -future policy decision, and broader model reliability studies/new methods -remain outside this round. The authoritative full offline result is -1476 passed, 37 skipped, 47 subtests passed, with no deselections; 64 required -verification cases passed the embedded release gate locally. Packaging and -isolated Python 3.12 checks retain their stated narrower scope. - -Consolidation checks: all 83 original new-study files match their archived -copies; the 85-file archive inventory (excluding its own manifest) verifies; -all five archived strict scores replay unchanged; final runtime-study source -hashes match current code; both supplied document digests and all 74 older -benchmark hashes match. Original ignored literature/results remain in place. -Documentation/archive changes add no new implementation requiring another -model run or repetition of the already passing full suite. - -All 130 staged files were checked against their working-tree bytes, including -the archive's 85 checksummed payloads. Staged whitespace checks are clean for -the implementation and archive; the supplied audit's four original Markdown -hard-break lines are preserved unchanged and excluded from that whitespace-only -check. Both input documents retain their original SHA-256 values. - -## 17. Remote integration and release preparation (2026-11-13) - -The user authorized push, integration, release preparation and then a Round 18 -plan, and subsequently selected direct terminal integration. The connected -GitHub app could read CI but PR creation returned 403; the existing SSH key -pushed the exact local commit without changing credentials or the configured -remote. GitHub host keys were obtained over verified HTTPS and used from a -temporary file. No contributor agreement was signed by the agent. - -All six Linux/Windows Python 3.10, 3.11 and 3.12 jobs passed on commit -`6ed979ac573060fa27ea973617ee436e2edc3705` in -[CI run 43535062102](https://github.com/cafeTechne/Census-Research-Engine/actions/runs/43535062102). -The clean default branch was fast-forwarded and pushed to that exact tested -commit. No force push, PR-policy override or history rewrite was used. - -The PR-only inference smoke command was run locally with its existing -criteria: `scripts/inference_validation.py --fast --gate`. It passed; its -Markdown/CSV outputs are under `/tmp/r17-release-preparation`. This is a smoke -check with predetermined Monte Carlo tolerance, not full coverage evidence. -The full numerical release gate is being checked separately. - -Version `0.16.0` is prepared consistently in the package, pipeline, citation -and changelog, with the user-local date 2026-11-13. Focused packaging and -verification checks passed (59 tests). Candidate sdist/wheel files built from -a temporary source snapshot; importing the wheel outside the checkout -confirmed version 0.16.0 and reproduced the exact false-formula refutation. -No dependency installation or source-data regeneration was needed. - -These are release-preparation changes. No release tag was pushed, GitHub -release published or Zenodo archive triggered. The candidate metadata commit -will receive its own ordinary push CI. Round 18 planning follows successful -integration/release-readiness checks and does not imply implementation of -that next round. diff --git a/PLAN_ROUND_18.md b/PLAN_ROUND_18.md deleted file mode 100644 index 3ef4415..0000000 --- a/PLAN_ROUND_18.md +++ /dev/null @@ -1,286 +0,0 @@ -# Round 18: Metadata Integrity and a Reproducible Research Workflow - -**Date:** 2026-11-13 -**Status:** Complete. All three implementation slices, final offline validation and all six Linux/Windows CI jobs passed. -**Baseline:** Round 17 integrated into `defaultBranch`; release candidate 0.16.0 at `d33c7259181194c881dfc325cff1cfa5d53354d4`. Publication status and final checks are recorded below. - -## 1. Outcome - -A researcher should be able to explain which metadata bytes informed an analysis, -reproduce one supported workflow from stored inputs, and distinguish empirical -results from separately reviewed mathematical evidence. Deliver that through the -existing loaders, workflows and documentation, with a small maintenance scope. - -This round resolves Q17-1 by making bundled metadata checksums a runtime -contract under the bounded defaults adopted with the implementation request. -This is an input integrity policy, not a newly confirmed mathematical bug. -The scope remains the existing loaders and researcher path, without a new -research platform or estimator suite. - -## 2. Preserve the completed foundation - -| Existing capability | Reuse in this round | -| :--- | :--- | -| Round 12/13 reference workflow and acceptance tests | Extend only where the metadata policy or Round 17 behavior needs coverage; do not rebuild the workflow | -| Product/vintage safeguards, extraction provenance and crosswalk validation | Preserve source identity, missing-vintage behavior and existing comparison rules | -| Price-table integrity loaders and typed errors | Reuse the design where appropriate; metadata repair guidance must identify metadata, not price files | -| Round 17 proposition binding, review, proof/refutation and evidence replay | Preserve the protected operator context and narrow domain; never add model-facing approval | -| Fresh bundles, filename confinement, complete inventories and figure collision guards | Retain the existing tests and destination policy | -| Archived actual model study | Keep four primary strict failures and separate JSON-mode success unchanged; neither model is established as reliable | -| CLI/MCP workflow adapters and the 14-tool core profile | Reuse the existing routes and discovery boundaries | - -No evidence or earlier plan is superseded by omission. Original datasets, -checksummed fixtures, supplied architecture documents and benchmark records -remain unchanged. New reproductions use temporary synthetic archive copies. - -## 3. Decisions to freeze before coding - -| Decision | Recommended bounded policy | Required clarification/evidence | -| :--- | :--- | :--- | -| D18-1 Bundled archives | Verify expected checksum on first actual read, before parsing/normalization/use | Confirm this is the supported runtime contract and distinguish absent archive from present-but-corrupt archive | -| D18-2 Custom metadata directories | Record actual consumed-byte hashes and an honest caller-supplied/unverified status; do not imply official verification from directory selection | Decide how an explicitly supplied custom manifest is treated; preserve documented custom-data use | -| D18-3 Hash and cache identity | Hash the decompressed JSON bytes already read, matching the current manifest convention; cache verification with that source | Check resolved archive-root identity and cache clearing/configuration behavior; do not introduce a cache-manager service | -| D18-4 Failure semantics | Corrupt bundled metadata fails before extraction or fallback; missing archive retains its documented missing/API behavior | Keep source missing, integrity failure, malformed JSON and transport failure distinct; no silent API substitution for corruption | -| D18-5 Workflow boundary | One existing stored-input workflow plus a separately supervised evidence appendix | No mandatory proof step for every analysis; no prose translation or review automatically approved by a model | - -The instruction to implement this plan adopted D18-1 through D18-5. Custom -manifest checksums are recorded as caller-supplied expectations, never promoted -to official verification. Resolved selection of the bundled root still enforces -the bundled contract. The implementation and compatibility policy are recorded -in [docs/metadata_integrity.md](docs/metadata_integrity.md). - -## 4. Implementation slices - -### 18.A — Metadata integrity contract and minimal enforcement (P0) - -Inspect `census/metadata.py`: `_read_archive`, `_load_json`, -`_archived_variables`, `load_manifest` and `verify_archive_file`. Trace metadata -provenance through extraction and concordance consumers. Currently the explicit -verification utility reads through `_read_archive`; avoid introducing recursion -by having that reader call the utility back. - -1. Reproduce the current behavior using a valid temporary gzip archive and - manifest, then change a plausible label/concept without updating its checksum. - Record what the public variable/group loader and extraction provenance report. -2. Freeze D18-1 through D18-4 and implement at the shared read seam. All archive - consumers must receive the same verified bytes or the same explicit failure. -3. Preserve public lookup return shapes and supported valid data. A checksum is - integrity against the trusted manifest, not authentication of the manifest or - a guarantee of Census semantic correctness. -4. Reuse typed error handling and actionable repair paths across library, CLI, - MCP and applicable GUI extraction paths. Do not silently downgrade an integrity - failure into ordinary missing data or generic successful provenance. -5. Hash already-read decompressed bytes once per cached source. Avoid a second - gzip read solely for verification. Define how archive-root/cache identity is - retained; do not promise hot reload unless it is implemented and tested. - -**Done when:** healthy bundled inputs preserve their results; present-but-corrupt -or unverifiable bundled inputs fail before use; custom-source status and actual -hashes follow the chosen policy; no provenance claims an unchecked expected -hash as verification of different consumed bytes. - -### 18.B — Pressure-test the supported researcher path (P1) - -Reuse `docs/reference_workflow.md`, `tests/test_reference_workflow.py`, existing -manifest examples and frozen inputs. The reference workflow already checks -reproducibility and library/CLI/MCP agreement; add only new contract cases. - -- Run the existing healthy path into a fresh directory and compare substantive - results with its established independent expectations. -- Exercise an integrity failure through a consuming entry point, not only a - helper. Assert no extraction API call, current-run bundle or success claim is - produced after a corrupt bundled archive is detected. -- Preserve ACS1/ACS5 identity and replicate restrictions and explicit legacy - compatibility. Matching geography/year alone does not establish a product. -- Link the existing verification guide as a separate operator-supervised - appendix. Its reviewed request and evidence retain their own identities; - empirical analysis status is never upgraded by a proof or model answer. -- Keep the reference document's existing CLI-first contract. Do not replace it - with a notebook, a second CLI application or an automatic approval command. - -**Done when:** another researcher can replay the stored-input workflow and its -failure case using the documented commands, identify the metadata source/hash -and empirical status, and separately inspect/replay any included exact evidence. - -### 18.C — Fix only observed friction and consolidate the handoff (P1) - -Keep a short list of concrete problems found in 18.A/B: source file/function, -reproducer, expected behavior, affected interfaces and validation. Fix the small -adapter/documentation issues needed by that path. Questions remain questions; -unrelated features return to a separate backlog. - -Update the existing guide, CHECK_MAP and changelog. Record compatibility changes -and the agreed metadata/cache policy in one short note. Reuse current provenance -fields/helpers where possible. A breaking manifest migration, new dependency or -wide refactor requires a separate proposal, not silent expansion of this slice. - -## 5. Regression and validation requirements - -- Temporary healthy, checksum-mismatched, missing-checksum, missing-file, - malformed-gzip and malformed-JSON inputs; include valid JSON with scientifically - plausible altered labels so parsing success cannot mask a mismatch. -- Correct decompressed-byte convention: gzip-header/recompression differences - must not be confused with changed JSON payloads. -- Public variable and group consumers; custom-directory behavior; cache/source - identity and the documented repair/restart path. A failed read must not poison - a later healthy run or reuse a different archive root's approved bytes. -- Mock the API fallback and assert corruption causes zero calls. Missing-source - fallback is tested separately under the existing policy. -- Library/CLI/MCP parity and applicable GUI error propagation. Preserve meaningful - metadata-specific repair guidance and useful source information. -- Existing source identity, reference workflow, bundle, proof/evidence and release - controls remain passing. Check archived evidence hashes and strict replay scores - without rewriting records or reinterpreting model formatting failures. - -Use independent expected bytes/values and consuming APIs. Do not create tests -that merely repeat a new helper's implementation or weaken scientific tolerances. -Run focused tests first, then the existing offline suite and Linux/Windows -Python 3.10–3.12 CI. Run applicable numerical studies only if a statistical method -or assumption changes; metadata work is not a reason to rerun every estimator. -Record representative first/warm-read cost without a flaky absolute timing gate. - -## 6. Explicit exclusions - -No general proof engine, new estimators/spatial models, literature retrieval, -vector store, workflow service, job database, GUI redesign, new model sweep, -automatic natural-language formalization approval or default model promotion. -No live Census requests, archive regeneration or source-data changes are needed -for the proposed reproductions. Local model calls are unnecessary for this round's -acceptance path; existing authorization is not a reason to spend model time. -No old plan, fixture, source document or archived experiment is rewritten. - -## 7. Order and stop conditions - -1. Confirm the baseline and freeze D18-1–D18-4 with the temporary reproducer. -2. Implement the shared loader/provenance/error contract and focused controls. -3. Extend the existing consuming workflow and fix the resulting narrow friction. -4. Run the appropriate suite/CI, record results and make a reviewable integration. - -Complete the round when those three slices pass and the policy is documented. -Do not use open-ended research-tool ambition to add unrelated functionality. -If custom-data policy or compatibility cannot be reconciled within a small change, -stop that dependent slice with a concrete decision question; continue independent -reproduction/documentation work. Do not mark an unresolved implementation complete. - -## 8. Baseline validation and publication state - -The following checks actually completed before this plan was written: - -- [Integration CI 43535062102](https://github.com/cafeTechne/Census-Research-Engine/actions/runs/43535062102): all six Linux/Windows Python 3.10–3.12 jobs passed at `6ed979a`. -- [Release-candidate CI 43537388947](https://github.com/cafeTechne/Census-Research-Engine/actions/runs/43537388947): all six jobs passed at `d33c7259181194c881dfc325cff1cfa5d53354d4`. -- Full offline baseline: 1476 passed, 37 skipped, 47 subtests passed, no exclusions. Release metadata/packaging checks: 59 passed. These counts overlap and are not additive. -- The predefined local inference smoke gate passed. The full numerical release gate also passed: 114 rows, all 11 gated rows passing under the unchanged full criteria, seed 20261102, Python 3.14.4, 17.9 minutes. Informational methods remain informational; criteria were not relaxed. -- Candidate 0.16.0 sdist/wheel assets built offline. Version consistency and an exact refutation were checked from the wheel outside the checkout. This is not a claim of general non-editable data-resource support. - -Round 17 was fast-forwarded into `defaultBranch` and pushed through the terminal. -Version 0.16.0 is prepared with a local annotated tag at the tested candidate. -No release tag was pushed; no public GitHub release or Zenodo archive was -triggered. Publishing the tag is a separate action from the requested release -preparation and from this Round 18 planning document. - -The release notes, candidate assets, smoke/full numerical reports and readiness -checks are retained under `/tmp/r17-release-preparation/` on this workstation. -That temporary directory is a local handoff, not a versioned publication. -The permanent Round 17 experiment remains in -[the versioned evidence archive](docs/audits/round17_bound_protocol/README.md). -No older plan, source document, benchmark record or research dataset was changed -while writing this plan. - -## 9. Round 18 implementation and evidence - -### Delivered slices - -- **18.A:** One root-aware cached archive read hashes decompressed JSON before - parsing, enforces bundled checksums and retains actual hashes for custom data. - Variable and group consumers, including the older `census.tables` path, use - that seam. A linked payload inside a bundled archive cannot bypass its checksum. - Normalized-variable cache keys include the consumed digest, preventing stale - values after eviction of a different bounded byte-cache entry. -- **18.B:** Existing stored-input reference commands and independent scientific - expectations remain. Acceptance checks independently hash variable/group bytes. - Corruption tests consume the library, CLI, MCP (including actual schema - validation), and headless GUI worker, asserting zero API calls and no extraction - output. Manifest validation checks present archives before geography probes; - runs fail before creating current-run artifacts. Supervised exact evidence is - linked as a separate optional appendix. -- **18.C:** The CLI/MCP adapters preserve the typed failure and metadata-specific - repair step. Legacy lookup caches distinguish source paths and clear with the - shared reset. Removed the GUI worker's redundant, unguarded default-metadata - read so the pipeline reports corruption through its existing error queue. - Workflow manifest adapters retain integrity failures instead of relabeling them - as ordinary missing data. CHECK_MAP, changelog and policy/reference guides updated. - -### Reproduction and frozen policy - -Before implementation, a temporary valid gzip with `Total population` changed to -`Adult population` returned the altered public variable label while -`verify_archive_file` returned false. Extraction provenance used the manifest's -expected SHA rather than the consumed bytes. No repository archive was modified. -The new regression cases reproduce that plausible alteration for variables and -universes. Missing files still permit existing fallback; present corrupt sources -fail explicitly. Repair uses `census.metadata.clear_cache()` or process restart; -no file watcher, cache service or automatic archive regeneration was added. - -Successful first-read snapshots are cached by resolved source root. Custom -manifest matches and mismatches both remain `caller_supplied_unverified`. -Existing lookup tuple shapes and healthy values are unchanged. Provenance additions -are additive; older replay records do not retroactively claim runtime verification. -The MCP table-search success value remains a list; its output type also permits -structured integrity failures. No dependency, estimator, release number or archive -format changed. Prepared plans must not be reused after changing input configuration. - -### Validation record - -- Final-source full offline suite: **1,507 passed, 37 skipped, 47 subtests - passed**, Python 3.14.4, 252.55 seconds. All 30 new metadata integrity - cases passed. The existing reference document executes twice into fresh - directories and retains its independent scientific expectations and route parity. -- `python -m census validate-crosswalks`: valid. Focused changed-module lint - (E4/E7/E9/F) and `git diff --check`: passed. Unrelated pre-existing lint - findings in the CLI/manifest were not reformatted or folded into this round. -- Implementation CI: [43545297502](https://github.com/cafeTechne/Census-Research-Engine/actions/runs/43545297502), - implementation commit `16acbbc0f2fdafbba41afddf09318dc8bf7108e8`. All six - Linux/Windows Python 3.10–3.12 offline jobs **passed**, including crosswalk - validation. Live validation was intentionally skipped. Windows skips the - symlink-creation test because it requires optional privileges; that test passes - on Linux. Existing optional/version-dependent skips account for other platform - count differences. No workflow dispatch or release/tag publication. -- Representative actual bundled ACS5 2023 variable read, Python 3.14.4 on this - workstation: **28,299 variables**, first verified read **0.0757 seconds**; - mean warm lookup over 1,000 calls **9.04 microseconds**. Informational only, - with no timing threshold. A synthetic control asserts one gzip read for lookup, - checksum verification and provenance together. -- All **74** original benchmark JSON records and both supplied architecture/audit - documents match their pre-Round-18 hashes. All **85** versioned Round 17 archive hashes verified. All five archived strict - protocol scores replayed unchanged: four primary failures and one JSON-mode - follow-up success. No model calls, approvals or external Census requests. -- No numerical method changed, so no new estimator simulation was required. - Round 17's numerical release evidence remains unchanged. - -### Handoff - -Implementation is committed and pushed on `phase18/metadata-integrity` at -`16acbbc0f2fdafbba41afddf09318dc8bf7108e8`; the following documentation-only -commit records these validation results and does not change tested source. -All Round 18 slices are complete, with no implementation stubs left open. -The default branch and prepared local v0.16.0 release tag remain at the Round 17 -baseline; this round does not merge or publish a release. - -### Review and release (2026-11-14) - -`/code-review high` of `faae550..b2f06f0` found 10 issues. Nine were fixed and one was rejected: - -| # | Finding | Outcome | -| :--- | :--- | :--- | -| 1 | Integrity errors subclass `ValueError`, so input handlers report corruption as INVALID_INPUT | Fixed: base class is `Exception`; every MCP tool is registered behind `integrity_errors` | -| 2 | Analysis ids are not checked as filenames until the bundle step | Fixed in `validate_manifest` (portable component, case-insensitive uniqueness) | -| 3 | `tables.clear_cache` empties dicts callers hold | Fixed: rebinds | -| 4 | Verification bundle-destination errors reported as EVIDENCE_INVALID | Fixed: INVALID_INPUT, verification result kept | -| 5 | Benchmark rejects omitted arguments | Fixed only for tools whose schema requires none; Round 17's "never substitute {}" rule otherwise holds | -| 6 | `_archive_at` caches raw + parsed payloads (128 entries) | Fixed: no raw bytes, 32 entries | -| 7 | Occupied output folder reported as "Manifest invalid" | Fixed: `output_dir unusable`, next step names the folder | -| 8 | Per-row regex in `_acs_product` | Fixed: distinct labels only | -| 9 | Redundant `digest` key in `_archived_variables_at` | **Rejected:** the key prevents stale normalized data after independent eviction (`test_normalized_cache_cannot_outlive_different_consumed_bytes`); now commented | -| 10 | Integrity re-raises patched per call site | Addressed by 1; the remaining explicit re-raise sits before an `except Exception` and stays | - -Regression tests: `tests/test_round18_review_fixes.py`. Released as v0.17.0. diff --git a/PLAN_ROUND_19.md b/PLAN_ROUND_19.md deleted file mode 100644 index e12f21d..0000000 --- a/PLAN_ROUND_19.md +++ /dev/null @@ -1,138 +0,0 @@ -# Round 19: One Complete Study, End to End - -**Date:** 2026-11-14 -**Status:** Implemented; study reviewed (author review delegated by the owner; report section 13) -**Baseline:** v0.17.0 (Round 18 plus review fixes) -**Owner role:** independent reviewer of the finished study (section 6) - -## 0. Why this round - -Since Round 12 the engine has gained the pieces a study needs: -- preregistered manifests, several states, several releases; -- CPI and RPP adjustment; -- Census VRE replicates; -- spatial methods with measured limits; -- exact verification and metadata integrity. - -It has never carried a real research question from preregistration to a finished, reviewable result. The county studies in `docs/case_study_mi_ny.md` are **worked examples**: their manifests were registered *after* exploratory analysis of the same data (its Limitations section says so), and they stop at county level. - -This round runs one study properly: -- the hypotheses are registered and committed **before** any of the study's data are pulled; -- it runs at **tract** level, where "where did change happen" is answerable; -- it produces a reviewable report; -- every point where the engine gets in the way is logged as evidence for the backlog, in place of speculative proposals. - -## 1. The study - -**Question.** Between 2015–2019 and 2020–2024, did real household income rise across Michigan and New York neighbourhoods (census tracts), and did it rise more where the share of adults with a bachelor's degree was higher, i.e. did the neighbourhood income–education gradient steepen? - -| Item | Choice | Why | -| :--- | :--- | :--- | -| Releases | ACS 5-year 2019 and 2024 | Non-overlapping periods, so the two samples are independent and replicates may be combined (overlap rule) | -| Units | All tracts in Michigan (26) and New York (36), on **2020 tract geography** | 2019 is published on 2010 tracts; it must be harmonized to compare the same areas | -| Outcome | Mean household income = aggregate household income `B19025_001` / households `B19001_001` | Both tables are on the VRE list at tract level, so the outcome has exact replicate SEs (median `B19013` has none) | -| Covariate | Bachelor's degree or higher, % of adults 25+: `B15002` (`_015`..`_018` + `_032`..`_035`) / `_001` x 100 | On the VRE list; standard area-level human-capital measure | -| Prices | R-CPI-U-RS to 2024 dollars, then BEA RPP of the tract's county area (MSA, or the state's nonmetro part), each release's own RPP year | As in case study section 8; county-metro areas need the A3 extension at tract level | -| Replicates | `census_vre`: 80 replicates per release, apportioned to 2020 tracts with the same weights as the estimates (A2) | Exact sampling SE for the apportioned 2019 values | -| Reliability | CV screen at 30% on the outcome, **flag** only | Preregistered: no tract is dropped for noise; sensitivity analysis drops flagged tracts | -| Missing | Complete case (e.g. tracts with no households) | | -| Correction | Holm across all three hypotheses, 90% intervals | ACS convention | - -**Hypotheses** (`examples/manifests/r19_mi_ny_tract_income_change.json`): -- **H1** (m1, `mean_hh_income ~ C(Year)`, target `these_areas`): real mean household income was higher in 2020–2024 than in 2015–2019. The estimand is the unweighted mean of tract changes, one-sided `greater`. -- **H2** (m2, `mean_hh_income ~ C(Year) * pct_bachelors + C(state)`, target `conservative`): the cross-tract slope of real mean income on the bachelor's share was steeper in 2020–2024. The test is on the interaction `C(Year)[T.2024]:pct_bachelors`, one-sided `greater`. `conservative` adds model and sampling variance because the claim is about a relationship, not only these tracts. -- **H3** (m3, `mean_hh_income ~ C(Year) * C(state)`, target `these_areas`): the mean tract change differed between New York and Michigan. Two-sided. - -**What the authors had seen before registering.** County-level change in mean household income for the same states and releases (case study section 8: +$3,100), and the tract-level structure of ACS sampling-error correlation (Round 16 C1). Neither shows tract-level income change, the tract education gradient, or the state difference. The report states this. - -## 2. Engine work the study needs (P0, offline tests first) - -**A1. Tract harmonization in manifests.** -- `geography.harmonize_tracts_to: 2020 | 2010`, valid only for `level: "tract"`. -- Estimates of releases on the other tract vintage are apportioned with the existing `census.geo_crosswalk` machinery: - - block **housing-unit** weights for household/housing tables and population weights otherwise (`weight_for_variable`); - - per state; - - targets covered below 99% dropped and reported. -- Validation refuses non-additive variables (medians, percentages): only counts can be apportioned. It also refuses `aggregation` combined with harmonization. -- `run.json`/`bundle.json` record the crosswalk source and hash, the weight type per variable, the notes, and dropped or unmatched tracts. `methods.md` states it. - -**A2. Replicate apportionment (the methodological piece).** -- Each of the 80 VRE replicates is a count, so it is apportioned with the **same crosswalk weights** as its estimate. The apportioned replicates then give the exact replicate SE of the apportioned estimate, including the covariance between source tracts that the published-MOE route ignores: `apportion` combines MOEs as sqrt(sum (w*MOE)^2), assuming independence. -- Implement `census.replicate.apportion_replicates` on top of `geo_crosswalk.apportion`, so the arithmetic is shared, not duplicated. -- In `run_manifest`, the 2019 VRE rows are apportioned *before* being matched to the analysis units. -- Self-check, recorded in `run.json`: the apportioned VRE `estimate` column must reproduce the harmonized frame estimate (published values, so rounding-level agreement). -- Offline tests: - - an apportioned estimate via the replicate path equals `harmonize_panel` exactly; - - weight-1 targets keep their source replicate SE; - - under simulated independent replicates, a split target's replicate SE converges to sqrt(sum w^2 SE^2); - - an end-to-end manifest on a frozen two-state toy crosswalk with network refused. - -**A3. County-metro RPP areas for tracts.** A tract lies in one county (GEOID[:5]), so `areas: "county_metro"` is allowed at tract level and maps each tract to its county's RPP area. Validation and the methods text are updated. - -**A4. Study outputs outside the tree.** Run outputs (bundle, frames, replicates) are attached to the GitHub release as an archive, not committed. The repository keeps the manifest, the report, small summary tables and the figures. This follows the Round 17 repo-hygiene note. - -## 3. Study protocol - -1. A1–A3 merged and released (v0.18.0 dev on the branch); full gate green. -2. Write the manifest, `create_manifest` (resolve on), and **commit it alone** (the preregistration commit). Record the commit SHA and manifest hash in this plan and the report. No study data are pulled before this commit. -3. `run_manifest` live with the owner's API key into `studies_output/r19/` (ignored by git). -4. Pre-specified **exploratory** analyses, labelled exploratory with the manifest hash: - - **E1:** maps of tract change in each state, with CV-flagged tracts hatched or marked. - - **E2:** local Moran's I of the change (Round 16 defaults: two-sided, 9,999 permutations, BH 0.10), with the Round 16 caveats on heterogeneous sampling variances. - - **E3:** sensitivity: - - drop CV-flagged tracts; - - household-weighted (WLS by households); - - nominal dollars; - - state price areas instead of county-metro. - - **E4:** for apportioned tracts, the replicate SE against the independence-approximation SE. This shows how wrong the approximation is in practice. -5. Report `docs/studies/r19_mi_ny_tract_income_change.md`: - - question; - - preregistration (commit, hash, what was seen before); - - data and provenance; - - methods; - - confirmatory results; - - exploratory results (labelled); - - limitations; - - reproduction commands; - - reviewer checklist. -6. **Friction log** in the report: every workaround, wrong error, slow step or missing tool hit while running the study, each marked fixed-in-round or backlog. - -## 4. Acceptance - -- Manifest committed before the first study data request (git history shows it). -- `analysis_status` is `confirmatory`, or `confirmatory_incomplete` reported as such with its deviations. A failed or blocked hypothesis is reported, never dropped. -- Every number in the report traces to a bundle file or an exploratory output with its hash. -- A1–A3 have offline tests. Fast gate and full suite pass on Python 3.14 and 3.10 (excluding the known local-only 3.10 spatial λ test, which fails identically on the baseline). CI is green. - -## 5. Out of scope - -- New estimators; -- spatial regression in the confirmatory set; -- SSH or remote access; -- more verification rules; -- model benchmarking. - -The friction log decides what comes next. - -## 6. Owner review - -The owner reviews the report as independent reviewer: question, preregistration timing, deviations, interpretation, limitations. Review comments go into the report's review section before release. - -## 7. Release - -v0.18.0 with a Zenodo DOI after `/code-review high`, the fixes, the full gate, and Python 3.10 and 3.14 runs. The study output archive is attached to the release. - ---- - -## 8. Implementation status (2026-11-14) - -| Item | Result | -| :--- | :--- | -| A1–A3 | `geography.harmonize_tracts_to`, replicate apportionment (`census.tract_harmonization`), tract `county_metro` RPP areas; offline tests in `tests/test_tract_harmonization.py` and `tests/test_manifest_multistate.py` | -| Preregistration | Manifest SHA-256 `8f205f51…fb126`, committed alone at `3f0b1fe` (2026-11-14T20:33:49Z) and pushed before any study data request | -| Run | Run 1 confirmatory, 0 deviations. Code review then found a GEOID-recode defect: 11 renumbered NY tracts were dropped from 2019 without a deviation. Engine fixed and run 3 of the **unchanged** manifest reported: confirmatory, 0 deviations. Both runs disclosed in the report. Apportioned VRE estimates match the API estimates exactly | -| Results (run 3) | H1 +$3,469 [3,104, 3,834] supported; H2 −$64.8/pp [−121.9, −7.7] not supported (opposite sign); H3 NY − MI +$3,895 [3,259, 4,530] supported | -| Exploratory | Sensitivity agrees across all real-dollar variants. Per-tract change mostly noise (< 1% significant after BH). Weak global autocorrelation of the change. Independence-approximation SEs right on average but ±6–7% off for genuinely merged tracts | -| Friction | F1–F4 and F11 fixed in round; F5–F10 and F12 backlog (report section 12) | -| Review | Owner delegated the review to the author (not independent). Checks recorded in report section 13. Corrections: `B19001` pre-registration exposure disclosed fully; the exploratory "flatter gradient" withdrawn after an errors-in-variables check (`scripts/r19_review_eiv.py`) | -| Report | `docs/studies/r19_mi_ny_tract_income_change.md` | diff --git a/PLAN_ROUND_2.md b/PLAN_ROUND_2.md deleted file mode 100644 index a587165..0000000 --- a/PLAN_ROUND_2.md +++ /dev/null @@ -1,72 +0,0 @@ -# PLAN_ROUND_2.md — US Census ACS Data Batch Downloader - -**Planning Round:** 2 -**Date:** 2026-11-01 -**Branch:** phase2/research-ux -**Status:** Phase 2 COMPLETE. Ready to begin Phase 3. -**Next Round:** See PLAN_ROUND_3.md for Phase 3 (Headless Library Layer). - ---- - -## Phase 2 Completion Summary - -All Phase 2 tasks are complete and verified on branch `phase2/research-ux`. - -### Implemented Enhancements - -1. **2a — Threading & Freeze Prevention (COMPLETE)** - - Moved the fetch loop from `on_submit()` into `_download_worker()`. - - Executed in a background daemon thread (`threading.Thread(target=..., daemon=True)`). - - Submit button is disabled and its label changed to "Downloading..." during operation to prevent duplicate execution. - - Status label updates dynamically per entity and per year. - - Main thread polls a thread-safe `queue.Queue` every 50ms via `root.after()`, guaranteeing 100% thread safety for Tkinter widget updates. - -2. **2b — Progress / Status Log (COMPLETE)** - - Added a scrollable Dracula-themed `tk.Text` log (`SELECTION_COLOR` background, `FOREGROUND_COLOR` text) with vertical scrollbar. - - Read-only (`state=tk.DISABLED`), automatically scrolling to `tk.END` on new entries. - - Clears automatically at the start of each submission. - - Logs timestamped status updates per entity, per year, rows/columns processed, and skipped years (e.g. unavailable ACS years). - -3. **2c — Multi-Place Selection (COMPLETE)** - - Replaced single Combobox with a multi-select Listbox (`selectmode=tk.EXTENDED`). - - Added real-time text filter entry to quickly search and isolate places. - - Added helper buttons: "Select All" and "Clear". - - Added real-time counter label showing selected count (e.g., "5 selected"). - - Worker loops across all selected places and years. - -4. **2d — Year Range UX Hardening & Call Estimator (COMPLETE)** - - Added real-time calculation and display of total estimated API calls: `entities x years`. - - Color-coded indicator: GREEN (≤ 50), ORANGE (51–150), RED (> 150). - - Added safety confirmation dialog before starting if total calls exceed 50 to prevent unintentional API throttling. - -5. **2e — Output Format Options (COMPLETE)** - - Added format dropdown supporting: - - "Both Wide & Melted CSVs" (default) - - "Melted CSV only" - - "Wide CSV only" - - "Excel Workbook (.xlsx)" (using `openpyxl`, creates `Wide_Data` and `Melted_Data` sheets). - - In both wide and melted data, a `Year` column is now prepended at index 0, resolving multi-year data disambiguation in compiled files. - -6. **2f — Variable Browser Enhancement (COMPLETE)** - - Selecting a table now lazily loads `VARIABLE_DEFINITIONS` without UI lag. - - Description pane displays Table Code, Title, Total Variables count, and formatted preview of variable codes and labels. - -7. **2g — Geography Level Selector (COMPLETE)** - - Added radio buttons for: - - "Place (Cities/CDPs)": `for=place:{place_code}&in=state:{state_code}` - - "Entire State": `for=state:{state_code}` - - "All Counties in State": `for=county:*&in=state:{state_code}` - - Dynamic UI hides or shows place picker depending on chosen geography level. - -8. **Architecture, Quality & Tests (COMPLETE)** - - Refactored `acs_downloader.py` into a clean `ACSDownloaderApp` class with `if __name__ == "__main__":` entry point, making all functions importable headlessly. - - Added comprehensive unit test suite in `tests/test_downloader.py` (13 tests covering melting, duplicate column handling, table loading, place parsing, file sanitization, export formats, and API parameter generation). - - Strict adherence to `black` formatting and explicit type annotations. - ---- - -## Known Remaining Issues / Transition to Phase 3 - -1. **Monolithic Architecture**: Non-GUI functions (`fetch_census_data`, `load_places_from_local`, `format_dataframe`, `process_data_and_save`) remain in `acs_downloader.py`. They should be extracted into a standalone headless `census/` package in Phase 3. -2. **API Resilience**: The app currently skips failed requests with a log entry; Phase 3 will introduce exponential backoff / retry logic in `census/api.py`. -3. **MCP Server**: Once the headless library is extracted, Phase 4 will expose tools over the Model Context Protocol. diff --git a/PLAN_ROUND_20.md b/PLAN_ROUND_20.md deleted file mode 100644 index 006ad87..0000000 --- a/PLAN_ROUND_20.md +++ /dev/null @@ -1,203 +0,0 @@ -# Round 20: Public-health determinants: disability, housing cost burden, vehicle access - -**Date:** 2026-11-14 -**Status:** Implemented; study run and reviewed (section 9) -**Baseline:** v0.18.0 (Round 19, first complete preregistered study) -**Owner direction:** move the engine towards public-health research. The test study uses disability, housing cost burden, vehicle access and the social and economic determinants themselves, not insurance. - -## 0. Why this round, and the three-step roadmap - -The engine is strong on area-level social and economic measures with exact ACS sampling error. Public-health studies add three things it does not yet have: - -| Step | What it adds | Round | -| :--- | :--- | :--- | -| **1. Health studies from ACS alone** | Age standardization of prevalences, a measurement-error-corrected regression for ACS-estimated determinants, and a preregistered disability study | **20 (this round)** | -| **2. External health outcomes** | Connectors for model-based and survey health data (CDC PLACES, USALEEP life expectancy, County Health Rankings and others) with provenance, uncertainty and circularity guards | 21 (section 7) | -| **3. Count-based outcomes** | Deaths, births and cases: rates with population denominators, age-adjusted rates with exact intervals, suppression, count models, small-area smoothing | 22+ (section 8) | - -Steps 2 and 3 are specified here so that the roadmap is tracked in one place. Their detailed plans are written in their own rounds, informed by this round's friction log. - -## 1. The study - -**Question.** Across Michigan and New York census tracts in 2020–2024, is age-standardized disability prevalence higher where more households lack a vehicle and more households are housing-cost burdened, beyond what tract poverty and education account for? - -Both determinants are standard social determinants of health: -- vehicle access limits reaching care, work and food; -- cost burden competes with health spending. - -Disability is both a consequence and a cause of economic hardship. This is a **cross-sectional ecological association** study. It does not estimate causal effects in either direction. - -| Item | Choice | Why | -| :--- | :--- | :--- | -| Release | ACS 5-year 2024 (2020–2024) only | One period on 2020 tracts: no harmonization, so this round's new methods are tested in isolation | -| Units | All tracts in Michigan (26) and New York (36) | Preferred test states | -| Outcome | `asdp`: age-standardized disability prevalence, % of the civilian noninstitutionalized population, `B18101` | Disability rises steeply with age, so crude prevalence mostly measures tract age structure. Direct standardization removes that | -| Standard | 2000 U.S. standard population (NCHS/SEER, Census P25-1130), single ages bundled with checksum. Bands 0–17, 18–34, 35–64, 65–74, 75+ (the `B18101` age groups, with under-5 and 5–17 combined, sexes combined) | The public-health convention, so rates compare with published age-adjusted rates | -| Determinants | `no_vehicle`: % of occupied housing units with no vehicle (`B25044`). `cost_burden`: % of households with housing costs ≥ 30% of income, renters (`B25070`) and owners with and without a mortgage (`B25091`), among those with the ratio computed. `pov_rate`: % below poverty (`B17001`). `pct_bachelors`: % of adults 25+ with a bachelor's degree or higher (`B15002`) | All on the tract VRE list for 2024, so each has exact replicate SEs | -| Replicates | `census_vre`, 80 replicates per table | Needed both for the SEs and for the measurement-error correction (A2) | -| Estimator | `fit_eiv` (A2): least squares corrected for the determinants' sampling error and for its correlation with the outcome's sampling error, using replicate error covariances. Sandwich SE | Round 19's review found covariate noise flattened a slope by about a third, and outcome–covariate error correlation was the same size. Determinants here have CVs of 15–60% at tract level | -| Reliability | CV screen at 30% on the outcome, **flag** only | As in Round 19 | -| Missing | Complete case; a tract with an empty age band has no age-standardized rate (A1) and is dropped and counted | | -| Correction | Holm across the three hypotheses, 90% intervals | | - -**Hypotheses** (manifest `examples/manifests/r20_mi_ny_disability_determinants.json`): -- **H1** (m1: `asdp ~ pov_rate + C(state)`, `fit_eiv`): age-standardized disability prevalence is higher in tracts with higher poverty. Coefficient of `pov_rate`, one-sided `greater`. -- **H2** (m2: `asdp ~ pov_rate + pct_bachelors + no_vehicle + cost_burden + C(state)`, `fit_eiv`): conditional on poverty, education and state, disability prevalence is higher where more households lack a vehicle. `no_vehicle`, one-sided `greater`. -- **H3** (m2): conditional on the same, disability prevalence is higher where more households are cost burdened. `cost_burden`, one-sided `greater`. - -H1 is expected from the literature and serves as a check on the pipeline. H2 and H3 are open: -- poverty and education absorb much of the shared variation; -- in New York City, vehicle ownership mostly reflects transit, not deprivation; -- cost burden is high both in poor tracts and in expensive ones. - -**What the authors had seen before registering:** -- county-level income and education in MI/NY (case study); -- Round 19's tract income and education; -- Round 16's tract poverty-rate sampling-error correlations, which used tract `B17001` replicate files for both releases. - -No tract disability, vehicle or cost-burden data have been downloaded or computed (checked against the cache before registration, and recorded in the report). - -## 2. Engine work (P0, offline tests first) - -**A1. Age standardization (`census.standardize`) and manifest derivation `age_standardized`.** -- **Bundled standard populations.** `data/standard_populations/us2000.csv`: the 2000 U.S. standard population by single year of age, 0–84 and 85+. It is built from SEER's `singleages.pops.tables.csv` by `scripts/update_standard_populations.py`. The raw file is kept in `data/standard_populations/source/`. The sidecar JSON carries the source URL and the SHA-256 of both the raw file and the CSV, and loads are integrity-checked like the price tables. -- **Functions:** - - `standard_weights(bands, standard="us2000")` gives weights for arbitrary age bands, checked to tile the bands without gaps or overlaps. - - `direct_standardize(cases, populations, weights)` computes the direct rate sum_a w_a × cases_a / pop_a. - - `age_adjusted_interval(...)` is the Fay–Feuer (1997) gamma interval. It is not used by this study, which has replicate SEs, but Step 3 needs it for registry counts. -- **Manifest derivation:** - - ```json - {"name": "asdp", "op": "age_standardized", "as_percent": true, "standard": "us2000", - "bands": [[0, 17], [18, 34], [35, 64], [65, 74], [75, null]], - "groups": [{"num": [...], "den": [...]}, ...]} - ``` - - - There is one group per band. A list sums its members, e.g. the male and female cells. - - `create_manifest` fills `weights` from the bundled standard before hashing, as it does for `rpp_year`. A manifest with explicit `weights` must match the standard, or declare `"standard": "custom"`. -- **Replicates.** On the replicates, the same formula is evaluated on each replicate, so the SE is the exact replicate SE. The frame path gives an approximate MOE from the per-band proportion MOEs. -- **Empty bands.** A band with zero population in the full-sample estimate leaves the rate undefined, so the unit is dropped as complete-case and counted. A zero replicate denominator follows the VRE documentation's ratio convention, as proportions already do. -- **Provenance.** The run records the weights, the standard and its table's SHA-256. `methods.md` states them. - -**A2. `fit_eiv`: least squares corrected with replicate error covariances (friction F12).** -- **Estimator.** Let x_i be the design row of unit i built from the estimates, and x_i^(r) the same row built from replicate r (same patsy design information, so factor coding is fixed). The error covariances are S_i = 4/80 Σ_r (x_i^(r) − x_i)(x_i^(r) − x_i)ᵀ and s_i = 4/80 Σ_r (x_i^(r) − x_i)(y_i^(r) − y_i). Then - - β̂ = (Σ x_i x_iᵀ − Σ S_i)⁻¹ (Σ x_i y_i − Σ s_i). - - This is Fuller's (1987, *Measurement Error Models*, sections 2.2 and 3.1) moment estimator with known, heteroscedastic error covariance. Its new feature is that the covariances, including the outcome–covariate cross term, come from the ACS replicates. Interaction and transformed columns get their error covariance directly from their replicate design rows, and error-free columns (factors, constants) get zero. -- **Variance.** The sandwich estimator V = M⁻¹ (Σ ψ_i ψ_iᵀ) M⁻¹, with ψ_i = x_i (y_i − x_iᵀβ̂) − (s_i − S_i β̂) and M = Σ x_i x_iᵀ − Σ S_i. It treats tracts as independent. Round 16 measured that neighbouring tracts' sampling errors are nearly uncorrelated. The SE is reported as `se_total`; `se_model` is the naive model SE, and the naive estimates are reported alongside. -- **Diagnostics.** - - Each noisy column's reliability, 1 − mean S_jj / Var(x_j), with x_j residualized on the error-free columns. - - The smallest eigenvalue of M relative to that of X'X. - - A note when reliability is below 0.5, where moment corrections become unstable. The fit is refused when M is not positive definite. -- **Scope.** - - OLS only, with an optional `weights` column for WLS. - - Targets `population` and `conservative` only. `these_areas` is refused: a measurement-error correction is a claim about the relationship, not about these tracts. - - A unit with an incomplete replicate set raises the same strict-policy error as `fit_replicate`. -- **Manifest.** Method `fit_eiv` (`formula`, optional `args.weights`, `args.target`). Methods text and references are added. -- **Validation.** Offline simulation, with tests and an audit table (`docs/audits/eiv_validation.md`): heteroscedastic covariate error, outcome–covariate error correlation ±0.5, reliabilities 0.95, 0.8 and 0.6, synthetic replicates whose 4/80 sum reproduces the error covariance in expectation. Acceptance: |bias| < 10% of the naive bias, and 90% interval coverage of 0.87–0.93 at reliability ≥ 0.8. Naive OLS and `regression_calibration` are shown for comparison, the latter without the correlated-error term. - -**A3. Reliability warning in `fit_replicate` (F12, part two).** `fit_replicate` reports in its diagnostics each numeric covariate's replicate reliability. When any is below 0.9, or two groups of a factor in the formula differ by more than 0.02, it adds a note recommending `fit_eiv`. The run record and `methods.md` carry it. This is a note, not a deviation. - -**A4. Study outputs** follow Round 19 A4: the bundle and `analysed/` go to the release as an archive. The repository keeps the manifest, report, figures and summaries. - -## 3. Study protocol - -1. A1–A3 implemented with offline tests and the simulation validation; fast gate green. -2. Check the cache for tract `B18101`, `B25044`, `B25070` and `B25091` data, and record the result. Write the manifest, run `create_manifest` (fills the weights), and **commit it alone** (the preregistration commit), then push. -3. `run_manifest` live into `studies_output/r20/`. -4. Pre-specified **exploratory** analyses (labelled, with the manifest hash): - - **E1:** naive `fit_replicate` (`conservative`) and `regression_calibration` versions of m1 and m2, to show what the correction changes. - - **E2:** m2 without New York City (counties 005, 047, 061, 081, 085), and m2 per state. This tests whether the vehicle association is a transit artefact. - - **E3:** crude rather than age-standardized prevalence, to show what standardization changes. - - **E4:** drop CV-flagged tracts; household-weighted WLS. - - **E5:** Moran's I of m2 residuals (queen contiguity), as the check on the independence assumption. - - **E6:** maps of `asdp`, `no_vehicle` and `cost_burden`, with reliability marked. -5. The report `docs/studies/r20_mi_ny_disability_determinants.md` uses the same structure as Round 19, including the friction log and the review section. - -## 4. Acceptance - -- The manifest is committed before the first study data request. -- `confirmatory`, or `confirmatory_incomplete` reported with its deviations. -- Every reported number traces to a bundle file or a hashed exploratory output. -- A1–A3 have offline tests, and A2's simulation meets its criteria. The full suite passes on Python 3.14 and 3.10, except the known local-only 3.10 spatial λ test. - -## 5. Out of scope this round - -External data sources (Step 2), count outcomes (Step 3), change over time for disability (a later study can reuse Round 19's harmonization), spatial regression and causal designs. - -## 6. Release - -v0.19.0 with a Zenodo DOI, after `/code-review high` and its fixes, the full suite on Python 3.14 and 3.10, and the review section of the report. The owner delegates the review; it is labelled as an author self-review. - ---- - -## 7. Step 2 (next round): external health-outcome data - -### 7.1 Do we need APIs? - -No API keys are needed. Every source below offers bulk file downloads (CSV, XLSX or zipped CSV) over plain HTTPS. The engine should treat them like the Census VRE and relationship files: download once into a cache, record the URL, retrieval time and SHA-256, parse locally, and refuse a file whose hash no longer matches the manifest's pinned value. - -| Source | What | Geography / vintage | Access (verify URLs when implementing) | Uncertainty | -| :--- | :--- | :--- | :--- | :--- | -| **CDC PLACES** | About 40 model-based adult health measures: diabetes, obesity, COPD, depression, disability types, checkups, smoking and more | Tract, county, place, ZCTA. 2025 release mostly from BRFSS 2023 | data.cdc.gov (Socrata). Full CSV export per dataset, no key (an app token only raises rate limits). 2025 tract GIS-friendly dataset `yjkw-uj5s`, county `swc5-untb`. A state filter can be applied in the export query to avoid the full national file (about 800 MB) | 95% CIs, model-based (not sampling replicates) | -| **USALEEP** (NCHS) | Life expectancy at birth | Tract, **2010 tracts**, 2010–2015 | Static per-state CSV files from NCHS | Published SE | -| **County Health Rankings** | Compiled county measures: premature death, low birthweight, preventable hospitalizations and others | County, annual | Annual national CSV from countyhealthrankings.org | CIs for most measures | -| **CDC/ATSDR SVI** | Social Vulnerability Index | Tract, county | CSV per state or nation from ATSDR | Built from ACS: the engine can recompute it with exact replicate SEs, which is preferable to importing it | -| **USDA Food Access Research Atlas** | Low-income/low-access tract flags, distance to supermarkets | **2010 tracts** (2019 edition) | XLSX download from USDA ERS | None published | -| **HRSA shortage areas** | Primary care, dental and mental health HPSAs | Designation areas (county, tract groups) | CSV from data.hrsa.gov | None (administrative) | -| **EPA air quality** | Annual PM2.5 and ozone summaries | Monitors to county | Pre-generated annual files from EPA AirData (aqs.epa.gov) | Monitor coverage; no area SE | - -### 7.2 Engine work (Round 21 sketch) - -- **`census.health_sources`.** A registry of connectors, each pinning a dataset id and release and recording URL, licence, retrieval time and SHA-256. It provides a cache under `.cache/health/`, state-filtered downloads where the portal supports them, and parsers into the analysis-frame convention: estimate, SE from published CIs (half-width / z), `method` (`model_based`, `survey`, `administrative`), and geography vintage. -- **Manifest references to external variables,** e.g. `{"name": "diabetes", "ref": "places:2025:DIABETES:CrudePrev"}`. The source version is in the hash, and validation checks the source registry offline. -- **Vintage handling.** 2010-tract sources (USALEEP, Food Atlas) go through Round 19's harmonization, but only for counts. Rates need population-weighted apportionment, a new and documented approximation. Connecticut planning regions and the 2022 county change stay refused or explicit. -- **Uncertainty.** - - Outcomes with known non-replicate variance use `fh_regression` (existing). - - Determinants keep `fit_eiv` (A2) for their ACS error. - - The outcome's error is independent of the ACS error, as a different survey, so the cross term is zero. The manifest declares this and the validator enforces it. -- **Circularity guard.** PLACES estimates come from multilevel regression and post-stratification of BRFSS, using demographic and area-level covariates. Regressing a PLACES outcome on those same covariates partly recovers the model, not the world. Each connector records its source model's covariates (to be verified against CDC's PLACES methodology), and `create_manifest` warns when a formula covariate overlaps them. The report template states it. -- **Spatial dependence of model-based outcomes.** PLACES tract estimates share county and state random effects, so their errors are correlated. Fits must say so, and the sandwich SE should cluster by county. - -### 7.3 Pilot study for Step 2 (proposal) - -PLACES 2025 tract disability measures (any disability, mobility) compared with Round 20's ACS age-standardized disability prevalence: -- This is a validation study: two independent measurement systems for the same construct. -- It tests the connector, the vintage join and the uncertainty treatment on a question where the answer is checkable. - -## 8. Step 3 (after Step 2): count-based outcomes - -### 8.1 Data access - -- **CDC WONDER** (mortality, natality). County data are available through the web interface only: WONDER's API serves **national** data only, per NCHS vital-statistics data-sharing policy. Counts below 10 are suppressed, and rates from fewer than 20 deaths are flagged unreliable. The engine therefore ingests **exported WONDER text files** that the user downloads. It parses the query parameters from the export footer, records the file hash, and refuses edited files. It does not automate the web form. -- **State vital statistics** (Michigan MDHHS, New York State DOH): published downloadable tables, county or ZIP, per state. Connectors are written per state. -- Unsuppressed or record-level data need an NCHS Research Data Center application and are **out of scope**. The engine stays aggregate and public. - -### 8.2 Methods to add - -- Crude and age-adjusted rates with the population denominator chosen explicitly: ACS, Census Population Estimates, or WONDER's bridged or single-race estimates. The provenance records which, because the denominators differ. -- Age-adjusted rate intervals by Fay–Feuer gamma (A1 helper) and Tiwari et al. (2006) modification. Poisson exact intervals for crude rates. -- **Suppression handled as censoring** (count in [0, 9]), never zero-filled. Rate models use interval-censored likelihoods or bounds analyses. -- Count models (`poisson`, `negbin` with `exposure`, existing in `census.models`) in manifests with offsets, with overdispersion diagnostics. -- Small-area smoothing: empirical Bayes (Marshall 1991) first. Spatial BYM/ICAR models come later, and only if a study needs them, with validation like Round 16. -- Combining count outcomes with ACS determinants: `fit_eiv` generalizes to GLMs only approximately. A corrected-score or regression-calibration route for Poisson models, with simulation validation, is part of this step. - -### 8.3 Pilot study for Step 3 (proposal) - -County premature mortality (WONDER exports, ages < 75, age-adjusted) against ACS disability, vehicle and cost-burden determinants for MI and NY counties. This study carries the Round 20 hypotheses to a hard outcome. - ---- - -## 9. Implementation status (2026-11-15) - -| Item | Result | -| :--- | :--- | -| A1 | `census.standardize` (bundled 2000 U.S. standard, SHA-256 checked; `standard_weights`, `direct_standardize`, Fay–Feuer `age_adjusted_interval`); manifest derivation `age_standardized` (weights filled by `create_manifest`, exact replicate SEs) | -| A2 | `census.models.fit_eiv` (manifest method `fit_eiv`, `fit_model` method `eiv`). Validation `docs/audits/eiv_validation.md`: passed, coverage 0.89–0.92, bias removed in all 9 cells. Regression calibration coverage fell to 0.04–0.51 with correlated errors | -| A3 | `fit_replicate` reports covariate reliability per factor group and notes low or uneven reliability | -| Review | `/code-review high` before registration: 10 findings fixed. Most serious: the standard table was git-ignored; reliability was grouped by tract instead of state | -| Preregistration | Manifest `b67232d5…f833`, committed alone at `bb53959` (01:58:54Z); first study request 01:59:05Z | -| Results | H1 supported (+0.32 per point of poverty); H2 not supported pooled, but clear outside New York City (exploratory); H3 not supported (opposite sign, concentrated in New York State) | -| Lesson | Residuals are spatially autocorrelated; county-clustered SEs are 2–3× the registered ones (no conclusion changes). `fit_eiv(cluster=...)` added; validating it is the first item for the next health manifest | -| Report | `docs/studies/r20_mi_ny_disability_determinants.md`; friction F1–F5 fixed, F6 partly, F7–F10 backlog | diff --git a/PLAN_ROUND_21.md b/PLAN_ROUND_21.md deleted file mode 100644 index c6aced1..0000000 --- a/PLAN_ROUND_21.md +++ /dev/null @@ -1,286 +0,0 @@ -# Round 21: External health outcomes (Step 2): CDC PLACES connector and a validation study - -**Date:** 2026-11-14 -**Status:** Part 1 implemented, study run and reviewed (section 9). Part 2 (sections 10-17, the section 8 work plus the circularity rule, at the owner's request of 2026-11-15) in progress -**Baseline:** v0.19.0 (Round 20, DOI 10.5281/zenodo.24238647) -**Roadmap:** Step 2 of PLAN_ROUND_20 section 0. Step 3 (count outcomes, PLAN_ROUND_20 section 8) stays next. - -## 0. Why this round - -Round 20 showed the engine can run a public-health study from ACS alone. Most health outcomes, though, are not in the ACS: diabetes, depression and life expectancy come from other systems. Step 2 lets a manifest bring in an external health measure with its provenance, uncertainty and caveats, starting with CDC PLACES, the most used tract-level source. - -Round 20's lesson comes first. Tract residuals were spatially autocorrelated and the independence sandwich understated SEs by 2–3× (friction F6). A clustered sandwich was added but not validated. PLACES estimates also share county random effects, so every PLACES model needs clustered SEs. Validating them is a precondition for the study. - -## 1. What PLACES is, and what that means for analysis (verified 2026-11-14) - -- **Datasets.** Socrata on data.cdc.gov, public domain: - - 2025 release, tract `cwsq-ngmh` (long format: one row per tract and measure); - - county `swc5-untb`; - - GIS-friendly tract `yjkw-uj5s`. -- **Model.** Measures are adult (18+) crude prevalences from BRFSS 2023 (2022 for five measures). They come from multilevel regression and post-stratification: - - individual age, sex, race/ethnicity and education; - - the county-level share of adults below 150% of the poverty line (ACS 5-year); - - state and county random effects; - - applied to 2020 decennial block population counts and aggregated to tracts. - - Source: CDC PLACES methodology page. -- **Uncertainty.** 95% intervals are the 2.5th and 97.5th percentiles of 1,000 Monte Carlo draws. They are model-based, not sampling replicates. -- **Tract vintage.** 2020 tracts. Michigan has 2,904 tracts with PLACES rows, which matches the 2020 count of populated tracts, not the 2,813 tracts of 2010; the run checks this against the ACS frame. - -**Consequence: the circularity problem.** Within a county, a tract's PLACES value varies only through its age, sex, race/ethnicity and education composition. The model contains no tract-level information about health. So: -- Regressing a PLACES outcome on ACS composition measures (age, race, education, and through the county term, poverty) partly recovers CDC's model, not the world. -- A PLACES association with a determinant the model does not contain (vehicle access, housing cost) can arise only through that determinant's correlation with composition. - -The engine must say this wherever PLACES is used. - -## 2. Engine work - -**B1. Validate the clustered sandwich of `fit_eiv` (F6).** -- **Simulation** (`scripts/eiv_cluster_validation.py`, 500 replications per cell): - - G clusters of m tracts (G = 30, 60 and 150; n ≈ 3,000); - - a cluster-shared component in the true covariate and in the model error (within-cluster correlation 0, 0.2 and 0.5); - - heteroscedastic ACS-type covariate error with synthetic replicates (as in Round 20); - - the outcome either ACS-type, with correlated error and replicates, or external-type, with independent error and no replicates (B3). -- **Acceptance:** - - clustered 90% coverage of 0.87–0.93 for G ≥ 60, reported for G = 30; - - bias removed as in Round 20; - - the independence sandwich's under-coverage under clustering is shown for comparison. -- Result: `docs/audits/eiv_cluster_validation.md`. -- **Manifest:** `args.cluster` takes a frame column or `GEOID.str[:N]`, for example `GEOID.str[:5]` for county, and validation checks the form. - -**B2. `census.health_sources`: external data registry and PLACES connector.** -- **Registry.** `SOURCES["places"]` records: - - the dataset ids per release and level; - - licence and attribution; - - measure ids, units and value type (crude prevalence at tract level); - - geography vintage; - - the source model's covariates, as text and as ACS table families (age-sex B01001, race/ethnicity B02001/B03002, education B15002/B15003, poverty B17001/C17002/B17024, used through the county term); - - the uncertainty type (`model_based`, Monte Carlo 95% interval). -- **Download:** - - Socrata CSV export over plain HTTPS: no key and no app token. - - Filtered to the manifest's states and measures, paged, and sent with the engine's generic User-Agent through `census.http.polite_get`. - - Each page is cached under `.cache/health/places///` with a sidecar recording the URL, SHA-256, size and retrieval time. - - The dataset's `rowsUpdatedAt` is recorded, because Socrata datasets can be revised in place. - - Offline mode (`CENSUS_HEALTH_OFFLINE`) refuses to download. -- **Parser:** - - output is a wide frame with `GEOID`, ``, `_se`, `_ci_lower` and `_ci_upper`; - - SE = (upper − lower) / (2 × 1.96), recorded as an approximation, since Monte Carlo percentile intervals need not be symmetric; - - malformed rows (wrong GEOID length, non-numeric values, duplicates) raise. -- Offline tests use an injected fetcher. - -**B3. Manifest `external` variables.** - - ```json - "external": [{"name": "places_dis", "source": "places", "release": 2025, "measure": "DISABILITY"}] - ``` - -- **Validation:** - - the source, release and measure are in the registry; - - the geography level has a dataset for that release; - - `states` are given; - - names are unique across variables and derivations; - - every ACS release uses the source's tract vintage (2020 tracts: ACS 2020 and later), or `harmonize_tracts_to` equals it; - - `aggregation` is refused; - - derivations may not reference external variables. -- **Use limits.** External variables may appear **only as the outcome** of `fit_eiv`, `fit_replicate` or `fh_regression`. As a covariate, their non-sampling model error would go uncorrected. -- **Run:** - - external data are fetched after the ACS analysis frame and joined on GEOID; - - the run records each file's URL and SHA-256, `rowsUpdatedAt`, matched units, frame units without a value, and source units outside the frame; - - a fetch failure is a blocking `external_unavailable` deviation; - - more than 5% of frame units with complete ACS data lacking an external value is a blocking `external_join_failed` deviation, which signals a vintage mismatch. -- **Fixed in replicate refits.** `fit_eiv` and `fit_replicate` hold external variables fixed (new `fixed_variables` for `fit_eiv`). Their error is independent of the ACS sampling error, so the cross term s_i is zero and their variance enters the residual, which the sandwich captures. Run records and methods state this. -- **Circularity guard.** For an analysis with an external outcome whose formula uses a variable from the source model's covariate tables, `create_manifest` returns a note naming the overlap. `methods.md` always carries the source's model description and the within-county caveat. The citation and data-source statement name the external source. - -**B4. Housekeeping from the Round 20 friction log.** F10: the analysis frame builds its columns in one concat, which removes the pandas fragmentation warnings. F7 (empty age bands) and F8/F9 stay in the backlog. - -## 3. The study: does CDC PLACES see tract disability? - -**Question.** Across Michigan and New York tracts, how far do PLACES model-based disability prevalences agree with the ACS direct survey estimates? And does that agreement survive within counties, and beyond age structure, where only demographic composition drives PLACES? - -This validates the measurement system. It is the PLAN_ROUND_20 section 7.3 pilot, and it tests the connector, the vintage join and the uncertainty treatment on a question where the answer is checkable. - -| Item | Choice | Why | -| :--- | :--- | :--- | -| Units | All MI (26) and NY (36) tracts, 2020 vintage | Test states. PLACES 2025 is on 2020 tracts | -| ACS | 5-year 2024 (2020–2024), `B18101` adult bands (18–34, 35–64, 65–74, 75+, both sexes), Census VRE replicates | Exact replicate SEs; overlaps BRFSS 2023 in time | -| Outcome | `places_dis`: PLACES 2025 `DISABILITY`, crude % of adults with any of six disability types (BRFSS 2023) | The external measure under test | -| Covariate | `acs_dis`: % of civilian noninstitutionalized adults 18+ with any disability (ACS six questions) | Same construct, independent system; crude, as PLACES tract values are crude | -| Composition | `pct65`: share of those adults aged 65+ (same table) | Age is the strongest driver of both measures | -| Estimator | `fit_eiv` (ACS error corrected; the PLACES outcome is external and held fixed), SEs **clustered by county** (`GEOID.str[:5]`, 145 clusters) | B1. PLACES errors share county random effects | -| Reliability | CV screen on `acs_dis` at 30%, flag only | As before | -| Correction | Holm across three hypotheses, 90% intervals | | - -**Hypotheses** (`examples/manifests/r21_mi_ny_places_disability_validation.json`): -- **H1** (m1: `places_dis ~ acs_dis + C(state)`): PLACES disability is higher where ACS disability is higher. `acs_dis`, one-sided `greater`. -- **H2** (m2: `places_dis ~ acs_dis + C(county)`): the association holds **within counties**. `acs_dis`, `greater`. -- **H3** (m3: `places_dis ~ acs_dis + pct65 + C(county)`): it holds within counties **beyond adult age structure**. `acs_dis`, `greater`. - -**Expectations, stated before the data:** -- H1 is close to certain: both measures rise with age. -- H2 is likely, because tract age composition varies within counties. -- H3 is open. Within counties, after age, PLACES varies only with sex, race/ethnicity and education composition, so H3 asks whether those carry ACS-measured disability. -- The two surveys differ in level: BRFSS adult disability is around 28% nationally, against roughly 15% for ACS adults. So the slope is not expected to be 1, and no hypothesis tests a level. - -**Seen before registration:** -- tract `B18101` estimates (Round 20 analysed them, including crude prevalence in E3); -- the PLACES measure list and per-state row counts for the 2025 release. - -No PLACES estimate values have been downloaded or viewed. The cache is checked and recorded. - -## 4. Protocol - -1. B1–B4 with offline tests. B1's simulation passes. Fast gate green. -2. `/code-review high`; fix findings. -3. Write the manifest, run `create_manifest` (records the circularity note), **commit it alone** and push. -4. `run_manifest` live into `studies_output/r21/confirmatory`. -5. Pre-specified **exploratory** analyses (labelled, with the manifest hash), in `scripts/r21_exploratory.py`: - - **E1:** the independence sandwich versus county-clustered, and naive versus corrected slopes. - - **E2:** Round 20's determinant models (m1, m2) refitted with `places_dis` as outcome, on the Round 20 analysed frame. Does PLACES reproduce the ACS determinant pattern, or only the composition-driven part? This is the circularity check in practice. - - **E3:** PLACES `LACKTRPT` (lack of reliable transportation, a health-related social need) against ACS `no_vehicle`. - - **E4:** H1–H3 without New York City, and per state. - - **E5:** maps of `places_dis`, `acs_dis` and their standardized gap. - - **E6:** Moran's I of m3 residuals. -6. Report `docs/studies/r21_mi_ny_places_disability_validation.md`, with the same structure as Round 20, the friction log and the review section (an author self-review, as the owner delegates). - -## 5. Acceptance - -- The manifest is committed before the first PLACES value download. -- The run is `confirmatory`, or `confirmatory_incomplete` with its deviations reported. -- Every reported number traces to the bundle or a hashed exploratory output. -- B1–B4 have offline tests, and B1 meets its criteria. -- The full suite passes on Python 3.14 and 3.10, except the known 3.10 spatial λ test. - -## 6. Out of scope - -Other Step 2 connectors (USALEEP, County Health Rankings, SVI, Food Atlas, HRSA, EPA), which reuse the registry next. Also out of scope: external covariates with model error; vintage apportionment of rates (USALEEP and Food Atlas are on 2010 tracts); and Step 3 count outcomes. - -## 7. Release - -v0.20.0 with a Zenodo DOI, after the review, its fixes, both suites and the report's review section. Study outputs are attached as an archive. - -## 8. Next (Round 22 sketch) - -- **Step 2, continued.** USALEEP life expectancy, on 2010 tracts, needs rate apportionment through the Round 19 crosswalk with population weights. That is a new, documented approximation, validated on tract splits where both vintages exist. County Health Rankings bring a county-level connector. -- **Step 3.** Ingest WONDER exports (PLAN_ROUND_20 section 8): county premature mortality against the Round 20 determinants. - ---- - -## 9. Implementation status - -| Item | Result | -| :--- | :--- | -| B1 | Clustered sandwich validated (`docs/audits/eiv_cluster_validation.md`, 500 per cell): G >= 60 coverage 0.870-0.914, passed; the independence sandwich fell to 0.46-0.88 under clustering. `args.cluster` takes `GEOID.str[:N]` | -| B2 | `census.health_sources` with the CDC PLACES connector (crude-only, deterministic paging, SHA-256 sidecars, revision stamp tied to the download, offline mode) | -| B3 | Manifest `external` variables (outcome-only, single release, vintage check, blocking deviations, circularity notes, methods text); `fit_eiv(fixed_variables=...)` | -| B4 | `analysis_frame` without fragmentation (F10) | -| Review | `/code-review high` before registration: 10 findings fixed (vintage check under `harmonize_tracts_to`, multi-release externals, duplicate measures, paging, revision stamp, cache helper reuse, methods wording) | -| Preregistration | Manifest `11294774...1f76`, committed alone at `2af16e0` (04:34:25Z); first PLACES download 04:34:46Z | -| Results | H1-H3 supported (corrected slopes 0.82, 0.77, 0.82; county-clustered). PLACES levels about 13 points above ACS. E2: PLACES reproduces model inputs (education doubled) but not cost burden, the circularity problem in data | -| Report | `docs/studies/r21_mi_ny_places_disability_validation.md`; friction F1-F7 | - ---- - -# Part 2 (owner request, 2026-11-15): enforce the circularity lesson, then do section 8 - -The owner asked that the MCP state, and the engine enforce, the lesson of the Part 1 study: **determinant studies keep using the ACS's own measure (for disability: B18101), and use PLACES for measures the ACS lacks, such as diabetes**. Then implement the rest of the plan (section 8, and the Part 1 backlog that blocks it) on this branch before merging. - -## 10. What the sources are (verified 2026-11-15) - -- **USALEEP** (NCHS, NAPHSIS, RWJF): tract life expectancy at birth e(0) and its SE for 2010-2015 deaths, on **2010 tracts** (Census 2010 codes). Files per state: `https://ftp.cdc.gov/pub/Health_Statistics/NCHS/Datasets/NVSS/USALEEP/CSV/{ST}_A.CSV` (columns `Tract ID, STATE2KX, CNTY2KX, TRACT2KX, e(0), se(e(0)), Abridged life table flag`). 65,662 tracts (88.8%); tracts under 5,000 people over the period are excluded. New York City has its own file (`NYC_A.CSV`), separate from the rest of New York (found when the first X4 run left 54% of NY tracts unmatched; fixed before any X4 result was read). Maine and Wisconsin were added in a second phase (files exist; an early registry entry wrongly marked them missing). - - **It is partly model-based.** Age groups with missing deaths take **predicted** death rates from a zero-truncated Poisson or negative-binomial model. Its covariates are tract quartiles of median family income, population density, % non-Hispanic Black, % Hispanic and % bachelor's or higher (ACS 2011-2015 and the 2010 census), plus a reservation-area indicator (PRCSDA) and region. - - Abridged life table flag (record layout): 1 = observed rates for all ages; 2 = **predicted for all ages**; 3 = combination. Nationally 4,639 tracts are flag 1, 4,872 flag 2 and 56,915 flag 3 (Arias et al. 2018, Vital Health Stat 2(181)). - - Consequence: a flag-2 tract's e(0) is a function of its income, race/ethnicity and education composition; flag-3 tracts are partly so. The same circularity rule applies as for PLACES. -- **County Health Rankings & Roadmaps (CHR&R)** analytic data, 2025 release: one CSV, `https://www.countyhealthrankings.org/sites/default/files/media/document/analytic_data2025_v2.csv` (796 columns; row 1 names, row 2 codes `vNNN_rawvalue`, `_numerator`, `_denominator`, `_cilow`, `_cihigh`, `_flag`; state and national rows mixed in). - - Mortality measures come from NCHS death records with Census Population Estimates denominators: premature death (v001, YPLL before 75 per 100,000, age-adjusted), premature age-adjusted mortality (v127, deaths under 75 per 100,000, age-adjusted), life expectancy (v147), child (v128) and infant (v129) mortality, drug overdose (v138) and injury deaths (v135); 2020-2022 for the premature measures in the 2025 release. Counties with fewer than 20 deaths are missing. - - Its BRFSS measures (diabetes v060, frequent physical v144 and mental v145 distress, % disability v180) are **PLACES model estimates**: the same circularity, at county level. % disability (v180) has a direct ACS equivalent. - - Terms of use: non-commercial research and public-health use; cite "University of Wisconsin Population Health Institute. County Health Rankings & Roadmaps 2025"; no suggested endorsement. The engine caches the file and does not redistribute it. -- **CDC WONDER**: unchanged from PLAN_ROUND_20 section 8.1. County data only through the web form; the engine reads exported text files the user supplies. - -## 11. Engine work, Part 2 - -**C1. The circularity rule, enforced (validation errors, not notes).** Registry: every measure records `kind` (`direct`, `model_based` or `partly_model_based`), its source model's covariate tables, and its **ACS equivalent** (tables that measure the same construct directly: PLACES DISABILITY and the six disability types: B18101-B18107; CHR v180: B18101; PLACES ACCESS2: B27010; PLACES FOODSTAMP: B22001 at household level). In `create_manifest`: -- **R1. Same construct.** An external outcome whose measure has an ACS equivalent is allowed only in a *measurement-comparison* analysis: the model contains a variable built from the equivalent tables, and every hypothesis on that analysis tests such a term. Otherwise: error, with the rule's text ("use the ACS's own measure, B18101, as the outcome; use the external source for measures the ACS lacks, such as diabetes"). Part 1's manifest passes (it is a comparison). -- **R2. Model inputs.** A hypothesis whose tested term draws on the source model's covariate tables (PLACES: age-sex, race/ethnicity, education, poverty; USALEEP: income, race/ethnicity, education) with an external outcome from that model is an error unless the hypothesis carries a written `circularity_justification`, which the methods text repeats. Controls that overlap stay notes. -- **R3. Composition confound stated.** Every analysis with a `model_based` or `partly_model_based` outcome gets the within-county (PLACES) or covariate-imputation (USALEEP) caveat in methods.md, as now. -- **MCP.** The server instructions, the core profile's instructions, `docs/agent_protocol.md` (new section 16, *External health data*), a recipe `external_health_outcome`, and a new tool `list_health_sources` (registry, measure kinds, ACS equivalents, the rule) state the rule in these words. - -**C2. USALEEP connector (tract).** `SOURCES["usaleep"]` (release 2018, the publication year; data 2010-2015; tract vintage 2010; `partly_model_based`). -- Per-state A files, cached with SHA-256 sidecars; GEOIDs converted from Census 2010 codes to ACS codes (the existing `census2010_to_acs_geoid`). -- Parser: ``, `_se`, and `_flag`. **Flag 2 (fully predicted) tracts are dropped by default** (`"include_predicted": true` keeps them, and R2 then applies to every tested term); the drop is recorded in run.json. -- **Rate apportionment to 2020 tracts** (`"apportion_to": 2020` on the entry): value_t = sum_s m_st v_s / sum_s m_st, with m_st the 2010 block population in source tract s and target tract t (block-weight crosswalk, Round 13). SE: sqrt(sum_s a_st^2 se_s^2) with a_st = m_st / sum m_st (independent tract estimates). A target needs at least 90% of its population mass covered by sources with a value, else missing. This is an approximation (life expectancy is not additive; a split tract inherits its parent's value), recorded in methods. - - **Validation:** apportion ACS 2015-2019 crude disability prevalence (2010 tracts) to 2020 tracts and compare with ACS 2016-2020 (2020 tracts) for MI and NY: z-scores of the difference for 1:1, split and merged tracts (`docs/audits/rate_apportionment_validation.md`). Acceptance: median |z| for split and merged tracts no worse than 1.25 times the 1:1 tracts'. - - **Result (2026-11-15): FAILED for split tracts** (median |z| ratio 1.33; merged 1.05). Consequence: `split: "missing"` is the default (split children are left missing); `split: "inherit"` is available and documented as not validated. A post hoc breakdown suggests children holding 95% or more of the parent's population behave like 1:1 tracts; not adopted, since it was found after the test. -- Join tolerance per source: USALEEP leaves tracts under 5,000 people unpublished (11.2% nationally), and flag-2 tracts are dropped, so its `external_join_failed` limit is 25% (registry `join_missing_limit`; 5% for PLACES, CHR&R and WONDER); the dropped counts are reported in run.json. -- R2 applies to USALEEP outcomes whether or not flag-2 tracts are kept (flag-3 tracts are partly predicted too). - -**C3. County Health Rankings connector (county).** `SOURCES["chr"]` (release 2025, county only; `direct` mortality measures; `model_based` BRFSS measures with PLACES covariates and ACS equivalents where they exist). -- The file is cached with a sidecar; HTTP `Last-Modified` and `ETag` are recorded, since CHR&R reissues files. -- Parser: county rows only; ``, `_se` = (cihigh - cilow) / (2 x 1.96), `_numerator`, `_denominator`; flag 2 (suppressed) is missing and flag 1 (unreliable) is recorded. -- Measures the ACS itself supplies (CHR's ACS-derived columns) are not registered: the error says to use the ACS table. - -**C4. WONDER export reader (Step 3).** `census.vital_records`: -- `read_wonder_export(path)` parses the tab-delimited export, the footer's dataset and query parameters, and the `Suppressed`, `Unreliable` and `Not Applicable` markers. Suppressed counts are kept as censored [0, 9], never zero. It refuses files without the footer (edited or partial). -- Crude rates get exact Poisson (Garwood) intervals; age-adjusted rates use WONDER's own SE when exported, else Fay-Feuer intervals need age-specific counts and are refused. -- Manifest `external` entries with `"source": "wonder"` take `"file"` and `"sha256"` (the export is pinned by hash; a different file is a validation error at run time). - -**C5. Delete-one-cluster jackknife for `fit_eiv` (Part 1 F3).** -- `args.cluster_se`: `"cr1"` (default) or `"jackknife"` (CR3: delete-one-cluster refits from the per-cluster moment sums; exact, no re-estimation of covariances). -- Validated in `scripts/eiv_cluster_validation.py` beside CR1 (same cells). Acceptance: coverage of 0.87-0.93 at G >= 60 and at least CR1's at G = 30. - -**C6. Diverging map scheme (Part 1 F1).** `make_map(..., scheme="diverging")`: symmetric breaks around 0 (or `center=`) with a red-blue palette, for residuals and gaps. - -**C7. `run_variant` (Part 1 F2, Round 20 F8).** Exploratory refits of a completed run without Python: `run_variant(run_dir, analysis, formula=None, subset=None, cluster=None, cluster_se=None)` checks the analysed files' SHA-256 against run.json, refits, and writes the result labelled `exploratory` with the manifest hash. `subset` keeps or drops GEOID prefixes. Exposed as an MCP tool. - -F4 (spatial HAC) and F5 (Round 20 age bands, NYC insets) stay in the backlog. - -## 12. Study (Part 2): county premature mortality and the Round 20 determinants - -The section 8 Step 3 pilot, with a direct outcome that has no model inputs: does the Round 20 determinant pattern for disability carry over to a hard outcome? - -| Item | Choice | Why | -| :--- | :--- | :--- | -| Units | All MI (83) and NY (62) counties | Test states; the outcome is county-level | -| Outcome | `prem_mort`: CHR&R 2025 premature age-adjusted mortality (v127), deaths under 75 per 100,000, NCHS 2020-2022 | Direct vital records; age-adjusted; no ACS inputs (Population Estimates denominator) | -| ACS | 5-year 2022 (2018-2022), county, Census VRE replicates | Period overlaps 2020-2022 | -| Determinants | `pov_rate`, `pct_bachelors`, `no_vehicle`, `cost_burden`, `asdp` (age-standardized disability, all ages, 2000 standard), built as in Round 20 | Same constructs as Round 20 | -| Estimator | `fit_eiv`, outcome held fixed (independent error); units independent (no clustering unit with >= 20 groups above the county) | | -| Correction | Holm across four hypotheses, 90% intervals | | - -**Hypotheses** (`examples/manifests/r21b_mi_ny_county_premature_mortality.json`), all `greater`: -- **H1** (m1: `prem_mort ~ pov_rate + C(state)`): premature mortality is higher where poverty is higher. -- **H2** (m2: `prem_mort ~ pov_rate + pct_bachelors + no_vehicle + cost_burden + C(state)`): conditional on the others, higher where more households lack a vehicle. -- **H3** (m2): conditional on the others, higher where more households are cost-burdened. -- **H4** (m3: `prem_mort ~ asdp + C(state)`): higher where age-standardized disability prevalence is higher. - -**Expectations stated before the data.** H1 and H4 are likely. H2 and H3 are open: among counties, no-vehicle and cost-burden shares are highest in New York City, whose premature mortality is not the highest in New York, and the county count (145) is small. - -**Circularity.** None by construction; `create_manifest` must report no R1 or R2 finding. This is the worked example of the rule. - -**Seen before registration:** the CHR&R file's header rows (names and codes) only. No CHR&R values, no county ACS values for these variables. - -**Exploratory (labelled, `scripts/r21b_exploratory.py`):** -- **X1:** the same models with v001 (YPLL) and v147 (life expectancy, sign reversed). -- **X2:** jackknife versus CR1 standard errors on Part 1's H1-H3 (C5). -- **X3:** Part 1 E2 re-run with the crude adult ACS measure (`acs_dis`), the follow-up the Part 1 report asked for. -- **X4:** USALEEP tract life expectancy against Round 20 determinants on ACS 2011-2015 (native 2010 tracts, flag-2 tracts dropped, county-clustered), the first tract-level mortality look. Tested terms on education overlap the USALEEP model (R2), so it is reported with that caveat. -- **X5:** a map of m2 residuals (diverging scheme) and Moran's I. - -## 13. Protocol, acceptance and release - -- C1-C7 come with offline tests; C2's apportionment validation and C5's simulation pass; then `/code-review high` and its fixes. -- Then the manifest is committed alone, run, the exploratory analyses run, and the report written (`docs/studies/r21b_mi_ny_county_premature_mortality.md`, author self-review). -- Both suites run. The release stays **v0.20.0** (not yet tagged), covering Parts 1 and 2; the merge waits for the owner. - -## 14. Part 2 implementation status - -| Item | Result | -| :--- | :--- | -| C1 | Rule enforced: R1 (same construct) and R2 (tested model inputs) are `create_manifest` errors. MCP: instructions, core profile line 7, protocol section 16, recipe `external_health_outcome`, tool `list_health_sources` | -| C2 | USALEEP connector (NYC file, flag-2 drop). Rate apportionment validation **FAILED** for split tracts (1.33 > 1.25; merged 1.05), so split targets are missing by default | -| C3 | CHR&R 2025 connector (direct mortality; PLACES-based flagged; ACS-derived refused; per-measure source text) | -| C4 | `census.vital_records` WONDER reader (footer required, SHA-256 pinned and checked at registration, censored suppression, Garwood intervals) | -| C5 | `cluster_se="jackknife"`: validation PASSED (G >= 60 coverage 0.872-0.916; at G = 30, mean 0.891 against CR1's 0.880) | -| C6, C7 | `scheme="diverging"`; `run_variant` (MCP tool) | -| Review | `/code-review high`: 10 findings fixed before registration | -| Preregistration | Manifest `8df6b790...70e2` committed alone at `15ffb9e` (16:41:51Z); first CHR&R download 16:42:29Z | -| Results | H1 (poverty +13.0) and H4 (disability +22.4) supported; H2 (no vehicle, −0.90) and H3 (cost burden, +0.82) not supported. Without NYC, no-vehicle is +5.8 (exploratory). X3 corrects Part 1 E2 (education 1.4×, not 2×; cost burden confirmed absent in PLACES). X4: USALEEP life expectancy −0.19 years per poverty point | -| Report | `docs/studies/r21b_mi_ny_county_premature_mortality.md`; friction F1-F6; Part 1 addendum (section 12) | diff --git a/PLAN_ROUND_22.md b/PLAN_ROUND_22.md deleted file mode 100644 index a47fcd3..0000000 --- a/PLAN_ROUND_22.md +++ /dev/null @@ -1,611 +0,0 @@ -# Round 22: Count outcomes, spatial standard errors and a low birth weight study - -Status: **implemented and locally validated; PR creation blocked by connected GitHub integration permissions (403)** (written 2026-11-15, after v0.20.0 / DOI 10.5281/zenodo.24260044). -Implementer: an external agent. This plan is written to be followed literally: do what it says, in the order -it says, and nothing else. Where a decision is not covered here, choose the smallest change that satisfies -the acceptance criteria, and record the decision in section 9 (implementation status). Do not redesign -anything that is not listed. - ---- - -## 0. Ground rules (read first; these are owner requirements, not suggestions) - -1. **Branch.** Work on a new branch `phase22/count-outcomes` created from `phase22/plan` (this plan, one - commit on top of `defaultBranch` merge commit `07f58c6`, tag `v0.20.0`). The default branch is called `defaultBranch` (not `main`). -2. **No merging, no tags, no releases.** Push the branch and open one pull request against `defaultBranch`. - The owner decides about merging. Do not create tags or GitHub releases. -3. **No AI attribution anywhere.** Commits and the PR body must not contain `Co-Authored-By`, - `Generated with`, or any similar line, even if your tooling suggests one. -4. **Secrets.** The Census API key is in `.env` (`CENSUS_API_KEY`). Never print it, log it, commit it or put it in - any output file. Before committing any output folder, check that the key string does not occur in it (read - it with `dotenv_values`, search the bytes, print only the count of hits). -5. **No email or personal data** in User-Agent strings or requests. Use a generic client name. -6. **Do not automate CDC WONDER** (its terms forbid it) and do not circumvent bot checks or paywalls. This - round needs no new download source: everything comes from the Census API, TIGER boundary files (already - supported by `census/boundaries.py`) and the County Health Rankings file that is already cached at - `.cache/health/chr/2025/analytic_data2025_v2.csv`. -7. **Circularity rule (owner rule; already enforced as R1/R2 by `census/manifest.py:circularity_errors`).** - Determinant studies use the ACS's own measure where the ACS has one; model-based external estimates - (CDC PLACES, County Health Rankings BRFSS measures) are used only for constructs the ACS lacks, and never - tested against their model's inputs. Do not weaken or bypass R1/R2. The study in this plan uses a direct - vital-records outcome (low birth weight) that the ACS does not measure, so R1/R2 do not fire; if they fire, - something is wrong in your registration, not in the rule. -8. **Python.** Use `.venv/bin/python` (there is no `python` command). Supported versions are Python 3.11+ - with SciPy 1.16+. Run the test suite only on the project venv. Do not create other venvs, and do not run - GitHub CI. -9. **No new dependencies.** Everything below is numpy/scipy/pandas/statsmodels/geopandas code that already - exists in `requirements.txt`. Do not add PySAL estimators for this; do not add packages. -10. **No new MCP tools.** The tool count stays at 63. New features are new arguments of existing functions, - documented in the existing tool docstrings and instructions. -11. **Pre-registration discipline (section 5).** You must not compute, plot or look at any association between - low birth weight and the ACS determinants before the study manifest is committed and pushed. Looking at the - outcome's own distribution and missingness is allowed (and needed to check feasibility). -12. **Pre-set criteria stay fixed.** The validation criteria in A2 and A4 are fixed here. If a validation fails, - report FAILED, apply the stated consequence, and do not change the criterion, the simulation design or the - seeds to make it pass. A post hoc analysis, clearly labelled post hoc, is allowed after the FAILED result. - -**What the planner already saw (disclosure for the study's preregistration).** While writing this plan, the -planner looked only at the outcome's coverage in the cached file: 145 Michigan and New York counties; low -birth weight (`v037`) numerator missing for 1 county; median numerator 297 low-weight births; median -denominator 3,917 births. No association with any determinant was computed. Copy this paragraph into the -study report's preregistration section. - ---- - -## 1. Why this round - -- Round 21 Part 2 (county premature mortality) left a backlog in its friction log - (`docs/studies/r21b_mi_ny_county_premature_mortality.md`, section 10): - - F3: no valid clustering level for a two-state county study, with residual Moran's I 0.195. Standard - errors assume independence and are probably too small. - - F5: `run_manifest` labels a run of an exploratory manifest "confirmatory". - - F6: rate apportionment of split tracts failed validation. A post hoc split by parent-population share - suggested a rule worth re-testing on held-out states. -- The public-health roadmap (PLAN_ROUND_20 section 8.2) calls for count/proportion outcomes from vital - records, with overdispersion handled. County Health Rankings 2025 publishes numerators and denominators for - its vital-records measures, so a count model is possible without any new download. -- The owner's research focus is public health (disability, housing cost burden, vehicle access, social and - economic determinants), not insurance. Low birth weight is a direct vital-records outcome the ACS does not - measure, and fits the circularity rule. - -## 2. Scope - -In scope, in this order: - -| Id | Work | Section | -| :--- | :--- | :--- | -| A1 | Manifest flag `"exploratory": true` (friction F5) | 3.1 | -| A2 | Spatial HAC (Conley) standard errors for `fit_eiv` and for `fit_replicate` binomial/Poisson GLMs, with a simulation validation | 3.2 | -| A3 | External count/proportion outcomes in `fit_replicate` (binomial with trials, Poisson with exposure, quasi-likelihood scale), register CHR `v037` low birth weight | 3.3 | -| A4 | Pre-registered re-test of split-tract apportionment on held-out states; new `split: "dominant"` policy only if it passes | 3.4 | -| S | Study: low birth weight and the Round 20 determinants, Michigan and New York counties | 5 | - -Out of scope (do **not** do these, even if they look easy): - -- Measurement-error correction (EIV, corrected score, regression calibration) for GLMs. Covariate sampling - error in GLMs is reported through `se_sampling` from replicates only, as `fit_replicate` already does. -- Negative binomial models for external outcomes, BYM/ICAR or other spatial random-effects models, empirical - Bayes smoothing. -- New external sources or connectors; WONDER automation; any change to PLACES/USALEEP connectors except - what A4 requires. -- Map insets, empty age bands, pandas performance, refactors, renames, formatting sweeps. -- Changes to the circularity rule, to existing validation criteria, or to existing study outputs/reports - (other than the friction-log status column of `docs/studies/r21b_*.md`, see 6.3). - -## 3. Engine work - -Before writing code, read these to understand existing patterns (do not modify them beyond what is listed): -`census/manifest.py` (`validate_manifest` ~line 815, `circularity_errors`, `create_manifest` ~1224, -`_run_analysis` ~1393, `run_variant` ~1484, `run_manifest` ~1560 and the methods-text builder near ~2496/2796), -`census/models.py` (`_fit`, `_prepare`, `fit_replicate` ~371, `_cluster_jackknife` ~712, `fit_eiv` ~779), -`census/health_sources.py` (`_CHR_MEASURES`, `parse_chr`, `join_external`, `apportion_rates`, -`SPLIT_POLICIES`), `scripts/eiv_cluster_validation.py` and `docs/audits/eiv_cluster_validation.md` (the -validation pattern to copy), `scripts/rate_apportionment_validation.py` and -`docs/audits/rate_apportionment_validation.md`. - -**Important existing trap:** in `census/manifest.py`, the decorator `@integrity_errors` must stay directly on -`run_manifest`. Do not insert new functions between that decorator and `run_manifest` (this broke a test in -Round 21). - -### 3.1 A1: exploratory manifests - -Behaviour: - -- New optional top-level manifest key `"exploratory"` (boolean, default `false`). Any other type is a - validation error: `exploratory must be true or false`. -- When `true`, `run_manifest` sets `analysis_status` to `STATUS_EXPLORATORY` (`"exploratory"`) in `run.json`, in - the result dict it returns, and in every per-analysis provenance tag. Deviations are still detected and - recorded exactly as now, but they do not turn the status into `confirmatory_incomplete`. -- The methods text gets one sentence at the start: "This analysis was registered as exploratory (manifest - flag `exploratory: true`); its results are not confirmatory." Hypothesis results are still reported, with - the column/label "exploratory" instead of "confirmatory". -- `create_manifest` writes the flag through unchanged and adds a note "registered as exploratory". -- The flag is part of the manifest, so it is covered by the manifest hash. Nothing else changes. - -Tests (`tests/test_round22.py`, class `TestExploratoryFlag`): flag true → status "exploratory" in returned -dict and `run.json` and methods text sentence present; flag absent → unchanged behaviour (existing tests -must still pass); flag `"yes"` → validation error; a blocking deviation in an exploratory run → status still -"exploratory", deviation recorded. Use the same offline fixtures/mocks the existing manifest tests use -(look at `tests/test_round21.py` and `tests/test_round21_part2.py` for how runs are faked offline). - -### 3.2 A2: spatial HAC (Conley) standard errors - -New module `census/spatial_hac.py` (numpy only), with: - -```python -EARTH_RADIUS_KM = 6371.0088 - -def haversine_matrix(lon, lat) -> np.ndarray: - """n x n great-circle distances in km (inputs in degrees).""" - -def conley_meat(scores: np.ndarray, dist_km: np.ndarray, cutoff_km: float) -> np.ndarray: - """M = sum_i sum_j K(d_ij) s_i s_j', K(d) = max(0, 1 - d / cutoff) (Bartlett). scores is n x k.""" - -def conley_covariance(scores, bread_inv, dist_km, cutoff_km) -> tuple[np.ndarray, dict]: - """V = bread_inv @ M @ bread_inv * n / (n - k). Returns (V, info) where info has - {"kernel": "bartlett", "cutoff_km": c, "n": n, "mean_neighbours": mean count of j != i with d_ij < c, - "min_eigenvalue": smallest eigenvalue of V, "psd": bool}.""" -``` - -Rules: - -- Kernel: Bartlett only. Cutoff must be a finite number > 0, else `ValueError`. -- The radial Bartlett kernel is not guaranteed positive semi-definite in two dimensions. If any diagonal - element of V is ≤ 0, raise `ValueError("Conley covariance is not positive at cutoff ...")`. If V merely has a - negative eigenvalue but positive diagonal, keep it and set `psd: False` (reported, not fatal). -- Small-sample factor n/(n − k), k = number of coefficients. No other adjustment. - -Coordinates: - -- New function `census/spatial_hac.py:unit_coordinates(geoids, level, acs_year)`. It returns a frame - `GEOID, lon, lat` from TIGER cartographic boundary files via the existing - `census.boundaries.get_boundaries(level, boundary_vintage(acs_year, level), state)` for each state in the - GEOIDs. Use `representative_point()` after projecting to EPSG:4269 (as `census/boundaries.py:626` does). Missing - GEOIDs → `ValueError` listing up to 10. -- It also returns the boundary files' SHA-256 values (the boundary cache already keeps sidecars; reuse them, - do not recompute if a sidecar exists) so `run_manifest` can record them in provenance. -- Supported levels: `county` and `tract`. Anything else → `ValueError`. - -`fit_eiv` (census/models.py): - -- New keyword `spatial_se: Optional[Mapping] = None`, e.g. `{"cutoff_km": 150}`, plus keyword - `coords: Optional[pd.DataFrame] = None` (GEOID, lon, lat). `spatial_se` requires `coords` and is mutually - exclusive with `cluster` (`ValueError` if both). -- Use the same per-observation moment contributions that the CR1 path already sums per cluster. The Conley - meat replaces the cluster meat; the bread is unchanged. Do not change the CR1 or jackknife code paths. -- Diagnostics: `diagnostics["spatial_se"] = info` (from `conley_covariance`), plus - `diagnostics["se_independent"]`, the existing independence SE, for comparison. Add a note: - "Spatial HAC (Conley) SEs, Bartlett kernel, cutoff C km". The interval/p-value use the Conley SE in place of - the independence SE (same place where the cluster SE is used today). - -`fit_replicate` (census/models.py), GLMs only: - -- Same two keywords. Allowed only for `model` in (`binomial`, `poisson`); for other models raise - `ValueError("spatial_se is supported for binomial and poisson models")`. -- Scores and bread at the full-sample fit (logit/log links, statsmodels defaults): - - binomial with trials t_i and fitted μ_i: score s_i = x_i t_i (y_i − μ_i); bread A = Σ t_i μ_i (1 − μ_i) x_i x_iᵀ. - - Poisson with exposure: s_i = x_i (y_i − μ_i); A = Σ μ_i x_i x_iᵀ. - - V = A⁻¹ M A⁻¹ · n/(n − k). The sandwich does not use the dispersion scale (it is robust to overdispersion). -- The Conley SE replaces `se_model` (the "population" target). `se_sampling` from the replicates is - unchanged; `se_total` is recomputed from the new `se_model`. Record `diagnostics["se_model_unadjusted"]` - (the GLM model SE) and `diagnostics["spatial_se"]`. - -Manifest (`census/manifest.py`): - -- `fit_eiv` allowed args gain `spatial_se`. `fit_replicate` gets an explicit allowed-args check (it has none - today). The allowed set is the union of what existing code passes through plus the new keys. Before writing - it, collect the existing args with - `grep -rn "fit_replicate" tests/ examples/ docs/recipes/ census/recipes.py` and the manifests in - `examples/manifests/`. Then run `validate_manifest` on every file in `examples/manifests/*.json` to confirm - none breaks. -- `spatial_se` value: an object with exactly one key `cutoff_km` (number > 0). Anything else → validation error. -- When any analysis has `spatial_se`, `run_manifest` calls `unit_coordinates` once for the analysed units and - passes `coords`. The provenance records level, boundary year, files and SHA-256, and the cutoff. -- `run_variant` must accept `spatial_se` in `args` (so exploratory cutoffs can be refitted). Check that - its existing argument validation passes the new key. -- Methods text: one sentence per analysis with spatial SEs: "Standard errors are spatial HAC (Conley 1999) - with a Bartlett kernel and a C km cutoff, using great-circle distances between unit representative points - (TIGER/Line cartographic boundaries, year Y); validated in docs/audits/spatial_hac_validation.md." - Cite: Conley, T. G. (1999). GMM estimation with cross sectional dependence. Journal of Econometrics, 92(1), - 1–45. - -Exact-identity unit tests (`tests/test_round22.py`, class `TestConley`; offline, synthetic coordinates): - -1. **Independence limit.** With a cutoff below the minimum pairwise distance, `conley_meat` equals - Σ s_i s_iᵀ (HC0 meat) to 1e-12. -2. **Cluster limit.** Put the units of each of G groups at one identical point, with groups ≥ 10 × cutoff apart. - Then `conley_meat` equals the CR0 cluster meat Σ_g (Σ_{i∈g} s_i)(Σ_{i∈g} s_i)ᵀ to 1e-10. Do this - for `fit_eiv` (compare against the meat its CR1 path builds, before CR1's own small-sample factor) - and for the binomial `fit_replicate` path. -3. Haversine against two known distances: (0,0)–(0,1°) = 111.195 km ± 0.01; Detroit - (−83.0458, 42.3314) to New York City (−74.0060, 40.7128) ≈ 775 km ± 5. -4. Errors: cutoff ≤ 0; `spatial_se` together with `cluster`; `spatial_se` with `model="ols"` in - `fit_replicate`; a non-positive diagonal. -5. Manifest: `spatial_se` validation messages; `run_variant` accepts it. - -Simulation validation (`scripts/spatial_hac_validation.py` → `docs/audits/spatial_hac_validation.md`; -copy the structure of `scripts/eiv_cluster_validation.py` and its audit): - -- Locations: the 145 real Michigan and New York county representative points (2020 cartographic boundary - files via `unit_coordinates`, cached after the first run). -- Spatial fields: a Gaussian field with exponential covariance exp(−d / r), drawn by Cholesky of the - 145×145 matrix (add 1e-10 to the diagonal), with range r ∈ {0 (independent), 50 km, 100 km}. x and the error - are independent fields with the same r. -- Linear design (Conley used through the `fit_eiv` OLS sandwich path with no measurement error; if - `fit_eiv` cannot run without replicates, call the internal sandwich function directly and say so in the - audit): y = 1 + 0.5 x + e, with x and e at unit variance. -- Binomial design: trials t_i drawn once (seeded) from a lognormal with median 3,900 and log-sd 1.0, rounded, - minimum 50; logit p_i = −2.6 + 0.3 x_i + u_i with u a spatial field (same r) of SD 0.15 (overdispersion); - y_i ~ Binomial(t_i, p_i)/t_i. -- Estimators per replicate: independence SE (OLS HC0 / GLM model SE with scale "X2") and Conley at cutoffs - 50, 100, 150, 200 km. Target: the slope (0.5 or 0.3). Nominal 90% Wald intervals. -- 2,000 replicates per (design, r) cell, `numpy.random.default_rng(20261122 + cell_index)`. Report - coverage, its Monte Carlo SE, mean SE / empirical SD, and the share of fits with `psd: False`. -- **Acceptance (pre-set, do not change): at the default cutoff 150 km, coverage ≥ 0.86 in all six cells - (2 designs × 3 ranges).** -- **If PASSED:** manifests may use `spatial_se`; the recommended default in docs is 150 km for county studies. -- **If FAILED:** keep the functions and arguments for exploratory use (`run_variant`, direct calls), but - `validate_manifest` refuses `spatial_se` in confirmatory manifests (allowed when `"exploratory": true`) with - the message "spatial_se failed its validation (docs/audits/spatial_hac_validation.md); exploratory only". - The study (section 5) is then registered without spatial SEs. -- Runtime: precompute the Cholesky factor once per r; the whole script should run in minutes. It may use the - network only for the boundary files. - -### 3.3 A3: external count and proportion outcomes in `fit_replicate` - -Registry (`census/health_sources.py`, `_CHR_MEASURES`): add - -```python -"v037": {"label": "Low birth weight: share of live births under 2,500 grams", "kind": "direct", - "unit": "proportion of live births (0-1)", "count": {"numerator": "low birth weight births", - "denominator": "live births"}}, -``` - -and the per-measure source text that the other CHR vital-records measures use, with the data source -"National Center for Health Statistics, Natality files (NVSS), 2017–2023". Before writing it, confirm the years -against the CHR 2025 low birth weight page -(https://www.countyhealthrankings.org/health-data/population-health-and-well-being/quality-of-life/physical-health/low-birth-weight) -and the 2025 data dictionary (https://www.countyhealthrankings.org/sites/default/files/media/document/DataDictionary_2025.pdf). -If the cached `analytic_data2025_v2.csv` documents different years, use the file's years and record the -discrepancy in section 9. Do not register any other new measure. - -Note: the CHR file's `v037_rawvalue` is already a proportion (e.g. 0.0705). `parse_chr` already produces -`_numerator` and `_denominator`, and `join_external` already joins them. Check this in a test. -Do not re-implement either. - -`fit_replicate` / `_fit` (census/models.py): - -- New keyword `scale: Optional[str] = None`. `None` keeps today's behaviour exactly. `"X2"` passes - `scale="X2"` to the statsmodels GLM `.fit()` (quasi-binomial / quasi-Poisson: SEs multiplied by the square - root of Pearson χ²/df). It is allowed only with `binomial` and `poisson`, else `ValueError`. -- For `binomial` and `poisson`, always record `diagnostics["pearson_dispersion"]` = Pearson χ² / residual df - of the full-sample fit, whether or not `scale` is set. Add a note when it exceeds 1.5 and `scale` is not "X2" - and `spatial_se` is not set: "Pearson dispersion D: model SEs assume no overdispersion; use scale='X2'". -- Replicate refits use the same `scale` (it does not change point estimates, so `se_sampling` is unaffected; - check this in a test). - -Manifest rules (`validate_manifest`), for a `fit_replicate` analysis whose outcome is an external variable: - -1. `args.model` must be `ols`, `binomial` or `poisson`. `negbin` and `wls` with an external outcome are - refused with a message naming this plan's out-of-scope reason ("not validated for external outcomes"). -2. `binomial` needs `args.trials` naming `_denominator`, where `` is that analysis's - external outcome. The effective-trials fallback (from an SE) is refused for external outcomes. - `poisson` needs `args.exposure` naming `_denominator`, and the outcome must be the count, so - for Poisson the formula outcome is `_numerator`. Allow the `_numerator` and `_denominator` - column names of a declared external variable in formulas/args only in exactly these two roles. - Everywhere else they stay unknown names. -3. `binomial` or `poisson` with an external outcome must set `args.scale: "X2"` or `args.spatial_se` (both - allowed). Otherwise refuse: "overdispersion would be ignored: set scale 'X2' or spatial_se". -4. Units whose outcome is missing (suppressed or absent) are dropped by complete-case, as now. They count - toward the existing join-missing limit. Low birth weight's "unreliable" flag (numerator < 20) is not a - suppression: keep those units, and report their count in the methods text ("N counties had fewer than 20 - low-weight births (flagged unreliable by CHR&R); they were kept, weighted by their births"). -5. R1/R2 (`circularity_errors`) apply unchanged. Add a test that `v037` with poverty, education, vehicle and - cost-burden terms raises no R1/R2 error. Add a test that CHR `v060` (diabetes, PLACES-based) tested - against `pov_rate` still raises R2. - -Methods text for these analyses: "Coefficients were estimated by quasi-binomial regression (logit link, -statsmodels GLM, dispersion from Pearson χ²) of the low birth weight proportion with live births as trials" -(Poisson variant analogous: "quasi-Poisson regression of counts with the denominator as exposure"). Report -coefficients on the logit scale and add an odds-ratio column (`exp(estimate)` and exp of the interval ends) -to the hypotheses table in the bundle for binomial analyses. Do not change the table format for other models. - -Tests (`tests/test_round22.py`, class `TestCountOutcomes`): scale None vs "X2" (same estimates, SEs ratio = -sqrt(dispersion)); `pearson_dispersion` recorded; overdispersion note; each manifest rule above (accept and -refuse); `v037` registry entry, kind "direct", source text; `parse_chr` on a small synthetic CHR-format file -yields `lbw`, `lbw_numerator`, `lbw_denominator`; an end-to-end offline manifest run with a synthetic external -binomial outcome (follow the offline run pattern of `tests/test_round21_part2.py`). - -### 3.4 A4: held-out re-test of split-tract apportionment - -Pre-registered hypothesis (record it in the audit before running, and commit the script change and this -hypothesis before the run): **split targets that hold at least 95% of their parent tract's 2010 population -behave like 1:1 targets.** - -- Extend `scripts/rate_apportionment_validation.py` with `--parent-share-min 0.95` and `--out`. Do not change its - default behaviour or its existing output file. -- Held-out states: **Ohio (39) and Pennsylvania (42)**. Do not use Michigan or New York (they generated the - hypothesis). Same data and method as the Round 21 run: ACS 2015–2019 B18101 crude disability on 2010 - tracts apportioned to 2020 tracts, against ACS 2016–2020 native. -- Output: `docs/audits/rate_apportionment_retest.md` (a new file; the Round 21 audit stays unchanged). -- **Acceptance (pre-set): median |z| of split targets with parent share ≥ 0.95, divided by the median |z| of - 1:1 targets, ≤ 1.25 (the Round 21 criterion).** Also report the other share bands and merged/complex - targets, as in Round 21. -- **If PASSED:** add `"dominant"` to `SPLIT_POLICIES` in `census/health_sources.py`. `apportion_rates(..., split="dominant")` - gives a split target its parent's value only when the target holds ≥ 0.95 of the parent's population mass, - else missing. Allow `split: "dominant"` for USALEEP `apportion_to` in manifests, with methods text citing the - re-test. **The default stays `"missing"`.** Add tests for the new policy. -- **If FAILED:** no engine change. Record the result in the audit and in section 9. - ---- - -## 4. Documentation and MCP (after A1–A4, before the study) - -- `mcp_server.py` INSTRUCTIONS (the "External health data..." block) and the docstrings of `create_manifest` - and `run_variant` in `census/workflows_inference.py`. Add, in at most 6 short lines: - - the `exploratory` flag; - - `spatial_se` (with the validation result and 150 km default for county studies, or "exploratory only" if - A2 failed); - - count outcomes (binomial with `_denominator` trials and `scale: "X2"`). - - Keep the existing circularity text verbatim. -- `census/mcp_profiles.py` CORE_INSTRUCTIONS: no change unless a line is now false. -- `docs/agent_protocol.md`: in section 16, add a subsection "Count and proportion outcomes" (the A3 rules) and - one paragraph on spatial standard errors. Do not renumber sections. -- `docs/recipes/external_health_outcome.md` and the matching `census/recipes.py` entry: add a binomial - example (the m1 analysis of section 5). -- README "External health data" section: one bullet each for count outcomes and spatial SEs. -- CHANGELOG: new heading `## 0.21.0 (Round 22): count outcomes and spatial standard errors` with bullets - for A1–A4 and the study. Bump the version to `0.21.0` in `census/__init__.py` (`__version__`), - `census/pipeline.py` (`PACKAGE_VERSION`), and `CITATION.cff` (`version`, `date-released` = the day you - finish). `tests/test_packaging.py` checks they match. Do not tag. - ---- - -## 5. Study: low birth weight and the Round 20 determinants, Michigan and New York counties - -### 5.1 Order (strict) - -1. A1–A4 are done, with tests passing and the A2 and A4 validations run and committed. -2. Write the manifest `examples/manifests/r22_mi_ny_county_low_birth_weight.json`. Run `create_manifest` on - it (offline validation plus resolution), and commit and push it. The commit message must include the - manifest SHA-256 (see the commit `15ffb9e` of Round 21 for the format: "Preregister Round 22 study: ... - (manifest SHA-256 )"). -3. Only then call `run_manifest`. -4. Then run the exploratory analyses (5.5), write the report (5.6), and update the friction log. - -### 5.2 Data - -- Outcome: County Health Rankings & Roadmaps 2025, measure `v037` (low birth weight), external name `lbw`, - source `chr`, release 2025, county level. Numerator: low-weight births; denominator: live births; NCHS - natality 2017–2023 (or as confirmed in A3). -- Determinants: ACS 5-year 2018–2022 (references `2022:...`), the same as Round 21 Part 2. -- Geography: `{"level": "county", "states": ["26", "36"]}`; replicates `{"source": "census_vre"}`; - `confidence_level` 0.90; correction Holm. -- Variables, derivations and the `cv_screen` exclusion (drop `asdp` from its variable list): copy verbatim from `examples/manifests/r21b_mi_ny_county_premature_mortality.json` - the definitions of `pov_rate`, `pct_bachelors`, `no_vehicle` and `cost_burden`, with all the raw variables - they need. Do **not** copy the B18101 disability variables or `asdp` (not used). Keep the variable names - identical so results are comparable with Round 21 Part 2. - -### 5.3 Analyses - -Use `fit_replicate` (not `fit_eiv`, which is linear only): - -| id | formula | args | -| :--- | :--- | :--- | -| m1 | `lbw ~ pov_rate + C(GEOID.str[:2])` | `{"model": "binomial", "trials": "lbw_denominator", "scale": "X2", "target": "population"}` | -| m2 | `lbw ~ pov_rate + pct_bachelors + no_vehicle + cost_burden + C(GEOID.str[:2])` | same as m1 | - -If A2 PASSED, add `"spatial_se": {"cutoff_km": 150}` to both analyses' args (keep `"scale": "X2"`; the -Conley sandwich ignores the scale, but the rule in 3.3 is satisfied either way). If A2 FAILED, do not add -it, and say in the report's limitations that the SEs assume spatial independence. - -### 5.4 Hypotheses (all `"statistic": "coefficient"`, `"direction": "greater"`, Holm across H1–H3, 90%) - -- **H1 (m1, term `pov_rate`).** "Across Michigan and New York counties, the County Health Rankings & - Roadmaps 2025 low birth weight proportion (v037: live births under 2,500 g; NCHS natality 2017–2023) is - higher where the ACS 2018–2022 poverty rate is higher, with state intercepts (quasi-binomial regression, - logit link, live births as trials)." -- **H2 (m2, term `cost_burden`).** "Conditional on the poverty rate, the bachelor's share, the no-vehicle - share and state, the county low birth weight proportion is higher where a larger share of households spends - 30% or more of income on housing costs." -- **H3 (m2, term `no_vehicle`).** "Conditional on the poverty rate, the bachelor's share, the cost-burden - share and state, the county low birth weight proportion is higher where a larger share of occupied housing - units has no vehicle available." - -No other confirmatory hypotheses. Do not add a race/ethnicity term to the confirmatory models. It is the -exploratory X2 below, because a planner-added control after seeing Round 21 results should not change the -registered models. - -### 5.5 Exploratory analyses (labelled exploratory everywhere; hashes in `docs/studies/r22/exploratory_summary.json`) - -- **X1: without New York City.** A second manifest - `examples/manifests/r22_x1_without_nyc.json` with `"exploratory": true` (A1). It is identical to the main - one except for an exclusion of GEOIDs 36005, 36047, 36061, 36081 and 36085 (add `{"rule": "geoid_exclude", "geoids": [...]}` to - `exclusions`, keeping the main manifest's `cv_screen` entry). Round 21 Part 2 found that these five counties reverse the vehicle - association. -- **X2: race/ethnicity composition.** `run_variant` can only refit with columns already in the run's - analysed data, so do X2 as a second exploratory manifest, `examples/manifests/r22_x2_race_composition.json` - (`"exploratory": true`). It is identical to the main manifest plus raw variables `2022:B03002_001` and - `2022:B03002_004`, a derivation `pct_nh_black` (proportion, `as_percent: true`), and m2 with `+ pct_nh_black`. -- **X3: spatial SE sensitivity.** Use `run_variant` on m1 and m2 with `spatial_se` cutoffs 50, 100 and 200 km. - Do this even if A2 failed, labelled "not validated". -- **X4: residual map and Moran's I.** A diverging residual map of m2 (`make_map` with `scheme="diverging"`) - and Moran's I with queen contiguity, permutation p-value with 999 permutations and seed 20261122, as in - Round 21 Part 2 X5. -- **X5: Poisson cross-check.** `lbw_numerator ~ ... ` with `model: "poisson"`, `exposure: "lbw_denominator"`, - `scale: "X2"`, on m2's terms. The coefficients should be close to the logit ones because LBW is rare. Report - both. - -Use `scripts/r22_exploratory.py`, modelled on `scripts/r21b_exploratory.py`, for X3–X5 (X1 and X2 run as manifests). Save outputs -under `studies_output/r22/` (git-ignored, as before), and copy the figures and a sanitized -`exploratory_summary.json` to `docs/studies/r22/`. - -### 5.6 Report - -`docs/studies/r22_mi_ny_county_low_birth_weight.md`. Copy the section structure of -`docs/studies/r21b_mi_ny_county_premature_mortality.md` exactly: Summary; 1 Question and hypotheses; 2 Data; -3 Preregistration (manifest hash, commit, the planner disclosure paragraph from section 0); 4 Engine work; -5 Results (table with estimate, 90% CI, odds ratio per 10 percentage points = exp(10 × estimate); all four -determinants are derived with `as_percent: true`, so one unit is one percentage point; Holm p, Supported / Not -supported); 6 Exploratory; 7 Interpretation; 8 Limitations; 9 Reproduction (exact commands); 10 Friction log; -11 Review. - -- Review section heading: "Review (author self-review at the owner's request, not independent)". It is a - self-review. Never call it independent. -- Required limitations: ecological (county) inference; LBW years 2017–2023 against ACS 2018–2022; no correction - for covariate measurement error in a GLM (replicate `se_sampling` only); spatial dependence (with or without - Conley, per A2); suppressed counties. -- Report results whatever they are. A non-supported hypothesis is a result, not a problem to fix. - -### 5.7 Friction log - -Keep a friction log while working (anything that needed a workaround, Python code outside the tools, or a -surprising behaviour). Put it in report section 10 with a Status column (Fixed / Backlog). Do not fix friction -items that are out of scope. Log them. - ---- - -## 6. Verification, review and PR - -### 6.1 Tests - -- New tests live in `tests/test_round22.py` (A1–A3) and `tests/test_round22_a4.py` (A4 policy, only if A4 - passed). Tests must be offline (no network). Mock boundaries and the Census API the way existing tests do. -- Full suite: `.venv/bin/python -m pytest -q -p no:cacheprovider`. It takes about 20 minutes, so run it in the - background. It must end with 0 failures. Report the exact pass, skip and fail counts in section 9. -- Also run `create_manifest` (validation) on every `examples/manifests/*.json` and report that all are valid, - except those that were already invalid before your change. Check this on `defaultBranch` first. - -### 6.2 Self-review before the study registration - -After A1–A4 and before step 5.1.2, review your own diff (`git diff defaultBranch...HEAD`) for correctness: -off-by-one errors in kernel distances, wrong small-sample factors, scores with the wrong sign or weights, the -external-name handling of `_numerator` and `_denominator`, and anything that could change existing -behaviour when the new arguments are absent. Fix what you find, and list the findings and fixes in section 9. - -### 6.3 Docs updates in earlier reports - -Only this one: in `docs/studies/r21b_mi_ny_county_premature_mortality.md` section 10, change the Status cell -of F3 (spatial HAC) and F5 (exploratory flag), and of F6 if A4 ran, to "Addressed in Round 22 (see -PLAN_ROUND_22.md)", adding the A2/A4 result in a few words. Change nothing else in earlier reports. - -### 6.4 PR - -- Push `phase22/count-outcomes` and open a PR against `defaultBranch` titled - "Round 22: count outcomes, spatial standard errors, low birth weight study". -- PR body: - - what was done (A1–A4, study); - - the validation results (PASSED/FAILED, with the key numbers); - - the study results table; - - test counts; - - the friction backlog; - - anything you could not do and why. -- No attribution lines (rule 0.3). Do not merge. - ---- - -## 7. Acceptance checklist (all must be true before opening the PR) - -- [x] A1 implemented and tested; existing tests unchanged and passing. -- [x] A2 implemented; identity tests pass; `docs/audits/spatial_hac_validation.md` committed with PASSED or - FAILED and the stated consequence applied. -- [x] A3 implemented; manifest rules enforced; `v037` registered with confirmed years. -- [x] A4 re-test run on OH/PA with `docs/audits/rate_apportionment_retest.md` committed; `"dominant"` added - only if PASSED; default still `"missing"`. -- [x] Docs/MCP updated per section 4. MCP tool count is still 63. -- [x] Manifest committed with its hash before `run_manifest`; study run, exploratory run, report written. -- [x] Full suite 0 failures; all example manifests still valid. -- [x] No API key, email or attribution lines in any commit, output or PR text. -- [x] Section 9 of this plan filled in. - -## 8. What to do when stuck - -- If an existing function does not behave as this plan assumes (an argument is missing, a column is named - differently), adapt with the smallest change. Record it in section 9 with the file and line. Do not - restructure the module. -- If a requirement here conflicts with an existing test, do not delete or weaken the test. Stop working on that - item, record the conflict in section 9 and in the PR body, and continue with the next item. -- If the Census API or the boundary download fails, retry later. Do not switch data sources or years. -- If the study cannot run as registered (for example, too many counties missing), do not edit the committed - manifest. Record the deviation as the engine reports it, and report the run as `confirmatory_incomplete`. - -## 9. Implementation status (to be filled in by the implementer) - -| Item | Result | Notes (decisions, deviations, file:line) | -| :--- | :--- | :--- | -| A1 | Implemented; focused tests passed | Boolean-only exploratory flag; retained blocking deviations and tagged hypotheses/analyses. | -| A2 code | Implemented; identity tests passed | census/spatial_hac.py; EIV moment scores and full-fit GLM scores; cached boundary hashes; confirmatory guard. | -| A2 validation | FAILED | 2,000 draws per cell, fixed seeds; 150-km coverages linear 0.8735/0.8170/0.7070, binomial 0.8345/0.7835/0.7285. Exploratory only; study omits spatial_se. | -| A3 | Implemented; focused tests passed | CHR page confirms 2025 release uses 2017–2023; dictionary confirms proportion/count columns and unreliable flag. Explicit trials/exposure and Pearson scale; numerator aliases preserve R1/R2. | -| A4 | PASSED | OH/PA held out; preregistration d8ad360; 499 high-share split targets, ratio 0.912797 <=1.25. dominant opt-in only; default missing. | -| Docs/MCP | Updated | Version 0.21.0; no added tools; no circularity text weakened; no CI, tags or releases. | -| Self-review findings | Reviewed before study registration | Fixed an OR-loop name shadowing the data-source record, a misplaced dominant annotation caught by USALEEP regressions, count outcomes in RHS roles, and moved unreliable-count reporting after exclusions. Scores use trials-weighted binomial residuals / exposure-adjusted Poisson mu; n/(n-k) only; coordinate rows align to actual fit rows. | -| Study registration (hash, commit) | Committed and pushed before fitting | Canonical SHA-256 `10c46acedc6eff827d8a41df34d272f839ed0d75804666e81b77ca75eab83ea6`, commit `7b967d1ebfeaa369b9dabeb7f81cecdf4d1ec037`; registered manifest bytes unchanged. | -| Study results | Complete, confirmatory; no blocking deviations | 144/145 counties used. H1 poverty supported (Holm p=2.65109e-12); H2 cost burden supported (p=0.00498352); H3 no-vehicle not supported (p=0.923584). Pearson dispersion 21.99/21.02. Independence SEs, as A2 requires. | -| Exploratory X1–X5 | Complete, labelled exploratory | X1 without NYC n=139; X2 composition n=144; X3 all six cutoff refits not validated; X4 residual Moran I=0.183552, p_sim=0.003, 999 permutations, seed 20261122; X5 Poisson n=144. Report and hashed summary in docs/studies/r22*. | -| Test suite counts | 1,628 passed; 37 skipped; 0 failed; 48 subtests passed | Full local project-venv suite, 555.84 seconds, Python 3.14 / SciPy 1.18.1. Final invalid-cutoff focused check: 15 passed. All 11 current example manifests validate; all 8 baseline examples validated before implementation. No GitHub CI. | -| Planner review | Done | Post hoc X6 (scripts/r22_posthoc_robust_se.py): quasi-binomial X2 SEs under-cover (A2 validation, range 0: 0.728); robust SEs 1.8–2.6x larger; H2 would not survive Holm (0.117). Added to report summary, X6, limitations and friction F6. Planner's error: section 3.3 treated X2 as sufficient. | -| Robust-SE fix | Done | `glm_robust_covariance` (HC3) and `fit_replicate(scale="robust")`; criterion committed before the run (4110a42); validation PASSED (HC3 0.879–0.897). Manifests require `robust` for external count outcomes; R22 manifests grandfathered by hash. HC3 rerun of the study: H1 Holm 0.049 (supported), H2 0.320, H3 0.639. | -| PR handoff | Blocked externally, not created | Branch pushed. GitHub create_pull_request returned 403 Resource not accessible by integration. No terminal API credentials/gh configuration available. Prepared description: /tmp/round22-pr-body.md; compare defaultBranch...phase22/count-outcomes. No merge/tag/release/CI performed. | - - -### Decisions and execution record - -- No preregistration association has been inspected. Synthetic-data tests and the - prescribed spatial simulation do not use the real low-birth-weight outcome. -- A2 simulation uses the explicit internal OLS sandwich because fit_eiv requires - replicates even for zero measurement error; binomial fits are computed once per - draw, then the exact tested score/bread feed every cutoff. No fitted covariance - failure is dropped from the coverage denominator. Criteria and seeds unchanged. -- A4 script/hypothesis were committed before the held-out run. Its network-bound - validation overlapped the final offline A3 checks; no study association was read. -- New manifest allowed arguments were checked against existing fit_replicate uses - and all eight pre-existing example manifests. No dependencies or venvs added. -- Validation output key scan: zero occurrences of the configured Census key. - Commit markers suppress GitHub CI as required by ground rule 0.8. -- Source confirmation: [CHR measure page](https://www.countyhealthrankings.org/health-data/population-health-and-well-being/quality-of-life/physical-health/low-birth-weight) - explicitly identifies 2017–2023 for the 2025 release; [2025 dictionary](https://www.countyhealthrankings.org/sites/default/files/media/document/DataDictionary_2025.pdf) - lists v037 raw proportion, numerator, denominator and unreliability indicator. - No different years were documented in the cached analytic CSV. - - -### Final implementation references and handoff - -- Exploratory validation/status: `census/manifest.py:815` (`validate_manifest`) and - `census/manifest.py:1620` (`run_manifest`); checks in `tests/test_round22.py:16`. -- Spatial covariance and coordinate provenance: `census/spatial_hac.py:22` and - `census/spatial_hac.py:68`; model integration `census/models.py:372` and - `census/models.py:809`. `SPATIAL_HAC_VALIDATED` remains false, matching the - committed fixed validation result. No confirmatory spatial argument is permitted. -- Count roles, fitting and fixed external outcomes: `census/manifest.py:1444`, - `census/models.py:372`; source/role/circularity and end-to-end controls in - `tests/test_round22.py:123`. -- Held-out dominant policy: `census/health_sources.py:786`; threshold/default - checks in `tests/test_round22_a4.py`. The original Round 21 audit is unchanged. - The validation script already had `--out`; it was preserved rather than duplicated. -- Only the F3/F5/F6 Status cells were changed in the older study report. All new - study results, limits, exact reproduction commands and the friction backlog are - in `docs/studies/r22_mi_ny_county_low_birth_weight.md`. -- Main registration preceded all LBW associations. Two allowed exploratory - manifests retain their own hashes and exploratory provenance. The versioned - summary SHA-256 is `818543c462d2f3f1f1ce01f91335fde315b764e15e79359dcf23aff016a6bab3`. -- One post-registration review fix standardized ValueError for invalid/nonnumeric - spatial cutoffs. It changes no accepted numeric input, score, coefficient, - validation criterion or study result; all 15 focused controls passed afterward. - This is disclosed in the study self-review. No registered manifest was edited. -- The residual map was visually inspected. Generated SVG whitespace is preserved - as renderer output to retain its recorded hash; source/doc checks exclude that - generated vector file. No formatting sweep or map redesign was performed. -- Literal-key scan: zero hits across the complete local study outputs and copied - docs artifacts, and zero hits in newly generated validation outputs. The raw - cached CHR file is not redistributed. MCP tool count verified as 63. -- Every new commit carries `[skip ci]`, including the final handoff. The default - branch, existing tags and releases remain unchanged. One PR is the remaining - external handoff; no merge is authorized or attempted. GitHub rejected the PR - creation with HTTP 403 (`Resource not accessible by integration`). Terminal - SSH push succeeded, but that does not grant REST pull-request permissions. - The current connection cannot create the PR, and no terminal GitHub API - credentials are configured. The complete proposed PR text is retained at - `/tmp/round22-pr-body.md`; no PR was fabricated or reported as created. - [Manual compare/create route](https://github.com/cafeTechne/Census-Research-Engine/compare/defaultBranch...phase22/count-outcomes?expand=1). diff --git a/PLAN_ROUND_23.md b/PLAN_ROUND_23.md new file mode 100644 index 0000000..13befd6 --- /dev/null +++ b/PLAN_ROUND_23.md @@ -0,0 +1,173 @@ +# Round 23 plan: split `census/manifest.py` into a package + +**Goal.** `census/manifest.py` (3,010 lines, 180 KB) is the engine's most-edited file. Every change makes an +agent read parts of it. Split it into a package of focused modules with **no behaviour change**, so later +rounds read one module of a few hundred lines. + +**Non-goals.** No new features, no bug fixes, no renamed functions, no changed messages, no reformatting of +moved code, no new dependencies, no changes outside the files named here. If you find a bug, write it in +section 7 (status table) and leave it alone. + +## 0. Ground rules + +1. Branch `phase23/split-manifest` from `phase23/plan`. Push it. Do not merge, tag or release. +2. **No AI attribution anywhere.** Commits and the PR body must not contain `Co-Authored-By`, + `Generated with`, or any similar line, even if your tooling suggests one. +3. Never print, log or commit `.env` or any API key or token. +4. Use `.venv/bin/python` (there is no `python` on PATH). Set `OPENBLAS_NUM_THREADS=1` for the suite. +5. Do not run GitHub CI. Commits may carry `[skip ci]`. +6. Read only what you need: `census/manifest.py`, the files named in sections 3–4, `pyproject.toml`, + `tests/test_packaging.py`. Do not read `docs/studies/`, `docs/audits/`, `studies_output/` or old plans. +7. If something in this plan is impossible as written, stop, record it in section 7, and choose the option + that changes the least. + +## 1. Baseline (before touching any code) + +On a clean checkout of `phase23/plan`: + +```bash +CENSUS_HEALTH_OFFLINE=1 .venv/bin/python scripts/refactor_golden.py /tmp/golden_before.json +.venv/bin/python -c "import census.manifest as m; print('\n'.join(sorted(n for n in vars(m) if not n.startswith('__'))))" > /tmp/names_before.txt +OPENBLAS_NUM_THREADS=1 .venv/bin/python -m pytest -q 2>&1 | tail -1 > /tmp/suite_before.txt +``` + +Expected suite line: `1634 passed, 37 skipped` (warnings and subtests may vary). Record the actual line in +section 7. `refactor_golden.py` must report `478 snapshots from 11 manifests`. + +## 2. Target layout + +Convert the module into a package. Start with `git mv census/manifest.py census/manifest/__init__.py`, then +cut code out of `__init__.py` into the submodules below. Move each definition **verbatim**: same body, +same comments, same docstring. Only import lines may change. + +| Module | Contents (exact names) | +| :--- | :--- | +| `schema.py` | All module-level constants from `SCHEMA_VERSION` through `REFERENCES` (including `ANALYSIS_METHODS`, `BLOCKING_DEVIATIONS`, the `STATUS_*` names, `FORMULA_ID_COLUMNS`, `FORMULA_FUNCTIONS`, `_STATE_PREFIXED_LEVELS`), `LEGACY_X2_MANIFESTS`, `CONFIRMATORY_REPLICATE_POLICY`, `REPLICATE_METHODS`, `_SPEC_KEYS`; and `canonical_json`, `manifest_hash`, `load_manifest`, `_table_of`, `_dataset_for`, `manifest_states`, `manifest_entities`, `_is_int`, `manifest_releases`, `release_overlap_message`, `_normalize_analysis`, `_duplicate_spec_keys`, `_formula_names`, `_variable_tables`, `tables_and_refs`, `_derivation_refs` | +| `external.py` | `external_names`, `external_circularity`, `_ext_source_text`, `_has_equivalent`, `circularity_errors` | +| `adjustments.py` | `default_rpp_areas`, `latest_rpp_year`, `_check_adjustments`, `_check_rpp_years`, `_check_dollar_year`, `_FILL_REASONS`, `_fill_adjustment_defaults`, `_scale_rows`, `_apply_adjustments` | +| `derivations.py` | `_age_standardized_errors`, `_fill_derivation_defaults` | +| `replicates.py` | `_vre_level`, `census_vre_source`, `_check_replicates`, `vre_unavailable`, `_declared_covariates_only`, `_replicate_overlap`, `_has_replicate_rows`, `_replicate_rows`, `_derive_replicates`, `_census_vre_replicates`, `_canonical_column`, `replicates_digest`, `read_replicates`, `replicates_sha256` | +| `datasource.py` | `effective_data_source`, `observed_data_source`, `data_source_record`, `resolve_data_source` | +| `validate.py` | `_check_geography`, `_check_variables`, `validate_manifest` (and the section 5 helpers) | +| `create.py` | `create_manifest` | +| `methods_text.py` | `_METHOD_TEXT`, `_age_standardized_text`, `_multi_release_adjustment_text`, `methods_section` (and the section 5 helpers) | +| `run.py` | `label_exploratory`, `analysis_status`, `_codebook_columns`, `_wide_from_panel`, `_one_sided`, `_run_analysis`, `_statistic`, `_tag`, `run_variant`, `run_manifest`, `_write_analysed_data`, `_write_bundle` | +| `__init__.py` | The original module docstring (lines 1–174, unchanged) and re-exports only (section 3). No logic. | + +Any definition not listed: put it in the module whose functions use it, and list it in section 7. + +**Import rules (to avoid cycles).** Submodules import each other with explicit relative imports +(`from .schema import manifest_hash`). A submodule never imports from `census.manifest` or from `.` (the +package `__init__`). Allowed direction: `schema` ← everything; `external`, `adjustments`, `derivations`, +`replicates`, `datasource` ← `validate`, `create`, `methods_text`, `run`; `validate` ← `create`, `run`; +`methods_text` ← `run`. Keep a function-local import inside a function body as a function-local import. + +**Packaging.** `pyproject.toml` lists `packages = ["census"]` explicitly, so the new subpackage would be +left out of installs. Change it to `packages = ["census", "census.manifest"]`. Nothing else in +`pyproject.toml` changes. + +## 3. Facade: `census/manifest/__init__.py` + +Re-export **every** name in `/tmp/names_before.txt` that was defined in the old module (functions, classes, +constants, including `_private` ones). Names that were only imported into the old module (`np`, `pd`, `json`, +`Path`, `Any`, `integrity_errors`, `IncompleteReplicatesError`, etc.) should also stay reachable: re-export +them too, so that `vars(census.manifest)` is a superset of the baseline. Write the re-exports as explicit +`from .schema import (a, b, ...)` lists, grouped by submodule, alphabetical within a group. No `import *`. + +## 4. Tests: only these lines may change + +Moving a function changes where `mock.patch` must point: a patch on `census.manifest.X` no longer affects +a call to `X` made inside a submodule. These are the only patches on this module: + +| File | Current target | +| :--- | :--- | +| `tests/test_review_fixes_r15.py` | `mock.patch("census.manifest.latest_rpp_year", ...)` | +| `tests/test_review_fixes_r12.py` | `mock.patch("census.manifest.replicates_digest", ...)` | +| `tests/test_metadata_integrity.py`, `tests/test_review_fixes_r14.py` | `patch("census.manifest.run_manifest", ...)` | +| `tests/test_manifest_releases.py` | `mock.patch("census.manifest.validate_manifest", ...)` | +| `tests/test_audit_decisions.py` | `patch.object(manifest, "_run_analysis", ...)` | +| `tests/test_review_fixes_r14.py` | `patch.object(M, "default_rpp_areas", ...)` | +| `tests/test_tract_harmonization.py` | `patch.object(mf, "_write_analysed_data", ...)` | + +For each one: run its test first. If it still passes unchanged (the caller looks the name up through +`census.manifest`, as `mcp_server.py` and `census/workflows*.py` do), leave it. If it fails, re-point the +patch at the submodule where the name is **looked up at call time** (for example +`census.manifest.run._run_analysis`). If a name is called from two submodules and the test needs both, +patch both. Change nothing else in any test file, and record each change in section 7. + +No other file outside `census/manifest/`, `pyproject.toml`, those test lines and the docs in section 6 may +change. In particular, `mcp_server.py` and other `census/*.py` modules keep importing from +`census.manifest` exactly as now. + +## 5. Splitting the three long functions (golden-covered only) + +After sections 2–4 pass section 6's checks, commit. Then split **only** these two functions, which the golden +snapshot covers byte for byte: + +- `validate_manifest` (about 300 lines): extract its consecutive blocks into private helpers in + `validate.py`, called in the same order with the same `errors` list, so the output order is unchanged. + Natural cuts: top-level keys/schema; variables; derivations; the per-analysis loop body + (`_check_analysis(i, a, ...)`); hypotheses; exclusions/missing/correction. Target: the + `validate_manifest` body is at most 80 lines. +- `methods_section` (about 320 lines): extract the paragraph builders (data, derivations, adjustments, + replicates, per-analysis text, references) into private helpers in `methods_text.py`, joined in the same + order. Target: the `methods_section` body is at most 80 lines. + +Do **not** split `run_manifest` this round. The golden snapshot does not cover it byte for byte; record its +natural stages in section 7 for a later round. + +If a split cannot keep the golden snapshot identical, revert that function to the verbatim move and record +it in section 7. + +## 6. Acceptance (all required) + +```bash +CENSUS_HEALTH_OFFLINE=1 .venv/bin/python scripts/refactor_golden.py /tmp/golden_after.json +cmp /tmp/golden_before.json /tmp/golden_after.json # must print nothing +.venv/bin/python -c "import census.manifest as m; print('\n'.join(sorted(n for n in vars(m) if not n.startswith('__'))))" > /tmp/names_after.txt +comm -23 /tmp/names_before.txt /tmp/names_after.txt # must print nothing (superset) +OPENBLAS_NUM_THREADS=1 .venv/bin/python -m pytest -q 2>&1 | tail -1 # same passed/skipped counts as baseline +.venv/bin/python -c "import mcp_server" # server module imports +wc -l census/manifest/*.py # every module <= 900 lines; __init__.py has no defs +grep -n "^def \|^class " census/manifest/__init__.py # must print nothing +``` + +Also run each check after the section 5 commit. Then update the docs: +`grep -rn "census/manifest.py\|manifest\.py" docs README.md CLAUDE.md STATUS.md CONTRIBUTING.md`, and change +each path reference to the package or submodule (no other wording changes). Add a three-line CHANGELOG +entry under a new `## Unreleased` heading: "Internal: `census.manifest` split into a package (no behaviour +change; golden snapshot identical)". + +## 7. Status table (fill in; keep it short) + +| Item | Status | Notes | +| :--- | :--- | :--- | +| Baseline suite line | PASS | 1634 passed, 37 skipped, 4 warnings, 48 subtests passed in 292.07s (0:04:52). Initial sandbox-only socket/MCP failures were rerun outside that restriction before code changed. | +| Golden snapshot count | PASS | 478 snapshots from 11 manifests; baseline namespace 122 names. | +| Unlisted definitions placed | None | All 68 original functions and 31 constants were listed; 23 imported compatibility names re-exported from schema. | +| Packaging change | PASS; least-change exception (0.7) | pyproject packages now ["census", "census.manifest"]. Updated only the identical expected list in tests/test_packaging.py; its old hard-coded assertion conflicted with required packaging. | +| Test patch targets changed | Three listed failures; one omitted target (0.7) | r15 latest_rpp_year -> adjustments; r12 replicates_digest -> run; r14 default_rpp_areas -> adjustments. Full suite also exposed tests/test_manifest_multistate.py::_run_analysis (omitted from section 4): retarget only its patch to run._run_analysis. Assertions unchanged. | +| Section 5: validate_manifest | PASS | 15 body lines; six private helpers; identical ordered errors and golden snapshots; checkpoint b full suite 1634 passed / 37 skipped. | +| Section 5: methods_section | PASS | 30 body lines; nine paragraph helpers; identical rendered bytes; post-commit checkpoint c full suite 1634 passed / 37 skipped. | +| `run_manifest` stages (for later) | Not split | Plan/extract; harmonize/aggregate/frame; fetch/derive replicates; external join; exclusions/adjustments; fit; save analysed data; corrected hypotheses; bundle. | +| Bugs noticed (not fixed) | Fault-injection coverage caveat | test_analysed_data_failure_does_not_lose_the_run passes without verifying its facade mock was called; after the move that mock is ineffective. Left unchanged per section 4. No production bug fix included. | +| Module line counts | PASS | __init__ 318; schema 256; external 113; adjustments 341; derivations 79; replicates 331; datasource 74; validate 498; create 129; methods_text 480; run 824. Facade has only its unchanged docstring and explicit imports. | +| Final suite line | PASS | 1634 passed, 37 skipped, 4 warnings, 48 subtests passed in 300.36s (0:05:00). Same pass/skip counts as baseline and a/b/c; golden byte-identical, no lost names, MCP imports, caps satisfied. | + +## 8. Deliverables + +1. Commits on `phase23/split-manifest`, in this order: (a) package conversion and verbatim moves, + packaging, test targets; (b) the `validate_manifest` split; (c) the `methods_section` split; (d) docs, + CHANGELOG, status table. Each commit passes section 6. +2. Push the branch. If you cannot open a PR, say so. The owner's agent opens it. +3. PR body (if you open it): what moved where (one table), the section 6 outputs pasted, the status table, + no attribution. + +Notes: Section 0.7 exceptions are the single packaging assertion and the omitted +`tests/test_manifest_multistate.py` spy target. Three listed failing targets were +retargeted; every listed test that passed unchanged was left unchanged. No test +assertions were weakened. Documentation was scanned for path tokens only, without +reviewing study/audit content; only `STATUS.md` needed a source-path replacement. +Actual `test_manifest.py` filenames remain valid and were not renamed. Checkpoints +a/b/c retained all 478 golden snapshots (11 manifests), all 122 baseline attributes +(the facade now has 132), MCP import and the <=900-line module cap. diff --git a/PLAN_ROUND_3.md b/PLAN_ROUND_3.md deleted file mode 100644 index fe5ef6b..0000000 --- a/PLAN_ROUND_3.md +++ /dev/null @@ -1,59 +0,0 @@ -# PLAN_ROUND_3.md — US Census ACS Data Batch Downloader - -**Planning Round:** 3 -**Date:** 2026-11-01 -**Branch:** phase3/headless-library -**Status:** Phase 3 COMPLETE. Ready to begin Phase 4. -**Next Round:** See PLAN_ROUND_4.md for Phase 4 (Model Context Protocol / MCP Server). - ---- - -## Phase 3 Completion Summary - -All Phase 3 tasks have been executed, verified, and committed on branch `phase3/headless-library`. - -### Implemented Architecture - -1. **3a — Headless Library Package (`census/`)** - - **`census/geo.py`**: - - Embedded reference data: `STATE_FILES`, `STATE_MAPPING`, `FIPS_MAPPING`. - - `extract_state_abbr_from_filename(filename: str) -> str`: FIPS code parser. - - `get_local_places_file(state_abbr: str, base_dir: Optional[str] = None) -> Optional[str]`: Resolves state file paths in dev or PyInstaller bundles. - - `load_places_from_local(path: str) -> List[Tuple[str, str, str, str, str]]`: Reads pipe-delimited Census places. - - **`census/tables.py`**: - - `load_tables_from_local(path: str) -> Dict[str, str]`: Reads table codes and descriptions from `groups.json`. - - `load_variable_definitions(filepath: Optional[str] = None, force_reload: bool = False) -> Dict[str, str]`: Cached reader for `variables.json`. - - `get_variable_descriptions(table_code: str, definitions: Optional[Dict[str, str]] = None) -> Dict[str, str]`: Filters variable descriptions by table prefix. - - **`census/api.py`**: - - `CensusAPIClient`: Configurable class managing connection pooling and exponential backoff retry via `requests.adapters.HTTPAdapter` and `urllib3.util.Retry` for status codes 429, 500, 502, 503, and 504. - - Method `fetch(year, state_code, table_code, geo_level, place_code, county_code)` supporting Place, State, and County queries. - - Convenience function `fetch_census_data(...)` maintaining signature backwards compatibility. - - **`census/export.py`**: - - `format_dataframe(df, table_code, variable_descriptions)`: Renames raw variable codes to human-readable labels. - - `handle_duplicate_columns(df)`: Deduplicates column names with sequential suffixes. - - `enhanced_melt(df, id_vars, var_name, value_name)`: Reshapes from wide to long format with missing column checks. - - `sanitize_filename(name)`: Punctuation and whitespace cleaner. - - `process_data_and_save(...)`: Writes wide/melted CSVs with explicit `Year` column at index 0. - - `save_excel_workbook(...)`: Compiles wide and melted data into `.xlsx` with `Wide_Data` and `Melted_Data` sheets. - - **`census/__init__.py`**: - - Exposes all core classes and functions at package level. - -2. **3b — Refactored `acs_downloader.py`** - - Converted into a thin Tkinter view layer importing from `census`. - - Maintains exact Dracula UI styling, threading queue, and backward compatibility. - -3. **3c — Test Suite Modularization** - - Expanded unit tests into targeted modules: - - `tests/test_geo.py`: Geographic parsing and mapping tests. - - `tests/test_tables.py`: Table and variable definition caching tests. - - `tests/test_api.py`: Client initialization, parameter construction, and mock queries. - - `tests/test_export.py`: Deduplication, melting, CSV, and Excel export tests. - - `tests/test_downloader.py`: End-to-end integration and GUI initialization tests. - - Total tests: 33 tests, all passing, 100% formatted to `black` conventions. - ---- - -## Known Remaining Issues / Transition to Phase 4 - -1. **MCP Server**: With the headless `census/` package complete, the next phase is building `mcp_server.py` to expose Census tools to Claude Desktop, Cursor, and agentic workflows. -2. **MCP SDK**: `mcp` library will need to be installed in Phase 4. diff --git a/PLAN_ROUND_4.md b/PLAN_ROUND_4.md deleted file mode 100644 index 8056079..0000000 --- a/PLAN_ROUND_4.md +++ /dev/null @@ -1,52 +0,0 @@ -# PLAN_ROUND_4.md — US Census ACS Data Batch Downloader - -**Planning Round:** 4 -**Date:** 2026-11-01 -**Branch:** phase4/mcp-server -**Status:** Phase 4 COMPLETE. Production Readiness Achieved. - ---- - -## Phase 4 Completion Summary - -Phase 4 successfully implemented the Model Context Protocol (MCP) server, targeted specifically for **Anthropic's Claude Code** and **Claude Desktop**. - -### Implemented Enhancements - -1. **MCP Server Core (`mcp_server.py`)** - - Built on official `mcp.server.fastmcp.FastMCP` with stdio transport. - - Clean integration with the underlying headless `census` library (`census.api`, `census.geo`, `census.tables`, `census.export`). - - Tools implemented: - - `list_states`: Lists states and territories with 2-digit FIPS codes. - - `list_places`: Resolves cities, towns, and CDPs for a state with substring filtering. - - `search_tables`: Searches ACS 5-year table groups by code or keyword in `groups.json`. - - `list_variables`: Retrieves variable definitions and labels from `variables.json`. - - `fetch_table_data`: Queries live Census ACS5 API and formats output into human-readable JSON records. - - `batch_download`: Downloads multi-year data across entities and exports CSV or multi-sheet Excel files. - -2. **Dependencies** - - Added `mcp>=1.0.0` to `requirements.txt`. - -3. **Automated Testing Suite** - - Added `tests/test_mcp.py` covering state listing, place filtering, table search, variable previews, live API query formatting, and multi-year batch downloads. - - Total test count: 39 tests across 6 modules (`test_mcp`, `test_api`, `test_geo`, `test_tables`, `test_export`, `test_downloader`). All passing in ~0.17s. - - 100% compliant with `black` formatting standards. - -4. **Claude Code & Claude Desktop Integration Guides** - - Created `docs/claude_setup.md` documenting: - - `claude mcp add` CLI command for Claude Code. - - Project-level configuration in `.claude/config.json`. - - User-global configuration in `~/.claude.json`. - - Claude Desktop configuration at `%APPDATA%\Claude\claude_desktop_config.json`. - - Practical example research prompts. - ---- - -## Retrospective Across All Rounds - -| Round / Phase | Key Accomplishments | -| :--- | :--- | -| **Phase 1: Cleanup & Bug Fixes** | Fixed DataFrame formatting crash, fixed place/state column filter bug, secured API key via `.env`, added dynamic year spinboxes, cleaned dead files. | -| **Phase 2: Research UX Overhaul** | Asynchronous daemon worker threading, Dracula progress/status log, multi-place selection, dynamic call estimator, output format options (CSV / Excel), variable explorer, geography selector. | -| **Phase 3: Headless Library** | Decoupled core logic into standalone `census/` package (`api`, `geo`, `tables`, `export`) with exponential backoff HTTP retry logic. Converted `acs_downloader.py` into a thin UI consumer. | -| **Phase 4: Anthropic MCP Integration** | Created `mcp_server.py` with 6 MCP tools, added `test_mcp.py`, configured setups for Claude Code & Claude Desktop. | diff --git a/PLAN_ROUND_5.md b/PLAN_ROUND_5.md deleted file mode 100644 index 2e4a416..0000000 --- a/PLAN_ROUND_5.md +++ /dev/null @@ -1,252 +0,0 @@ -# PLAN_ROUND_5.md — US Census ACS Data Batch Downloader - -**Planning Round:** 5 -**Date:** 2026-11-01 -**Target Milestone:** Research-Grade Econometric & Demographic Data Pipeline -**Branch:** `phase5/research-grade-pipeline` (to branch off `defaultBranch`) -**Status:** Planning Complete. Ready for Implementation Roadmap. - ---- - -## Executive Overview - -Phases 1 through 4 established a robust, freeze-free desktop GUI and headless Python library (`census/`) with an Anthropic FastMCP server (`mcp_server.py`) for Claude Code and Claude Desktop. - -Phase 5 elevates this toolkit from a data fetcher to an **end-to-end econometric and demographic research engine**. It addresses fundamental scientific requirements for empirical research: statistical sampling error handling (MOEs, standard errors, coefficients of variation), high-demand spatial scales (Census Tracts, ZCTAs/ZIP, MSAs), curated Census endpoints (Data Profiles DP02–DP05), longitudinal price harmonization (CPI-U inflation deflators), Census sentinel code sanitization, and high-performance panel data outputs (Tidy Panel, Codebooks, Parquet). - ---- - -## Architectural Pillars & Detailed Specifications - -```text - CENSUS RESEARCH ENGINE - ┌──────────────────────────────────────────────────────────────────────────────────┐ - │ │ - │ 1. STATISTICAL RIGOR 2. EXTENDED GEOGRAPHIES 3. PROFILE ENDPOINTS │ - │ • MOE & SE Derivation • Census Tracts (11-digit) • DP02 Social │ - │ • CV Reliability Flags • ZCTAs / ZIP Codes • DP03 Economic │ - │ • Z-Test Significance • MSAs / CBSAs • DP04 Housing │ - │ • Sentinel (-666M) Filtering • National Benchmark (US:1) • DP05 Demographics │ - │ │ - │ 4. TIME-SERIES HARMONIZATION 5. DATA ENGINEERING 6. INTERFACES │ - │ • CPI-U Inflation Deflator • Tidy Panel (GEOID, Year) • Desktop GUI Pass │ - │ • 2010-2020 Boundary Warning • Automated Codebooks • Claude MCP Server │ - │ • SQLite Query Cache • Parquet Columnar Export • Headless Python SDK│ - │ │ - └──────────────────────────────────────────────────────────────────────────────────┘ -``` - ---- - -### Pillar 1: Statistical Rigor & Demographic Methodology - -ACS 5-year data represents sample estimates subject to sampling variability. Research standards (OMB, Census Bureau, academic journals) require reporting and analyzing this uncertainty. - -#### 1.1 Pairwise Estimate & MOE Pairing -- The Census API returns estimates (`E`) and Margins of Error (`M`) at the 90% confidence level. -- `census.export` will systematically pair estimates with their corresponding MOE rather than treating them as disconnected variables. -- Structure: For each variable code `B01001_001`, maintain both `B01001_001E` (Estimate) and `B01001_001M` (Margin of Error). - -#### 1.2 Automated Derivation of Standard Errors (SE) & Coefficients of Variation (CV) -- **Standard Error Calculation**: - $$SE = \frac{MOE_{90}}{1.645}$$ -- **Coefficient of Variation (CV)**: - $$CV = \left(\frac{SE}{\text{Estimate}}\right) \times 100\%$$ -- **Reliability Categorization (Census Standard)**: - - $CV \le 15\%$: **High Reliability** (Green flag, standard reporting). - - $15\% < CV \le 30\%$: **Moderate Reliability** (Yellow flag, interpret with caution). - - $CV > 30\%$: **Low Reliability** (Red flag, extreme sampling error, avoid standalone policy conclusions). - -#### 1.3 Statistical Significance Hypothesis Testing ($Z$-Test) -- When comparing two estimates $\hat{X}_1$ and $\hat{X}_2$ (e.g., Austin median income in 2015 vs. 2022, or County A vs. County B): - $$Z = \frac{|\hat{X}_1 - \hat{X}_2|}{\sqrt{SE_1^2 + SE_2^2}}$$ -- Provide two-tailed significance determination: - - $|Z| \ge 1.645$: Statistically significant at $\alpha = 0.10$ (90% confidence). - - $|Z| \ge 1.960$: Statistically significant at $\alpha = 0.05$ (95% confidence). - - $|Z| \ge 2.576$: Statistically significant at $\alpha = 0.01$ (99% confidence). - -#### 1.4 Additive Margin of Error Propagation -- When researchers aggregate categories (e.g. summing age brackets $18\text{--}24$ and $25\text{--}34$): - $$\text{Combined Estimate} = \sum_{i=1}^k \hat{X}_i$$ - $$\text{Combined MOE} = \sqrt{\sum_{i=1}^k MOE_i^2}$$ -- Add an aggregation helper function in `census.export` ensuring proper quadrature error propagation. - -#### 1.5 Census Sentinel Value & Missing Code Filtering -The Census Bureau encodes missing, top-coded, or non-applicable values as large negative integers: -- `-666666666`: Estimate or margin of error not applicable or cannot be calculated. -- `-888888888`: Not applicable or estimate missing. -- `-999999999`: Top-coded (e.g., Median Income `$250,000+`) or bottom-coded interval. -- `-555555555`: Annotation flag. -- `-222222222`, `-333333333`: Controlled or missing margin of error. - -**Requirement**: `census.export.clean_census_sentinels(df)` will replace these sentinels with `NaN` (or categorical indicators like `>250,000`) so researchers do not accidentally calculate negative median incomes or negative ages. - ---- - -### Pillar 2: Extended Research Geographies & Spatial Identifiers - -Researchers need spatial granularity beyond cities and whole states. - -#### 2.1 Census Tracts (Summary Level 140) -- **Importance**: The universal geographic unit for spatial econometrics, neighborhood studies, redlining research, and health disparities. -- **API Pattern**: - `for=tract:*&in=state:{state_code}&in=county:{county_code}` -- **Coverage**: All ~85,000 tracts across all 3,143 US counties. - -#### 2.2 ZIP Code Tabulation Areas / ZCTAs (Summary Level 860) -- **Importance**: Necessary for merging Census demographics with external datasets indexed only by 5-digit ZIP codes (e.g., clinical health registries, voter files, business licenses). -- **API Pattern**: - `for=zip code tabulation area:*` or `for=zip code tabulation area:{zip_code}` -- **Integration**: Leverage the existing workspace [`zip2fips.json`](file:///c:/Users/hobo/Desktop/US-Census-5-Year-ACS-Data-Batch-Downloader/zip2fips.json) to allow users to input a ZIP code or County and resolve corresponding boundaries. - -#### 2.3 Metropolitan Statistical Areas / CBSAs (Summary Level 310) -- **Importance**: Standard boundary for regional labor market, wage, and housing cost studies. -- **API Pattern**: - `for=metropolitan statistical area/micropolitan statistical area:*` - -#### 2.4 National US Benchmark (Summary Level 010) -- **Importance**: Provides the reference baseline against which state, county, or place values are indexed. -- **API Pattern**: - `for=us:1` - -#### 2.5 Standard 11-Digit Composite `GEOID` Construction -In raw API responses, `state`, `county`, `tract`, and `place` are returned as disparate columns. Researchers require the standard Census `GEOID` string to join with TIGER/Line GIS shapefiles in GeoPandas, QGIS, or ArcGIS: -- **Tract `GEOID` (11 digits)**: `{state:02d}{county:03d}{tract:06d}` -- **County `GEOID` (5 digits)**: `{state:02d}{county:03d}` -- **Place `GEOID` (7 digits)**: `{state:02d}{place:05d}` -- **ZCTA `GEOID` (5 digits)**: `{zcta:05d}` - ---- - -### Pillar 3: Specialized Endpoints (Data Profiles & Subject Tables) - -Detailed tables (`/acs/acs5`) only provide raw counts; researchers must manually locate denominators and compute percentages. The Census Bureau provides dedicated pre-calculated endpoints: - -#### 3.1 Data Profile Endpoint (`/acs/acs5/profile`) -Provides pre-computed rates, percentages, and median estimates with built-in denominators: -- `DP02`: **Selected Social Characteristics** (Educational attainment %, veteran status, foreign-born %, disability rate, language spoken at home). -- `DP03`: **Selected Economic Characteristics** (Unemployment rate, poverty rate %, median household income, industry shares, health insurance coverage %). -- `DP04`: **Selected Housing Characteristics** (Homeownership rate %, gross rent as % of income, housing units, vacant units). -- `DP05`: **Demographic and Housing Estimates** (Age pyramid, sex ratio, race/ethnicity percentages). - -#### 3.2 Subject Tables Endpoint (`/acs/acs5/subject`) -Pre-aggregated research tables on specific domains: -- `S1501`: Educational Attainment -- `S1701`: Poverty Status in the Past 12 Months -- `S1901`: Income in the Past 12 Months -- `S2701`: Health Insurance Coverage Status - -#### 3.3 Comparison Profiles (`/acs/acs5/cprofile`) -Includes built-in statistical significance flags (`*` indicator) comparing 5-year periods directly from the Census Bureau. - ---- - -### Pillar 4: Time-Series & Longitudinal Harmonization - -#### 4.1 CPI-U Inflation Deflator Engine -- **Problem**: In ACS 5-year datasets, financial figures (median income `B19013`, rent `B25064`, home value `B25077`) are reported in dollars adjusted to the final year of that specific 5-year period (e.g. 2012 ACS is in 2012 dollars; 2022 ACS is in 2022 dollars). -- **Solution**: Embed a Bureau of Labor Statistics (BLS) **CPI-U** (Consumer Price Index for All Urban Consumers) annual deflator lookup table: - $$\text{Real Dollars}_{\text{base}} = \text{Nominal Dollars}_t \times \left(\frac{\text{CPI}_{\text{base}}}{\text{CPI}_t}\right)$$ -- Allow researchers to select a target base year (e.g., "Adjust all financial values to 2022 constant dollars"). - -#### 4.2 2010 vs. 2020 Decennial Census Boundary Shift Advisory -- **Problem**: In 2020, Census tract and county subdivision boundaries underwent decennial redistricting. Many tracts were split, merged, or renumbered between the 2019 ACS and 2020 ACS. -- **Solution**: Add an automated advisory in logs and metadata when a multi-year query spans across the 2019/2020 divide, warning researchers that geographic boundaries may not be strictly contiguous. - -#### 4.3 SQLite Local Query Cache -- Store API responses keyed by `(year, state, geo_level, entity_code, table_code)` in a local SQLite database (`.cache/census_cache.db`). -- Re-running analysis scripts or refining panel models avoids re-requesting identical data from the Census API, eliminating rate limit risks and enabling offline reproducibility. - ---- - -### Pillar 5: Research Toolchain Output Formats - -#### 5.1 Tidy Longitudinal Panel Format -- Format optimized for panel data regression in R (`fixest`, `plm`), Stata, and Python (`statsmodels`, `linearmodels`): - - Primary Index: `(GEOID, Year)` - - Columns: Sanitized short slugs (e.g., `pop_total`, `med_hh_income_est`, `med_hh_income_moe`, `poverty_rate_pct`). - -#### 5.2 Automated Codebook / Data Dictionary Generation (`codebook.csv`) -For every batch extraction, automatically output a companion codebook file: -- `Variable_Code`: Raw Census code (e.g., `B19013_001E`). -- `Variable_Slug`: Sanitized identifier (e.g., `median_household_income`). -- `Measurement_Type`: `Estimate`, `Margin of Error`, `Percentage`, or `Annotation`. -- `Full_Description`: Human-readable label from `variables.json`. -- `Universe`: Target universe (e.g., `Households`, `Total Population`). -- `Unit`: Dollars, Count, Percent, or Ratio. - -#### 5.3 Apache Parquet & Arrow Export -- For high-volume queries (e.g., all 85,000 tracts across 10 years): - - CSV files can reach 10+ GB and consume excessive memory. - - Export to `.parquet` format with snappy compression: reduces file size by 85–92%, retains native column datatypes (int64, float64, string), and integrates directly with DuckDB, Polars, and Pandas. - ---- - -### Pillar 6: Interface Upgrades (GUI, CLI & Claude MCP Tools) - -#### 6.1 Claude Code & Claude Desktop MCP Tools Expansion -Expose the new research capabilities directly to Claude via `mcp_server.py`: -- `list_tracts(state_code, county_code)`: Lists all Census tracts in a county. -- `list_zctas(state_code)`: Lists ZIP Code Tabulation Areas in a state or resolves ZIP codes using `zip2fips.json`. -- `calculate_significance(est1, moe1, est2, moe2, confidence_level)`: Runs two-tailed $Z$-test and returns $Z$-statistic, $p$-value, and significance verdict. -- `calculate_cv(estimate, moe)`: Calculates standard error, CV %, and reliability flag. -- `query_data_profile(profile_code, state_code, geo_level, ...)`: Direct query for `DP02`, `DP03`, `DP04`, `DP05`. -- `adjust_inflation(value, from_year, to_year)`: Returns CPI-U constant dollar adjusted value. -- `generate_codebook(table_code)`: Emits machine-readable data dictionary. - -#### 6.2 Desktop GUI Enhancements ([acs_downloader.py](file:///c:/Users/hobo/Desktop/US-Census-5-Year-ACS-Data-Batch-Downloader/acs_downloader.py)) -- Geography selector: Add `Tracts in County` and `ZCTA (ZIP)`. -- Format selector: Add `Tidy Panel CSV`, `Parquet (.parquet)`, and `Include Codebook`. -- Statistical toggle: Checkbox to `Calculate Standard Errors & CVs`. -- Inflation toggle: Checkbox to `Adjust Financial Values to Constant Dollars (CPI-U)`. - ---- - -## Implementation Roadmap - -```text -PHASE 5a: Statistical Engine & Data Profiles -├── Pair E & M variables, calculate SE, CV, and Z-test statistics -├── Implement sentinel cleaner (-666666666 -> NaN) -└── Implement Data Profile endpoints (DP02, DP03, DP04, DP05) - -PHASE 5b: Extended Geographies & Composite GEOIDs -├── Implement Census Tract query handler (Summary Level 140) -├── Implement ZCTA query handler (Summary Level 860) with zip2fips.json -└── Generate standard 11-digit / 5-digit composite GEOID columns - -PHASE 5c: Longitudinal Harmonization & Output Formats -├── Integrate BLS CPI-U annual deflator table & constant dollar converter -├── Add 2019-2020 decennial boundary shift advisory flag -├── Implement Tidy Panel data generator & codebook.csv exporter -└── Implement Apache Parquet export (.parquet) - -PHASE 5d: Claude MCP Server & Desktop GUI Expansion -├── Register new tools in mcp_server.py for Claude Code & Claude Desktop -├── Update acs_downloader.py GUI with new controls and toggles -└── Add comprehensive unit test suite in tests/ -``` - ---- - -## Testing & Validation Strategy - -1. **Statistical Verification**: - - Verify SE formula against Census Bureau published tables. - - Verify $Z$-test p-values against standard normal distribution values ($Z = 1.96 \implies p = 0.05$). -2. **Geography Validation**: - - Validate 11-digit tract GEOID generation for known counties (e.g. New York County 36061, Travis County TX 48453). - - Validate ZCTA queries using `zip2fips.json`. -3. **Data Profile Tests**: - - Verify `DP03` economic queries return percentages and medians correctly. -4. **Inflation Verification**: - - Verify 2010 nominal dollar conversion to 2022 constant dollars against official BLS CPI-U series. -5. **Format Validation**: - - Validate Parquet schemas, tidy panel index uniqueness `(GEOID, Year)`, and codebook completeness. - ---- - -## Ready to Proceed - -This plan covers all five core pillars and tangential gaps. When you are ready, we will branch `phase5/research-grade-pipeline` and begin implementing **Phase 5a** (Statistical Engine & Data Profiles). - diff --git a/PLAN_ROUND_6.md b/PLAN_ROUND_6.md deleted file mode 100644 index 9037b76..0000000 --- a/PLAN_ROUND_6.md +++ /dev/null @@ -1,362 +0,0 @@ -# PLAN_ROUND_6.md: Vintage-Safe Variables, CPI-U-RS, and an Agent-First Interface - -**Planning Round:** 6 -**Date:** 2026-11-01 -**Status:** Implemented on `phase6/vintage-safe-pipeline` (see section 13). No owner actions outstanding. -**Revised:** 2026-11-02 (R-CPI-U-RS data imported; licensing and copyright holder confirmed) -**Branch:** `phase6/vintage-safe-pipeline` (from `phase5/research-grade-pipeline`) - ---- - -## 1. Summary - -Round 5 produced a working research pipeline, but two of its assumptions are unsafe for multi-year research: - -1. **A variable code is assumed to mean the same thing in every year.** Panels are keyed by variable code, and labels come from one metadata vintage (bundled 2021 `variables.json` for detailed tables, the final requested year for profile and subject tables). If the Census Bureau renumbers a Data Profile or Subject table line, or reuses a code for a different measure, the panel silently merges different quantities into one column, or splits one quantity across two columns. Nothing in the output reveals this. -2. **Constant dollars use CPI-U.** The Census Bureau uses R-CPI-U-RS for all ACS years (its CPS-based income reports switched to C-CPI-U in 2023; ACS did not). Our adjusted values will therefore not match Census-published ACS comparisons. - -Round 6 fixes both and treats agents (MCP and CLI) as the primary interface. The GUI moves to maintenance mode. - -| Priority | Workstream | -| :--- | :--- | -| P0 | Vintage-aware metadata and a variable concordance that panels are built from | -| P0 | R-CPI-U-RS as the default deflator, with sourced and checksummed index data | -| P1 | Agent-first interface: dry-run planning, concordance tools, structured errors, CLI | -| P1 | Live API contract tests and real-data fixtures | -| P2 | GUI maintenance mode | - ---- - -## 2. Problem: variable meaning drifts across ACS releases - -### 2.1 How it fails today - -| Change between releases | Example shape | Current behavior | Consequence | -| :--- | :--- | :--- | :--- | -| Line renumbered | Measure X is `DP02_0067PE` in year A and `DP02_0068PE` in year B | Two columns, each half-empty | Panel looks like the measure stops and a new one starts | -| Code reused | `DP02_0068PE` is X in year A and Y in year B | One column | Different measures pooled; regressions are silently wrong | -| Label reworded | Same code, edited wording | One column, latest label | Usually fine, but a wording change can hide a definition or universe change | -| Universe changed | Same code and label, different denominator | One column | Rates not comparable across years | -| Added or discontinued | Code exists only in some years | Partially empty column | Mostly visible, but undocumented | - -Detailed tables (`B`/`C`) change less often, but they are exposed to the same failure because every year is labelled from one bundled vintage. Comparison Profiles embed the period in the code (`CP03_2017_001E`), so codes never match across releases and panels from them are currently not usable. - -The examples above are illustrative. We have not yet measured how often each case occurs. The first deliverable (6a.1) measures it. - -### 2.2 Design principle - -The unit of a panel column is a **concept** (one measured quantity), not a variable code. Every (year, code) cell is mapped to a concept through an explicit, inspectable concordance. Matches are only automatic when they are exact. Anything else is reported and needs an explicit decision, either a reviewed override file or a user flag. The pipeline never guesses silently. - ---- - -## 3. Workstream A (P0): vintage-aware metadata and concordance - -### A1. Vintage discovery - -- Read the Census API catalog (`https://api.census.gov/data.json`) to list the available years per dataset (`acs/acs5`, `acs/acs5/profile`, `acs/acs5/subject`, `acs/acs5/cprofile`). -- This replaces hard-coded year bounds (`MIN_YEAR`, `MAX_YEAR`, the `2022` default year in MCP tools) and makes new releases appear without code changes. -- New function `census.metadata.available_vintages(dataset) -> list[int]`. Results are cached with a time-to-live, because the catalog changes when new releases are published. - -### A2. Per-vintage metadata registry - -- **Offline archive (decided, done).** `metadata///{variables,groups}.json.gz` holds every published vintage: acs5 and profile 2009–2024, subject 2010–2024, cprofile 2015–2024 (114 files, 19 MB). Source URL, retrieval time, and SHA-256 of each raw response are in `metadata/manifest.json`. `scripts/fetch_metadata_archive.py` adds new releases, and with `--refresh` it reports, rather than overwrites, any published file whose checksum changed. -- Vintages missing from the archive fall back to the API (cached in SQLite), and the source used is recorded per year. -- `resolve_metadata(table, dataset, year)` always returns metadata for **that** year. -- Offline prefetch: `python -m census metadata prefetch --dataset profile --years 2012-2024`. - -**Bundled metadata policy (decided, revised).** The root `variables.json` (2021 release) and `groups.json` (2020 release) were two different vintages. They are removed; identical content lives in the archive. The archive is now the offline fallback for every vintage. It supports table search and lookup without network access and keeps the test suite deterministic. A single vintage is never enough on its own for a multi-year panel in `fail` mode. - -### A3. Label normalization - -`canonical_label(label, concept, dataset)` produces the key used for matching: - -- Strip measurement prefixes (`Estimate!!`, `Percent!!`, `Margin of Error!!`) and comparison-period prefixes (`2012-2016 Estimates!!`). -- Remove vintage-specific dollar text (`(in 2021 inflation-adjusted dollars)`). -- Normalize case, whitespace, trailing colons (`Total:` vs `Total`), and separator variants (`--`, `!!`). -- Keep the full hierarchy path, not only the leaf. `Total!!Male!!Under 5 years` and `Total!!Female!!Under 5 years` must stay distinct. -- Normalization rules are a versioned, tested list. Each rule has a test with a real before/after label pair. - -### A4. Concordance builder - -`build_concordance(table, dataset, years) -> Concordance` assigns every (year, code) to a `concept_id` and a status: - -| Status | Rule | Auto-merged? | -| :--- | :--- | :--- | -| `stable` | Same code, same canonical label, same universe in every year | Yes | -| `renumbered` | Same canonical label and universe, different code | Yes, under one concept | -| `redefined` | Same code, different canonical label | No: split into separate concepts | -| `relabeled` | Same code, labels differ but similarity is above threshold | No: `needs_review` | -| `universe_changed` | Same code and label, different table universe | No: `needs_review` | -| `ambiguous` | Canonical label matches more than one code in one vintage | No: `needs_review` | -| `added` / `discontinued` | Present in only some years | n/a: recorded | - -Details: - -- A `concept_id` is a stable hash of (dataset, table, canonical label). Slugs are derived from the concept, not the code. -- Estimate, MOE, percent, and annotation suffixes travel together. `E`/`M` and `PE`/`PM` pairs are matched as a unit, so an estimate can never be paired with another year's MOE. -- Comparison Profiles: the period segment is removed from the code before matching, and the `_SS` significance flags attach to their concept. -- Similarity scoring uses `difflib` token ratios, which adds no dependencies. It is used only to rank candidate pairs for review, never to merge. - -### A5. Reviewed overrides - -- Directory `crosswalks//.csv` with columns `year, code, concept_id, decision, rationale, source_url, evidence_id, drafted_by, reviewed_by, reviewed_on`. -- Decisions: `merge` (accept a `needs_review` pair), `split`, `exclude`. When in doubt, `split`. Leaving two concepts apart never corrupts data; a wrong merge does. -- Overrides take precedence over automatic rules. - -**Review requirements (decided).** A `merge` row is valid only if all of these hold, and CI rejects the file otherwise: - -1. **Machine evidence.** `evidence_id` references an evidence record generated by `check_variable_concordance`. It contains both vintages' labels, concepts, universes, and similarity score. The evidence is generated by code, never typed by hand. -2. **Primary-source citation.** `source_url` points to Census Bureau documentation that supports the decision, such as the release's table and geography changes documentation or the table shells. Third-party crosswalks (for example, NHGIS) may be cited as supporting evidence but never as the only source. -3. **Continuity check passes** (A9), or the rationale explains a real-world reason for the discontinuity. -4. **Independent sign-off (decided, revised).** Agents are expected to make these decisions. `drafted_by` and `reviewed_by` are identities of the form `human:` or `agent:/`, and they must differ. An agent reviewer must be a different session from the drafter, and it re-derives the conclusion from the evidence and the cited source instead of relying on the drafter's rationale. The workflow is documented in the MCP server instructions, in `get_research_protocol`, and in `docs/agent_protocol.md`. -5. **Codes exist.** Every referenced (year, code) exists in that vintage's metadata. - -### A6. Pipeline integration - -- `run_batch` builds the concordance before downloading, using metadata only, which needs no data calls. -- New argument `variables=[...]` on `run_batch`, `batch_download`, and the CLI. It lists the variables (codes in any one vintage, or concept IDs) that an analysis depends on. Omitted means the whole table. -- New option `ResearchOptions.concordance_mode`: - - `fail` (default): return `success: False` with `error_code: CONCORDANCE_REVIEW_REQUIRED` if any `needs_review` item, failed continuity check, or fallback-only metadata affects the selected variables. Drift in unselected lines of the table is reported but does not block. - - `strict`: build columns by concept. `needs_review` items stay split by vintage segment (for example, `..._est__2015_2018` and `..._est__2019_2022`) and are listed under `concordance_issues`. The run succeeds. Intended for exploratory whole-table pulls. - - `code`: Round 5 behavior (key by code). Allowed only for single-year requests, or with an explicit `allow_unverified_concordance=True`, which is recorded in the metadata. - -**Why `fail` is the default (decided).** The panels feed hypothesis tests. A silently pooled column changes test results without any visible symptom, while a blocked run costs only a review step. Scoping the check to `variables=` makes this strictness practical: you name the variables your hypothesis uses, those are verified, and unrelated drift elsewhere in the table does not get in the way. This also works as lightweight pre-registration, because the selected variables and their concordance are recorded in the extraction metadata. -- New output `
_concordance.csv` with columns `concept_id, slug, year, code, label, status, decided_by`. -- The codebook gains `Concept_ID`, `Years_Available`, `Source_Codes_By_Year`, and `Concordance_Status`. -- Extraction metadata records the metadata checksum for each vintage used and the override files applied. - -### A7. Scale audit (first deliverable) - -`scripts/audit_concordance.py` sweeps every profile, subject, and comparison-profile table, and a sample of detailed tables, across all vintages. It reports counts per status and lists every `renumbered`, `redefined`, and `needs_review` case. Outputs: - -1. A report committed under `docs/audits/` that quantifies the problem. -2. Real cases that become test fixtures (A8). -3. Evidence for whether the similarity threshold and the default mode are reasonable. - -### A9. Continuity check (independent safety net) - -Label matching can be fooled: an identical label can hide a definitional change. So every merged concept, whether matched automatically or through an override, is also checked against the data: - -- Fetch the concept at the national level (`us`) for each vintage. This is one cached call per year. -- For each adjacent pair of years, run `census.stats.z_test` with the overlapping-period adjustment, and compute the relative change. -- Flag a concept when the change is both statistically significant and implausibly large. The default threshold is relative change > 25% in one year for counts and > 10 percentage points for percentages; the audit (A7) will calibrate these. -- Flags appear in `concordance_issues` with `status: continuity_flag` and block selected variables in `fail` mode. - -This turns a matching error into a visible numeric discontinuity instead of a silent one. It does not prove that two vintages measure the same thing, so primary-source review is still required for overrides. - -### A8. Tests - -- Fixtures: recorded real `variables.json` excerpts for at least two vintages of one table with a confirmed renumbering, and one with a confirmed redefinition, both taken from the audit. -- Property tests: a column never mixes concepts; every panel cell traces to exactly one (year, code); E/M pairs are never split across concepts. -- Regression: a panel built from fixtures with a known renumbering produces one continuous column under `strict`, and a code reuse produces two columns. - ---- - -## 4. Workstream B (P0): R-CPI-U-RS deflator - -### B1. Rationale - -The Census Bureau adjusts historical income for inflation with the CPI-U-RS research series (published by BLS as R-CPI-U-RS), not with CPI-U. Using the same index keeps our constant-dollar figures consistent with Census-published comparisons. Task B5 confirms the exact ACS guidance wording and cites it in the docs and code. - -### B2. Index data as sourced artifacts - -- `data/price_indexes/r_cpi_u_rs.csv` and `data/price_indexes/cpi_u.csv` with columns `year, value`. -- Each has a sidecar `.json` recording the source URL, series identifier, base period, retrieval date, and SHA-256 of the raw download. -- `scripts/update_price_indexes.py` downloads from BLS, parses, and writes both files. Values are never hand-typed. This replaces the hard-coded `CPI_U_ANNUAL` dictionary in `census/inflation.py`, which was entered from memory. -- A test verifies that the vendored CSV matches its recorded checksum and that years are contiguous. - -### B3. API changes - -- `ResearchOptions.price_index: Literal["r-cpi-u-rs", "cpi-u"] = "r-cpi-u-rs"`. -- `adjust_inflation(value, from_year, to_year, index="r-cpi-u-rs")`. The MCP tool gains an `index` parameter and returns the index used and its source. -- Codebook `Unit` and extraction metadata name the index, for example `Dollars (constant 2022 dollars, R-CPI-U-RS)`. -- `cpi_table=` remains available for custom indexes. - -### B4. Year coverage - -R-CPI-U-RS annual values are published the spring after each year, before the corresponding ACS 5-year release, so coverage should not lag. If a needed year is missing, the run fails with `PRICE_INDEX_UNAVAILABLE`, naming the update script. - -**Incomplete years.** BLS did not publish October 2025 (lapse in appropriations) and never will. The R-CPI-U-RS workbook nonetheless shows a 2025 average computed from 11 months. The importer excludes any year with an unpublished month, for both indexes, so 2025 is currently unavailable. This must be resolved before the 2025 ACS 5-year release can be deflated; see section 8. - -### B5. Verification - -- Confirmed: Census's [current vs. constant dollars guidance](https://census.gov/topics/income-poverty/income/guidance/current-vs-constant-dollars.html) states that for ACS "the Census Bureau continues to use the R-CPI-U-RS for all years", while CPS ASEC income uses C-CPI-U from 2000 onward ([Census blog, 2023](https://census.gov/newsroom/blogs/research-matters/2023/09/inflation-income-and-earnings-estimates.html)). -- Open: cross-check at least two adjustment factors against a Census-published ACS constant-dollar comparison (acceptance criterion 6). Carried to Round 7. - ---- - -## 5. Workstream C (P1): agent-first interface - -The GUI predates agentic workflows. The primary users are now agents that call MCP tools or run scripts. Agents benefit most from being able to plan before acting, from structured errors, and from compact results. - -### C1. Dry-run planning - -`plan_extraction(...)`, exposed as a library function and an MCP tool, takes the same arguments as `batch_download` and returns, without downloading data: - -- the number of API calls, and how many are already cached; -- the vintages requested and their availability (from A1); -- the concordance summary and any `needs_review` items (from A4); -- the boundary and comparability advisories; -- the estimated row count and output size. - -### C2. New MCP tools - -| Tool | Purpose | -| :--- | :--- | -| `list_vintages(dataset)` | Available years from the API catalog | -| `plan_extraction(...)` | Dry run as described in C1 | -| `check_variable_concordance(table_code, start_year, end_year)` | Concordance report with statuses, candidate pairs, and evidence (labels, universes, similarity) | -| `describe_variable(code, year)` | Label, concept, and universe for one code in one vintage | - -### C3. Structured results and errors - -- Every tool returns `success`, plus `error_code` and `error` on failure. Error codes are a documented enumeration (for example `INVALID_GEO_LEVEL`, `VINTAGE_UNAVAILABLE`, `PRICE_INDEX_YEAR_MISSING`, `CONCORDANCE_REVIEW_REQUIRED`, `API_HTTP_ERROR`). -- Large data stays on disk. Tools return paths and summaries, and a row preview capped by `max_rows`. - -### C4. Command-line interface - -`python -m census` with subcommands `extract`, `plan`, `concordance`, `metadata prefetch`, and `cache stats|clear`. - -- `--json` prints a single JSON result object to stdout. Progress lines go to stderr. -- Exit codes: `0` success, `2` invalid input, `3` concordance review required (in `fail` mode), `4` API failure. - -### C5. Provenance - -Extraction metadata adds the package version and git commit (when available), the metadata checksum for each vintage, the concordance and override files used, and the price index source. Two runs with the same inputs and cache produce identical outputs, apart from timestamps. - ---- - -## 6. Workstream D (P1): validation - -- **Live contract tests** behind a pytest marker (`pytest -m live`), skipped by default. They cover each geography level, each dataset, a ZCTA list containing a missing code, catalog discovery, and metadata fetches. Round 5's real bugs (duplicate `NAME`, comparison-profile code shapes, bracket counts treated as dollars) were all found only against the live API. -- **Recorded fixtures** of real responses for the offline suite, refreshed by a script rather than edited by hand. - ---- - -## 7. Workstream E (P2): GUI maintenance mode - -- No new features. Keep the existing tests passing and keep the GUI working on top of `run_batch`. -- When Workstreams A and B land, the GUI inherits the new defaults (`strict` concordance, R-CPI-U-RS) and shows `concordance_issues` and advisories in its log. It gets no new controls. -- README labels it as legacy. - ---- - -## 8. Out of scope (candidates for Round 7) - -- **Geographic crosswalks across the 2010/2020 boundary change.** This means apportioning tract-level estimates using Census relationship files (or LTDB/NHGIS weights), including MOE propagation for apportioned values. It is a large methodological project, and today's advisory is the interim measure. -- ACS 1-year data and the decennial census. -- **Table-scoped rule overrides** for the subject-table redesigns (about 12,000 `relabeled` items; see section 13). -- **2025 price-index policy.** Before the 2025 ACS 5-year release, decide whether to accept BLS's 11-month 2025 R-CPI-U-RS average with an advisory (likely what Census itself will use) or keep 2025 unavailable. Recommendation: accept it with an advisory, once Census documents what it used for the 2025 ACS. -- **Optional C-CPI-U index**, for users who want to match CPS ASEC income reports. -- **Acceptance criterion 6:** cross-check factors against a Census-published ACS constant-dollar comparison. -- `CITATION.cff` and a README attribution section. -- Derived ratio and proportion MOE formulas beyond sums. - ---- - -## 9. Implementation roadmap - -```text -PHASE 6.0: Interim guard (superseded: the full concordance shipped in this round) - - multi-year panel/parquet requests for profile, subject, and cprofile tables - return error_code UNVERIFIED_CONCORDANCE unless allow_unverified_concordance=True - - single-year requests unaffected; detailed tables get an advisory only - - legacy CSV/Excel outputs (one file per year) get an advisory only - -PHASE 6a: Measure and model drift - a.1 audit_concordance.py and docs/audits report (quantifies the problem) - a.2 vintage discovery (A1) and per-vintage metadata registry (A2) - a.3 canonical_label rules and tests (A3) - -PHASE 6b: Concordance in the pipeline - b.1 build_concordance with statuses (A4) and override files (A5) - b.2 concept-keyed panels, concordance_mode, concordance.csv, codebook columns (A6) - b.3 fixture and property tests from audit cases (A8) - -PHASE 6c: Price index - c.1 update_price_indexes.py and vendored R-CPI-U-RS / CPI-U with checksums (B2) - c.2 price_index option, MCP parameter, codebook units (B3, B4) - c.3 cross-check against Census-published factors (B5) - -PHASE 6d: Agent interface and validation - d.1 plan_extraction, list_vintages, check_variable_concordance, describe_variable (C1, C2) - d.2 structured error codes (C3) and the python -m census CLI (C4) - d.3 provenance fields (C5) and live contract tests (D) -``` - -6.0 ships first because it closes the silent-error path in Round 5 at low cost. 6a.1 comes next because its findings may change the thresholds and defaults in 6b. - ---- - -## 10. Acceptance criteria - -1. No panel column contains values from two different concepts. This is enforced by a property test and by a runtime assertion in `build_tidy_panel`. -2. Every panel cell traces to one (year, code) through `
_concordance.csv`. -3. A known renumbering from the audit yields one continuous column. A known code reuse yields two columns. -4. Under the default `fail` mode, any unresolved drift item or continuity flag affecting the selected variables stops the run. Under `strict`, every such item appears in `concordance_issues`. -5. Every `merge` override in `crosswalks/` passes the five review requirements in A5, enforced by CI. -6. Constant-dollar values match a Census-published R-CPI-U-RS factor to within rounding. (Open; carried to Round 7.) -7. Price-index data is reproducible from `update_price_indexes.py` and matches its recorded checksum. -8. An agent can go from a plain-language request to a planned, reviewed, and executed extraction through MCP or the CLI alone. - ---- - - -## 11. Decisions - -| # | Question | Decision | Section | -| :--- | :--- | :--- | :--- | -| 1 | Default concordance mode | `fail`, scoped to the variables named in `variables=`; `strict` is opt-in for exploration | A6 | -| 2 | Bundled `variables.json` / `groups.json` | Keep as a vintage-stamped offline fallback; never sufficient alone for multi-year panels in `fail` mode | A2 | -| 3 | Override review | Agents (or humans) draft and review; the drafter and reviewer identities must differ. Five enforced requirements: generated evidence, census.gov source, continuity check, independent sign-off, codes exist. `split` when in doubt. Documented in the MCP server | A5, A9 | -| 4 | Interim behavior | Superseded: the full concordance shipped in this round. Single-year pulls never block, and agents may collate them, checking codes with `describe_variable` / `check_variable_concordance` against the archive | A6 | -| 5 | License | AGPL-3.0-or-later with a Contributor License Agreement (`LICENSE`, `CLA.md`, `CONTRIBUTING.md`) | 12 | -| 6 | Bare codes in `variables=` | Resolve in the latest requested year; `YEAR:CODE` pins a vintage, because a code can denote different measures in different years | A6 | -| 7 | Copyright holder and CLA | Copyright holder is `cafeTechne` (the project's GitHub account). `CLA.md` is adopted as written | 12 | -| 8 | Years with an unpublished month | Excluded from both price indexes, including BLS's 11-month 2025 R-CPI-U-RS average. Revisit before the 2025 ACS release (section 8) | B4 | - -## 12. Licensing (decided: AGPL-3.0-or-later with a CLA; not legal advice) - -Goal as stated: accept contributions while preventing others from taking the project and presenting it as their own. - -Every common license, including permissive ones, already requires forks to keep the original copyright notice, so misattribution is a license violation under any option below. The real choice is whether forks may be closed or commercialized. - -| Option | Forks can be closed or proprietary? | Commercial use | Effect on contributions | -| :--- | :--- | :--- | :--- | -| No license (current) | No one may legally use or modify the code | Not permitted | Contributions are legally unclear; discourages use | -| MIT / Apache-2.0 | Yes, with attribution | Allowed | Easiest to contribute to | -| GPL-3.0 | No, distributed forks must stay GPL | Allowed if source is shared | Common in research software | -| AGPL-3.0 | No, and it also covers software offered over a network (for example, a hosted MCP server) | Allowed if source is shared | Same as GPL, with stronger reciprocity | -| PolyForm Noncommercial | Source-available, not open source | Prohibited | Fewer outside contributors | - -**Recommendation:** AGPL-3.0-or-later, plus: - -- a **Contributor License Agreement** (or at minimum a DCO `Signed-off-by` requirement), so the owner keeps the right to relicense later; -- a **`CITATION.cff`** file, so academic users cite the project, which matters more for research credit than the license itself; -- a `NOTICE` or README attribution section naming the original author. - -**Adopted (2026-11-02):** AGPL-3.0-or-later (`LICENSE`), the CLA in `CLA.md` as written, and `CONTRIBUTING.md`. The copyright holder is `cafeTechne`. `CITATION.cff` and a README attribution section remain recommended and are listed for Round 7. - -## 13. Implementation status - -| Item | Status | Notes | -| :--- | :--- | :--- | -| A1 vintage discovery | Done | Archive manifest plus optional live catalog (`list_vintages refresh=True`) | -| A2 per-vintage metadata | Done | `census/metadata.py`, 114-file archive | -| A3 normalization | Done | Version 2: prefixes ("Estimate", "Percent Estimate", period), mid-label markers, dollar year, hyphens, "Subject" placeholder, order-insensitive match key | -| A4 concordance | Done | `census/concordance.py`; code reassignments explained by renumbering are non-blocking | -| A5 overrides | Done | `census/review.py`; evidence is hash-identified and regenerated during validation | -| A6 pipeline | Done | `fail` / `strict` / `code`; `variables=` with `YEAR:CODE`; `concordance.csv`; codebook concept columns | -| A7 audit | Done | `docs/audits/concordance_audit.md` (detail CSV is regenerable and gitignored) | -| A8 tests | Done | Real fixtures: DP02 2018→2019 renumbering and code reuse; B27001 2016→2017 age-band change | -| A9 continuity | Done | National Z-test plus magnitude thresholds; unverifiable merged concepts block | -| B R-CPI-U-RS | Done | `data/price_indexes/r_cpi_u_rs.csv` (2000–2024) imported from the BLS workbook, which is kept in `data/price_indexes/source/` with its SHA-256 in the sidecar. 2025 excluded (October 2025 unpublished; the workbook's 11-month average is not used). CPI-U comes from the BLS API (computed values match published ones to ±0.001). Example: 2019 to 2024 factor is 1.2307 (R-CPI-U-RS) vs. 1.2270 (CPI-U) | -| B5 verification | Partly done | ACS index choice confirmed in Census guidance; factor cross-check against a Census publication still open | -| B5 home values | Done | Advisory for pre-2022 home-value tables (ACS user note 2023-07) | -| C1–C5 agent interface | Done | `plan_extraction`, 22 MCP tools with server instructions, error codes, `python -m census`, provenance | -| D validation | Done | `tests/test_live.py` (opt-in); all live contracts pass | -| E GUI | Done | Maintenance mode; inherits defaults | -| Bug found in passing | Fixed | Legacy `compiled_unmelted_data.csv` appended rows under the first year's header when columns differed | - -**Audit findings that shape follow-up work:** the subject tables' 2013 and 2017 redesigns produce about 12,000 `relabeled` items, many of them added header parts such as "total population". Variable-scoped review handles individual analyses, but resolving a table as a whole would need **table-scoped rule overrides**: reviewed, regex-like label rewrites for one table and year boundary, validated by regenerated evidence and continuity checks. These are proposed for Round 7, together with the 2010/2020 geographic crosswalk. - diff --git a/PLAN_ROUND_7.md b/PLAN_ROUND_7.md deleted file mode 100644 index 8f9cfba..0000000 --- a/PLAN_ROUND_7.md +++ /dev/null @@ -1,428 +0,0 @@ -# PLAN_ROUND_7.md: Correct Derived Statistics, Verified Against Census - -**Planning Round:** 7 -**Date:** 2026-11-02 -**Status:** Complete: implemented, validated, reviewed, and merged (sections 16, 16a, 16b) -**Branch:** `phase7/inference-ready-statistics` (from `phase6/vintage-safe-pipeline`) - ---- - -## 1. Summary - -Rounds 5 and 6 made the *inputs* trustworthy: every value traces to one release, one code, and one reviewed concept, and dollar values use the Census Bureau's own index. Round 7 makes the *calculations* trustworthy and proves it. - -Researchers rarely test hypotheses on published estimates directly. They compute rates, combine areas, compare many places at once, and build medians for regions Census does not publish. Today the toolkit can only sum estimates and compare two of them. An agent asked for "the poverty rate of these twelve tracts, and whether it changed" would have to hand-roll four formulas, each with documented pitfalls. - -Round 7 has three goals: - -1. **Derived statistics, done the Census way.** Proportions, ratios, products, percent changes, custom-geography aggregation, and medians from distributions, each with a 90% margin of error. Where Census publishes variance replicate estimates, use them for exact variances instead of approximations. -2. **Proof against Census-published values.** An automated validation suite that recomputes numbers Census itself publishes (profile percentages, comparison-profile inflation adjustments and significance flags, replicate-based MOEs, published medians, geographic totals) and fails when we disagree beyond documented tolerances. It runs offline against frozen fixtures in every test run and live on a schedule. -3. **Close the Round 6 follow-ups:** table-scoped concordance rules, the 2010/2020 tract crosswalk, nationwide extraction, extraction replay, the 2025 price-index policy, C-CPI-U, `CITATION.cff`, and continuous integration. - -Round 8 will build an inference framework (regression, small-area models, and hypothesis-test workflows on top of `scipy` and `statsmodels`). Round 7 defines the data contract that framework will consume (section 14), so that work starts from verified numbers. - -| Priority | Workstream | Section | -| :--- | :--- | :--- | -| P0 | A. Derived estimates with approximation formulas | 3 | -| P0 | B. Exact variance from Census replicate tables | 4 | -| P0 | F. Validation against Census-published values | 8 | -| P1 | C. Multiple-comparison correction and batch testing | 5 | -| P1 | D. Custom geography aggregation | 6 | -| P1 | E. Medians from distribution tables | 7 | -| P1 | G. Table-scoped concordance rule overrides | 9 | -| P1 | I. Nationwide extraction and extraction replay | 11 | -| P2 | H. 2010/2020 tract crosswalk | 10 | -| P2 | J. Price indexes: 2025 policy, C-CPI-U | 12 | -| P2 | K. Project infrastructure: CI, `CITATION.cff`, attribution | 13 | -| P0 | L. Agent interface for all of the above | 15 | - ---- - -## 2. Principles - -1. **Every derived number carries its method.** Each result is a `DerivedEstimate` (`census/variance.py`) or a DataFrame with the same fields: `estimate`, `moe`, `se`, `cv_percent`, `method`, `notes`. A user can always tell an approximation from a replicate-exact value. -2. **Prefer exact over approximate, and say which was used.** Replicate variance when Census publishes it for the table, geography, and release; Census approximation formulas otherwise. -3. **Census documentation is the specification.** Formulas cite the ACS General Handbook (2020), chapter 8, "Calculating Measures of Error for Derived Estimates", and the Variance Replicate Estimate documentation. Where guidance changed between editions, the current edition wins and the change is noted in code. -4. **Calculation errors must be caught by comparison with Census, not by reasoning.** Every formula has at least one validation check against a number Census computed independently (section 8). -5. **Refuse rather than mislead.** Medians cannot be summed, ratios of unrelated universes are flagged, apportioned medians are refused, and estimates with undefined MOEs (sentinels) propagate as undefined. -6. **Pure functions first, pipeline second.** New statistics live in pure, vectorized modules with no network access. Network access is confined to explicit fetch functions with caching. - ---- - -## 3. Workstream A (P0): derived estimates (`census/derived.py`) - -### A1. Formulas (ACS General Handbook 2020, ch. 8) - -All inputs are estimates with published 90% MOEs. Functions accept scalars or aligned NumPy/pandas arrays and return `DerivedEstimate` (scalar) or a DataFrame (vectorized). - -| Function | Estimate | MOE | -| :--- | :--- | :--- | -| `sum_estimates(estimates, moes)` | ΣX | sqrt(Σ MOE²), with the zero-estimate rule in A2 | -| `difference(x, moe_x, y, moe_y)` | X − Y | sqrt(MOE_X² + MOE_Y²) | -| `proportion(num, moe_num, den, moe_den)` | P = X/Y, X a subset of Y | (1/Y)·sqrt(MOE_X² − P²·MOE_Y²); if the radicand is negative, use the ratio formula and record `proportion_ratio_fallback` | -| `ratio(num, moe_num, den, moe_den)` | R = X/Y, X not a subset of Y | (1/Y)·sqrt(MOE_X² + R²·MOE_Y²) | -| `product(x, moe_x, y, moe_y)` | X·Y | sqrt(X²·MOE_Y² + Y²·MOE_X²) | -| `percent_change(current, moe_c, prior, moe_p)` | 100·(C − P)/P | 100 × ratio MOE of C/P | -| `scale(x, moe_x, k)` | k·X for a constant k | \|k\|·MOE_X | - -Percentages are returned on the scale requested (`as_percent=True` multiplies estimate and MOE by 100). - -### A2. Edge cases - -- **Zero estimates in sums.** Round 5 implemented "include only the largest MOE among zero estimates". **Finding:** that rule appears in 2016 Census training slides but not in the 2018 or 2020 Handbook ch. 8, the 2025 Worked Examples, or the 2025 Instructions for Statistical Testing, all of which sum every component's squared MOE. The default is now `zero_estimate_rule="all"` (also in `census.stats.aggregate_estimates`), with `"largest"` available. -- **Zero denominator.** Estimate and MOE are NaN with a note; never infinity. -- **Undefined component MOE** (Census sentinel or NaN): the result MOE is NaN, the note names the component. -- **Controlled estimates** (MOE published as `-555555555`, meaning the estimate is controlled and has no sampling error): the MOE is treated as 0, and the note says so. -- **Correlated components.** The approximation formulas ignore covariance. Results carry a note when both inputs come from the same table and are not in a subset relationship. Workstream B gives the exact alternative. -- **Approximation quality.** The handbook warns that approximations get worse as more estimates are combined. `sum_estimates` notes when more than 4 components are combined and replicate data is not used. - -### A3. DataFrame helpers - -- `derive_columns(df, spec)` adds derived columns to a wide frame. `spec` is a list of `{"name", "op", "num", "den"}` entries using estimate codes (MOE columns are found by the `E`/`M` suffix convention in `census.stats`). -- `subset_relationship(num_code, den_code, metadata)` returns `True` when the numerator's label path lies under the denominator's in the table's label hierarchy. This chooses `proportion` or `ratio` automatically and is overridable. - -### A4. Tests - -- Handbook worked examples, reproduced to the published precision. -- Property tests: proportion MOE ≤ ratio MOE when the radicand is non-negative; scale invariance; symmetry of `difference`. -- Validation checks in section 8 (F2, F4). - ---- - -## 4. Workstream B (P0): exact variance from replicate tables (`census/replicate.py`) - -### B1. Background - -Census publishes **Variance Replicate Estimate (VRE) tables** for a subset of ACS 5-year detailed tables, from the 2010–2014 release onward (verify the first year). Each row holds an estimate, its MOE, and 80 replicate estimates `Var_Rep1`…`Var_Rep80`. The variance of any statistic f is computed by evaluating f on each replicate: - -Var(f) = 4/80 · Σ_r (f_r − f)². - -This captures covariance between components, so ratios and sums of correlated estimates are exact under the survey design. The primitives already exist in `census/variance.py`. - -### B2. Discovery and download - -- `vre_tables(year)`: the list of tables Census publishes VREs for in a release, read from the release's published table list. Cached in `.cache/vre/`. -- `vre_available(table, year, geo_level)`: boolean. -- `fetch_vre(table, year, geo_level, state_code=None)`: downloads (or reads from cache) and returns a **replicate frame** (B3). Tract and block-group files are per state; other summary levels are national files. -- Sources come from the release's documentation page (`https://www.census.gov/programs-surveys/acs/data/variance-tables.html` and the per-year pages). File URLs and layouts must be confirmed per release and recorded in a registry (`census/replicate.py: VRE_LAYOUTS`) rather than guessed. -- Downloads go to `.cache/vre//` with a SHA-256 recorded in a sidecar. Nothing under `.cache/` is committed. A small frozen fixture (one table, a handful of geographies) is committed under `tests/fixtures/vre/` for offline tests. - -### B3. Replicate frame contract - -A long DataFrame with one row per (geography, variable): - -| Column | Meaning | -| :--- | :--- | -| `GEOID` | Standard GEOID matching `census.geo.build_geoid` (strip the `GEO_ID` summary-level prefix) | -| `NAME` | Geography name | -| `variable` | API variable code without suffix, e.g. `B19001_002` | -| `estimate` | Published estimate | -| `moe` | Published 90% MOE | -| `rep_1` … `rep_80` | Replicate estimates (`census.variance.REPLICATE_COLUMNS`) | - -### B4. Exact derived statistics - -- `replicate_sum(frame, variables, geoids=None)`, `replicate_ratio(frame, num, den)`, `replicate_proportion(...)`, `replicate_difference(...)`: evaluate the statistic on the estimate and on every replicate, then apply `census.variance.replicate_moe`. Returns a DataFrame with the `DerivedEstimate` fields and `method="replicate"`. -- `replicate_apply(frame, func)`: general hook. `func` maps a (geography × variable) array to a statistic; it is applied to the estimate column and to each replicate column. Medians (E) and aggregation (D) use this. -- Self-check on load: recomputing each row's MOE from its own replicates must match the published MOE to within rounding (this is check F3, also run as an assertion with a warning on load). - -### B5. Tests - -- Fixture parse, GEOID normalization, MOE self-reproduction on the fixture. -- Exact replicate ratio equals the approximation formula when replicates are perfectly uncorrelated (synthetic data). -- Live opt-in test downloads one real table for one state and reproduces published MOEs. - ---- - -## 5. Workstream C (P1): multiple comparisons and batch testing (`census/multitest.py`) - -### C1. Corrections - -`adjust_pvalues(pvalues, method)` with `method` in `bonferroni`, `holm`, `bh` (Benjamini–Hochberg FDR), `by` (Benjamini–Yekutieli). Pure NumPy, NaN-aware (NaN p-values are excluded from the family and returned as NaN). Results are cross-checked in tests against `statsmodels.stats.multitest.multipletests` when `statsmodels` is installed. - -### C2. Batch significance - -`batch_z_test(df, est1, moe1, est2, moe2, year1=None, year2=None, confidence_level=0.90, correction="holm")` returns one row per geography: difference, its MOE, z, raw p, adjusted p, `significant_raw`, `significant_adjusted`, the overlap factor applied, and the family size. - -- Default correction is `holm`, since it controls family-wise error and is uniformly more powerful than Bonferroni; `bh` is recommended for exploratory screening. Census's own published comparisons use Bonferroni within a family for some products; the docstring says so. -- Overlapping-period adjustment comes from `census.stats.overlap_factor`. Comparisons of overlapping 5-year periods carry the note Census attaches: overlapping periods share sample, and Census advises comparing non-overlapping periods where possible. - -### C3. Tests - -Known p-value vectors with hand-computed adjusted values; agreement with `statsmodels` to 1e-12 when available; batch test on a small frame. - ---- - -## 6. Workstream D (P1): custom geography aggregation (`census/aggregate.py`) - -- `aggregate_geographies(df, mapping, estimate_cols, method="auto", replicate_frame=None)`: `mapping` is GEOID → group ID (a dict or a two-column frame). Sums estimates within groups and computes MOEs: - - `method="replicate"` when a replicate frame covering every member is supplied (exact). - - `method="approximate"` uses `derived.sum_estimates` per group. - - `"auto"` picks replicate if available, otherwise approximate, recording which. -- `aggregate_then_derive(df, mapping, spec, ...)`: aggregate counts first, then compute proportions or ratios on the aggregates (the correct order; averaging tract rates is wrong and is refused). -- Refuses medians, means, percentages, and other non-additive measures, detected from codebook units (`census.panel.infer_unit`) and labels. The error names Workstream E for medians. -- Output keeps `group_id`, `member_count`, `members` (semicolon-joined GEOIDs), and the `DerivedEstimate` fields per variable. - -Tests: synthetic sums; refusal of non-additive columns; replicate path equals sum of replicate columns. - ---- - -## 7. Workstream E (P1): medians from distribution tables (`census/medians.py`) - -### E1. Method - -Census computes medians from finer internal distributions. For user-defined areas only the published distribution tables are available (for example B19001 household income brackets, B25063 gross rent, B25075 home value). - -- `median_from_distribution(lower_bounds, upper_bounds, counts)`: locate the bracket containing the 50th percentile and interpolate. **Finding:** the ACS Subject Definitions state that ACS medians are "calculated using linear interpolation" from Census's internal median standard distributions (household income in $2,500 intervals to $250,000+); no ACS source specifies Pareto interpolation, which appears only for older CPS medians. Linear is the default; Pareto is offered for comparison only. -- Open-ended top bracket: if the median falls in it, return the bracket's lower bound with a "+" note, mirroring Census's own `250,000+` convention, and an undefined MOE. -- `distribution_brackets(table, year)`: bracket bounds parsed from that release's variable labels in the metadata archive (labels such as "$10,000 to $14,999", "Less than $10,000", "$200,000 or more"). Bounds come from labels, never from hard-coded lists, because brackets changed across releases. - -### E2. MOE - -1. **Replicate method (exact, preferred):** compute the median on each of the 80 replicate distributions using `replicate.replicate_apply`, then apply the replicate variance formula. -2. **Approximation (fallback):** the Census "standard error of a 50 percent proportion" method from the PUMS Accuracy of the Data (form 50% ± SE on the cumulative distribution, interpolate both bounds, SE = half the width; the three worked examples reproduce exactly). **Finding:** design factors are published only for PUMS, and the formula's 19/B term reflects a 5% PUMS sample, so it overstates table-based MOEs (median ratio to published MOE about 1.5–1.6). The default instead fits the same variance function to the published bracket MOEs (g = ΣSE²/ΣY(1−Y/B)); this is a documented adaptation, not a Census formula, tracking published county MOEs at a median ratio of about 1.07 (income) and 1.12 (rent). The design-factor method is used only when the caller passes one. - -### E3. Tests - -Hand-computed interpolation examples; open-ended bracket behavior; agreement with published county medians (validation check F5). - ---- - -## 8. Workstream F (P0): validation against Census-published values - -This is the ultimate test of the toolkit's calculations. Comparing our API output with the API itself would be circular. Instead, every check recomputes something **Census computed independently** from the same underlying data, and compares. - -### F1. Check catalogue - -| ID | What we recompute | Census's own value | What it proves | Tolerance | -| :--- | :--- | :--- | :--- | :--- | -| F1 | Raw values through our pipeline (tidy panel, Parquet, CSV) | The same cells read directly from the API response | No value is lost, shifted, or mislabeled by parsing, concordance, pivoting, or export | Exact | -| F2 | Proportions and their MOEs from detailed-table counts (`derived.proportion`) | Data Profile percentages and percent MOEs (`DPxx_xxxxPE`/`PM`) for the same concept | Proportion formula, subset logic, rounding | Estimate within 0.1 point; MOE within the larger of 0.2 points or 25% (Census uses replicate MOEs; approximation differs) | -| F3 | MOEs from 80 replicates (`census.variance.replicate_moe`) | The MOE published in the same VRE file and in the API | Replicate variance formula and VRE parsing | Within 1 unit of rounding | -| F4 | Exact replicate proportion MOE | Data Profile percent MOE | Replicate-based derivation matches Census to rounding | Within 0.1 point | -| F5 | Medians from distribution tables (`medians.median_from_distribution`) | Published medians: `B25064` from `B25063` (same brackets Census uses) and `B19013` from `B19001` (coarser than Census's internal $2,500 distribution) | Interpolation | Rent: within $1. Income: median error under 1%, and at least 85% of counties within 2% (measured nationally: median 0.61%, 87% within 2%, maximum 19.9%) | -| F6 | Prior-period dollars adjusted with R-CPI-U-RS (`inflation.adjust_inflation`) | Comparison Profile values, which Census publishes in current-period dollars for the prior period | The price index file and deflation formula; closes Round 6 acceptance criterion 6 | Within 0.5% or the published rounding | -| F7 | Significance of change (`stats.z_test`) between two non-overlapping 5-year periods | Comparison Profile significance flags (`CPxx_xxxx_..SS` or equivalent) | Z-test, confidence level, MOE handling | Agreement on ≥ 98% of tested items; every disagreement listed with z near the threshold | -| F8 | Sums of counties (`aggregate.aggregate_geographies`) | The state estimate | Aggregation and GEOID mapping | Estimate exact for additive counts; MOE ratio reported, not asserted | -| F9 | Concordance continuity (Round 6) | National values for merged concepts | Already in Round 6 continuity check; included in the report for completeness | As Round 6 | -| F10 | Optionally, Summary File cells | The ACS 5-year table-based Summary File on `www2.census.gov` | An independent Census delivery channel agrees with the API values we extracted | Exact | - -F10 uses a different publication channel (bulk files) than the API. data.census.gov itself serves the same database as the API and has no documented public API, so it is not used as an oracle; the Summary File is the independent bulk equivalent. - -### F2. Implementation - -- `census/validation.py`: one function per check, each returning a `ValidationResult` (`check_id`, `description`, `n_compared`, `n_failed`, `max_abs_error`, `max_rel_error`, `tolerance`, `failures` sample, `census_source` URLs, `passed`). -- `scripts/validate_against_census.py`: runs a configurable battery (default: a fixed list of states, tables, releases) and writes `docs/audits/validation_report.md` plus a JSON report. Exit code 1 on any failure. -- **Frozen fixtures** under `tests/fixtures/validation/`: small API responses and VRE extracts captured once with SHA-256 sidecars, so `tests/test_validation.py` runs every check offline in every test run. -- **Live mode**: `CENSUS_LIVE_TESTS=1 pytest tests/test_validation_live.py` and the scheduled CI job (K1) run the same checks against the live API and current files. -- Each check's tolerance and the reason for it are documented in the report, so a widened tolerance is a visible decision. - -### F3. Default battery - -At minimum: states 06, 36, 48, 17, 56 (large, medium, small); releases 2019 and 2023; tables B17001/DP03 (poverty), B27001/DP03 (insurance), B19001/B19013 (income median), B25064/B25063 (rent median), CP03 for F6/F7. The agent implementing F finalizes exact variable pairings and records each pairing's Census definition in the report. - ---- - -## 9. Workstream G (P1): table-scoped concordance rule overrides - -### G1. Problem - -The Round 6 audit found about 12,000 `relabeled` items, mostly from the 2013 and 2017 subject-table redesigns. Many differ only by systematic header changes (for example a header part such as "Total population" added before every label). Reviewing them one by one is impractical. - -### G2. Design - -- New override decision type `rule`, stored in `crosswalks//
_rules.csv` with columns: `table`, `from_year`, `to_year`, `pattern`, `replacement`, `rationale`, `source_url`, `evidence_id`, `drafted_by`, `reviewed_by`, `reviewed_on`. -- A rule is a regular-expression rewrite applied to canonical labels of the **earlier** vintage, for one table and one year boundary, before match keys are computed. Rules never span tables or boundaries. -- Evidence for a rule is generated: the full list of (code, before, after) label pairs it rewrites and the resulting matches, hashed into the `evidence_id`. If the archive changes, the hash changes and validation fails. -- Rule application can only produce `merge` candidates. Every merged concept still goes through the continuity check (A9 in Round 6); a rule does not exempt anything. -- Guardrails: a rule that maps two earlier codes to one later concept, or that changes any digit or numeric bound in a label (for example age or dollar bounds), is rejected at validation. Numbers changing is exactly the case that must stay blocked. -- Drafting and review use the same two-identity process as Round 6 (`draft_rule_override`, `record_rule_review`), and `validate_all_crosswalks` validates rule files too. - -### G3. Tests - -A rule that resolves a real subject-table header change (from the audit) produces merges that pass continuity; a rule that would change a numeric bound is rejected; stale evidence is rejected. - ---- - -## 10. Workstream H (P2): 2010/2020 tract crosswalk (`census/geo_crosswalk.py`) - -### H1. Data - -- Census **2020 Census Tract to 2010 Census Tract Relationship File** (national, pipe-delimited) from the Census geography relationship-files page. It gives land and water area of each intersection. -- `scripts/fetch_tract_relationships.py` downloads it, records SHA-256 and source URL in a sidecar, and writes a compact gzipped extract to `data/geo_crosswalks/` (committed if under 10 MB; otherwise fetched on demand into `.cache/`). - -### H2. Method - -- `tract_crosswalk(direction="2010_to_2020", weight="land_area")`: a weights table (`source_geoid`, `target_geoid`, `weight`), where weights from each source tract sum to 1. -- `weight="population"` accepts a user-supplied weight file (for example NHGIS block-based crosswalks, which are more accurate but have their own terms of use), validated for the same columns and row sums. -- `apportion(df, crosswalk, estimate_cols)`: target estimate = Σ w·X. MOE for each target: sqrt(Σ (w·MOE)²). This treats components as independent and ignores weight uncertainty. Results use `method="apportioned"` with a note naming the weight type, and the advisory lists the limitation. -- Refuses non-additive measures (medians, means, percentages); for those, apportion the underlying counts and re-derive. -- Pipeline integration: `ResearchOptions.harmonize_tracts_to: Optional[int]` (2010 or 2020). When set on a tract panel spanning the boundary, earlier-vintage tracts are apportioned to the target vintage, and the existing boundary advisory is replaced by a harmonization record in the metadata. - -### H3. Tests - -Weights sum to 1; a tract that was unchanged maps with weight 1; a known split tract apportions to its successors; non-additive refusal. - ---- - -## 11. Workstream I (P1): nationwide extraction and replay - -### I1. Nationwide extraction - -- `state_code="all"` in `plan_extraction`, `run_batch` (via entity expansion), the CLI, and MCP. Expands to the 50 states plus DC (Puerto Rico opt-in with `include_pr=True`) using `census.geo.STATE_MAPPING`. -- County and place levels use one request per year; tract level loops over states. **Finding:** `in=state:*` works for county and place but includes Puerto Rico, and tracts reject a state wildcard, so nationwide requests pass the explicit list of 51 state codes. -- The plan reports the number of calls; the run resumes from the query cache after interruption, and `failed_calls` lists anything to retry. - -### I2. Replay - -- `python -m census replay
_extraction_metadata.json [--output-dir DIR] [--verify]` re-runs an extraction from its metadata (table, entities, years, variables, options, archive SHAs). -- `--verify` compares output file hashes and, if they differ, the data cell by cell, reporting which values changed. Census occasionally revises published data, so a mismatch is reported precisely rather than treated as a bug. -- Metadata gains whatever fields replay needs (for example `variables` and the exact entity list) and a `replay_command` string. -- MCP: `replay_extraction(metadata_path, verify=True)`. - ---- - -## 12. Workstream J (P2): price indexes - -- **2025 policy.** `scripts/update_price_indexes.py` records years with an unpublished month in a separate `provisional` section of the sidecar, with BLS's published (11-month) average where one exists. `load_price_index` ignores provisional years unless `allow_provisional_index=True` (a new `ResearchOptions` field and MCP parameter), which adds an advisory naming the missing months. Default stays strict until Census documents what it used for the 2025 ACS. -- **C-CPI-U.** Add `c-cpi-u` (BLS series `SUUR0000SA0`, chained CPI, not seasonally adjusted, December 1999 = 100) from the BLS API. Annual averages computed from 12 months. The sidecar notes that C-CPI-U values are revised for about 10–12 months after first release, and records whether each year is final. Purpose: match CPS ASEC income reports, which use C-CPI-U from 2000 onward. -- **Factor cross-check** (Round 6 acceptance criterion 6) is validation check F6. - ---- - -## 13. Workstream K (P2): project infrastructure - -- **K1. Continuous integration** (`.github/workflows/ci.yml`): offline test suite on Linux and Windows with Python 3.10–3.12 on every push and pull request, plus `validate-crosswalks`. A scheduled weekly job runs the live validation battery (F) with a `CENSUS_API_KEY` repository secret and uploads the report as an artifact. -- **K2. `CITATION.cff`** naming the project, `cafeTechne`, the license, and the repository URL. -- **K3. README attribution** section: original author, how to cite, and a pointer to `CITATION.cff`. - ---- - -## 14. Data contract for the Round 8 inference framework - -So that inference code can be built directly on verified outputs: - -- Every derived or published estimate available to analysis carries `estimate`, `moe`, `se`, `cv_percent`, `method`, and provenance (release, code or concept ID, geography). -- Where replicate data exists, the tidy panel can optionally carry the 80 replicate values (`include_replicates=True`), enabling replicate-weighted regression and design-consistent variance for any statistic. -- `se` is always on the estimate's scale (not percent of estimate), which is what `statsmodels` weights and measurement-error models expect. -- Round 8 candidates: regression with known sampling variance (weighted least squares, measurement-error-aware models), Fay–Herriot small-area estimation, spatial models, and pre-registered hypothesis manifests that lock variables, geographies, releases, tests, and correction method before data are pulled. - ---- - -## 15. Workstream L (P0): agent interface - -New MCP tools (all return `DerivedEstimate` fields and `method`): - -| Tool | Purpose | -| :--- | :--- | -| `derive_estimate` | Proportion, ratio, product, difference, percent change, or sum from given estimates and MOEs | -| `aggregate_geographies` | Combine geographies from a downloaded file or inline data, then optionally derive rates | -| `median_from_distribution` | Median and MOE from a distribution table for one geography or a group | -| `check_replicate_availability` | Whether Census publishes replicate estimates for a table, release, and geography level | -| `batch_significance` | Many-geography comparison with multiple-comparison correction | -| `apportion_tracts` | 2010↔2020 tract apportionment | -| `run_validation` | Run the Census validation battery (or one check) and return the report | -| `replay_extraction` | Re-run and verify an extraction from its metadata | -| `draft_rule_override`, `record_rule_review` | Table-scoped concordance rules | - -CLI subcommands mirror these (`derive`, `aggregate`, `median`, `validate`, `replay`, `rules`). `docs/agent_protocol.md` gains sections on choosing proportion vs ratio, when replicate variance is required (recommended for any derived estimate feeding a hypothesis test, when available), multiple-comparison defaults, aggregation order, and the validation suite. - ---- - -## 16. Implementation plan and status - -Work was split into parallel streams that owned disjoint files; integration (package exports, MCP, CLI, docs) happened last. - -| Stream | Files | Status | Key results | -| :--- | :--- | :--- | :--- | -| Shared contract | `census/variance.py` | Done | `DerivedEstimate`, replicate variance primitives | -| A Derived estimates | `census/derived.py` | Done | Handbook 2020 ch. 8 examples 1–4 and the product example reproduce to published precision, as do the 2025 Worked Examples 2–5 | -| B Replicate tables | `census/replicate.py` | Done | Releases 2014–2024; URL patterns, layouts, and table lists recorded with sources; 769,533 real rows across 9 files reproduce published MOEs with 0 mismatches (max gap 0.49999). Zero-variance cells use Census model-based MOEs and are returned as NaN | -| C Multiple comparisons | `census/multitest.py` | Done | Agrees with `statsmodels` to 1e-12 for all four corrections | -| D Aggregation | `census/aggregate.py` | Done | Replicate and approximate paths; non-additive measures refused | -| E Medians | `census/medians.py` | Done | Linear interpolation per ACS Subject Definitions; rent reproduces B25064 to $1; income from B19001 within 2% for 87% of counties | -| G Rule overrides | `census/concordance.py`, `census/review.py` | Done | Rules link concepts across a boundary; S1701 2016→2017: two rules resolve 165 of 171 changed lines; an S1601 rule is rejected for changing numbers in 24 labels | -| H Tract crosswalk | `census/geo_crosswalk.py`, `data/geo_crosswalks/` | Done | Census national relationship file (2.1 MB gzipped extract); ACS tract renumberings and county recodes after 2010; Connecticut 2022 planning-region recode (879 tracts); pipeline option `harmonize_tracts_to` | -| I Nationwide and replay | `census/pipeline.py`, `census/__main__.py` | Done | `state_code="all"`; county nationwide = 1 call per year (3,144 counties); replay with file-hash and cell-level verification | -| J Price indexes | `census/inflation.py`, `scripts/update_price_indexes.py` | Done | C-CPI-U (2000–2024); provisional 2025 R-CPI-U-RS behind `allow_provisional_index` with advisory | -| K Infrastructure | `.github/workflows/ci.yml`, `CITATION.cff`, README | Done | Linux and Windows, Python 3.10–3.12; weekly live validation | -| L Interfaces | `census/workflows.py`, `mcp_server.py` (32 tools), CLI, docs | Done | One implementation shared by MCP and CLI | -| F Validation | `census/validation.py`, `scripts/validate_against_census.py`, fixtures, report | Done | All ten checks pass; two toolkit bugs found and fixed (section 16a) | - -### 16a. Validation results (live, 2026-11-02) - -All ten checks pass live and offline (`docs/audits/validation_report.md`). - -| Check | Compared | Result | -| :--- | :--- | :--- | -| F1 pipeline cells vs API | 9,800 cells (B01001, WY + 2 CA counties, 2019 and 2023, CSV panel and Parquet) | Exact | -| F2 DP percentages from detailed tables | 5,040 (5 states, 499 counties, 5 constructions, 2 releases) | Estimates within 0.05 points (DP rounding); toolkit MOEs equal an independent coding of the Handbook formulas in every row | -| F3 replicate MOEs | 29,572 (8 tables) | All within 0.5 (rounding); every VRE cell equals the API cell | -| F4 replicate proportions vs DP MOEs | 1,038 | Estimates within 0.05, MOEs within 0.098 points | -| F5 medians | 1,983 counties | Rent: exact to $1 in both releases. Income: median error 0.49% (2019), 0.64% (2023); 98.8% and 99.8% of counties within the published MOE | -| F6 CP inflation | 125 | Census's implied factors exceed ours by 0.008–0.010% (consistent with Census using the index rounded to one decimal). Closes Round 6 criterion 6 | -| F7 CP significance | 616 tests, 442 decidable | 100% agreement on every test whose outcome is determined by the published (rounded) values | -| F8 county sums vs state | 110 | Estimates exact | -| F9 continuity | 48 concepts | No flags | -| F10 Summary File vs API | 960 cells (2019–2023 table-based Summary File) | Identical | - -**Bugs found by validation and fixed:** - -1. `census/medians.py`: when exactly 50% of the distribution lay at a bracket boundary followed by empty brackets, the toolkit returned the gap's lower edge; Census publishes the gap midpoint (2019 B25064, Sterling and Stonewall counties, TX). Fixed; both now match. -2. `census/continuity.py`: thresholds meant for adjacent releases were applied to any requested pair, so 2019+2023 extractions of B19001, B17001, and B27010 were blocked although every concept was stable. Thresholds now scale with the gap between releases (relative threshold compounds, percentage-point threshold scales linearly). - -**Criteria decided during validation (disclosed, not hidden):** - -- **F2 MOE band.** The Handbook approximation ignores covariance, so 215 county owner-occupancy MOEs fall outside the max(0.2 point, 25%) band (median ratio 0.88). For every one of them the exact replicate MOE reproduces the published MOE within 0.1 point (F4), so they are counted as explained, not as failures. The approximate/published MOE ratios are reported per construction (poverty 1.00, uninsured 0.89, owner-occupied 0.88, age 65+ 4.4, bachelor's 1.02); the age 65+ ratio reflects summing 12 correlated cells, which is exactly when replicate variances should be used. -- **F5 income criterion, changed after the first live run.** The plan's criterion (85% of counties within 2%) was set before measurement; 2023 measured 84.1%, because B19001's brackets are much coarser than Census's internal $2,500 distribution in small counties. Since a calculation error would push medians outside Census's own sampling error, the criterion is now: median relative error under 1% and at least 98% of counties within the published MOE. The 2% share is still reported. This change was made after seeing the result and is recorded here so reviewers can judge it. -- **F6** asserts the 0.5% tolerance only for items Census converts with one constant factor; in the 2019 Comparison Profile, Census re-derived medians rather than scaling them (implied factors 1.0747–1.0838), which no deflator can reproduce; those items are reported, not asserted. -- **F7** asserts the 98% agreement rate on decidable tests only; 174 tests straddle z = 1.645 within the rounding of the published values and are listed with their z ranges. - -### 16b. Code review and CI (PR #1) - -CI on its first run found two environment problems, both fixed: MCP SDK 2.x renamed `FastMCP` (pinned `mcp>=1.2.0,<2`), and pandas 3 renamed its string dtype (test made version-agnostic). A high-effort code review of the full branch found ten issues, all fixed with regression tests in `tests/test_review_fixes.py`: - -- replicate median MOEs summed every row of a whole-state replicate file instead of the selected geographies; -- same-period comparisons applied an overlap factor of sqrt(0), marking every difference significant; -- `z_test` mishandled controlled, sentinel, and missing MOEs; -- aggregated zero-variance groups and zero replicate denominators did not follow the VRE documentation (Examples 5 and 7); -- panels could be renamed with another table's codebook; -- the exact-boundary median rule used a relative tolerance that misfired for large totals; -- the 2010 boundary-change check ignored GEOIDs that had lost leading zeros; -- the Connecticut 2022 planning-region change produced no advisory unless a span also crossed 2020; -- the `mcp` lower bound allowed versions without `FastMCP`. - -Final state: 460 tests pass offline on Linux and Windows, Python 3.10–3.12; the validation battery passes live and offline. - -## 17. Acceptance criteria - -1. Every derived-statistic function returns its method, and the method appears in output files and MCP results. -2. Handbook worked examples reproduce to published precision. -3. Replicate MOEs reproduce Census-published MOEs to rounding on the fixture and in live validation (F3). -4. Data Profile percentages and MOEs are reproduced from detailed tables within the stated tolerances (F2, F4). -5. Comparison Profile inflation-adjusted values are reproduced with R-CPI-U-RS (F6), closing Round 6 criterion 6. -6. Comparison Profile significance flags are reproduced at the stated agreement rate (F7). -7. The offline test suite runs every validation check against frozen fixtures; the live battery passes on its first run, or every failure is explained in the report. -8. Non-additive measures are refused by aggregation and apportionment with a message naming the correct approach. -9. A rule override that changes a numeric bound is rejected. -10. A nationwide county extraction plans and runs from a single call; an extraction replays from its metadata and verifies. -11. CI runs the offline suite on Linux and Windows; the price-index checksum tests pass on both. - ---- - -## 18. Out of scope (Round 8 and later) - -- The inference framework (section 14). -- PUMS microdata and custom tabulations. -- Boundary files and mapping output. -- ACS 1-year and decennial census data. -- Block-group geographies and the 2000/2010 crosswalk. - ---- - -## Sources - -- [ACS General Handbook (2020), chapter 8: Calculating Measures of Error for Derived Estimates](https://www.census.gov/programs-surveys/acs/library/handbooks/general.html) -- [ACS Variance Replicate Estimate Tables](https://www.census.gov/programs-surveys/acs/data/variance-tables.html) -- [Census: current vs. constant dollars](https://census.gov/topics/income-poverty/income/guidance/current-vs-constant-dollars.html) -- [Census geography relationship files](https://www.census.gov/geographies/reference-files/time-series/geo/relationship-files.html) diff --git a/PLAN_ROUND_8.md b/PLAN_ROUND_8.md deleted file mode 100644 index 58c6989..0000000 --- a/PLAN_ROUND_8.md +++ /dev/null @@ -1,352 +0,0 @@ -# PLAN_ROUND_8.md: An Inference Framework That Respects ACS Sampling Error - -**Planning Round:** 8 -**Date:** 2026-11-02 -**Status:** Implemented (section 15) -**Branch:** `phase8/inference-framework` (from `defaultBranch` after the Rounds 1–7 merge) - ---- - -## 1. Summary - -Rounds 5–7 made the data and the arithmetic trustworthy: every value traces to one release and one reviewed concept, every derived estimate carries a margin of error and the method that produced it, and the calculations reproduce numbers the Census Bureau publishes. Round 8 builds the layer researchers actually test hypotheses with: models. - -The central problem is that ACS estimates are not data in the usual sense. Each one is a survey estimate with known, often large, sampling error, and standard statistical software ignores that: - -1. **Outcomes with sampling error.** A regression of tract poverty rates on covariates treats a rate with a 15-point MOE the same as one with a 2-point MOE. Standard errors are wrong and small-area noise dominates. -2. **Covariates with sampling error.** Measurement error in a predictor biases its coefficient toward zero (attenuation). Ordinary regression cannot detect this. -3. **Overlapping periods are correlated.** Consecutive 5-year releases share 4 of 5 sample years. A panel of annual 5-year estimates looks like 10 observations per tract but contains far less independent information; naive panel models overstate precision. -4. **Many tests at once.** Round 7 added corrections for batch comparisons; models need the same discipline across hypotheses. -5. **Analytic flexibility.** With thousands of tables and geographies, results can be found by searching. Pre-registration is the standard defense, and nothing in the toolkit supports it yet. - -Round 8 delivers models that account for these, a pre-registration workflow that locks an analysis before data are pulled, and, as in Round 7, proof: simulation studies showing that the intervals cover the truth at the stated rate, and reproduction of published reference results. - -| Priority | Workstream | Section | -| :--- | :--- | :--- | -| P0 | A. Analysis data contract (panels with SEs and optional replicates) | 3 | -| P0 | B. Regression with known sampling error (replicate and model-based) | 4 | -| P0 | F. Inference validation: coverage simulations and reference results | 8 | -| P1 | C. Small-area estimation (Fay–Herriot) | 5 | -| P1 | D. Pre-registered hypothesis manifests | 6 | -| P1 | E. Planning tools: minimum detectable differences and reliability screens | 7 | -| P2 | G. Time and overlap-aware designs | 9 | -| P0 | H. Agent interface and reporting | 10 | -| P1 | I. Publication outputs: figures, correlation matrices, tables | 10a | -| P0 | J. Local-model readiness: an MCP server small open models can use | 10b | - -The toolkit must remain usable after development without Claude: by researchers directly (library and CLI) and by agents running on local open-weights models through MCP. Workstream J makes that a tested requirement, not a hope. - ---- - -## 2. Principles - -1. **Never discard the MOE.** Every model input keeps its standard error; every model output reports which sources of uncertainty its standard errors include (sampling error of inputs, model error, or both). -2. **Prefer design-based variance.** Where Census publishes replicate estimates, sampling uncertainty in model coefficients comes from refitting on all 80 replicates, which requires no modeling assumptions about measurement error. Model-based corrections are the fallback, and results say which was used. -3. **Validate inference, not just arithmetic.** A confidence-interval method is accepted only after simulation shows it covers the true value at close to the nominal rate under ACS-like error, and after it reproduces a published reference result where one exists. -4. **Pre-register, then run.** Confirmatory analyses run from a locked manifest; anything not in the manifest is labeled exploratory in outputs. -5. **Established libraries only.** Statistics come from `scipy` and `statsmodels`, figures from `matplotlib`: widely cited, peer-reviewed or long-standing projects with stable APIs (decision 1). We implement only what they lack (replicate variance, ACS-specific corrections, Fay–Herriot) and cross-check our code against them wherever they overlap. -6. **Same discipline as before.** Census and published methodological sources are the specification, every result records its method and provenance, and agents can run every step through MCP or the CLI. - ---- - -## 3. Workstream A (P0): analysis data contract - -Round 7 section 14 defined the contract; Round 8 implements it. - -- **A1. `analysis_frame(...)`** builds a model-ready table from one or more extractions: one row per geography (and release, for panels), with `` (estimate), `_se`, `_cv`, `_method`, and provenance columns (release, concept ID, GEOID). Derived variables (rates, aggregates, medians) come through `census.derived` / `census.aggregate` / `census.medians` with their SEs. -- **A2. Replicate attachment.** `include_replicates=True` attaches the 80 replicate values for every variable where Census publishes them (`census.replicate`), stored in a companion long frame (`GEOID, variable, rep_1..rep_80`) or Parquet file to keep the main table narrow. Derived variables get replicate values by evaluating the derivation on each replicate. -- **A3. Coverage report.** For each variable: whether replicates exist, the share of rows with undefined SEs, and the CV distribution, so an agent can see before modeling which variables are noisy or lack exact variances. -- **A4. Panel outputs gain SE columns** consistently (`*_se` in addition to `*_moe`), so `statsmodels` weights and measurement-error models can be applied directly. - ---- - -## 4. Workstream B (P0): regression with known sampling error - -### B1. Replicate-based inference (preferred) - -For a model with coefficient vector β fit on full-sample estimates, refit it on each of the 80 replicate datasets (all ACS variables replaced by their r-th replicate values) and compute - -Var_sampling(β) = 4/80 · Σ_r (β_r − β)². - -This captures sampling error in every ACS variable, outcome and covariates, including covariances, without assuming a measurement-error model. It is the regression analogue of the Round 7 replicate statistics, and it is standard practice for ACS PUMS regression with replicate weights. - -- `fit_replicate(model_spec, frame, replicates)`: wraps any `statsmodels` formula model (OLS, WLS, GLM, logit on proportions) and returns coefficients, sampling SEs from replicates, model SEs from the full-sample fit, and a combined SE. -- How sampling and model variance combine is a methodological choice (decision 2). Default: report both components and a total that adds them, with a note on the interpretation. - -### B2. Model-based corrections (when replicates are unavailable) - -- **Known-variance outcome (meta-regression form).** Outcome y_i observed with known sampling variance s_i²: fit y_i = x_iβ + u_i + e_i with Var(e_i) = s_i² known and Var(u_i) = σ_u² estimated (the Fay–Herriot structure; shares estimation code with workstream C). Weights 1/(σ_u² + s_i²), not 1/s_i², which over-trusts precise areas. -- **Covariates with error.** Two documented options: - - *Regression calibration*: replace each noisy covariate by its best linear predictor given its estimate and SE. - - *SIMEX* (simulation-extrapolation, Cook & Stefanski 1994): add extra simulated error at increasing multiples of each covariate's known variance, refit, and extrapolate back to zero error. - Output includes the naive coefficient alongside the corrected one, so attenuation is visible. -- **Reliability screen.** Before fitting, list observations whose CV exceeds a threshold (default 30%, Census's "low reliability"); the model runs either way, but the screen is reported (decision 4). - -### B3. Supported models - -OLS/WLS, binomial-family GLM for proportions (with an effective sample size derived from the proportion and its SE), Poisson/negative binomial for counts with exposure, and fixed-effects models for panels (subject to section 9). Formula interface (`patsy`), so agents write `"poverty_rate ~ median_rent + unemployment_rate"`. - -### B4. Diagnostics - -Residual plots are out of scope (no plotting in the core), but numeric diagnostics are reported: the share of residual variance attributable to known sampling error, influence of high-CV observations, and coefficient stability across replicates. - ---- - -## 5. Workstream C (P1): small-area estimation (Fay–Herriot) - -Small areas (tracts, block groups, small counties) often have estimates too noisy to use. The Fay–Herriot area-level model borrows strength from covariates: - -- Model: ŷ_i = θ_i + e_i, θ_i = x_iβ + u_i, with e_i ~ N(0, s_i²) known (from the ACS MOE) and u_i ~ N(0, σ_u²). -- Estimation of σ_u² by REML (default), ML, or Fay–Herriot moment method. -- EBLUP: θ̃_i = γ_i ŷ_i + (1 − γ_i) x_iβ̂, γ_i = σ_u² / (σ_u² + s_i²). -- MSE by the Prasad–Rao analytic approximation (with the Datta–Lahiri correction under REML), with a parametric bootstrap option. -- Outputs: EBLUP, its MSE and 90% interval, the shrinkage factor γ_i, and the direct estimate, so users can see how much each area was pulled toward the model. -- Guardrails: refuse areas with undefined sampling variance; report when σ_u² is estimated at zero (complete shrinkage to the synthetic estimate), a known failure mode. -- Validation (section 8): reproduce published reference results for the Fay–Herriot EBLUP and MSE (for example the worked examples accompanying Molina and Marhuenda's `sae` package paper, R Journal 2015), cited during implementation. - ---- - -## 6. Workstream D (P1): pre-registered hypothesis manifests - -A manifest is a JSON (or YAML) file that fixes an analysis before data are pulled: - -- hypotheses, each with a direction (two-sided or one-sided) and the statistic that tests it; -- variables (`YEAR:CODE` or concept IDs), geographies, releases, and any aggregation groups; -- models and tests (from workstreams B, C, and Round 7), confidence level, multiple-comparison correction and family definition; -- exclusion rules (for example the CV screen) and the handling of missing or undefined estimates; -- authors (same identity convention as overrides) and a creation timestamp. - -Workflow: - -1. `create_manifest` validates the manifest (every variable resolves, every test is supported) and records its SHA-256. Optionally the hash is committed to the repository or a timestamping service before any data are pulled (decision 3). -2. `run_manifest` executes it exactly: extraction (with `plan_extraction` first), derivations, models, corrections. Results are written with the manifest hash. -3. Anything run outside the manifest on the same data is labeled **exploratory** in outputs. -4. Deviations (an API failure forcing a different release, a variable blocked by concordance review) are recorded explicitly, never silently substituted. -5. A generated **methods section** describes the data sources (releases, tables, concordance decisions), the derivations and their MOE methods, the models, and the corrections, with citations, so the write-up matches what was run. - ---- - -## 7. Workstream E (P1): planning tools - -- **Minimum detectable difference.** Given two estimates' MOEs (or typical MOEs for a table and geography level), the smallest difference detectable at a chosen confidence level and power. Answers "can these data detect a 2-point change in tract poverty?" before an analysis is designed. -- **Precision forecast for aggregation.** How much combining tracts into groups of size k reduces the CV, using approximate and (where available) replicate variances. -- **Reliability profile.** For a table and geography level, the distribution of CVs across all areas, so a manifest can choose a geography level where the variables are usable. - ---- - -## 8. Workstream F (P0): inference validation - -As in Round 7, nothing is accepted on reasoning alone. - -### F1. Coverage simulations - -Simulate populations where the truth is known, generate ACS-like estimates by adding sampling error with realistic SEs (drawn from real tract and county MOE distributions, including correlated replicate structure), run each method, and measure: - -- coverage of 90% intervals for coefficients and small-area means (target 88–92%); -- bias of coefficients (naive vs corrected, demonstrating attenuation and its correction); -- Fay–Herriot MSE accuracy (estimated vs empirical MSE). - -Simulations are seeded and small enough to run in CI in reduced form; the full study runs on demand and writes `docs/audits/inference_validation.md`. - -### F2. Reference results - -- Reproduce published Fay–Herriot results (section 5). -- Reproduce replicate-based regression SEs on a published ACS PUMS example where Census provides worked results, or cross-check against `statsmodels` survey-weighted estimates on synthetic data. -- Cross-check SIMEX and regression calibration against published worked examples (cited at implementation). - -### F3. Round 7 battery stays green - -The Round 7 validation battery continues to run in CI and weekly live. - ---- - -## 9. Workstream G (P2): time and overlapping periods - -- **Design checks.** Panel models over consecutive 5-year releases are flagged: observations share sample years, so the effective number of independent periods is roughly the span divided by five. The tooling suggests non-overlapping releases (for example 2013, 2018, 2023) for confirmatory trend tests. -- **Trend tests** on non-overlapping periods using the Round 7 z-tests, extended to linear trends across three or more periods with known variances. -- **Overlap-aware variance for differences across overlapping periods** remains the Census approximation (Round 7); exact covariance across releases is not published and is out of scope. - ---- - -## 10. Workstream H (P0): agent interface and reporting - -New MCP tools and CLI commands mirror each workstream: - -| Tool | Purpose | -| :--- | :--- | -| `build_analysis_frame` | Model-ready table with SEs, optional replicates, and a coverage report | -| `correlation_matrix` | Correlations with intervals, replicate SEs, optional disattenuation | -| `make_figure`, `make_table` | Publication figures and tables with captions and provenance | -| `get_recipe` | Step-by-step workflow recipes for smaller models | -| `fit_model` | Regression with replicate or model-based sampling error; returns coefficients, SE components, method | -| `small_area_estimates` | Fay–Herriot EBLUPs with MSE, intervals, and shrinkage | -| `minimum_detectable_difference`, `reliability_profile` | Planning | -| `create_manifest`, `run_manifest` | Pre-registration and execution | -| `run_inference_validation` | Coverage simulations and reference checks | - -`docs/agent_protocol.md` gains an inference section: when to use replicate inference, how to read SE components, the CV screen, overlap rules for trends, and the manifest workflow. Model results are written with the same provenance metadata as extractions, plus the model specification and the manifest hash when one was used. - ---- - -## 10a. Workstream I (P1): publication outputs - -Figures and tables suitable for journal submission, generated from results objects so that every number in a figure traces to a result and its provenance. - -### I1. Figures (`census/figures.py`, matplotlib) - -| Figure | Shows | -| :--- | :--- | -| Estimates with intervals | Point estimates with 90% intervals from MOEs (dot-and-whisker), sorted, with low-reliability estimates marked | -| Coefficient plot | Model coefficients with intervals; naive vs corrected side by side to show attenuation | -| Correlation matrix | Heatmap of correlations among ACS variables, with significance or interval information (I2) | -| Forest plot | Many-area comparisons from `batch_significance`, highlighting results significant after correction | -| Small-area shrinkage | Direct estimates vs Fay–Herriot EBLUPs with intervals, showing how much each area moved | -| Trend plot | Non-overlapping periods with intervals and the trend test result | -| Reliability profile | CV distribution by geography level (planning) | - -Requirements: - -- **Vector output** (PDF, SVG, EPS) plus PNG at a chosen DPI; sizes for one- and two-column journal widths; embedded fonts. -- **Accessible defaults:** colorblind-safe palettes, line styles and markers that survive grayscale printing, readable minimum font sizes. -- **Every figure ships with** a CSV of the plotted values, a caption draft stating the data source (releases, tables), interval level, and methods, and a JSON sidecar with provenance (result ID, manifest hash when used). -- **Reproducible bytes:** fixed metadata (no creation timestamps; matplotlib's SVG hash salt fixed) so regenerating a figure from the same results gives an identical file. -- **Tested by content, not by eye:** tests check plotted data, axis labels, interval endpoints, and caption text, and include a small set of image-regression baselines. - -### I2. Correlation matrices that respect sampling error - -Correlations between noisy estimates are biased toward zero (attenuation), and their uncertainty includes sampling error. - -- `correlation_matrix(frame, variables, method="pearson"|"spearman")` via `scipy.stats`, returning the matrix, pairwise n, p-values, and confidence intervals (Fisher z for Pearson). -- With replicates: the correlation recomputed on each of the 80 replicates gives a design-based standard error for every coefficient. -- Optional disattenuation (Spearman's correction using each variable's reliability, computed from its known sampling variance), reported alongside, never instead of, the observed correlation. -- Multiple-comparison correction across the matrix (Round 7 `adjust_pvalues`). - -### I3. Tables - -- Regression, descriptive, and comparison tables as LaTeX (booktabs), Markdown, and CSV, with standard errors or intervals, significance markers after correction, notes on SE components, and the data citation. -- A results bundle per analysis: tables, figures, captions, the methods section (workstream D), and provenance in one folder. - -## 10b. Workstream J (P0): local-model readiness - -Goal: an agent running on a local open-weights model (via Ollama, LM Studio, llama.cpp, or vLLM) can complete the standard workflows through MCP without Claude. - -Small models struggle with long tool lists, long descriptions, nested parameters, and ambiguous next steps. Today's server has 32 tools with long docstrings, written for a frontier model. - -### J1. Tool profiles - -- **`core` profile (default for local models, about 10 tools):** find a table, describe a variable, plan, download, derive, compare, fit a model, make a figure, run a manifest, get help. Each tool covers one task end to end. -- **`full` profile:** everything, as today. -- **`review` profile:** concordance review and rule tools only, for reviewer agents. -- Selected with `python mcp_server.py --profile core` or an environment variable. - -### J2. Tool design rules (enforced by tests) - -- Descriptions: one sentence on what the tool does, one line per parameter, one example call; length budget enforced by a test. -- Flat parameters with enums instead of free text where possible; no nested objects in the core profile. -- Every result starts with a short `summary` sentence and, on success or failure, a `next_step` suggestion naming the tool to call next. -- Errors: a fixed set of codes (already in place) plus a plain-language fix. -- Large results are written to files and summarized; the tool returns the path and a preview, never thousands of rows. - -### J3. Workflow recipes - -Short, numbered recipes (for example "compare a rate between two counties", "map-free tract trend test", "pre-registered regression") returned by a `get_recipe` tool, so a small model follows steps instead of planning from scratch. - -### J4. Local-model evaluation harness - -- A scripted benchmark of tasks with known correct outcomes (for example "What was the poverty rate in Travis County in 2023, with its MOE?", "Is the change in median rent between 2018 and 2023 in county X significant?", "Fit the pre-registered model in this manifest"). -- Runs an MCP client against a local model through an OpenAI-compatible endpoint (Ollama and most local servers expose one), scores tool choice, argument correctness, and final answers, and writes `docs/audits/local_model_eval.md`. -- Target: at least one widely available open-weights model in the 7–14B range completes the core recipes reliably (decision 7). The harness is model-agnostic so results can be compared across models. -- The harness itself is optional to run (it needs a local model server) and is not part of CI. - -### J5. No Claude-specific dependencies - -The server, docs, and recipes avoid assumptions about a particular client: standard MCP over stdio, plain JSON results, and setup instructions for common local MCP clients alongside Claude Code. - -## 11. Out of scope (Round 9 and later) - -- **Spatial models and maps.** Spatial autocorrelation (Moran's I) and spatial regression need tract adjacency, which needs TIGER/Line boundaries. Planned with mapping output in Round 9. -- **PUMS microdata** and custom tabulations with person/household replicate weights. -- **ACS 1-year and decennial data.** -- **Bayesian hierarchical models** beyond Fay–Herriot (for example spatial Fay–Herriot, multivariate models). - ---- - -## 12. Implementation roadmap - -```text -PHASE 8a: Foundations - a.1 analysis_frame and replicate attachment (A) - a.2 simulation harness with realistic ACS error (F1 scaffolding) - -PHASE 8b: Models (parallel streams, disjoint files) - b.1 replicate regression and model-based corrections (B) - b.2 Fay-Herriot (C) - b.3 planning tools (E) - b.4 overlap design checks and trend tests (G) - -PHASE 8c: Discipline and proof - c.1 manifests and methods-section generation (D) - c.2 coverage studies and reference results for every method (F) - -PHASE 8d: Outputs and interface - d.1 figures, correlation matrices, tables, results bundles (I) - d.2 MCP tools, CLI, protocol docs, README (H) - d.3 tool profiles, design-rule tests, recipes (J1-J3) - d.4 local-model evaluation harness and first report (J4) -``` - -8a comes first because every model consumes the analysis frame and every method is validated with the simulation harness. - ---- - -## 13. Decisions - -| # | Question | Decision | -| :--- | :--- | :--- | -| 1 | Dependencies | **Decided:** lean on established libraries as required dependencies: `scipy` (statistics), `statsmodels` (regression, GLMs, mixed models, multiple testing), `matplotlib` (figures). Considered and not adopted as dependencies: `seaborn` (convenience over matplotlib; not needed), `scikit-learn` (prediction-oriented), `linearmodels` (panel/IV; revisit if needed), `samplics` (SAE and replicate weights; used only as a test-time cross-check if installed), `pingouin` (less established) | -| 2 | Combining replicate and model variance | **Decided, then revised by validation:** all three SEs are reported, but intervals use `target="population"` (model SE, coverage 0.896) by default, or `"these_areas"` (replicate SE, 0.895). The sum (`"conservative"`) double-counts outcome sampling error and over-covers (0.961) | -| 3 | Recording pre-registration hashes | **Decided:** commit the manifest to the repository; external timestamping (for example OSF) optional | -| 4 | High-CV observations | **Decided:** keep and report by default; manifests may exclude explicitly | -| 5 | Fay–Herriot validation | **Decided:** reproduce published `sae` worked examples in Python, without R | -| 6 | Figure style | Default: a neutral journal style (serif-free sans fonts, colorblind-safe palette, grayscale-safe markers) with presets for one- and two-column widths; journal-specific presets added on request | -| 7 | Local-model target | Default: evaluate with Ollama against at least one open-weights model in the 7–14B range with tool-calling support (for example a current Qwen or Llama instruct model), chosen at implementation from what is available; tell us if you already run a particular model | - ---- - -## 14. Acceptance criteria - -1. Every model result states which uncertainty sources its standard errors include and the method used. -2. Replicate regression reproduces exact replicate variances on synthetic data with known answers. -3. Coverage simulations show 88–92% coverage for nominal 90% intervals for each supported method under ACS-like error; methods that fail are documented and not exposed as defaults. -4. Corrected methods remove the attenuation bias that naive regression shows in simulation. -5. Fay–Herriot EBLUP and MSE reproduce a published reference result to its published precision. -6. A manifest runs end to end from MCP or the CLI, its hash appears in every output, and runs outside it are labeled exploratory. -7. The Round 7 validation battery still passes; CI stays green on Linux and Windows, Python 3.10–3.12. -8. Figures are produced as vector files with a values CSV, caption, and provenance sidecar, and regenerate byte-identically from the same results. -9. Correlation matrices report intervals and, when replicates exist, design-based standard errors. -10. The `core` MCP profile passes the design-rule tests, and the local-model evaluation reports a completion rate for each core recipe with at least one local model. - ---- - -## 15. Implementation status - -| Workstream | Files | Result | -| :--- | :--- | :--- | -| Contract | `census/results.py` | `ModelResult`, `SmallAreaResult`, `CorrelationResult`, analysis-frame convention | -| A Analysis frames | `census/analysis.py`, panel `*_se` columns | Derived variables evaluated on each replicate | -| B Models | `census/models.py` | Replicate refit regression (OLS/WLS/GLM), regression calibration, SIMEX, reliability screen; interval targets | -| C Fay–Herriot | `census/sae.py` | Reproduces the R `sae` milk example (sigma2_u 0.0185503 vs 0.01855022; betas and SEs within 2e-6) | -| D Manifests | `census/manifest.py`, `examples/manifests/` | Validate, hash, run, deviations, methods section | -| E, G Planning and trends | `census/planning.py`, `census/trends.py` | MDD, aggregation forecasts, overlap checks, GLS trend test | -| F Validation | `census/simulation.py`, `scripts/inference_validation.py` | Coverage studies; findings below | -| I Outputs | `census/correlation.py`, `census/figures.py`, `census/report_tables.py` | Reproducible journal figures, LaTeX/Markdown tables, results bundles | -| H, J Interfaces | `census/workflows_inference.py`, `census/mcp_profiles.py`, `census/recipes.py`, `scripts/local_model_eval.py` | 44 tools (13 in `core`), recipes, design-rule tests, local-model harness | - -**Validation findings that changed the plan.** (1) Adding model and replicate variance (decision 2) double-counts outcome sampling error: coverage 0.961. Intervals now default to the model SE for population inference (0.896) or replicate SE for these-areas inference (0.895). (2) Quadratic SIMEX under-corrects attenuation and under-covers (0.81; 0.29 at reliability 0.38); it is documented as not validated and regression calibration (0.89) is the recommended correction. (3) Replicate refits do not correct attenuation from noisy predictors (coverage 0.54 in that scenario). (4) Correlations among noisy estimates are attenuated: intervals cover the observed-estimate correlation (0.895) but not the true-value correlation (0.45); disattenuation is reported alongside. - -**Local-model evaluation.** Qwen3 8B (CPU with partial GPU offload) passed 5 of 6 tasks through the `core` profile, about 5–7 minutes per task. The failure (a multi-step question answered without the recipe) led to a recipe-first instruction; the run also exposed a misleading overlapping-trend error, now fixed. - -**Code review (PR #2).** Ten findings: nine fixed with regression tests (`tests/test_review_fixes_r8.py`), including a Fay–Herriot collapse when one area has zero sampling variance, non-JSON tool results, and SEs in tables that did not match the p-values. One is partial: pipeline outputs store controlled-estimate MOEs (-555555555) as missing rather than 0, because the Round 7 F1 check requires every sentinel to be removed; `census.stats.moe_to_se` now applies one rule everywhere it sees the raw code. - -**Carried to Round 9:** controlled-estimate MOEs preserved as 0 through the pipeline (with F1 updated to match); spatial models and maps; slimmer core wrapper for `batch_download` (26 parameters); native `summary`/`next_step` in Round 7 tools; tract-level CV profiles from real tract MOEs for the simulations (currently assumed); rerun of the local-model evaluation with an 8K context. - diff --git a/PLAN_ROUND_9.md b/PLAN_ROUND_9.md deleted file mode 100644 index c02687f..0000000 --- a/PLAN_ROUND_9.md +++ /dev/null @@ -1,174 +0,0 @@ -# PLAN_ROUND_9.md: Geography, Maps, and Spatial Inference - -**Planning Round:** 9 -**Date:** 2026-11-02 -**Status:** Complete: implemented, validated, reviewed, and merged (section 14) -**Branch:** `phase9/geography-and-maps` (from `defaultBranch` after the Round 8 merge) - ---- - -## 1. Summary - -Rounds 5–8 made ACS values, derived statistics, and models trustworthy. What they lack is **space**: boundaries to map, neighbors to model, and accurate ways to move estimates between changing geographies. Round 9 adds a geospatial layer built on the open-source scientific Python stack, fixes the remaining accuracy gap in the 2010/2020 crosswalk, and closes Round 8's carryovers. - -| Priority | Workstream | Section | -| :--- | :--- | :--- | -| P0 | A. Boundaries: official Census cartographic files, matched to each ACS release | 3 | -| P0 | B. Maps that show uncertainty (journal-ready static maps) | 4 | -| P0 | C. Spatial statistics and models with sampling error | 5 | -| P1 | D. Population-weighted crosswalks from decennial blocks | 6 | -| P1 | E. Block groups and custom areas from user polygons | 7 | -| P1 | F. ACS errata and data-quality flags | 8 | -| P1 | G. Round 8 carryovers | 9 | -| P2 | H. Packaging and release: pip install, versioned releases, DOI | 10 | - ---- - -## 2. Approach to maps and GIS - -**Why not ArcGIS as the engine.** ArcGIS is the institutional standard for producing and sharing maps, but it is proprietary, licensed per seat, and slow for batch geoprocessing because operations run through a GUI-oriented geoprocessing framework one feature class at a time. It also cannot be driven by agents or local models without a license and Windows desktop. - -**What we use instead.** The open-source stack that ArcGIS itself reads and writes: - -| Library | Role | Standing | -| :--- | :--- | :--- | -| GEOS via Shapely 2 | Geometry operations, vectorized in C | The same engine used by PostGIS and QGIS | -| GeoPandas | Tables with geometry, spatial joins, dissolves | De facto Python standard | -| pyogrio (GDAL) | Fast reading and writing of GIS formats | GDAL underlies nearly every GIS, including ArcGIS's format support | -| pyproj (PROJ) | Coordinate systems and projections | The reference projection library | -| PySAL (`libpysal`, `esda`, `spreg`, `mapclassify`) | Spatial weights, autocorrelation, spatial regression, map classification | Peer-reviewed, widely cited (Rey and Anselin) | -| matplotlib | Static maps for journals | Already used for figures | - -Operations that are slow in desktop GIS (a spatial join of every US tract, dissolving 85,000 tracts into custom regions, building contiguity for a national tract file) take seconds to tens of seconds here because they are vectorized and run without a GUI. - -**Interoperability with ArcGIS.** Outputs are written as **GeoPackage** (an open OGC standard that ArcGIS Pro and QGIS open natively), **GeoParquet** (fast, columnar), and shapefile only on request (field names truncated to 10 characters, so ACS codes collide). A researcher can do the analysis here and finish cartography in ArcGIS Pro or QGIS if a journal or agency requires it. - ---- - -## 3. Workstream A (P0): boundaries - -- **Source:** Census **cartographic boundary files** (`cb____500k`), generalized for mapping and small, from `www2.census.gov/geo/tiger/GENZ/`. Full TIGER/Line files when exact boundaries are needed (areal calculations, crosswalks). -- **Vintage matching:** each ACS release is joined to the boundary vintage Census used for it (ACS 5-year uses the geography of its final year), so 2019 releases get 2010-based tracts and 2020+ get 2020 tracts. A mismatch is an error, not a silent join. -- **Join key:** GEOID, which every output already carries. The join reports unmatched GEOIDs on either side. -- **Cache and integrity:** downloaded files go to `.cache/geo//` with SHA-256 sidecars, as with replicate files. -- **Projections:** stored in the source CRS (NAD83). Area and distance calculations use an equal-area projection (EPSG:5070 for the conterminous US; Alaska, Hawaii, and Puerto Rico have their own), chosen automatically. -- **Tools:** `get_boundaries(level, year, state=None)`, `join_boundaries(frame, ...)`, `write_geo(frame, path, format=...)`. - -## 4. Workstream B (P0): maps that show uncertainty - -A choropleth of ACS estimates is misleading when neighboring classes are not statistically distinguishable or many areas are unreliable. The literature offers concrete remedies (Sun and Wong 2010, *Cartography and Geographic Information Science*; Koo, Chun and Griffith 2017, on class separability with MOEs); we implement them: - -- **Classification:** quantile, natural breaks, equal interval, and **class-separability breaks** that maximize the share of areas whose 90% intervals do not overlap adjacent classes (via `mapclassify` plus an MOE-aware criterion). The legend reports how separable the classes are. -- **Reliability overlays:** hatching or reduced opacity for low-reliability estimates (CV > 30%), with the threshold in the legend. -- **Significance maps:** for comparisons (two periods, or each area vs a reference), show significant increases, decreases, and no significant change after multiple-comparison correction (Round 7), not raw differences. -- **Companion uncertainty map:** a CV or MOE map beside every estimate map, on request. -- **Cartographic basics:** scale bar, north arrow only when meaningful, inset handling for Alaska, Hawaii, and Puerto Rico, colorblind-safe sequential and diverging palettes, grayscale-safe hatching. -- **Same output contract as Round 8 figures:** PDF/SVG/EPS/PNG, values CSV, caption draft with data sources (ACS release and boundary vintage), provenance JSON, byte-identical regeneration. -- **Interactive maps** (HTML) are optional and P2 (decision 3). - -## 5. Workstream C (P0): spatial statistics and models with sampling error - -- **Spatial weights:** queen and rook contiguity, k-nearest neighbors, distance bands (`libpysal`), built from boundaries, cached, with reports of islands (areas with no neighbors). -- **Spatial autocorrelation:** global Moran's I and local indicators (LISA) via `esda`, with permutation inference and the Round 7 multiple-comparison corrections for local tests. -- **Sampling error in spatial statistics:** noisy small-area estimates can create or hide clusters. Two remedies: - - compute Moran's I on each of the 80 replicate maps (design-based SE for I), and - - test on Fay–Herriot EBLUPs (Round 8) as a sensitivity analysis. -- **Spatial regression:** spatial lag and spatial error models via `spreg` (ML and GMM), with diagnostics (Lagrange multiplier tests) that suggest which model fits; replicate refits for sampling SEs where feasible. -- **Spatial Fay–Herriot:** small-area estimates that borrow strength from neighbors as well as covariates (SAR random effects; Pratesi and Salvati 2008), validated against published reference results and by simulation, as in Round 8. -- **Validation:** the Round 8 simulation harness gains spatially correlated populations; coverage targets as before (88–92% for nominal 90%). - -## 6. Workstream D (P1): population-weighted crosswalks from decennial blocks - -Round 7 apportions tracts across the 2010/2020 boundary change by land area, which assumes people are spread evenly over land. Block-level population is published in the decennial census, so we can do better without NHGIS: - -- Use the Census **2010-to-2020 block relationship file** and **2010 and 2020 decennial block populations** (P1 table, `dec/pl` endpoint) to build population (and housing-unit) weights for every tract-to-tract piece. -- Choose weights by variable: population counts use population weights; household and housing variables use housing-unit weights. -- Validate: weights from blocks against land-area weights (report where they differ most), and apportioned 2019 tract totals against county totals (which did not change). -- Same mechanism yields **ZCTA 2010→2020** and **place** crosswalks. - -## 7. Workstream E (P1): block groups and custom areas - -- **Block groups (summary level 150):** API extraction, boundaries, and replicate files (Census publishes block-group VRE files per state). Many tables are not published at block-group level; the planner reports which. -- **Custom areas from polygons:** a user supplies polygons (neighborhoods, service areas, school catchments) as GeoPackage, GeoJSON, or shapefile; tracts or block groups are allocated by area overlap or by block-population weights (workstream D), then aggregated with MOEs (Round 7). Partial-overlap allocation is reported and its assumption stated. - -## 8. Workstream F (P1): ACS errata and data-quality flags - -The Census Bureau publishes errata and user notes for ACS releases (tables withdrawn, corrected, or with known problems). Researchers rarely check them. - -- Maintain a dataset of errata and user notes by release and table, generated from the Census errata and user-notes pages with source URLs and retrieval dates (`scripts/fetch_acs_errata.py`). -- `plan_extraction` reports any errata or user notes affecting the requested tables and releases; confirmatory manifests record them. -- The Round 7 validation battery gains a check that every cited note resolves. - -## 9. Workstream G (P1): Round 8 carryovers - -1. **Controlled-estimate MOEs** (`-555555555`) preserved as 0 through the pipeline (with an explicit `controlled` flag column), and the Round 7 F1 check updated to accept the documented representation. -2. **Slim `batch_download` for the core profile** (a few required parameters, sensible defaults) and native `summary`/`next_step` in Round 7 tools. -3. **Realistic tract CV profiles** for simulations from real tract MOEs (fixtures), replacing the assumed profile. -4. **Local-model re-evaluation** with an 8K context and the new geography tools, comparing at least two models. -5. **Map tools in the `core` profile** if the local-model evaluation shows small models can use them. - -## 10. Workstream H (P2): packaging and release - -- `pyproject.toml` so the toolkit installs with `pip install` (optional extra `dev`), with a console entry point (`census-acs`) and an MCP entry point. -- Tagged, versioned releases with a changelog; a Zenodo DOI per release, wired into `CITATION.cff`, so papers can cite the exact version used. -- A built documentation site from the existing Markdown (optional). - ---- - -## 11. Out of scope (Round 10 and later) - -- **PUMS microdata** (custom tabulations with person and household replicate weights, PUMAs). -- **ACS 1-year estimates** (larger areas, more timely, different variance properties). -- **Cost-of-living adjustment across places** (BEA Regional Price Parities) for real-income comparisons between areas. -- **Multilevel models** (tracts within counties) and difference-in-differences designs on non-overlapping periods. -- **Interactive web maps** beyond the optional HTML export. - ---- - -## 12. Decisions - -| # | Question | Decision | -| :--- | :--- | :--- | -| 1 | Geospatial libraries | **Required dependencies** (GeoPandas, pyogrio, pyproj, Shapely, PySAL: `libpysal`, `esda`, `spreg`, `mapclassify`) | -| 2 | Boundary detail | **Decided:** cartographic boundary files (1:500k) for maps; TIGER/Line for areal calculations and crosswalks | -| 3 | Map priority | **Decided:** static journal maps are the priority; optional self-contained HTML export (no external tile servers) is P2 | -| 4 | Default map classification | **Decided: follow research practice.** Quantile classes (5 by default) are the most common choice in social-science journals because classes are comparable across maps; natural breaks (Jenks) and equal interval are available. Because ACS estimates carry error, every map's legend reports class separability (the share of adjacent-class pairs whose 90% intervals do not overlap), and class-separability breaks (Sun and Wong 2010) are available for maps where distinguishability matters most | -| 5 | Crosswalk weights | **Decided:** block-population weights by default; land-area fallback with a note | -| 6 | Shapefile output | **Decided:** available on request; GeoPackage is the default | - ---- - -## 13. Acceptance criteria - -1. Every map states its ACS release, boundary vintage, classification method, and class separability; low-reliability areas are marked. -2. Boundary joins fail loudly on vintage mismatch and report unmatched GEOIDs. -3. A national tract map (about 85,000 tracts) with contiguity weights builds in under a minute on a laptop. -4. Moran's I and spatial regression results reproduce PySAL reference examples; replicate-based SEs for Moran's I are validated by simulation. -5. Spatial Fay–Herriot reproduces a published reference result and meets coverage targets in simulation. -6. Block-population crosswalk weights sum to one per source tract, and apportioned 2019 tract totals match unchanged county totals. -7. Errata affecting a requested table appear in the plan and in manifest outputs. -8. Outputs open in ArcGIS Pro and QGIS (GeoPackage), verified by reading them back with GDAL. -9. CI stays green on Linux and Windows, Python 3.10–3.12. - ---- - -## 14. Implementation status - -| Workstream | Files | Result | -| :--- | :--- | :--- | -| A Boundaries | `census/boundaries.py` | Cartographic and TIGER/Line URL patterns verified by year (national tract and block-group cartographic files exist from 2019; ZCTA cartographic files are `zcta510` 2013–2019 and `zcta520` for 2020); vintage checks; GeoPackage default | -| B Maps | `census/maps.py` | Quantile default, natural breaks, equal interval, exact separability-optimal breaks (85k tracts in 0.05 s); separability in every legend; reliability hatching; uncertainty and significance maps; insets | -| C Spatial | `census/spatial.py`, `census/sae_spatial.py` | PySAL references reproduced (SIDS, STL, Columbus, Baltimore); spatial Fay–Herriot reproduces R `sae` grapes (REML and ML to 1e-5); coverage 0.891. Moran's I replicate SE is conservative (coverage 0.987; SE 1.1–1.5x true SD), reported but not gated | -| D Block weights | `census/block_weights.py`, `census/geo_crosswalk.py` | 2010/2020 block relationship files and decennial counts; weights by population or housing units; Delaware: population-weight apportionment error 444 vs 624 (land area); county totals within 3 people | -| E Block groups, custom areas | `census/api.py`, `census/pipeline.py`, `census/custom_areas.py` | Block groups from 2013 (acs5 only; unpublished tables detected); polygon allocation by area or block population | -| F Errata | `census/errata.py`, `data/acs_errata.json` | 258 notes (145 errata, 113 user notes); included in every plan | -| G Carryovers | `census/stats.py`, `census/validation.py`, `census/workflows.py`, `census/mcp_profiles.py`, `census/simulation.py` | Controlled MOEs kept as 0 with a flag (F1 updated); `download_table` in the core profile; native summary/next_step in Round 7 tools; tract CV profile from 1,278 real tracts (median 13.4% vs assumed 22%) | -| H Packaging | `pyproject.toml`, `CHANGELOG.md`, `.github/workflows/release.yml` | Editable install from the clone (no generic top-level packages); release workflow; Zenodo DOI after the owner enables the integration | -| Interfaces | `census/workflows_geo.py` | Six geography tools (51 total); `make_map` added to the core profile pending the local-model rerun | - -**Not built:** ZCTA and place crosswalks from blocks (need block-to-ZCTA/place assignment files); interactive HTML maps (P2). - -**Inference validation rerun** (`docs/audits/inference_validation.md`) with the real tract CV profile and the spatial studies: every gated method passes. fit_replicate 0.907 (population target) and 0.897 (these-areas target); regression calibration 0.890; Fay–Herriot 0.897; correlations 0.895; trend test size 0.098; spatial Fay–Herriot 0.890 (10x10 lattice, rho 0.5). - -**Local-model rerun** (`docs/audits/local_model_eval.md`, Qwen3 8B, 8K context, CPU with partial integrated-GPU offload): 5 of 6 tasks, the same as Round 8. The recipe task still fails: with a larger context and a recipe-first instruction the model still answers multi-step questions from tool descriptions without calling `get_recipe`. This is treated as a limit of this model size, not tuned around. The evaluation has no map task yet, so `make_map` in the core profile is untested with a local model; adding map and spatial tasks to the harness is carried to Round 10. - diff --git a/STATUS.md b/STATUS.md new file mode 100644 index 0000000..1aebf6b --- /dev/null +++ b/STATUS.md @@ -0,0 +1,79 @@ +# Project status + +The single current summary of the Census Research Engine: what it does, which methods are validated, the +standing rules, and what is next. Read this instead of old plans. Updated each round when it merges. + +**Current release:** v0.21.0 (Round 22), DOI 10.5281/zenodo.24264540. All versions: concept record +[zenodo.org/records/23925301](https://zenodo.org/records/23925301). Per-round detail: `CHANGELOG.md`. + +## Where things are + +| Path | What | +| :--- | :--- | +| `census/` | Library. `manifest/` package (pre-registered studies: `schema`, `validate`, `create`, `run`, `methods_text`, plus `adjustments`, `replicates`, `external`, `derivations`, `datasource`; import from `census.manifest`, but patch tests at the submodule that looks the name up), `models.py` (`fit_replicate`, `fit_eiv`, robust GLM SEs), `health_sources.py` (PLACES, USALEEP, CHR&R, WONDER exports), `spatial*.py`, `sae*.py`, `simulation*.py` (validation studies) | +| `mcp_server.py` | MCP server (63 tools; profiles in `census/mcp_profiles.py`) | +| `docs/agent_protocol.md`, `docs/recipes/` | What agents are told; keep in step with engine rules | +| `docs/audits/` | Validation reports (pass/fail, reproducible scripts in `scripts/`) | +| `docs/studies/` | Pre-registered study reports; outputs are release attachments, not in the repo | +| `examples/manifests/` | Registered manifests (their bytes are fixed once registered) | +| `PLAN_ROUND_.md` | Only the current round's plan. Older plans are archived (below) | + +**Archived plans.** `PLAN_ROUND_1`–`22` were removed from the tree after Round 22. Code and docs still cite +them as labels (for example "PLAN_ROUND_16 A2"). To read one: `git show v0.21.0:PLAN_ROUND_16.md`, or +`https://github.com/cafeTechne/Census-Research-Engine/blob/v0.21.0/PLAN_ROUND_16.md`. + +## Methods: validation status + +Validated means a fixed, pre-set simulation or reference check passed. Failed methods stay exploratory only; +manifests refuse them in confirmatory analyses. + +| Method | Status | Evidence | +| :--- | :--- | :--- | +| ACS derived estimates and MOEs (proportions, ratios, sums, medians) | Validated against Census-published values | `docs/audits/validation_report.md` | +| `fit_replicate` (OLS/WLS; 80 VRE replicate refits) | Validated: `population` 0.907, `these_areas` 0.897; `se_total` conservative (0.952–0.961) | `docs/audits/inference_validation.md` | +| `fit_eiv` (errors-in-variables) and its cluster jackknife | Validated (0.87–0.93) | `eiv_validation.md`, `eiv_cluster_validation.md` | +| Binomial/Poisson GLM with `scale: "robust"` (HC3) | Validated (0.879–0.897); required for external count outcomes | `robust_se_validation.md` | +| Binomial/Poisson with `scale: "X2"` (quasi-likelihood) | Failed under between-area overdispersion (0.689–0.79); refused for new manifests | `robust_se_validation.md` | +| Conley spatial HAC SEs (`spatial_se`) | **Failed** (0.71–0.87 at 150 km); exploratory only | `spatial_hac_validation.md` | +| Spatial error model λ correction (ME-SEM) | Validated for λ > 0 (0.86–0.93); lag model at γ 0.6 under-covers | `spatial_regression_validation.md` | +| LISA under sampling error | Characterised, information only | `lisa_validation.md` | +| Tract 2010→2020 apportionment: 1:1 and merged | Validated | `rate_apportionment_validation.md` | +| Split tracts: default `missing`; opt-in `split: "dominant"` (≥0.95 share) | Dominant passed held-out OH/PA (ratio 0.913); general split failed | `rate_apportionment_retest.md` | + +## Standing rules + +- **Circularity (enforced, R1/R2):** determinant studies use the ACS's own measure (disability: B18101). + Model-based external estimates such as PLACES only for measures the ACS lacks (such as diabetes). +- **Pre-registration:** a manifest is hashed and committed before any outcome association is examined. + Registered results are never rewritten; later checks are labelled post hoc. +- **Studies:** MI (26) and NY (36) are the test states. Study reviews by an agent are labelled "author + self-review, not independent". +- **Releases:** every merged round gets a tag, GitHub release and Zenodo DOI. Study outputs are attached as + a tarball after a key scan. Merges and releases need the owner's explicit approval. +- **Secrets and attribution:** see `CLAUDE.md`. + +## Studies so far + +| Round | Study | Headline | +| :--- | :--- | :--- | +| 19 | MI/NY tract income change | `docs/studies/r19_mi_ny_tract_income_change.md` | +| 20 | Disability and its determinants | `r20_mi_ny_disability_determinants.md` | +| 21 | PLACES disability vs ACS | PLACES inflates the education gradient 1.4× | +| 21b | County premature mortality | Poverty, disability supported; vehicle, cost burden not | +| 22 | County low birth weight | Poverty supported (holds with robust SEs); cost burden fragile (fails with robust SEs) | + +## Open backlog + +1. Spatial dependence: residual Moran's I is about 0.18–0.20 in both county studies; Conley failed. + Candidates: spatial random effects or a spatial block bootstrap, tested on the Round 22 design. +2. Measurement-error correction for GLM covariates (none yet; `regression_calibration` is approximate). +3. Replication of the county findings in held-out states (OH/PA). +4. Split `run_manifest` into stages (Round 23 recorded them in `PLAN_ROUND_23.md` section 7). +5. A per-method registry so a new analysis method is defined in one place (today: validation args, + `_run_analysis`, methods text, recipes, agent protocol). + +## Plan format (keeps agent context small) + +A round plan stays under about 15 KB: goal, ground rules, exact names and files, fixed acceptance checks, +and a short status table. When the round merges, add its outcome here in a few lines, then delete the plan +from the tree (it stays at its release tag). diff --git a/census/__init__.py b/census/__init__.py index 58a0b94..3f8daca 100644 --- a/census/__init__.py +++ b/census/__init__.py @@ -1,7 +1,7 @@ # SPDX-License-Identifier: AGPL-3.0-or-later """Headless Census ACS data client and processing library.""" -__version__ = "0.21.0" +__version__ = "0.21.1" from census.api import ( DATASETS, diff --git a/census/manifest.py b/census/manifest.py deleted file mode 100644 index 65ec072..0000000 --- a/census/manifest.py +++ /dev/null @@ -1,3010 +0,0 @@ -# SPDX-License-Identifier: AGPL-3.0-or-later -"""Pre-registered analysis manifests (PLAN_ROUND_8 workstream D). - -A manifest is a JSON document that fixes an analysis before any data are -pulled. Schema (``schema_version`` 1):: - - { - "schema_version": 1, - "title": "Income and poverty in NY tracts", - "authors": ["human:jdoe", "agent:claude"], # identity convention of overrides - "created": "2026-11-02T12:00:00Z", # ISO 8601 - "releases": [2019], # ACS 5-year end years - "geography": {"level": "tract", "state": "36", "codes": ["061"]}, - # or several states, stacked: {"level": "county", "states": ["26", "36"]} - # tracts across the 2010/2020 redraw: add "harmonize_tracts_to": 2020 - "variables": [ # YEAR:CODE or concept IDs - {"name": "income", "ref": "2019:B19013_001"}, - {"name": "pop", "ref": "c_0123456789ab", "table": "B01003"} - ], - "aggregation": {"mapping": {"": ""}}, # optional - "derivations": [{"name": "rate", "op": "proportion", "num": "a", "den": "b"}], - "adjustments": [{"type": "rpp", "variables": ["income"], # optional; after exclusions - "rpp_year": 2019, "areas": "county_metro" | "state"}], - # several releases: {"type": "rpp", "variables": [...], "dollar_year": 2024, - # "price_index": "r-cpi-u-rs", "rpp_years": {"2019": 2019, "2024": 2024}} - "replicates": {"source": "census_vre"}, # optional; run_manifest fetches VRE files - "exclusions": [{"rule": "cv_screen", "threshold": 30, "variables": ["income"], - "action": "flag" | "exclude"}, - {"rule": "geoid_exclude", "geoids": ["..."]}], - "missing": "complete_case", - "confidence_level": 0.90, - "correction": {"method": "holm", "family": "all" | "by_family"}, - "analyses": [{"id": "m1", "method": "fit_replicate", "formula": "income ~ pop", - "args": {"model": "ols"}, - "replicate_scope": "all" | "covariates_only"}], # optional, default "all" - "hypotheses": [{"id": "H1", "statement": "...", "analysis": "m1", - "statistic": "coefficient", "term": "pop", - "direction": "two-sided" | "greater" | "less", "family": "primary"}], - "data_source": {"synthetic": true, # optional; default: real Census API data - "description": "...", "generator": "..."}, - "external": [{"name": "places_dis", "source": "places", # optional; external health outcomes - "release": 2025, "measure": "DISABILITY"}] - } - -``external`` (optional, PLAN_ROUND_21 B3 and Part 2): variables from a registered external source -(:mod:`census.health_sources`: CDC PLACES, USALEEP, County Health Rankings, or a pinned CDC WONDER -export read by :mod:`census.vital_records`), downloaded after the ACS analysis frame is built and -joined on GEOID, with each file's URL and SHA-256 and the dataset's revision stamp in ``run.json`` -(``external``). An external variable may only be the outcome of ``fit_eiv``, ``fit_replicate`` or -``fh_regression``: its error is independent of the ACS sampling error, so it is held fixed in -replicate refits and its error enters the residual (as a covariate it would go uncorrected). -Derivations cannot use external variables; aggregation is refused; the values' tract vintage must -match every release (``harmonize_tracts_to``, or ``apportion_to`` for USALEEP). A failed download -is a blocking ``external_unavailable`` deviation; more than the source's limit (5%; 25% for -USALEEP, whose small tracts are unpublished) of the units with complete ACS data lacking an -external value is a blocking ``external_join_failed``. - -The circularity rule is enforced (:func:`circularity_errors`): R1, an outcome whose construct the -ACS measures directly (PLACES DISABILITY: B18101) is allowed only in a comparison with that ACS -measure, testing only it (determinant studies use the ACS measure); R2, a tested term built from -the outcome model's own inputs (PLACES: age, sex, race/ethnicity, education, poverty; USALEEP: -income, race/ethnicity, education, density) needs a written ``circularity_justification`` on the -hypothesis. Overlapping controls are noted by :func:`create_manifest`; direct outcomes (vital -records) have no model inputs. The methods text states each source's model. - -``geography`` takes ``state`` (one state, or ``"all"``) or ``states`` (a list -of 2-digit FIPS codes), never both. With ``states`` each table is extracted -per state and the states are stacked into one frame; ``codes`` are then not -allowed (county codes are per state). Pooled models get no automatic state -term: a cross-state formula states it explicitly, e.g. ``C(GEOID.str[:2])``. - -``geography.harmonize_tracts_to`` (optional, ``2020`` or ``2010``; tract level only, -PLAN_ROUND_19): releases on the other tract vintage are apportioned to that vintage with -block housing-unit or population weights before the analysis frame is built, and their -Census VRE replicates go through the same crosswalk, so replicate SEs are exact for the -apportioned estimates (:mod:`census.tract_harmonization`). Only additive counts can be -apportioned (derive means and shares from them); Connecticut and ``aggregation`` are -refused; a crosswalk failure is a blocking ``harmonization_failed`` deviation. An RPP -adjustment at tract level may use ``areas: "county_metro"`` (each tract's county). - -``adjustments`` (optional) are pre-registered transformations applied to the -analysis frame after derivations and exclusions (so a pre-registered exclusion -can remove a unit without an RPP area; a remaining unit without one is a -blocking ``adjustment_failed``). ``"rpp"`` divides -dollar variables (``variables``: manifest variable or derivation names), their -SEs and any supplied replicates by the BEA Regional Price Parity / 100 of each -row's area (``areas``: ``"county_metro"``, the default for county geographies, -or ``"state"``), with the same code as the ``place_adjust_frame`` tool -(:func:`census.workflows_microdata.place_adjust`). ``rpp_year`` is required so the -manifest hash fixes the price year (:func:`create_manifest` fills in the latest -year in the RPP table when it is omitted, before hashing); the year used, every -RPP area and the RPP table's SHA-256 (bundled checksum, or the custom table's path and hash) are -recorded in ``run.json`` and ``bundle.json`` (``adjustments``). An RPP year -other than the release end year is a note, not a deviation. - -Several releases (PLAN_ROUND_15 A1, decision D15-2): an RPP adjustment then also -requires ``dollar_year``, ``price_index`` (one of census.inflation.PRICE_INDEXES; -``"r-cpi-u-rs"``, the index the Census Bureau uses for ACS dollars, is the default -create_manifest fills in) and ``rpp_years`` (``{"": RPP year}`` for every -release, instead of ``rpp_year``; filled with each release's own year). Each -release's dollars are first converted to ``dollar_year`` dollars with the price -index (CPI(dollar_year) / CPI(release)), then divided by the RPP of the release's -``rpp_years`` entry; estimates, SEs and replicates of that release alike. With one -release a ``dollar_year`` (plus ``price_index``) is optional. Not combinable with -``aggregation``. The run records the CPI values and factors, the price-index -table's SHA-256 and every release's RPP areas. - -``replicates`` (optional, PLAN_ROUND_15 A2): ``{"source": "census_vre"}`` makes -run_manifest fetch the Census VRE files for the manifest's tables, geography and -releases (:func:`census.replicate.fetch_vre` and its cache), as the replicate -frame of every replicate analysis; ``run.json`` and ``bundle.json`` record each -file's URL and SHA-256, and ``replicates.unavailable`` lists variables without -published replicates (e.g. B19013 medians). Under the strict confirmatory policy a -replicate analysis that uses such a variable blocks (``replicates_incomplete``) -unless the analysis declares ``"replicate_scope": "covariates_only"`` -(fit_replicate only; target ``population`` or ``conservative``): its outcome is -then held at its estimate in every refit, and the declaration is in the hash and -the methods text. The declaration is allowed whenever it is well-formed: an -outcome the VRE registry lists can still lack replicates for the analysed units, -so create_manifest only notes such an outcome, and a run that finds replicates -for it still holds it fixed as declared (the hashed declaration governs) and -records that (``replicates.covariates_only_outcome_had_replicates``). Census VRE -files of tract and block-group geographies are per state, so those levels need -``geography.state`` or ``states``. Caller-supplied ``replicates`` are an error -with census_vre. - -Releases whose ACS 5-year periods overlap (shared sample years) are never -combined through replicates (the trend module's overlap rule: published -replicates do not carry the covariance of overlapping periods): validation -refuses census_vre or an analysis declaring ``"replicates": "required"`` with -overlapping releases, and caller-supplied replicates spanning overlapping -releases stop every replicate analysis with a blocking -``replicates_overlapping_releases`` deviation. - -When replicates are supplied and the manifest has derivations, each -derivation absent from the replicate frame is evaluated on the replicates of -its inputs (replicate rows named by manifest variable name or ACS code), and -its SE becomes the exact replicate SE where its inputs have replicates. -``replicates.derived_on_replicates`` lists only derivations that received a -replicate SE, with per-derivation counts (``derived_on_replicates_rows``: -rows_replicate of rows in the analysed frame); a failure of that step is a -blocking ``replicates_incomplete`` deviation. - -``data_source`` (optional) *declares* where the input data come from. The -labels follow where the data actually came from (:func:`observed_data_source`): -data are synthetic when the client that served them carries a true -``synthetic`` attribute or its cache was marked with -``QueryCache.mark_synthetic``; otherwise they are real Census Data API data. -A declaration that disagrees with the observed source records a blocking -``data_source_mismatch`` deviation, and the citation and methods text follow -the observed source (real Census data are never called synthetic; synthetic -data never claim retrieval from the Census Data API). - -Run status (``analysis_status`` in results, ``results/run.json``, -``bundle.json`` provenance and ``methods.md``): - -- ``"confirmatory"`` (:data:`STATUS_CONFIRMATORY`): the pre-registered - analysis ran as registered, with no blocking deviation. -- ``"confirmatory_incomplete"`` (:data:`STATUS_CONFIRMATORY_INCOMPLETE`): at - least one blocking deviation (:data:`BLOCKING_DEVIATIONS`, e.g. a hypothesis - not tested, an analysis failed, replicates incomplete, an extraction - failure); the pre-registered analysis did not run as registered and its - results are not confirmatory. -- ``"exploratory"``: anything run outside a manifest (:func:`label_exploratory`). - -Supported analysis methods are :data:`ANALYSIS_METHODS`; hypothesis -statistics are :data:`STATISTICS`. :func:`create_manifest` validates (every -variable resolves in the offline metadata archive through the concordance, -every method is supported) and returns the SHA-256 of the canonical JSON -(:func:`manifest_hash`). :func:`run_manifest` executes it exactly and writes -the hash into every output; deviations are recorded, never substituted -silently. Anything run outside a manifest should be tagged with -:func:`label_exploratory`. -""" - -from census.data_integrity import integrity_errors, BundledDataIntegrityError - -import copy -import hashlib -import json -import math -import re -from datetime import datetime -from pathlib import Path -from typing import Any, Callable, Dict, List, Mapping, Optional, Sequence, Tuple, Union - -import numpy as np -import pandas as pd - -from .results import IncompleteReplicatesError, ModelResult, interval_description - -SCHEMA_VERSION = 1 -AUTHOR_RE = re.compile(r"^(human|agent):\S+$") -CONCEPT_RE = re.compile(r"^c_[0-9a-f]{12}$") -YEAR_CODE_RE = re.compile(r"^(\d{4}):([A-Z]+\d+[A-Z]*_\d+[A-Z]*)$") -NAME_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$") -DIRECTIONS = ("two-sided", "greater", "less") -STATUS_CONFIRMATORY = "confirmatory" -STATUS_CONFIRMATORY_INCOMPLETE = "confirmatory_incomplete" -INCOMPLETE_STATUS_TEXT = "pre-registered analysis did not run as registered; results are not confirmatory" -# deviations meaning the pre-registered analysis did not run as registered -BLOCKING_DEVIATIONS = ("blocked_variable", "extraction_failed", "variable_missing_from_extraction", - "aggregation_failed", "analysis_frame_failed", "analysis_not_run", - "replicates_unavailable", "analysis_failed", "replicates_incomplete", - "hypothesis_not_tested", "failed_calls", "data_source_mismatch", "adjustment_failed", - "replicates_overlapping_releases", "harmonization_failed", "harmonization_incomplete", - "harmonization_mismatch", "external_unavailable", "external_join_failed") -DATA_SOURCE_KEYS = ("synthetic", "description", "generator") -STATUS_EXPLORATORY = "exploratory" - -# method -> (required args, statistics it supports) -ANALYSIS_METHODS: Dict[str, Tuple[Tuple[str, ...], Tuple[str, ...]]] = { - "fit_replicate": (("formula",), ("coefficient",)), - "fit_eiv": (("formula",), ("coefficient",)), - "regression_calibration": (("formula", "error_covariates"), ("coefficient",)), - "simex": (("formula", "error_covariates"), ("coefficient",)), - "fh_regression": (("formula",), ("coefficient",)), - "fay_herriot": (("formula",), ("coefficient",)), - "correlation_matrix": (("variables",), ("correlation",)), - "linear_trend_test": (("variable", "geoid"), ("trend_slope",)), - "batch_z_test": (("variable_1", "variable_2"), ("z_difference",)), -} -STATISTICS = ("coefficient", "correlation", "trend_slope", "z_difference") -EXCLUSION_RULES = ("cv_screen", "geoid_exclude") -ADJUSTMENT_TYPES = ("rpp",) -RPP_ADJUSTMENT_KEYS = ("type", "variables", "rpp_year", "areas", "rpp_years", "dollar_year", "price_index") -REPLICATE_SOURCES = ("census_vre",) -REPLICATES_KEYS = ("source",) -REPLICATE_SCOPES = ("all", "covariates_only") -# frame identifier columns a formula may name (C(Year) for a change across releases, C(GEOID...) terms) -FORMULA_ID_COLUMNS = ("GEOID", "Year") -# geography levels whose GEOIDs start with the state FIPS (state RPP areas; usable with -# geography.states). ZCTA GEOIDs are not state-prefixed and ZCTAs can cross state lines. -_STATE_PREFIXED_LEVELS = ("state", "county", "tract", "block group", "place") -FIPS2_RE = re.compile(r"^\d{2}$") -FORMULA_FUNCTIONS = {"C", "I", "np", "log", "exp", "sqrt", "center", "standardize", "scale", "Q"} - -REFERENCES = { - "acs": "U.S. Census Bureau (2020). Understanding and Using American Community Survey Data: " - "What All Data Users Need to Know (ACS General Handbook), ch. 7-8.", - "vre": "U.S. Census Bureau. Variance Replicate Estimate Tables Documentation. " - "https://www.census.gov/programs-surveys/acs/data/variance-tables.html", - "design": "U.S. Census Bureau (2014). American Community Survey Design and Methodology, ch. 12-13.", - "testing": "U.S. Census Bureau (2025). Instructions for Applying Statistical Testing to ACS Data.", - "carroll": "Carroll, R.J., Ruppert, D., Stefanski, L.A. & Crainiceanu, C.M. (2006). Measurement " - "Error in Nonlinear Models: A Modern Perspective, 2nd ed. Chapman & Hall/CRC.", - "simex": "Cook, J.R. & Stefanski, L.A. (1994). JASA 89, 1314-1328; Stefanski, L.A. & Cook, J.R. " - "(1995). JASA 90, 1247-1256; Devanarayan, V. & Stefanski, L.A. (2002). Stat. Prob. " - "Letters 59, 219-225.", - "fuller": "Fuller, W.A. (1987). Measurement Error Models. Wiley, sections 2.2 and 3.1.", - "cluster": "Cameron, A.C. & Miller, D.L. (2015). A practitioner's guide to cluster-robust inference. " - "J. Human Resources 50, 317-372.", - "standard_pop": "Klein, R.J. & Schoenborn, C.A. (2001). Age adjustment using the 2000 projected U.S. " - "population. Healthy People 2010 Statistical Notes 20. National Center for Health Statistics.", - "fisher": "Fisher, R.A. (1921). On the probable error of a coefficient of correlation. Metron 1, 3-32.", - "holm": "Holm, S. (1979). Scand. J. Statist. 6, 65-70.", - "bh": "Benjamini, Y. & Hochberg, Y. (1995). JRSS-B 57, 289-300.", - "by": "Benjamini, Y. & Yekutieli, D. (2001). Ann. Statist. 29, 1165-1188.", - "bonferroni": "Dunn, O.J. (1961). JASA 56, 52-64.", - "statsmodels": "Seabold, S. & Perktold, J. (2010). statsmodels: Econometric and statistical " - "modeling with Python. Proc. 9th Python in Science Conf.", -} - - -# --------------------------------------------------------------------------- -# Hashing, loading -# --------------------------------------------------------------------------- - - -def canonical_json(manifest: Mapping[str, Any]) -> str: - """Canonical serialization: sorted keys, no whitespace, UTF-8 text.""" - body = {k: v for k, v in manifest.items() if k != "manifest_hash"} - return json.dumps(body, sort_keys=True, separators=(",", ":"), ensure_ascii=False) - - -# Manifests registered with scale 'X2' for an external count outcome before the robust-SE rule -# (Round 22: main study, X1, X2). They stay valid so the registered runs reproduce; new ones are refused. -LEGACY_X2_MANIFESTS = frozenset({ - "10c46acedc6eff827d8a41df34d272f839ed0d75804666e81b77ca75eab83ea6", - "0d5e657b1bd86658ed950fd32eda96832614ae0e2ad1932cf1eddd6daeeaeba9", - "9e83d6a0f361496ded30a95c8373f61fa924875ca62bd2252242312c8e8f6a79", -}) - - -def manifest_hash(manifest: Mapping[str, Any]) -> str: - """SHA-256 (hex) of :func:`canonical_json`.""" - return hashlib.sha256(canonical_json(manifest).encode("utf-8")).hexdigest() - - -def load_manifest(path_or_dict: Union[str, Path, Mapping[str, Any]]) -> Dict[str, Any]: - if isinstance(path_or_dict, Mapping): - return copy.deepcopy(dict(path_or_dict)) - with open(path_or_dict, encoding="utf-8") as f: - return json.load(f) - - -def _table_of(var: Mapping[str, Any]) -> Optional[str]: - if var.get("table"): - return str(var["table"]).upper() - m = YEAR_CODE_RE.match(str(var.get("ref", ""))) - return m.group(2).split("_")[0] if m else None - - -def _dataset_for(table: str) -> str: - from .api import infer_dataset - return infer_dataset(table) - - -def manifest_states(geography: Mapping[str, Any]) -> List[Optional[str]]: - """The geography's state selections: ``states`` as given, else ``[state]`` - (``[None]`` for levels that need no state).""" - if geography.get("states") is not None: - return [str(s) for s in geography["states"]] - return [geography.get("state")] - - -def manifest_entities(geography: Mapping[str, Any], years: Optional[List[int]] = None) -> List[Any]: - """Request entities for a manifest geography: one state's entities, or each - listed state's entities in order (extracted per state and stacked).""" - from .pipeline import build_entities - entities: List[Any] = [] - for st in manifest_states(geography): - entities.extend(build_entities(geography["level"], st, geography.get("codes"), years=years)) - return entities - - -def _check_geography(geo: Mapping[str, Any], errors: List[str]) -> None: - from .pipeline import expand_states, is_nationwide, normalize_geo_level - if not isinstance(geo, Mapping) or not geo.get("level"): - errors.append("geography.level is required") - return - state = geo.get("state") - if isinstance(state, str) and not is_nationwide(state): - bad = [p for p in state.split(",") if not FIPS2_RE.match(p.strip())] - if bad: - errors.append(f"geography.state: {bad[0]!r} must be a state FIPS code of exactly 2 digits " - "(e.g. \"06\", not \"6\"), or \"all\"") - return - states = geo.get("states") - if states is not None: - if "state" in geo: - errors.append("geography: give either 'state' or 'states', not both") - return - if not isinstance(states, list) or not states or not all(isinstance(s, str) for s in states): - errors.append("geography.states must be a non-empty list of 2-digit state FIPS strings, " - "e.g. [\"26\", \"36\"]") - return - for s in states: - if is_nationwide(s) or "," in s: - errors.append(f"geography.states: {s!r} must be one 2-digit state FIPS code (for every " - "state use \"state\": \"all\")") - return - if not FIPS2_RE.match(s): - errors.append(f"geography.states: {s!r} must be a state FIPS code of exactly 2 digits " - "(e.g. \"06\", not \"6\")") - return - try: - expand_states(s) - except ValueError as exc: - errors.append(f"geography.states: {exc}") - return - if len(set(states)) != len(states): - errors.append("geography.states lists a state more than once") - return - if geo.get("codes"): - errors.append("geography: 'codes' cannot be combined with 'states' (codes are per state); " - "list GEOIDs in a geoid_exclude exclusion or use one state per manifest") - return - try: - level = normalize_geo_level(geo["level"]) - except Exception: # noqa: BLE001 - reported by build_entities below - level = geo["level"] - if level == "zcta": - errors.append("geography.states does not apply to level 'zcta': ZCTA GEOIDs are not state-prefixed " - "and ZCTAs can cross state lines, so they cannot be extracted per state and stacked") - return - if level not in _STATE_PREFIXED_LEVELS: - errors.append(f"geography.states does not apply to level {geo['level']!r} (national geography)") - return - try: - manifest_entities(geo) - except Exception as exc: - errors.append(f"geography: {exc}") - - -def default_rpp_areas(level: Optional[str]) -> str: - """Default RPP ``areas`` of an adjustment for a (normalized) geography level: the - county's metro/nonmetro area for counties, else the state (validation and runs).""" - return "county_metro" if level == "county" else "state" - - -def latest_rpp_year() -> Optional[int]: - """Latest year in the active RPP table (filled in by create_manifest when ``rpp_year`` is omitted), or None.""" - from . import rpp - years = rpp._years() - return years[-1] if years else None - - -def _check_adjustments(manifest: Mapping[str, Any], names: List[str], errors: List[str]) -> None: - adjustments = manifest.get("adjustments") - if adjustments is None: - return - if not isinstance(adjustments, list): - errors.append("adjustments must be a list of adjustment steps") - return - from .pipeline import normalize_geo_level - geo = manifest.get("geography") or {} - try: - level = normalize_geo_level(geo.get("level", "")) - except Exception: # noqa: BLE001 - geography errors are reported by _check_geography - level = geo.get("level") - releases = manifest_releases(manifest) - if adjustments and manifest.get("aggregation") is not None: - errors.append("adjustments cannot be combined with aggregation (RPP areas are assigned per GEOID)") - adjusted: List[str] = [] - for i, adj in enumerate(adjustments): - if not isinstance(adj, Mapping): - errors.append(f"adjustments[{i}] must be an object") - continue - if adj.get("type") not in ADJUSTMENT_TYPES: - errors.append(f"adjustments[{i}]: type must be one of {ADJUSTMENT_TYPES}") - continue - for k in sorted(set(adj) - set(RPP_ADJUSTMENT_KEYS)): - errors.append(f"adjustments[{i}]: unknown key {k!r}; allowed {RPP_ADJUSTMENT_KEYS}") - vars_ = adj.get("variables") - if not isinstance(vars_, list) or not vars_: - errors.append(f"adjustments[{i}]: variables must be a non-empty list of manifest variable names") - vars_ = [] - for v in vars_: - if str(v) not in names: - errors.append(f"adjustments[{i}]: unknown variable {v!r}") - elif str(v) in adjusted: - errors.append(f"adjustments[{i}]: {v!r} is adjusted twice") - adjusted.append(str(v)) - areas = adj.get("areas", default_rpp_areas(level)) - if areas not in ("county_metro", "state"): - errors.append(f"adjustments[{i}]: areas must be 'county_metro' or 'state'") - elif areas == "county_metro" and level not in ("county", "tract"): - errors.append(f"adjustments[{i}]: areas 'county_metro' needs geography level 'county' or 'tract' " - "(a tract takes its county's RPP area)") - elif areas == "state" and level not in _STATE_PREFIXED_LEVELS: - errors.append(f"adjustments[{i}]: areas 'state' needs a state-based geography level " - f"{_STATE_PREFIXED_LEVELS}") - try: - from . import rpp - years = rpp._years() - except Exception as exc: # noqa: BLE001 - e.g. a bundled-data integrity failure - errors.append(f"adjustments[{i}]: RPP table unreadable ({type(exc).__name__}: {exc})") - continue - if not years: - errors.append(f"adjustments[{i}]: no RPP data available") - continue - _check_dollar_year(adj, i, releases, errors) - if len(releases) > 1 or "rpp_years" in adj: - _check_rpp_years(adj, i, releases, years, errors) - continue - year = adj.get("rpp_year") - if year is None: - errors.append(f"adjustments[{i}]: rpp_year is required (it fixes the price level in the " - f"pre-registration); the latest year in the RPP table is {years[-1]}. " - "create_manifest fills it in when omitted") - elif not _is_int(year) or year not in years: - errors.append(f"adjustments[{i}]: rpp_year must be an integer year in the RPP table " - f"({years[0]}-{years[-1]})") - - -def _is_int(v: Any) -> bool: - return isinstance(v, int) and not isinstance(v, bool) - - -def manifest_releases(manifest: Mapping[str, Any]) -> List[int]: - """The manifest's integer releases, in order (anything else is reported by validate_manifest).""" - releases = manifest.get("releases") - return [r for r in releases if _is_int(r)] if isinstance(releases, list) else [] - - -def release_overlap_message(releases: List[int], subject: str) -> Optional[str]: - """None when the ACS 5-year periods of ``releases`` share no sample years, else why replicate - SEs across them are refused (the trend module's overlap rule, census.trends.overlap_design_check).""" - if len(set(releases)) < 2: - return None - from .trends import overlap_design_check - design = overlap_design_check(releases) - if not design["overlapping"]: - return None - return (f"{subject} with overlapping 5-year periods: releases " - f"{[(p['year1'], p['year2']) for p in design['overlapping_pairs']]} share sample years, and their " - "replicates do not carry that covariance, so replicate SEs of analyses that combine or contrast them " - f"would be wrong; use non-overlapping releases {design['suggested_nonoverlapping']}") - - -def _check_rpp_years(adj: Mapping[str, Any], i: int, releases: List[int], years: List[int], - errors: List[str]) -> None: - """``rpp_years`` ({release: RPP year}, one entry per release) replaces ``rpp_year`` when a - manifest has several releases (each release's dollars get their own year's RPP, D15-2).""" - if "rpp_year" in adj: - errors.append(f"adjustments[{i}]: give rpp_years (one RPP year per release), not rpp_year, " - "when the manifest has several releases" if len(releases) > 1 - else f"adjustments[{i}]: give either rpp_year or rpp_years, not both") - return - ry = adj.get("rpp_years") - if ry is None: - errors.append(f"adjustments[{i}]: rpp_years is required with several releases, e.g. " - + json.dumps({str(r): r for r in releases}) - + " (each release's own year, D15-2; it fixes the price levels in the pre-registration). " - "create_manifest fills it in when omitted") - return - if not isinstance(ry, Mapping) or sorted(str(k) for k in ry) != sorted(str(r) for r in releases): - errors.append(f"adjustments[{i}]: rpp_years must map every release (as a string key) to an RPP year, " - "e.g. " + json.dumps({str(r): r for r in releases})) - return - for k, v in ry.items(): - if not _is_int(v) or v not in years: - errors.append(f"adjustments[{i}]: rpp_years[{k!r}] must be an integer year in the RPP table " - f"({years[0]}-{years[-1]})") - - -def _check_dollar_year(adj: Mapping[str, Any], i: int, releases: List[int], errors: List[str]) -> None: - """``dollar_year`` + ``price_index``: every release's dollars are converted to the dollar year - with the price index (CPI first, then the RPP; D15-2). Required with several releases.""" - from .inflation import PRICE_INDEXES, load_price_index - dy, index = adj.get("dollar_year"), adj.get("price_index") - if dy is None: - if len(releases) > 1: - errors.append(f"adjustments[{i}]: dollar_year is required with several releases (dollars of " - f"different years are converted to one year with the price index before the RPP); " - "create_manifest fills in the latest release") - if index is not None: - errors.append(f"adjustments[{i}]: price_index needs dollar_year") - return - if not _is_int(dy): - errors.append(f"adjustments[{i}]: dollar_year must be an integer year") - return - if index is None: - errors.append(f"adjustments[{i}]: price_index is required with dollar_year (one of {PRICE_INDEXES}; " - "create_manifest fills in the default r-cpi-u-rs)") - return - if index not in PRICE_INDEXES: - errors.append(f"adjustments[{i}]: price_index must be one of {PRICE_INDEXES}") - return - try: - table = load_price_index(index) - except Exception as exc: # noqa: BLE001 - e.g. a bundled-data integrity failure - errors.append(f"adjustments[{i}]: price index table unreadable ({type(exc).__name__}: {exc})") - return - missing = sorted({int(dy), *releases} - set(table)) - if missing: - errors.append(f"adjustments[{i}]: no complete {index} value for {missing} (available " - f"{min(table) if table else '-'}-{max(table) if table else '-'})") - - -def _vre_level(level: Any) -> str: - """A manifest geography level as census.replicate names it (``block group`` -> ``block_group``).""" - from .pipeline import normalize_geo_level - return normalize_geo_level(str(level)).replace(" ", "_") - - -def census_vre_source(manifest: Mapping[str, Any]) -> bool: - """True when the manifest declares ``"replicates": {"source": "census_vre"}``.""" - reps = manifest.get("replicates") - return isinstance(reps, Mapping) and reps.get("source") == "census_vre" - - -def _check_replicates(manifest: Mapping[str, Any], errors: List[str]) -> None: - """``replicates`` (optional): ``{"source": "census_vre"}`` makes run_manifest fetch the Census - VRE files for the manifest's tables, geography and releases (census.replicate and its cache).""" - reps = manifest.get("replicates") - if reps is None: - return - if not isinstance(reps, Mapping): - errors.append("replicates must be an object, e.g. {\"source\": \"census_vre\"}") - return - for k in sorted(set(reps) - set(REPLICATES_KEYS)): - errors.append(f"replicates: unknown key {k!r}; allowed {REPLICATES_KEYS}") - if reps.get("source") not in REPLICATE_SOURCES: - errors.append(f"replicates.source must be one of {REPLICATE_SOURCES}") - return - from . import replicate as vre - geo = manifest.get("geography") if isinstance(manifest.get("geography"), Mapping) else {} - try: - sumlev = vre.sumlev_for(_vre_level(geo.get("level", ""))) - except Exception as exc: # noqa: BLE001 - unsupported level for VRE files - errors.append(f"replicates.source census_vre: {exc}") - sumlev = None - if sumlev in vre.PER_STATE_SUMLEVS and not geo.get("state") and not geo.get("states"): - errors.append(f"replicates.source census_vre at level {geo.get('level')!r}: Census VRE files of summary " - f"level {sumlev} are published per state; give geography.state or geography.states") - releases = manifest_releases(manifest) - bad = [r for r in releases if r not in vre.VRE_LAYOUTS] - if bad: - errors.append(f"replicates.source census_vre: no Census VRE release for {bad} (5-year releases " - f"{vre.FIRST_VRE_YEAR}-{vre.LAST_KNOWN_VRE_YEAR})") - if manifest.get("aggregation") is not None: - errors.append("replicates.source census_vre cannot be combined with aggregation (VRE replicates are " - "per GEOID and are not aggregated by run_manifest)") - overlap = release_overlap_message(releases, "replicates.source census_vre") - if overlap: - errors.append(overlap) - - -def _derivation_refs(d: Mapping[str, Any]) -> List[str]: - """Every input name of a derivation (num/den and age_standardized groups; - census.derived._spec_codes, the one shared walk).""" - from .derived import _spec_codes - return _spec_codes(d) - - -def _age_standardized_errors(d: Mapping[str, Any], errors: List[str]) -> None: - """Validation of an ``age_standardized`` derivation (PLAN_ROUND_20 A1).""" - from .derived import standardization_weights - from .standardize import STANDARDS, normalize_bands, standard_weights - name = d.get("name") - groups = d.get("groups") - if not isinstance(groups, list) or not groups: - errors.append(f"derivation {name!r}: age_standardized needs 'groups', one {{num, den}} per age band") - return - for k, g in enumerate(groups): - if not isinstance(g, Mapping) or g.get("num") is None or g.get("den") is None: - errors.append(f"derivation {name!r}: groups[{k}] needs 'num' and 'den'") - return - for key in ("num", "den"): - if d.get(key) is not None: - errors.append(f"derivation {name!r}: age_standardized takes 'groups', not {key!r}") - standard = d.get("standard") - if standard is None: - errors.append(f"derivation {name!r}: 'standard' is required ({list(STANDARDS)} or 'custom' with 'weights')") - return - if standard != "custom" and standard not in STANDARDS: - errors.append(f"derivation {name!r}: unknown standard {standard!r}; expected one of {list(STANDARDS)} " - "or 'custom'") - return - if standard != "custom": - try: - bands = normalize_bands(d.get("bands")) - except ValueError as exc: - errors.append(f"derivation {name!r}: {exc}") - return - if len(bands) != len(groups): - errors.append(f"derivation {name!r}: {len(bands)} bands for {len(groups)} groups") - return - try: - expected = standard_weights(bands, standard) - except ValueError as exc: - errors.append(f"derivation {name!r}: {exc}") - return - given = d.get("weights") - if given is not None and (not isinstance(given, list) or len(given) != len(expected) or any( - not isinstance(x, (int, float)) or abs(float(x) - e) > 1e-12 for x, e in zip(given, expected))): - errors.append(f"derivation {name!r}: weights do not match the {standard} standard over these bands " - f"(expected {expected}); omit them so create_manifest fills them in, or declare " - "\"standard\": \"custom\"") - return - elif d.get("weights") is None: - errors.append(f"derivation {name!r}: standard 'custom' needs explicit 'weights'") - return - if d.get("weights") is not None: - try: - standardization_weights(d) - except ValueError as exc: - errors.append(f"derivation {name!r}: {exc}") - - -def _fill_derivation_defaults(manifest: Dict[str, Any]) -> List[Dict[str, Any]]: - """Fill omitted ``weights`` of ``age_standardized`` derivations (in place) from their - standard, so the hash fixes them; returns one ``{"derivation": i, "weights": [...]}`` - entry per filled derivation. Invalid entries are left for validation to report.""" - filled: List[Dict[str, Any]] = [] - derivations = manifest.get("derivations") - for i, d in enumerate(derivations if isinstance(derivations, list) else []): - if not (isinstance(d, dict) and d.get("op") == "age_standardized" and d.get("weights") is None - and d.get("standard") not in (None, "custom")): - continue - try: - from .standardize import standard_weights - d["weights"] = standard_weights(d.get("bands"), str(d["standard"])) - except (ValueError, TypeError): - continue - filled.append({"derivation": i, "weights": list(d["weights"])}) - return filled - - -def _variable_tables(manifest: Mapping[str, Any]) -> Dict[str, List[str]]: - """Manifest variable or derivation name -> the tables it is built from.""" - out: Dict[str, List[str]] = {} - variables = manifest.get("variables") - for v in variables if isinstance(variables, list) else []: - if isinstance(v, Mapping) and v.get("name") is not None and _table_of(v): # invalid entries: reported - out[str(v["name"])] = [str(_table_of(v))] - derivations = manifest.get("derivations") - for d in derivations if isinstance(derivations, list) else []: - if not isinstance(d, Mapping): # reported by validate_manifest - continue - refs = _derivation_refs(d) - out[str(d.get("name"))] = sorted({t for r in refs for t in out.get(str(r), [])}) - return out - - -def vre_unavailable(manifest: Mapping[str, Any]) -> List[Dict[str, Any]]: - """Variables and derivations without published Census VRE replicates for some release - (offline, from the census.replicate registry): [{"variable", "table", "release", "reason"}].""" - from . import replicate as vre - level = _vre_level((manifest.get("geography") or {}).get("level", "")) - out = [] - releases = manifest_releases(manifest) - for name, tables in _variable_tables(manifest).items(): - for table in tables: - for year in releases: - info = vre.vre_availability(table, year, level) - if not info["available"]: - out.append({"variable": name, "table": table, "release": year, "reason": info["reason"]}) - return out - - -def tables_and_refs(manifest: Mapping[str, Any]) -> Dict[str, List[str]]: - """table code -> variable refs (in manifest order).""" - out: Dict[str, List[str]] = {} - for v in manifest.get("variables", []): - t = _table_of(v) - if t: - out.setdefault(t, []).append(str(v["ref"])) - return out - - -# --------------------------------------------------------------------------- -# Validation -# --------------------------------------------------------------------------- - - -_SPEC_KEYS = ("formula", "error_covariates", "variables", "variable", "geoid", "variable_1", "variable_2") - - -def _normalize_analysis(a: Mapping[str, Any]) -> Dict[str, Any]: - """Return the analysis with spec keys (formula, variables, ...) lifted from - ``args`` to the top level, so either placement works the same way. - Duplicates (a key both top-level and in args) are a validation error.""" - out = dict(a) - args = dict(a.get("args") or {}) - for k in _SPEC_KEYS: - if k in args and k not in out: - out[k] = args.pop(k) - elif k in args: - args.pop(k) - out["args"] = args - return out - - -def _duplicate_spec_keys(a: Mapping[str, Any]) -> List[str]: - args = a.get("args") or {} - return [k for k in _SPEC_KEYS if k in a and k in args] - - -def _formula_names(formula: str) -> List[str]: - """Identifiers in a patsy formula that are not function calls or frame identifiers - (GEOID, Year: e.g. ``C(Year)`` for a change between releases).""" - names = [] - for m in re.finditer(r"[A-Za-z_][A-Za-z0-9_.]*", str(formula)): - tail = str(formula)[m.end():].lstrip() - if tail.startswith("(") or m.group(0) in FORMULA_FUNCTIONS or "." in m.group(0) \ - or m.group(0) in FORMULA_ID_COLUMNS: - continue - names.append(m.group(0)) - return names - - -def _check_variables(manifest: Mapping[str, Any], errors: List[str], resolve: bool, client: Any) -> None: - from .metadata import archived_vintages, vintage_variables - releases = [int(r) for r in manifest.get("releases", []) if str(r).isdigit()] - variables = manifest.get("variables") or [] - if not variables: - errors.append("variables: at least one variable is required") - return - if not isinstance(variables, list): - errors.append("variables must be a list of {name, ref} objects") - return - by_table: Dict[str, List[str]] = {} - for i, v in enumerate(variables): - if not isinstance(v, Mapping) or "name" not in v or "ref" not in v: - errors.append(f"variables[{i}]: needs 'name' and 'ref'") - continue - ref = str(v["ref"]) - m = YEAR_CODE_RE.match(ref) - if not m and not CONCEPT_RE.match(ref): - errors.append(f"variables[{i}] {ref!r}: ref must be YEAR:CODE or a concept ID (c_ + 12 hex)") - continue - table = _table_of(v) - if table is None: - errors.append(f"variables[{i}] {ref!r}: a concept ID needs 'table'") - continue - dataset = _dataset_for(table) - if m: - year, code = int(m.group(1)), m.group(2) - if year not in releases: - errors.append(f"variables[{i}] {ref!r}: year {year} is not among releases {releases}") - if resolve: - if year not in archived_vintages(dataset): - errors.append(f"variables[{i}] {ref!r}: {dataset} {year} not in the offline metadata archive") - continue - known, _ = vintage_variables(dataset, year) - if not any(k == code or k.startswith(code) and k[len(code):] in ("E", "PE") for k in known): - errors.append(f"variables[{i}] {ref!r}: {code} does not exist in {dataset} {year}") - continue - by_table.setdefault(table, []).append(ref) - if not resolve: - return - from .concordance import build_concordance - for table, refs in by_table.items(): - dataset = _dataset_for(table) - missing = [y for y in releases if y not in archived_vintages(dataset)] - if missing: - errors.append(f"{table}: releases {missing} not in the offline metadata archive for {dataset}") - continue - try: - conc = build_concordance(table, dataset, releases, client=client) - _, unknown = conc.resolve_selection(refs) - except BundledDataIntegrityError: - raise - except Exception as exc: # concordance build failures are validation errors - errors.append(f"{table}: concordance could not be built ({exc})") - continue - for ref in unknown: - errors.append(f"variable {ref!r} does not resolve in the {table} concordance for {releases}") - - -def validate_manifest(manifest: Mapping[str, Any], resolve: bool = True, client: Any = None) -> List[str]: - """Return a list of errors (empty when the manifest is valid). - - ``resolve``: check every variable against the offline metadata archive - and the concordance (no network when the archive covers the releases). - """ - errors: List[str] = [] - if "exploratory" in manifest and type(manifest["exploratory"]) is not bool: - errors.append("exploratory must be true or false") - if manifest.get("schema_version") != SCHEMA_VERSION: - errors.append(f"schema_version must be {SCHEMA_VERSION}") - if not str(manifest.get("title", "")).strip(): - errors.append("title is required") - authors = manifest.get("authors") or [] - if not authors: - errors.append("authors: at least one author ('human:' or 'agent:') is required") - for a in authors: - if not AUTHOR_RE.match(str(a)): - errors.append(f"author {a!r} must look like 'human:' or 'agent:'") - try: - datetime.fromisoformat(str(manifest.get("created", "")).replace("Z", "+00:00")) - except ValueError: - errors.append("created must be an ISO 8601 timestamp") - releases = manifest.get("releases") or [] - if not isinstance(releases, list) or not releases or not all(_is_int(r) for r in releases): - errors.append("releases must be a non-empty list of ACS 5-year end years (integers)") - # Present archive corruption must win over geography probes or API fallback. - variables = manifest.get("variables") - if resolve and isinstance(releases, list) and all(_is_int(r) for r in releases) and isinstance(variables, list): - from .metadata import preflight_archives - for variable in variables: - if isinstance(variable, Mapping) and "ref" in variable: - table = _table_of(variable) - if table: - preflight_archives(_dataset_for(table), releases) - _check_geography(manifest.get("geography") or {}, errors) - from .tract_harmonization import validation_errors as _harmonization_errors - errors.extend(_harmonization_errors(manifest.get("geography") or {}, manifest)) - level = manifest.get("confidence_level", 0.90) - if not isinstance(level, (int, float)) or not 0 < float(level) < 1: - errors.append("confidence_level must be in (0, 1), e.g. 0.90") - corr = manifest.get("correction") or {} - try: - from .multitest import _method - _method(corr.get("method", "holm")) - except ValueError as exc: - errors.append(f"correction: {exc}") - if corr.get("family", "all") not in ("all", "by_family"): - errors.append("correction.family must be 'all' or 'by_family'") - ds = manifest.get("data_source") - if ds is not None: - if not isinstance(ds, Mapping): - errors.append("data_source must be an object {synthetic, description, generator}") - else: - if not isinstance(ds.get("synthetic"), bool): - errors.append("data_source.synthetic is required and must be true or false") - for k in ("description", "generator"): - if k in ds and not isinstance(ds[k], str): - errors.append(f"data_source.{k} must be a string") - for k in sorted(set(ds) - set(DATA_SOURCE_KEYS)): - errors.append(f"data_source: unknown key {k!r}; allowed {DATA_SOURCE_KEYS}") - if manifest.get("missing", "complete_case") != "complete_case": - errors.append("missing: only 'complete_case' is supported") - - _check_variables(manifest, errors, resolve, client) - variables = manifest.get("variables") - names = [str(v.get("name")) for v in (variables if isinstance(variables, list) else []) if isinstance(v, Mapping)] - derivations = manifest.get("derivations") or [] - for i, d in enumerate(derivations if isinstance(derivations, list) else [derivations]): - if not isinstance(d, Mapping): - errors.append(f"derivations[{i}] must be an object {{name, op, num, den}}") - continue - if d.get("op") == "age_standardized": - _age_standardized_errors(d, errors) - for r in _derivation_refs(d): - if r not in names: - errors.append(f"derivation {d.get('name')!r}: input {r!r} is not an earlier variable name") - names.append(str(d.get("name"))) - for n in names: - if not NAME_RE.match(n): - errors.append(f"name {n!r} must be a Python identifier (used in formulas)") - if len(set(names)) != len(names): - errors.append("variable/derivation names must be unique") - agg = manifest.get("aggregation") - if agg is not None and not isinstance(agg.get("mapping"), Mapping): - errors.append("aggregation.mapping must be an object GEOID -> group id") - ext_names = external_names(manifest) - if manifest.get("external") is not None: - from .health_sources import validate_external - errors.extend(validate_external(manifest.get("external"), manifest.get("geography") or {}, - [int(r) for r in releases if _is_int(r)] if isinstance(releases, list) - else [], names)) - if agg is not None: - errors.append("external variables cannot be combined with aggregation (external values are per unit)") - _check_adjustments(manifest, names, errors) - _check_replicates(manifest, errors) - - for i, ex in enumerate(manifest.get("exclusions") or []): - if ex.get("rule") not in EXCLUSION_RULES: - errors.append(f"exclusions[{i}]: rule must be one of {EXCLUSION_RULES}") - if ex.get("rule") == "cv_screen" and ex.get("action", "flag") not in ("flag", "exclude"): - errors.append(f"exclusions[{i}]: action must be 'flag' or 'exclude'") - for v in ex.get("variables") or []: - if v not in names: - errors.append(f"exclusions[{i}]: unknown variable {v!r}") - - analyses = manifest.get("analyses") or [] - if not analyses: - errors.append("analyses: at least one analysis is required") - from .report_tables import _artifact_name - ids = {} - folded_ids: Dict[str, Any] = {} - needs_replicates: List[str] = [] # replicate analyses declaring "replicates": "required" - for i, a in enumerate(analyses): - for k in _duplicate_spec_keys(a): - errors.append(f"analyses[{i}]: {k!r} given both at the top level and in args; give it once") - a = _normalize_analysis(a) - method = a.get("method") - if method not in ANALYSIS_METHODS: - errors.append(f"analyses[{i}]: unsupported method {method!r}; supported {sorted(ANALYSIS_METHODS)}") - continue - aid = a.get("id") - if aid in ids or not aid: - errors.append(f"analyses[{i}]: id must be present and unique") - elif str(aid).casefold() in folded_ids: - # results-bundle artifacts are named after ids; reject now, not after every model has run - errors.append(f"analyses[{i}]: id {aid!r} collides with {folded_ids[str(aid).casefold()]!r} on a " - "case-insensitive filesystem; ids must differ in more than letter case") - else: - try: - _artifact_name(str(aid)) - except ValueError: - errors.append(f"analyses[{i}]: id {aid!r} must be usable as a filename (letters, digits, '_', " - "'-', '.'; no path separators, ':', spaces at the end or reserved device names)") - if aid: - folded_ids.setdefault(str(aid).casefold(), aid) - ids[aid] = method - if method in REPLICATE_METHODS and a.get("replicates") == "required": - needs_replicates.append(str(a.get("id"))) - for req in ANALYSIS_METHODS[method][0]: - if req not in a: - errors.append(f"analyses[{i}] ({method}): {req!r} is required") - used = _formula_names(a["formula"]) if a.get("formula") else [] - for key in ("variables", "error_covariates"): - used += list(a.get(key) or []) - for key in ("variable", "variable_1", "variable_2"): - if a.get(key): - used.append(a[key]) - aargs = a.get("args") or {} - outcome_name = str(a.get("formula", "")).split("~")[0].strip() - count_outcome = (outcome_name if method == "fit_replicate" and aargs.get("model") == "poisson" - and outcome_name.endswith("_numerator") and outcome_name[:-10] in ext_names else None) - if count_outcome and count_outcome in _formula_names(str(a.get("formula", "")).split("~", 1)[1]): - errors.append("external count columns are allowed as the Poisson outcome only") - for u in used: - if u not in names and u not in ext_names and u != count_outcome: - errors.append(f"analyses[{i}] ({a.get('id')}): {u!r} is not a manifest variable") - ext_used = [u for u in used if u in ext_names or u == count_outcome] - if ext_used: - from .health_sources import EXTERNAL_OUTCOME_METHODS - outcome = _formula_names(str(a.get("formula", "")).split("~")[0]) if "~" in str(a.get("formula", "")) \ - else [] - if method not in EXTERNAL_OUTCOME_METHODS: - errors.append(f"analyses[{i}] ({a.get('id')}): external variables {ext_used} can only be the outcome " - f"of {list(EXTERNAL_OUTCOME_METHODS)}") - elif [u for u in ext_used if u not in outcome] or len(outcome) != 1: - errors.append(f"analyses[{i}] ({a.get('id')}): external variables {ext_used} may only be the outcome " - "(alone): as covariates their model-based error would go uncorrected") - elif method == "fit_replicate" and (a.get("args") or {}).get("target", "population") == "these_areas": - errors.append(f"analyses[{i}] ({a.get('id')}): an external outcome is held fixed in the refits, so " - "target 'these_areas' (replicate SE alone) is refused; use 'population' or " - "'conservative'") - if method == "fit_replicate": - allowed = {"model", "weights", "exposure", "trials", "cv_threshold", "max_failed_replicates", - "target", "scale", "spatial_se"} - unknown = sorted(set(aargs) - allowed) - if unknown: - errors.append(f"fit_replicate does not take args {unknown}") - glm = aargs.get("model", "ols") - if aargs.get("scale") is not None and (aargs["scale"] not in ("X2", "robust") - or glm not in ("binomial", "poisson")): - errors.append("scale must be 'robust' or 'X2' for binomial or poisson models only") - if aargs.get("scale") == "robust" and aargs.get("spatial_se") is not None: - errors.append("scale 'robust' and spatial_se are alternatives; choose one") - parent = outcome_name if outcome_name in ext_names else count_outcome[:-10] if count_outcome else None - if parent is not None: - if glm not in ("ols", "binomial", "poisson"): - errors.append(f"{glm} not validated for external outcomes (PLAN_ROUND_22 out of scope)") - if glm == "binomial" and aargs.get("trials") != parent + "_denominator": - errors.append(f"external binomial outcome requires trials '{parent}_denominator'; effective trials refused") - if glm == "poisson" and (count_outcome is None or aargs.get("exposure") != parent + "_denominator"): - errors.append(f"external poisson needs {parent}_numerator outcome and exposure '{parent}_denominator'") - if glm in ("binomial", "poisson") and aargs.get("scale") is None and aargs.get("spatial_se") is None: - errors.append("overdispersion would be ignored: set scale 'robust'") - elif (glm in ("binomial", "poisson") and aargs.get("scale") == "X2" - and manifest_hash(manifest) not in LEGACY_X2_MANIFESTS): - errors.append("external count outcome with scale 'X2' refused: Pearson X2 SEs under-cover when " - "areas differ beyond binomial/Poisson chance (validated coverage 0.69-0.79 at " - "nominal 0.90, docs/audits/robust_se_validation.md); set scale 'robust' (HC3)") - for role in ("weights", "exposure", "trials"): - column = aargs.get(role) - valid_count = parent and ((glm == "binomial" and role == "trials") or - (glm == "poisson" and role == "exposure")) and column == parent+"_denominator" - if column is not None and column not in names and column not in FORMULA_ID_COLUMNS and not valid_count: - errors.append(f"fit_replicate {role} {column!r} is not a manifest variable in this role") - spatial = (a.get("args") or {}).get("spatial_se") - if spatial is not None: - valid_spatial = (isinstance(spatial, Mapping) and set(spatial) == {"cutoff_km"} - and type(spatial["cutoff_km"]) in (int, float) - and np.isfinite(spatial["cutoff_km"]) and spatial["cutoff_km"] > 0) - if not valid_spatial: - errors.append("spatial_se must be an object with exactly cutoff_km (finite number > 0)") - from .spatial_hac import SPATIAL_HAC_VALIDATED - if not SPATIAL_HAC_VALIDATED and not manifest.get("exploratory", False): - errors.append("spatial_se failed its validation (docs/audits/spatial_hac_validation.md); exploratory only") - if (a.get("args") or {}).get("cluster"): - errors.append("spatial_se and cluster are mutually exclusive") - if method not in ("fit_eiv", "fit_replicate") or (method == "fit_replicate" and - (a.get("args") or {}).get("model", "ols") not in ("binomial", "poisson")): - errors.append("spatial_se is supported for fit_eiv and fit_replicate binomial/poisson") - if method == "fit_eiv": - eargs = a.get("args") or {} - unknown = sorted(set(eargs) - {"model", "weights", "cv_threshold", "target", "cluster", "cluster_se", "spatial_se"}) - if unknown: - errors.append(f"analyses[{i}] ({a.get('id')}): fit_eiv does not take args {unknown} (allowed: " - "model, weights, cv_threshold, cluster, cluster_se)") - if "cluster_se" in eargs and (eargs["cluster_se"] not in ("cr1", "jackknife") or "cluster" not in eargs): - errors.append(f"analyses[{i}] ({a.get('id')}): fit_eiv args.cluster_se must be 'cr1' or 'jackknife' " - "and needs args.cluster") - if "target" in eargs: - errors.append(f"analyses[{i}] ({a.get('id')}): fit_eiv takes no 'target': its intervals use the " - "sandwich se_total (a measurement-error correction is a claim about the population " - "relationship, not about these areas)") - if "cluster" in eargs: - cl = eargs["cluster"] - from .models import CLUSTER_PREFIX_RE - mm = CLUSTER_PREFIX_RE.match(str(cl)) if isinstance(cl, str) else None - if not isinstance(cl, str) or not (cl in names or cl in FORMULA_ID_COLUMNS - or (mm and 1 <= int(mm.group(1)) <= 11)): - errors.append(f"analyses[{i}] ({a.get('id')}): fit_eiv args.cluster must be a manifest variable " - "or GEOID.str[:N] (e.g. GEOID.str[:5] for counties of tracts)") - if str(eargs.get("model", "ols")).lower() not in ("ols", "wls"): - errors.append(f"analyses[{i}] ({a.get('id')}): fit_eiv supports model 'ols' or 'wls'") - if str(eargs.get("model", "ols")).lower() == "wls" and not eargs.get("weights"): - errors.append(f"analyses[{i}] ({a.get('id')}): fit_eiv with model 'wls' needs args.weights") - if not census_vre_source(manifest) and a.get("replicates") != "required": - errors.append(f"analyses[{i}] ({a.get('id')}): fit_eiv needs replicates: declare " - "replicates.source 'census_vre' or \"replicates\": \"required\"") - scope = a.get("replicate_scope", "all") - if scope not in REPLICATE_SCOPES: - errors.append(f"analyses[{i}] ({a.get('id')}): replicate_scope must be one of {REPLICATE_SCOPES}") - elif scope == "covariates_only": - outcome = _formula_names(a["formula"].split("~")[0]) if "~" in str(a.get("formula", "")) else [] - if method != "fit_replicate": - errors.append(f"analyses[{i}] ({a.get('id')}): replicate_scope 'covariates_only' applies to " - "fit_replicate only") - elif (a.get("args") or {}).get("target", "population") == "these_areas": - errors.append(f"analyses[{i}] ({a.get('id')}): replicate_scope 'covariates_only' cannot use " - "target 'these_areas' (its interval would rest on a sampling SE without the " - "outcome's sampling error); use 'population' or 'conservative'") - elif len(outcome) != 1: - errors.append(f"analyses[{i}] ({a.get('id')}): replicate_scope 'covariates_only' needs a formula " - "whose outcome is one manifest variable") - # An outcome the VRE registry lists may still lack replicates for the analysed units (a line - # missing from the file), so the declaration is allowed; create_manifest notes it and the run - # records whether the outcome had replicates (it is held fixed either way: the hash governs). - if needs_replicates and not census_vre_source(manifest): # census_vre: _check_replicates - overlap = release_overlap_message(manifest_releases(manifest), - f"analyses {needs_replicates} require replicates") - if overlap: - errors.append(overlap) - hyps = manifest.get("hypotheses") or [] - if not hyps: - errors.append("hypotheses: at least one pre-registered hypothesis is required") - hids = set() - for i, h in enumerate(hyps): - if not h.get("id") or h["id"] in hids: - errors.append(f"hypotheses[{i}]: id must be present and unique") - hids.add(h.get("id")) - if h.get("direction", "two-sided") not in DIRECTIONS: - errors.append(f"hypotheses[{i}]: direction must be one of {DIRECTIONS}") - method = ids.get(h.get("analysis")) - if method is None: - errors.append(f"hypotheses[{i}]: analysis {h.get('analysis')!r} is not defined") - continue - if h.get("statistic") not in ANALYSIS_METHODS[method][1]: - errors.append(f"hypotheses[{i}]: statistic {h.get('statistic')!r} not available from {method} " - f"(supported: {ANALYSIS_METHODS[method][1]})") - need = {"coefficient": ("term",), "correlation": ("variables",), "trend_slope": (), - "z_difference": ("geoid",)}.get(h.get("statistic"), ()) - for k in need: - if k not in h: - errors.append(f"hypotheses[{i}]: {k!r} is required for statistic {h.get('statistic')!r}") - just = h.get("circularity_justification") - if just is not None and not isinstance(just, str): - errors.append(f"hypotheses[{i}]: circularity_justification must be text") - if ext_names: - errors.extend(circularity_errors(manifest)) - return errors - - -def external_names(manifest: Mapping[str, Any]) -> List[str]: - """Names of the manifest's ``external`` variables (well-formed entries only).""" - ext = manifest.get("external") - return [str(e["name"]) for e in (ext if isinstance(ext, list) else []) - if isinstance(e, Mapping) and e.get("name") is not None] - - -def external_circularity(manifest: Mapping[str, Any]) -> List[Dict[str, Any]]: - """Analyses with an external outcome, with the covariates that measure the outcome model's inputs - (``overlap``) and those that measure the outcome's construct directly in the ACS (``equivalent``).""" - from .health_sources import circularity_overlap, equivalent_tables, measure_kind - ext = {str(e["name"]): e for e in manifest.get("external") or [] if isinstance(e, Mapping) and "name" in e} - if not ext: - return [] - tables = _variable_tables(manifest) - out = [] - for a in manifest.get("analyses") or []: - if not isinstance(a, Mapping) or "~" not in str(a.get("formula", "")): - continue - lhs, rhs = str(a["formula"]).split("~", 1) - ys = [y if y in ext else y[:-10] for y in _formula_names(lhs) - if y in ext or (y.endswith("_numerator") and y[:-10] in ext)] - if not ys: - continue - covs = [c for c in _formula_names(rhs) if c in tables] - for y in ys: - src, meas = ext[y].get("source"), ext[y].get("measure") - try: - kind = measure_kind(src, meas) - hit = {c: circularity_overlap(src, tables[c], meas) for c in covs} - eq = {c: equivalent_tables(src, meas, tables[c]) for c in covs} - except (ValueError, KeyError, TypeError): # unknown source or measure: reported by validation - continue - out.append({"analysis": a.get("id"), "outcome": y, "source": src, "measure": meas, "kind": kind, - "overlap": {c: v for c, v in hit.items() if v}, - "equivalent": {c: v for c, v in eq.items() if v}, - "has_equivalent": bool(_has_equivalent(src, meas))}) - return out - - -def _ext_source_text(entry: Mapping[str, Any]) -> str: - from .health_sources import SOURCES, source_text - try: - return source_text(str(entry["source"]), entry["measure"]) - except (KeyError, ValueError): - return str(SOURCES.get(entry.get("source"), {}).get("model_text", "")) - - -def _has_equivalent(source: Any, measure: Any) -> tuple: - from .health_sources import acs_equivalent - try: - return acs_equivalent(str(source), measure) - except ValueError: - return () - - -def circularity_errors(manifest: Mapping[str, Any]) -> List[str]: - """The enforced circularity rule (PLAN_ROUND_21 Part 2, C1): R1 same construct, R2 model inputs.""" - from .health_sources import CIRCULARITY_RULE, acs_equivalent - errors: List[str] = [] - by_analysis: Dict[Any, List[Dict[str, Any]]] = {} - for c in external_circularity(manifest): - by_analysis.setdefault(c["analysis"], []).append(c) - hyps = [h for h in manifest.get("hypotheses") or [] if isinstance(h, Mapping)] - tables = _variable_tables(manifest) - for aid, recs in by_analysis.items(): - tested = [h for h in hyps if h.get("analysis") == aid] - for c in recs: - if c["has_equivalent"]: - eq_tabs = "/".join(t.rstrip("*") + ("*" if t.endswith("*") else "") - for t in acs_equivalent(c["source"], c["measure"])[:4]) - if not c["equivalent"]: - errors.append( - f"analyses {aid}: CIRCULARITY R1: the outcome {c['outcome']} ({c['source']} " - f"{c['measure']}) measures a construct the ACS measures directly ({eq_tabs}). Use the " - "ACS measure as the outcome of a determinant study; an external measure of the same " - "construct is allowed only to compare the two (the model must contain the ACS measure " - f"and only it may be tested). {CIRCULARITY_RULE}") - continue - for h in tested: - term = [n for n in _formula_names(str(h.get("term", ""))) if n in tables] or [str(h.get("term"))] - if not all(t in c["equivalent"] for t in term): - errors.append( - f"hypotheses {h.get('id')}: CIRCULARITY R1: analysis {aid} compares the external " - f"{c['outcome']} with its ACS equivalent {sorted(c['equivalent'])}; only that " - f"comparison may be tested, not {h.get('term')!r} (a determinant study uses the ACS " - "measure as its outcome)") - if c["kind"] == "direct": - continue - for h in tested: - term = [n for n in _formula_names(str(h.get("term", ""))) if n in c["overlap"]] - just = h.get("circularity_justification") - if term and not (isinstance(just, str) and len(just.strip()) >= 20): - fams = "; ".join(f"{t}: " + ", ".join(f"{lab} {'/'.join(tabs)}" for lab, tabs in c["overlap"][t].items()) - for t in term) - errors.append( - f"hypotheses {h.get('id')}: CIRCULARITY R2: the tested term {h.get('term')!r} is built from " - f"inputs of the model that produced {c['outcome']} ({c['source']}, {c['kind']}: {fams}); " - "its coefficient partly recovers that model, not the world. Test a direct outcome (the " - "ACS measure, vital records), or add a written circularity_justification (20+ " - "characters) to the hypothesis") - return errors - - -def _declared_covariates_only(manifest: Mapping[str, Any]) -> Dict[str, List[str]]: - """{analysis id: outcome names} of analyses declaring ``replicate_scope: "covariates_only"``.""" - out: Dict[str, List[str]] = {} - for a in manifest.get("analyses") or []: - if not isinstance(a, Mapping): - continue - a = _normalize_analysis(a) - if a.get("replicate_scope") == "covariates_only" and "~" in str(a.get("formula", "")): - out[str(a.get("id"))] = _formula_names(str(a["formula"]).split("~")[0]) - return out - - -_FILL_REASONS = { - "rpp_year": "the latest year in the RPP table", - "dollar_year": "the latest release", - "price_index": "the default index (R-CPI-U-RS, the index the Census Bureau uses for ACS dollars)", - "rpp_years": "each release's own year (D15-2)", -} - - -def _fill_adjustment_defaults(manifest: Dict[str, Any]) -> List[Dict[str, Any]]: - """Fill omitted required adjustment fields (in place) so the hash fixes them; returns - one ``{"adjustment": i, : value}`` entry per filled field. - - One release: ``rpp_year`` (the latest RPP year; Round 14). Several releases (or a declared - ``dollar_year``): ``dollar_year`` (the latest release), ``price_index`` (R-CPI-U-RS) and - ``rpp_years`` (each release's own year, when the RPP table has it; otherwise left out and - validation asks for it).""" - filled: List[Dict[str, Any]] = [] - adjustments = manifest.get("adjustments") - if not isinstance(adjustments, list): - return filled - releases = manifest_releases(manifest) - multi = len(releases) > 1 - try: - from . import rpp - rpp_years, latest = rpp._years(), latest_rpp_year() - except Exception: # noqa: BLE001 - an unreadable RPP table is reported by validation - rpp_years = latest = None - for i, adj in enumerate(adjustments): - if not (isinstance(adj, dict) and adj.get("type") == "rpp"): - continue - if multi and adj.get("dollar_year") is None and releases: - adj["dollar_year"] = max(releases) - filled.append({"adjustment": i, "dollar_year": max(releases)}) - if adj.get("dollar_year") is not None and adj.get("price_index") is None: - from .inflation import DEFAULT_PRICE_INDEX - adj["price_index"] = DEFAULT_PRICE_INDEX - filled.append({"adjustment": i, "price_index": DEFAULT_PRICE_INDEX}) - if multi or "rpp_years" in adj: - if adj.get("rpp_years") is None and "rpp_year" not in adj and rpp_years \ - and all(r in rpp_years for r in releases): - adj["rpp_years"] = {str(r): int(r) for r in releases} - filled.append({"adjustment": i, "rpp_years": dict(adj["rpp_years"])}) - elif adj.get("rpp_year") is None and latest is not None: - adj["rpp_year"] = int(latest) - filled.append({"adjustment": i, "rpp_year": int(latest)}) - return filled - - -@integrity_errors -def create_manifest( - path_or_dict: Union[str, Path, Mapping[str, Any]], path: Optional[Union[str, Path]] = None, - client: Any = None, resolve: bool = True, -) -> Dict[str, Any]: - """Validate a manifest and record its SHA-256 before any data are pulled. - - When valid and ``path`` is given (or the input was a dict and ``path`` - is given) the manifest is written there as pretty JSON. Commit the file - to the repository to fix the pre-registration (PLAN_ROUND_8 decision 3). - - An RPP adjustment without ``rpp_year`` gets the latest year in the active RPP - table before validation and hashing, so the hash fixes the applied year (with - several releases: ``dollar_year``, ``price_index`` and ``rpp_years`` get their - defaults, :func:`_fill_adjustment_defaults`); the result says so (``filled``, ``notes``) and the completed manifest is written to - ``path``, or, when no ``path`` is given, to a new sibling file - ``.completed.json`` (the input file is never overwritten; a dict input - without ``path`` gets it in ``manifest``). Commit and run the completed file. - Returns ``{"hash", "path", "errors", "valid", "notes", "filled"}`` (and - ``"manifest"`` when something was filled in). - """ - manifest = load_manifest(path_or_dict) - filled = _fill_adjustment_defaults(manifest) + _fill_derivation_defaults(manifest) - errors = validate_manifest(manifest, resolve=resolve, client=client) - out_path = str(path) if path else (str(path_or_dict) if not isinstance(path_or_dict, Mapping) else None) - write_to = path - if not path and filled and out_path: - src = Path(out_path) - write_to = str(src.with_name(src.stem + ".completed" + (src.suffix or ".json"))) - out_path = write_to - if not errors and write_to: - Path(write_to).parent.mkdir(parents=True, exist_ok=True) - Path(write_to).write_text(json.dumps(manifest, indent=2, sort_keys=True, ensure_ascii=False) + "\n", - encoding="utf-8", newline="\n") - where = f" (written to {write_to})" if write_to and not errors else "; commit the completed manifest" - notes = [f"adjustments[{f['adjustment']}]: {k} was omitted; filled in with {json.dumps(v)}, " - + _FILL_REASONS[k] + ", before hashing" + where - for f in filled if "adjustment" in f for k, v in f.items() if k != "adjustment"] - notes += [f"derivations[{f['derivation']}]: weights were omitted; filled in with {json.dumps(f['weights'])} " - "from the derivation's standard population, before hashing" + where - for f in filled if "derivation" in f] - if census_vre_source(manifest): - missing = vre_unavailable(manifest) - if missing: - notes.append("replicates (census_vre): no published Census VRE replicates for " - + "; ".join(f"{u['variable']} ({u['table']}, {u['release']})" for u in missing) - + ". Under the strict confirmatory policy a replicate analysis that uses them blocks " - "(replicates_incomplete) unless it declares \"replicate_scope\": \"covariates_only\" " - "(outcome without replicates, held at its estimate in the refits)") - no_vre = {u["variable"] for u in missing} - for aid, held in _declared_covariates_only(manifest).items(): - listed = [v for v in held if v not in no_vre] - if listed: - notes.append(f"analyses {aid}: replicate_scope 'covariates_only' declared, but the Census VRE " - f"registry lists replicates for the outcome {listed} in every release. The outcome " - "will be held at its estimate in every refit as declared (the hashed declaration " - "governs; the run records whether its replicates were available); remove the " - "declaration before hashing to use the outcome's replicates") - if not errors: - from .health_sources import CIRCULARITY_RULE - hyps = [h for h in manifest.get("hypotheses") or [] if isinstance(h, Mapping)] - for c in external_circularity(manifest): - model = _ext_source_text(c) - if c["equivalent"]: - notes.append(f"analyses {c['analysis']}: MEASUREMENT COMPARISON: the outcome {c['outcome']} " - f"({c['source']} {c['measure']}) is compared with its direct ACS equivalent " - f"{sorted(c['equivalent'])} (allowed by rule R1; a determinant study would use the " - "ACS measure as its outcome)") - if c["kind"] == "direct": - notes.append(f"analyses {c['analysis']}: the outcome {c['outcome']} is a direct external estimate " - f"({c['source']}: {model}); no model inputs, so no circularity check applies") - continue - justified = [h.get("id") for h in hyps if h.get("analysis") == c["analysis"] - and h.get("circularity_justification")] - if c["overlap"]: - notes.append(f"analyses {c['analysis']}: CIRCULARITY: the outcome {c['outcome']} is {c['kind'].replace('_', '-')} " - f"({c['source']}: {model}); covariates " - + "; ".join(k + " (" + ", ".join(lab + ": " + "/".join(t) for lab, t in v.items()) + ")" - for k, v in c["overlap"].items()) - + " measure the model's own inputs, so their coefficients partly recover the source " - "model, not the world (allowed as controls; tested terms need a written " - "circularity_justification" - + (f", given for {justified}" if justified else "") + ")") - else: - notes.append(f"analyses {c['analysis']}: the outcome {c['outcome']} is {c['kind'].replace('_', '-')} ({c['source']}: " - f"{model}); associations with covariates outside the model arise only through their " - "correlation with the model's inputs") - if manifest.get("external"): - notes.append("external outcomes: " + CIRCULARITY_RULE) - if manifest.get("exploratory"): - notes.append("registered as exploratory") - out = {"hash": manifest_hash(manifest), "path": out_path, "errors": errors, "valid": not errors, - "notes": notes, "filled": filled} - if filled: - out["manifest"] = manifest - return out - - -# --------------------------------------------------------------------------- -# Exploratory labelling -# --------------------------------------------------------------------------- - - -def label_exploratory(result: Any, reason: str = "not in a pre-registered manifest", - manifest_hash_value: Optional[str] = None) -> Any: - """Tag a result (result object with ``provenance`` or a dict) as exploratory. - - Use for any analysis run outside a manifest, including on the same data - as a manifest run (pass its hash in ``manifest_hash_value``). - """ - tag = {"analysis_status": STATUS_EXPLORATORY, "exploratory_reason": reason, - "related_manifest_hash": manifest_hash_value} - if hasattr(result, "provenance") and isinstance(result.provenance, dict): - result.provenance.update(tag) - if hasattr(result, "notes") and isinstance(result.notes, list): - result.notes.append(f"EXPLORATORY: {reason}") - return result - if isinstance(result, dict): - result.update(tag) - return result - raise TypeError("label_exploratory needs a result object with .provenance or a dict") - - -def analysis_status(result: Any) -> str: - prov = getattr(result, "provenance", result if isinstance(result, dict) else {}) - return prov.get("analysis_status", STATUS_EXPLORATORY) if isinstance(prov, dict) else STATUS_EXPLORATORY - - -# --------------------------------------------------------------------------- -# Running -# --------------------------------------------------------------------------- - - -def _codebook_columns(codebook: pd.DataFrame, concept_id: str) -> Optional[Tuple[str, str, str]]: - """(estimate slug, moe slug, estimate code) for a concept in a run_batch codebook.""" - rows = codebook[codebook["Concept_ID"].astype(str) == concept_id] - est = rows[rows["Variable_Slug"].str.endswith(("_est", "_pct")) & ~rows["Variable_Slug"].str.endswith("_moe")] - if est.empty: - return None - e = est.iloc[0] - slug = str(e["Variable_Slug"]) - moe = f"{slug}_moe" if slug.endswith("_pct") else slug[: -len("_est")] + "_moe" - return slug, moe, str(e["Variable_Code"]) - - -def _wide_from_panel(panel: pd.DataFrame, columns: Mapping[str, Tuple[str, str, str]]) -> pd.DataFrame: - """Panel (slugged) -> wide frame keyed by ACS codes (``E``/``M``).""" - from .stats import parse_variable_code - out = pd.DataFrame({"GEOID": panel["geoid"].astype(str), "Year": panel["year"].astype("int64")}) - if "name" in panel.columns: - out["NAME"] = panel["name"] - for _, (est, moe, code) in columns.items(): - base, suffix = parse_variable_code(code) or (code.rstrip("E"), "E") - out[f"{base}{suffix}"] = pd.to_numeric(panel[est], errors="coerce").to_numpy() - out[f"{base}{'PM' if suffix == 'PE' else 'M'}"] = pd.to_numeric(panel[moe], errors="coerce").to_numpy() - return out - - -CONFIRMATORY_REPLICATE_POLICY = "strict" -# Manifest methods that use replicates; each runs with replicate_policy="strict". -# (Spatial, multilevel, DiD and PUMS analyses are not manifest methods; if added -# they must join this set so confirmatory runs refuse incomplete replicates.) -REPLICATE_METHODS = ("fit_replicate", "correlation_matrix", "fit_eiv") - - -def _one_sided(p: float, estimate: float, direction: str) -> float: - if direction == "two-sided" or p is None or not np.isfinite(p): - return p - agrees = (estimate > 0) if direction == "greater" else (estimate < 0) - return p / 2 if agrees else 1 - p / 2 - - -def _run_analysis(a: Mapping[str, Any], frame: pd.DataFrame, level: float, replicates: Any, - correction: str, external: Sequence[str] = (), coords=None) -> Any: - from . import correlation, models, multitest, sae, trends - a = _normalize_analysis(a) - method, args = a["method"], dict(a.get("args") or {}) - for k in ("error_covariates", "variables", "variable", "geoid", "variable_1", "variable_2"): - if k in a: - args.setdefault(k, a[k]) - if method in REPLICATE_METHODS: - # Confirmatory runs refuse incomplete replicates (PLAN_ROUND_11 Q2/D7); - # the failure is recorded as a "replicates_incomplete" deviation. - args["replicate_policy"] = CONFIRMATORY_REPLICATE_POLICY - if args.get("spatial_se") is not None: - args["coords"] = coords - held_external = [v for v in _formula_names(str(a.get("formula", "")).split("~")[0]) if v in set(external) or (v.endswith("_numerator") and v[:-10] in set(external))] - if method == "fit_replicate": - fixed_vars = list(held_external) - if a.get("replicate_scope") == "covariates_only": # pre-registered: the outcome is held fixed - fixed_vars += [v for v in _formula_names(a["formula"].split("~")[0]) if v not in fixed_vars] - if fixed_vars: - args["fixed_variables"] = fixed_vars - return models.fit_replicate(a["formula"], frame, replicates=replicates, confidence_level=level, **args) - if method == "fit_eiv": - if held_external: - args["fixed_variables"] = held_external - return models.fit_eiv(a["formula"], frame, replicates=replicates, confidence_level=level, **args) - if method in ("regression_calibration", "simex"): - return getattr(models, method)(a["formula"], frame, confidence_level=level, **args) - if method in ("fay_herriot", "fh_regression"): - y = a["formula"].split("~")[0].strip() - args.setdefault("se_col", f"{y}_se") - return getattr(sae, method)(a["formula"], frame, confidence_level=level, **args) - if method == "correlation_matrix": - return correlation.correlation_matrix(frame, args.pop("variables"), confidence_level=level, - replicates=replicates, correction=correction, **args) - if method == "linear_trend_test": - v, g = args.pop("variable"), str(args.pop("geoid")) - sub = frame[frame["GEOID"] == g].sort_values("Year") - return trends.linear_trend_test(sub["Year"].tolist(), sub[v].tolist(), ses=sub[f"{v}_se"].tolist(), - confidence_level=level, **args) - if method == "batch_z_test": - v1, v2 = args.pop("variable_1"), args.pop("variable_2") - df = frame[["GEOID"]].copy() - for v, tag in ((v1, "1"), (v2, "2")): - df[f"e{tag}"] = frame[v].to_numpy(float) - df[f"m{tag}"] = frame[f"{v}_se"].to_numpy(float) * 1.645 - out = multitest.batch_z_test(df, "e1", "m1", "e2", "m2", confidence_level=level, - correction=correction, **args) - return out - raise ValueError(f"unsupported method {method}") - - -def _statistic(h: Mapping[str, Any], res: Any) -> Dict[str, Any]: - stat = h["statistic"] - if stat == "coefficient": - c = res.coefficients.set_index("term") - if h["term"] not in c.index: - raise KeyError(f"term {h['term']!r} not in model (terms: {list(c.index)})") - row = c.loc[h["term"]] - from .models import interval_se_column - se_col = interval_se_column(res) # the SE behind p_value and the interval - return {"estimate": float(row["estimate"]), "se": float(row[se_col]), "se_basis": se_col, - "p_two_sided": float(row["p_value"]), - "ci_lower": float(row["ci_lower"]), "ci_upper": float(row["ci_upper"])} - if stat == "correlation": - a, b = h["variables"] - return {"estimate": float(res.matrix.loc[a, b]), "se": None, "p_two_sided": float(res.p_values.loc[a, b]), - "ci_lower": float(res.ci_lower.loc[a, b]), "ci_upper": float(res.ci_upper.loc[a, b])} - if stat == "trend_slope": - return {"estimate": res["slope"], "se": res["se_slope"], "p_two_sided": res["p_value"], - "ci_lower": res["ci_lower"], "ci_upper": res["ci_upper"]} - if stat == "z_difference": - row = res[res["GEOID"] == str(h["geoid"])] - if row.empty: - raise KeyError(f"GEOID {h['geoid']} not in the test frame") - r = row.iloc[0] - diff = float(r.get("difference", r["e1"] - r["e2"])) - return {"estimate": diff, "se": None, "p_two_sided": float(r["p_value"]), "ci_lower": None, "ci_upper": None} - raise ValueError(stat) - - -def _tag(result: Any, h: str, analysis_id: str, status: str = STATUS_CONFIRMATORY) -> Any: - tag = {"analysis_status": status, "manifest_hash": h, "manifest_analysis_id": analysis_id} - if hasattr(result, "provenance") and isinstance(result.provenance, dict): - result.provenance.update(tag) - return result - if isinstance(result, pd.DataFrame): - return {**tag, "rows": result} - if isinstance(result, dict): - return {**result, **tag} - return result - - -def run_variant( - run_dir: Union[str, Path], - analysis: str, - formula: Optional[str] = None, - keep_prefixes: Optional[Sequence[str]] = None, - drop_prefixes: Optional[Sequence[str]] = None, - args: Optional[Mapping[str, Any]] = None, - out_dir: Optional[Union[str, Path]] = None, - label: Optional[str] = None, -) -> Dict[str, Any]: - """Exploratory refit of one analysis of a completed run (PLAN_ROUND_21 Part 2, C7). - - Loads ``/manifest.json`` and the analysed data recorded in run.json (refusing files - whose SHA-256 differs), applies the variant (``formula``; ``keep_prefixes``/``drop_prefixes``: - GEOID prefixes, e.g. ["36"] or ["36005", "36047"]; ``args`` merged over the analysis args, e.g. - {"cluster": "GEOID.str[:5]", "cluster_se": "jackknife"}), refits with the confirmatory code path - and writes ``/variant__