diff --git a/CHANGELOG.md b/CHANGELOG.md index 2fe12d1..90cb530 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,25 @@ each tagged release (`v`) is archived on Zenodo with its own DOI (see [Releasing](#releasing)). Plans and acceptance criteria for each round are in `PLAN_ROUND_.md`. +## 0.22.0 (Round 24): method registry and public-health inequality + +- Centralize manifest execution, method-specific validation, statistics and methods text in + `MethodSpec`; preserve all 478 refactor snapshots and the existing public constant ordering. +- Add weighted area ridits, SII, log RII and concentration with independent-area HC3 and strict + ACS replicate uncertainty; concentration ranks from least to most deprived (the reverse of + the usual income-ranked sign). Warn about rank-error attenuation above median CV 15%. +- Add indirect SMR, pooled counts/expected counts, fixed-E Garwood intervals and combined + Poisson/ACS log intervals; require explicit per-person reference rates and a hashed citation. + Negative ACS band replicates are refused under the fixed validation domain, even with positive E. +- Both pre-specified validations passed: inequality conservative coverage 0.876–0.905 in + required CV<=15% cells; SMR combined coverage 0.893–0.909. High-error inequality cells + remain outside the validated range (coverage as low as 0.638 at rank CV 30%). +- Register both manifest methods without adding MCP tools (63). Extend circularity to ranked + external outcomes, add log-scale ratio columns, explicit complete-case exclusion records, + public library exports and the `health_inequality` recipe across agent guidance. +- Cache-only exploratory MI/NY LBW example: 144 complete counties, rank CV 6.63%; all + conservative intervals include their null. Outputs remain a separate local archive. + ## 0.21.1 (Round 23): internal package split Internal: `census.manifest` split into a package (no behaviour change; golden snapshot identical) diff --git a/CITATION.cff b/CITATION.cff index 518a17f..a23ad26 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -12,7 +12,7 @@ authors: - alias: cafeTechne license: AGPL-3.0-or-later repository-code: "https://github.com/cafeTechne/Census-Research-Engine" -version: "0.21.1" +version: "0.22.0" doi: "10.5281/zenodo.23925301" # doi: Zenodo concept DOI, added after the first archived release (see CHANGELOG.md "Releasing"). date-released: "2026-11-15" diff --git a/CLAUDE.md b/CLAUDE.md index 29210e0..ca958b8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,8 +1,8 @@ # Agent guide Start with `STATUS.md` (current state, validated methods, rules, backlog). Then read only the current -`PLAN_ROUND_.md` if you are implementing a round. Older plans are archived at tag `v0.21.0` -(`git show v0.21.0:PLAN_ROUND_.md`); read one only when a docstring citation is not enough. +`PLAN_ROUND_.md` if you are implementing a round. Older plans are archived at their release tags +(`git show v0.21.0:PLAN_ROUND_.md` for rounds 1–22; `v0.21.1` for 23; `v0.22.0` for 24); read one only when a docstring citation is not enough. ## Commands diff --git a/PLAN_ROUND_23.md b/PLAN_ROUND_23.md deleted file mode 100644 index 13befd6..0000000 --- a/PLAN_ROUND_23.md +++ /dev/null @@ -1,173 +0,0 @@ -# Round 23 plan: split `census/manifest.py` into a package - -**Goal.** `census/manifest.py` (3,010 lines, 180 KB) is the engine's most-edited file. Every change makes an -agent read parts of it. Split it into a package of focused modules with **no behaviour change**, so later -rounds read one module of a few hundred lines. - -**Non-goals.** No new features, no bug fixes, no renamed functions, no changed messages, no reformatting of -moved code, no new dependencies, no changes outside the files named here. If you find a bug, write it in -section 7 (status table) and leave it alone. - -## 0. Ground rules - -1. Branch `phase23/split-manifest` from `phase23/plan`. Push it. Do not merge, tag or release. -2. **No AI attribution anywhere.** Commits and the PR body must not contain `Co-Authored-By`, - `Generated with`, or any similar line, even if your tooling suggests one. -3. Never print, log or commit `.env` or any API key or token. -4. Use `.venv/bin/python` (there is no `python` on PATH). Set `OPENBLAS_NUM_THREADS=1` for the suite. -5. Do not run GitHub CI. Commits may carry `[skip ci]`. -6. Read only what you need: `census/manifest.py`, the files named in sections 3–4, `pyproject.toml`, - `tests/test_packaging.py`. Do not read `docs/studies/`, `docs/audits/`, `studies_output/` or old plans. -7. If something in this plan is impossible as written, stop, record it in section 7, and choose the option - that changes the least. - -## 1. Baseline (before touching any code) - -On a clean checkout of `phase23/plan`: - -```bash -CENSUS_HEALTH_OFFLINE=1 .venv/bin/python scripts/refactor_golden.py /tmp/golden_before.json -.venv/bin/python -c "import census.manifest as m; print('\n'.join(sorted(n for n in vars(m) if not n.startswith('__'))))" > /tmp/names_before.txt -OPENBLAS_NUM_THREADS=1 .venv/bin/python -m pytest -q 2>&1 | tail -1 > /tmp/suite_before.txt -``` - -Expected suite line: `1634 passed, 37 skipped` (warnings and subtests may vary). Record the actual line in -section 7. `refactor_golden.py` must report `478 snapshots from 11 manifests`. - -## 2. Target layout - -Convert the module into a package. Start with `git mv census/manifest.py census/manifest/__init__.py`, then -cut code out of `__init__.py` into the submodules below. Move each definition **verbatim**: same body, -same comments, same docstring. Only import lines may change. - -| Module | Contents (exact names) | -| :--- | :--- | -| `schema.py` | All module-level constants from `SCHEMA_VERSION` through `REFERENCES` (including `ANALYSIS_METHODS`, `BLOCKING_DEVIATIONS`, the `STATUS_*` names, `FORMULA_ID_COLUMNS`, `FORMULA_FUNCTIONS`, `_STATE_PREFIXED_LEVELS`), `LEGACY_X2_MANIFESTS`, `CONFIRMATORY_REPLICATE_POLICY`, `REPLICATE_METHODS`, `_SPEC_KEYS`; and `canonical_json`, `manifest_hash`, `load_manifest`, `_table_of`, `_dataset_for`, `manifest_states`, `manifest_entities`, `_is_int`, `manifest_releases`, `release_overlap_message`, `_normalize_analysis`, `_duplicate_spec_keys`, `_formula_names`, `_variable_tables`, `tables_and_refs`, `_derivation_refs` | -| `external.py` | `external_names`, `external_circularity`, `_ext_source_text`, `_has_equivalent`, `circularity_errors` | -| `adjustments.py` | `default_rpp_areas`, `latest_rpp_year`, `_check_adjustments`, `_check_rpp_years`, `_check_dollar_year`, `_FILL_REASONS`, `_fill_adjustment_defaults`, `_scale_rows`, `_apply_adjustments` | -| `derivations.py` | `_age_standardized_errors`, `_fill_derivation_defaults` | -| `replicates.py` | `_vre_level`, `census_vre_source`, `_check_replicates`, `vre_unavailable`, `_declared_covariates_only`, `_replicate_overlap`, `_has_replicate_rows`, `_replicate_rows`, `_derive_replicates`, `_census_vre_replicates`, `_canonical_column`, `replicates_digest`, `read_replicates`, `replicates_sha256` | -| `datasource.py` | `effective_data_source`, `observed_data_source`, `data_source_record`, `resolve_data_source` | -| `validate.py` | `_check_geography`, `_check_variables`, `validate_manifest` (and the section 5 helpers) | -| `create.py` | `create_manifest` | -| `methods_text.py` | `_METHOD_TEXT`, `_age_standardized_text`, `_multi_release_adjustment_text`, `methods_section` (and the section 5 helpers) | -| `run.py` | `label_exploratory`, `analysis_status`, `_codebook_columns`, `_wide_from_panel`, `_one_sided`, `_run_analysis`, `_statistic`, `_tag`, `run_variant`, `run_manifest`, `_write_analysed_data`, `_write_bundle` | -| `__init__.py` | The original module docstring (lines 1–174, unchanged) and re-exports only (section 3). No logic. | - -Any definition not listed: put it in the module whose functions use it, and list it in section 7. - -**Import rules (to avoid cycles).** Submodules import each other with explicit relative imports -(`from .schema import manifest_hash`). A submodule never imports from `census.manifest` or from `.` (the -package `__init__`). Allowed direction: `schema` ← everything; `external`, `adjustments`, `derivations`, -`replicates`, `datasource` ← `validate`, `create`, `methods_text`, `run`; `validate` ← `create`, `run`; -`methods_text` ← `run`. Keep a function-local import inside a function body as a function-local import. - -**Packaging.** `pyproject.toml` lists `packages = ["census"]` explicitly, so the new subpackage would be -left out of installs. Change it to `packages = ["census", "census.manifest"]`. Nothing else in -`pyproject.toml` changes. - -## 3. Facade: `census/manifest/__init__.py` - -Re-export **every** name in `/tmp/names_before.txt` that was defined in the old module (functions, classes, -constants, including `_private` ones). Names that were only imported into the old module (`np`, `pd`, `json`, -`Path`, `Any`, `integrity_errors`, `IncompleteReplicatesError`, etc.) should also stay reachable: re-export -them too, so that `vars(census.manifest)` is a superset of the baseline. Write the re-exports as explicit -`from .schema import (a, b, ...)` lists, grouped by submodule, alphabetical within a group. No `import *`. - -## 4. Tests: only these lines may change - -Moving a function changes where `mock.patch` must point: a patch on `census.manifest.X` no longer affects -a call to `X` made inside a submodule. These are the only patches on this module: - -| File | Current target | -| :--- | :--- | -| `tests/test_review_fixes_r15.py` | `mock.patch("census.manifest.latest_rpp_year", ...)` | -| `tests/test_review_fixes_r12.py` | `mock.patch("census.manifest.replicates_digest", ...)` | -| `tests/test_metadata_integrity.py`, `tests/test_review_fixes_r14.py` | `patch("census.manifest.run_manifest", ...)` | -| `tests/test_manifest_releases.py` | `mock.patch("census.manifest.validate_manifest", ...)` | -| `tests/test_audit_decisions.py` | `patch.object(manifest, "_run_analysis", ...)` | -| `tests/test_review_fixes_r14.py` | `patch.object(M, "default_rpp_areas", ...)` | -| `tests/test_tract_harmonization.py` | `patch.object(mf, "_write_analysed_data", ...)` | - -For each one: run its test first. If it still passes unchanged (the caller looks the name up through -`census.manifest`, as `mcp_server.py` and `census/workflows*.py` do), leave it. If it fails, re-point the -patch at the submodule where the name is **looked up at call time** (for example -`census.manifest.run._run_analysis`). If a name is called from two submodules and the test needs both, -patch both. Change nothing else in any test file, and record each change in section 7. - -No other file outside `census/manifest/`, `pyproject.toml`, those test lines and the docs in section 6 may -change. In particular, `mcp_server.py` and other `census/*.py` modules keep importing from -`census.manifest` exactly as now. - -## 5. Splitting the three long functions (golden-covered only) - -After sections 2–4 pass section 6's checks, commit. Then split **only** these two functions, which the golden -snapshot covers byte for byte: - -- `validate_manifest` (about 300 lines): extract its consecutive blocks into private helpers in - `validate.py`, called in the same order with the same `errors` list, so the output order is unchanged. - Natural cuts: top-level keys/schema; variables; derivations; the per-analysis loop body - (`_check_analysis(i, a, ...)`); hypotheses; exclusions/missing/correction. Target: the - `validate_manifest` body is at most 80 lines. -- `methods_section` (about 320 lines): extract the paragraph builders (data, derivations, adjustments, - replicates, per-analysis text, references) into private helpers in `methods_text.py`, joined in the same - order. Target: the `methods_section` body is at most 80 lines. - -Do **not** split `run_manifest` this round. The golden snapshot does not cover it byte for byte; record its -natural stages in section 7 for a later round. - -If a split cannot keep the golden snapshot identical, revert that function to the verbatim move and record -it in section 7. - -## 6. Acceptance (all required) - -```bash -CENSUS_HEALTH_OFFLINE=1 .venv/bin/python scripts/refactor_golden.py /tmp/golden_after.json -cmp /tmp/golden_before.json /tmp/golden_after.json # must print nothing -.venv/bin/python -c "import census.manifest as m; print('\n'.join(sorted(n for n in vars(m) if not n.startswith('__'))))" > /tmp/names_after.txt -comm -23 /tmp/names_before.txt /tmp/names_after.txt # must print nothing (superset) -OPENBLAS_NUM_THREADS=1 .venv/bin/python -m pytest -q 2>&1 | tail -1 # same passed/skipped counts as baseline -.venv/bin/python -c "import mcp_server" # server module imports -wc -l census/manifest/*.py # every module <= 900 lines; __init__.py has no defs -grep -n "^def \|^class " census/manifest/__init__.py # must print nothing -``` - -Also run each check after the section 5 commit. Then update the docs: -`grep -rn "census/manifest.py\|manifest\.py" docs README.md CLAUDE.md STATUS.md CONTRIBUTING.md`, and change -each path reference to the package or submodule (no other wording changes). Add a three-line CHANGELOG -entry under a new `## Unreleased` heading: "Internal: `census.manifest` split into a package (no behaviour -change; golden snapshot identical)". - -## 7. Status table (fill in; keep it short) - -| Item | Status | Notes | -| :--- | :--- | :--- | -| Baseline suite line | PASS | 1634 passed, 37 skipped, 4 warnings, 48 subtests passed in 292.07s (0:04:52). Initial sandbox-only socket/MCP failures were rerun outside that restriction before code changed. | -| Golden snapshot count | PASS | 478 snapshots from 11 manifests; baseline namespace 122 names. | -| Unlisted definitions placed | None | All 68 original functions and 31 constants were listed; 23 imported compatibility names re-exported from schema. | -| Packaging change | PASS; least-change exception (0.7) | pyproject packages now ["census", "census.manifest"]. Updated only the identical expected list in tests/test_packaging.py; its old hard-coded assertion conflicted with required packaging. | -| Test patch targets changed | Three listed failures; one omitted target (0.7) | r15 latest_rpp_year -> adjustments; r12 replicates_digest -> run; r14 default_rpp_areas -> adjustments. Full suite also exposed tests/test_manifest_multistate.py::_run_analysis (omitted from section 4): retarget only its patch to run._run_analysis. Assertions unchanged. | -| Section 5: validate_manifest | PASS | 15 body lines; six private helpers; identical ordered errors and golden snapshots; checkpoint b full suite 1634 passed / 37 skipped. | -| Section 5: methods_section | PASS | 30 body lines; nine paragraph helpers; identical rendered bytes; post-commit checkpoint c full suite 1634 passed / 37 skipped. | -| `run_manifest` stages (for later) | Not split | Plan/extract; harmonize/aggregate/frame; fetch/derive replicates; external join; exclusions/adjustments; fit; save analysed data; corrected hypotheses; bundle. | -| Bugs noticed (not fixed) | Fault-injection coverage caveat | test_analysed_data_failure_does_not_lose_the_run passes without verifying its facade mock was called; after the move that mock is ineffective. Left unchanged per section 4. No production bug fix included. | -| Module line counts | PASS | __init__ 318; schema 256; external 113; adjustments 341; derivations 79; replicates 331; datasource 74; validate 498; create 129; methods_text 480; run 824. Facade has only its unchanged docstring and explicit imports. | -| Final suite line | PASS | 1634 passed, 37 skipped, 4 warnings, 48 subtests passed in 300.36s (0:05:00). Same pass/skip counts as baseline and a/b/c; golden byte-identical, no lost names, MCP imports, caps satisfied. | - -## 8. Deliverables - -1. Commits on `phase23/split-manifest`, in this order: (a) package conversion and verbatim moves, - packaging, test targets; (b) the `validate_manifest` split; (c) the `methods_section` split; (d) docs, - CHANGELOG, status table. Each commit passes section 6. -2. Push the branch. If you cannot open a PR, say so. The owner's agent opens it. -3. PR body (if you open it): what moved where (one table), the section 6 outputs pasted, the status table, - no attribution. - -Notes: Section 0.7 exceptions are the single packaging assertion and the omitted -`tests/test_manifest_multistate.py` spy target. Three listed failing targets were -retargeted; every listed test that passed unchanged was left unchanged. No test -assertions were weakened. Documentation was scanned for path tokens only, without -reviewing study/audit content; only `STATUS.md` needed a source-path replacement. -Actual `test_manifest.py` filenames remain valid and were not renamed. Checkpoints -a/b/c retained all 478 golden snapshots (11 manifests), all 122 baseline attributes -(the facade now has 132), MCP import and the <=900-line module cap. diff --git a/STATUS.md b/STATUS.md index 1aebf6b..33b5ce0 100644 --- a/STATUS.md +++ b/STATUS.md @@ -3,14 +3,14 @@ The single current summary of the Census Research Engine: what it does, which methods are validated, the standing rules, and what is next. Read this instead of old plans. Updated each round when it merges. -**Current release:** v0.21.0 (Round 22), DOI 10.5281/zenodo.24264540. All versions: concept record +**Current release:** v0.22.0 (Round 24: method registry, inequality indices, indirect SMR). All versions: concept record [zenodo.org/records/23925301](https://zenodo.org/records/23925301). Per-round detail: `CHANGELOG.md`. ## Where things are | Path | What | | :--- | :--- | -| `census/` | Library. `manifest/` package (pre-registered studies: `schema`, `validate`, `create`, `run`, `methods_text`, plus `adjustments`, `replicates`, `external`, `derivations`, `datasource`; import from `census.manifest`, but patch tests at the submodule that looks the name up), `models.py` (`fit_replicate`, `fit_eiv`, robust GLM SEs), `health_sources.py` (PLACES, USALEEP, CHR&R, WONDER exports), `spatial*.py`, `sae*.py`, `simulation*.py` (validation studies) | +| `census/` | Library. `manifest/` package (pre-registered studies: `schema`, `validate`, `create`, `run`, `methods_text`, `methods` (single method registry), plus `adjustments`, `replicates`, `external`, `derivations`, `datasource`; import from `census.manifest`, but patch tests at the submodule that looks the name up), `models.py` (`fit_replicate`, `fit_eiv`, robust GLM SEs), `health_sources.py` (PLACES, USALEEP, CHR&R, WONDER exports), `spatial*.py`, `sae*.py`, `simulation*.py` (validation studies) | | `mcp_server.py` | MCP server (63 tools; profiles in `census/mcp_profiles.py`) | | `docs/agent_protocol.md`, `docs/recipes/` | What agents are told; keep in step with engine rules | | `docs/audits/` | Validation reports (pass/fail, reproducible scripts in `scripts/`) | @@ -18,10 +18,14 @@ standing rules, and what is next. Read this instead of old plans. Updated each r | `examples/manifests/` | Registered manifests (their bytes are fixed once registered) | | `PLAN_ROUND_.md` | Only the current round's plan. Older plans are archived (below) | -**Archived plans.** `PLAN_ROUND_1`–`22` were removed from the tree after Round 22. Code and docs still cite -them as labels (for example "PLAN_ROUND_16 A2"). To read one: `git show v0.21.0:PLAN_ROUND_16.md`, or +**Archived plans.** `PLAN_ROUND_1`–`22` are at tag `v0.21.0`, `PLAN_ROUND_23` at `v0.21.1`, `PLAN_ROUND_24` at `v0.22.0`. Code and docs still cite +them as labels (for example "PLAN_ROUND_16 A2"). To read one: `git show v0.21.0:PLAN_ROUND_16.md` (or the later tag), or `https://github.com/cafeTechne/Census-Research-Engine/blob/v0.21.0/PLAN_ROUND_16.md`. +**Round 23 (v0.21.1, internal):** `census/manifest.py` became the `census/manifest/` package with no +behaviour change (478 golden snapshots identical; `scripts/refactor_golden.py`). `run_manifest` stages for a +later split are listed in `PLAN_ROUND_23.md` section 7 (at `v0.21.1`). + ## Methods: validation status Validated means a fixed, pre-set simulation or reference check passed. Failed methods stay exploratory only; @@ -34,12 +38,20 @@ manifests refuse them in confirmatory analyses. | `fit_eiv` (errors-in-variables) and its cluster jackknife | Validated (0.87–0.93) | `eiv_validation.md`, `eiv_cluster_validation.md` | | Binomial/Poisson GLM with `scale: "robust"` (HC3) | Validated (0.879–0.897); required for external count outcomes | `robust_se_validation.md` | | Binomial/Poisson with `scale: "X2"` (quasi-likelihood) | Failed under between-area overdispersion (0.689–0.79); refused for new manifests | `robust_se_validation.md` | +| Area SII / log RII / concentration (`inequality`, Round 24 implementation) | Passed fixed 90% conservative-interval validation: 0.876–0.905 for rank CV<=15%; high-error attenuation persists (CV30% information only). Independent areas. | `inequality_validation.md` (criterion `60330e3`) | +| Indirect SMR (`smr`, Round 24 implementation) | Passed combined Poisson/ACS log-interval validation: 0.893–0.909; rates fixed, independent band errors; negative band replicates refused. Exact Garwood limits condition on E. | `smr_validation.md` (criterion `680707e`) | | Conley spatial HAC SEs (`spatial_se`) | **Failed** (0.71–0.87 at 150 km); exploratory only | `spatial_hac_validation.md` | | Spatial error model λ correction (ME-SEM) | Validated for λ > 0 (0.86–0.93); lag model at γ 0.6 under-covers | `spatial_regression_validation.md` | | LISA under sampling error | Characterised, information only | `lisa_validation.md` | | Tract 2010→2020 apportionment: 1:1 and merged | Validated | `rate_apportionment_validation.md` | | Split tracts: default `missing`; opt-in `split: "dominant"` (≥0.95 share) | Dominant passed held-out OH/PA (ratio 0.913); general split failed | `rate_apportionment_retest.md` | +**Round 24 (v0.22.0):** a new manifest method is one `MethodSpec` entry in `census/manifest/methods.py` +(golden identical for the refactor; 516 snapshots now). New methods `inequality` (SII, log RII, +concentration index; ridits least to most deprived, positive = worse in more deprived areas) and `smr` +(combined interval drives tests; fixed-E Garwood limits reported only). Both assume independent areas. +Exploratory example: `docs/studies/r24_lbw_inequality_example.md`. Suite: 1703 passed, 37 skipped. + ## Standing rules - **Circularity (enforced, R1/R2):** determinant studies use the ACS's own measure (disability: B18101). @@ -61,6 +73,7 @@ manifests refuse them in confirmatory analyses. | 21 | PLACES disability vs ACS | PLACES inflates the education gradient 1.4× | | 21b | County premature mortality | Poverty, disability supported; vehicle, cost burden not | | 22 | County low birth weight | Poverty supported (holds with robust SEs); cost burden fragile (fails with robust SEs) | +| 24 | LBW inequality by poverty (exploratory) | RII 1.25 (90% CI 1.00–1.57); all intervals include the null | ## Open backlog @@ -68,9 +81,7 @@ manifests refuse them in confirmatory analyses. Candidates: spatial random effects or a spatial block bootstrap, tested on the Round 22 design. 2. Measurement-error correction for GLM covariates (none yet; `regression_calibration` is approximate). 3. Replication of the county findings in held-out states (OH/PA). -4. Split `run_manifest` into stages (Round 23 recorded them in `PLAN_ROUND_23.md` section 7). -5. A per-method registry so a new analysis method is defined in one place (today: validation args, - `_run_analysis`, methods text, recipes, agent protocol). +4. Split `run_manifest` into stages (listed in `PLAN_ROUND_23.md` section 7, at tag `v0.21.1`). ## Plan format (keeps agent context small) diff --git a/census/__init__.py b/census/__init__.py index 3f8daca..6d9c568 100644 --- a/census/__init__.py +++ b/census/__init__.py @@ -1,7 +1,7 @@ # SPDX-License-Identifier: AGPL-3.0-or-later """Headless Census ACS data client and processing library.""" -__version__ = "0.21.1" +__version__ = "0.22.0" from census.api import ( DATASETS, @@ -199,7 +199,11 @@ from census.web_maps import web_choropleth from census.api import PRODUCTS, dataset_for_product, dataset_product +from census.inequality import ridits, slope_index, concentration_index, inequality_replicate +from census.standardize import indirect_standardize + __all__ = [ + "ridits", "slope_index", "concentration_index", "inequality_replicate", "indirect_standardize", # api "CensusAPIClient", "fetch_census_data", diff --git a/census/health_sources.py b/census/health_sources.py index 2dd7dae..d7ee59d 100644 --- a/census/health_sources.py +++ b/census/health_sources.py @@ -40,7 +40,7 @@ "wonder": ("file", "sha256")} # where an external variable may appear: only as the outcome of these methods (its error is independent # of the ACS sampling error, so it enters the residual; as a covariate it would go uncorrected) -EXTERNAL_OUTCOME_METHODS = ("fit_eiv", "fit_replicate", "fh_regression") +from .manifest.methods import EXTERNAL_OUTCOME_METHODS JOIN_MISSING_LIMIT = 0.05 KINDS = ("direct", "model_based", "partly_model_based") diff --git a/census/inequality.py b/census/inequality.py new file mode 100644 index 0000000..d2cff1e --- /dev/null +++ b/census/inequality.py @@ -0,0 +1,196 @@ +# SPDX-License-Identifier: AGPL-3.0-or-later +"""Population-weighted area inequality, not individual-level inequality. + +Areas have positive weights. Required ``higher_is_more_deprived`` declares +whether increasing x means greater deprivation. Weighted ridits run from least +(0) to most (1) deprived; ties share their block's weighted mid-rank. Positive +SII, log RII and C mean higher outcomes among more deprived areas. C therefore +has the reverse sign of the usual income-ranked concentration convention. +HC3 treats areas as independent; rank-error attenuation is not corrected. +""" +import numpy as np +import pandas as pd + +from .results import ModelResult, precheck_replicates + +TERMS = ("sii", "log_rii", "concentration_index") +DIRECTION_NOTE = ("Positive indices mean higher outcomes among more deprived areas; concentration " + "has the reverse sign of the usual income-ranked convention.") +RANK_ERROR_NOTE = "Median rank CV exceeds 15%; estimates may be attenuated by rank error." +# Set only from the pre-specified validation report, never from the example study. +INEQUALITY_VALIDATED = True # docs/audits/inequality_validation.md (60330e3 criterion) + + +def _vectors(y, x, w, hi): + if not isinstance(hi, (bool, np.bool_)): + raise ValueError("higher_is_more_deprived must be a boolean") + x, w = np.asarray(x, float), np.asarray(w, float) + if x.ndim != 1 or x.shape != w.shape or not x.size: + raise ValueError("rank and weights must be non-empty vectors of equal length") + if not np.isfinite(x).all() or not np.isfinite(w).all() or (w <= 0).any(): + raise ValueError("rank and weights must be finite; weights must be positive") + if y is not None: + y = np.asarray(y, float) + if y.shape != x.shape or not np.isfinite(y).all(): + raise ValueError("outcome must be a finite vector matching rank and weights") + return y, x, w + + +def _ridits_batch(x, w, hi): + """Rows are refits; sorting and tied-block mid-mass are vectorized.""" + order = np.argsort(x if hi else -x, axis=1, kind="stable") + sx, sw = np.take_along_axis(x, order, 1), np.take_along_axis(w, order, 1) + cum = np.cumsum(sw, axis=1) + starts = np.concatenate((np.ones((len(x), 1), bool), sx[:, 1:] != sx[:, :-1]), axis=1) + ends = np.concatenate((sx[:, 1:] != sx[:, :-1], np.ones((len(x), 1), bool)), axis=1) + below = np.maximum.accumulate(np.where(starts, cum - sw, 0), axis=1) + above = np.minimum.accumulate(np.where(ends, cum, np.inf)[:, ::-1], axis=1)[:, ::-1] + sorted_r = (below + above) / (2 * cum[:, -1:]) + r = np.empty_like(sorted_r) + np.put_along_axis(r, order, sorted_r, 1) + return r + + +def ridits(x, w, higher_is_more_deprived): + """Weighted mid-ranks from least to most deprived; tied blocks share a ridit.""" + _, x, w = _vectors(None, x, w, higher_is_more_deprived) + return _ridits_batch(x[None], w[None], higher_is_more_deprived)[0] + + +def _indices(y, x, w, hi, model_se=False): + """Batch WLS identities; HC3 uses the whitened WLS leverage and sandwich. + + Centering the design permits scalar bread/meat operations identical to + statsmodels WLS(cov_type='HC3'), without 80 Python model fits per draw. + """ + r = _ridits_batch(x, w, hi) + W = w.sum(axis=1) + rm, ym = (w*r).sum(axis=1)/W, (w*y).sum(axis=1)/W + dr = r-rm[:, None] + srr = (w*dr**2).sum(axis=1) + if (srr <= 0).any(): + raise ValueError("inequality needs at least two distinct deprivation ranks") + b = (w*dr*(y-ym[:, None])).sum(axis=1)/srr + a = ym-b*rm + valid = (a > 0) & (a+b > 0) + with np.errstate(divide="ignore", invalid="ignore"): + log_rii = np.where(valid, np.log((a+b)/a), np.nan) + k = np.where(ym != 0, 2*srr/W/ym, np.nan) + c = k*b + estimates = np.column_stack((b, log_rii, c)) + if not model_se: + return estimates + if y.shape[1] < 3: + raise ValueError("HC3 inequality requires at least three areas") + h = w*(1/W[:, None] + dr**2/srr[:, None]) + if (h >= 1-1e-12).any(): + raise ValueError("HC3 is undefined for unit leverage") + q2 = (w*(y-a[:, None]-b[:, None]*r)/(1-h))**2 + vb = (q2*dr**2).sum(axis=1)/srr**2 + vm = q2.sum(axis=1)/W**2 + cmb = (q2*dr).sum(axis=1)/(W*srr) + with np.errstate(divide="ignore", invalid="ignore"): + ga = 1/(a+b)-1/a + gb = 1/(a+b)-ga*rm + vl = ga**2*vm+gb**2*vb+2*ga*gb*cmb + se = np.column_stack((np.sqrt(vb), np.where(valid, np.sqrt(np.maximum(0, vl)), np.nan), + np.abs(k)*np.sqrt(vb))) + return estimates, se, a, b + + +def slope_index(y, x, w, higher_is_more_deprived): + """SII and log RII from weighted ridits, with HC3 and delta-method SEs. + + Returns estimates, SEs, intercept/slope and notes. log RII is undefined + when either fitted endpoint is nonpositive; no clipping is applied. + """ + y, x, w = _vectors(y, x, w, higher_is_more_deprived) + e, se, a, b = _indices(y[None], x[None], w[None], higher_is_more_deprived, True) + notes = [DIRECTION_NOTE] + if not np.isfinite(e[0, 1]): + notes.append("log RII undefined: a fitted endpoint is nonpositive.") + return dict(sii=float(e[0, 0]), se_sii=float(se[0, 0]), log_rii=float(e[0, 1]), + se_log_rii=float(se[0, 1]), intercept=float(a[0]), slope=float(b[0]), notes=notes) + + +def concentration_index(y, x, w, higher_is_more_deprived): + """C=2 cov_w(y,r)/mean_w(y), with approximate convenient-regression HC3 SE.""" + y, x, w = _vectors(y, x, w, higher_is_more_deprived) + e, se, _, _ = _indices(y[None], x[None], w[None], higher_is_more_deprived, True) + notes = [DIRECTION_NOTE, "Concentration SE is the convenient-regression HC3 approximation."] + if not np.isfinite(e[0, 2]): + notes.append("Concentration undefined: weighted outcome mean is zero.") + return dict(concentration_index=float(e[0, 2]), se_concentration_index=float(se[0, 2]), notes=notes) + + +def inequality_replicate(frame, outcome, rank_by, weights, higher_is_more_deprived, + replicates, fixed=(), level=0.90, target="conservative"): + """Three indices, refitting all 80 ACS replicates with fresh weighted ridits. + + Frame/long replicate keys are GEOID (+Year), as in fit_replicate. Missing + stochastic variables/rows and invalid refits are refused, never held fixed + silently. ``fixed`` explicitly declares external quantities held constant. + Target population uses HC3, these_areas replicate SE, conservative their + quadrature sum. No replicate source means model-only (reported in notes). + """ + from .models import _replicate_matrices, _table + from .results import stochastic_variables, IncompleteReplicatesError + if target not in ("population", "these_areas", "conservative") or not 0 < level < 1: + raise ValueError("invalid target or confidence level") + variables = list(dict.fromkeys((outcome, rank_by, weights))) + if set(fixed)-set(variables): + raise ValueError("fixed must name outcome, rank_by or weights") + y, x, w = _vectors(frame[outcome], frame[rank_by], frame[weights], higher_is_more_deprived) + estimates, model, _, _ = _indices(y[None], x[None], w[None], higher_is_more_deprived, True) + notes = [DIRECTION_NOTE, "HC3 assumes independent areas; no correction for rank-error attenuation."] + mats, held = {}, {} + active = [v for v in variables if v not in fixed] + if replicates is not None and len(replicates): + keys = ["GEOID"] + (["Year"] if "Year" in frame and "Year" in replicates else []) + if frame.duplicated(keys).any() or replicates.duplicated(keys+["variable"]).any(): + raise ValueError("duplicate area/variable replicate keys") + mats, held = _replicate_matrices(frame, replicates, active) + missing = [v for v in stochastic_variables(frame, active) if v not in mats] + precheck_replicates("strict", rows_held_fixed=held, variables_without_replicates=missing, + context="inequality") + sampling = np.full(3, np.nan) + if mats: + batch = [mats.get(v, np.broadcast_to(frame[v].to_numpy(float)[:, None], (len(frame), 80))).T + for v in (outcome, rank_by, weights)] + if not all(np.isfinite(b).all() for b in batch) or (batch[2] <= 0).any(): + raise IncompleteReplicatesError("inequality: invalid replicate values or nonpositive weights") + try: + rep = _indices(*batch, higher_is_more_deprived) + except ValueError as exc: + raise IncompleteReplicatesError(f"inequality: replicate refit failed: {exc}") from exc + # Undefined full-sample log RII/C remain undefined, rather than failing SII. + defined = np.isfinite(estimates[0]) + if not np.isfinite(rep[:, defined]).all(): + raise IncompleteReplicatesError("inequality: an estimable full-sample statistic has undefined refits") + sampling = np.sqrt(.05*np.sum((rep-estimates)**2, axis=0)) + else: + notes.append("No replicates: sampling SE unavailable; conservative intervals use model SE only.") + if target == "these_areas": + raise IncompleteReplicatesError("inequality: these_areas requires replicates") + if not np.isfinite(estimates[0, 1]): + notes.append("log RII undefined: a fitted endpoint is nonpositive.") + if not np.isfinite(estimates[0, 2]): + notes.append("Concentration undefined: weighted outcome mean is zero.") + cv = np.full(len(frame), np.nan) + if rank_by+"_se" in frame: + rank_se = frame[rank_by+"_se"].to_numpy(float) + elif rank_by in mats: + rank_se = np.sqrt(.05*np.sum((mats[rank_by]-x[:, None])**2, axis=1)) + else: + rank_se = np.zeros(len(frame)) # explicitly nonstochastic rank + np.divide(100*rank_se, np.abs(x), out=cv, where=x != 0) + cv[(x == 0) & (rank_se > 0)] = np.inf + median = float(np.nanmedian(cv)) if (~np.isnan(cv)).any() else float("nan") + if median > 15: + notes.append(RANK_ERROR_NOTE) + coefs = _table(pd.Series(estimates[0], index=TERMS), pd.Series(model[0], index=TERMS), + pd.Series(sampling, index=TERMS), level, target) + return ModelResult("inequality", f"{outcome} ~ ridit({rank_by})", len(frame), coefs, + "inequality", ["model"]+(["sampling_replicate"] if mats else []), level, + notes=notes, diagnostics={"interval_target": target, "rank_cv_median": median, + "fixed_variables": list(fixed)}) diff --git a/census/manifest/external.py b/census/manifest/external.py index 9364b8a..dc58f56 100644 --- a/census/manifest/external.py +++ b/census/manifest/external.py @@ -4,6 +4,7 @@ from typing import Any, Dict, List, Mapping from .schema import ( _formula_names, + _normalize_analysis, _variable_tables, ) @@ -26,9 +27,18 @@ def external_circularity(manifest: Mapping[str, Any]) -> List[Dict[str, Any]]: tables = _variable_tables(manifest) out = [] for a in manifest.get("analyses") or []: - if not isinstance(a, Mapping) or "~" not in str(a.get("formula", "")): + if not isinstance(a, Mapping): + continue + if a.get("method") in ("inequality", "smr"): + a = _normalize_analysis(a) + lhs = str(a.get("outcome", a.get("observed", ""))) + bands = a.get("population_bands") + rhs = str(a.get("rank_by", " ".join(v for v in bands if isinstance(v, str)) + if isinstance(bands, list) else "")) + elif "~" in str(a.get("formula", "")): + lhs, rhs = str(a["formula"]).split("~", 1) + else: continue - lhs, rhs = str(a["formula"]).split("~", 1) ys = [y if y in ext else y[:-10] for y in _formula_names(lhs) if y in ext or (y.endswith("_numerator") and y[:-10] in ext)] if not ys: @@ -75,6 +85,9 @@ def circularity_errors(manifest: Mapping[str, Any]) -> List[str]: hyps = [h for h in manifest.get("hypotheses") or [] if isinstance(h, Mapping)] tables = _variable_tables(manifest) for aid, recs in by_analysis.items(): + analysis = next(a for a in manifest["analyses"] if a.get("id") == aid) + if analysis.get("method") in ("inequality", "smr"): + analysis = _normalize_analysis(analysis) tested = [h for h in hyps if h.get("analysis") == aid] for c in recs: if c["has_equivalent"]: @@ -89,23 +102,25 @@ def circularity_errors(manifest: Mapping[str, Any]) -> List[str]: f"and only it may be tested). {CIRCULARITY_RULE}") continue for h in tested: - term = [n for n in _formula_names(str(h.get("term", ""))) if n in tables] or [str(h.get("term"))] + tested_term = analysis.get("rank_by") if analysis.get("method") == "inequality" else h.get("term") + term = [n for n in _formula_names(str(tested_term if tested_term is not None else "")) if n in tables] or [str(tested_term)] if not all(t in c["equivalent"] for t in term): errors.append( f"hypotheses {h.get('id')}: CIRCULARITY R1: analysis {aid} compares the external " f"{c['outcome']} with its ACS equivalent {sorted(c['equivalent'])}; only that " - f"comparison may be tested, not {h.get('term')!r} (a determinant study uses the ACS " + f"comparison may be tested, not {tested_term!r} (a determinant study uses the ACS " "measure as its outcome)") if c["kind"] == "direct": continue for h in tested: - term = [n for n in _formula_names(str(h.get("term", ""))) if n in c["overlap"]] + tested_term = analysis.get("rank_by") if analysis.get("method") == "inequality" else h.get("term") + term = [n for n in _formula_names(str(tested_term if tested_term is not None else "")) if n in c["overlap"]] just = h.get("circularity_justification") if term and not (isinstance(just, str) and len(just.strip()) >= 20): fams = "; ".join(f"{t}: " + ", ".join(f"{lab} {'/'.join(tabs)}" for lab, tabs in c["overlap"][t].items()) for t in term) errors.append( - f"hypotheses {h.get('id')}: CIRCULARITY R2: the tested term {h.get('term')!r} is built from " + f"hypotheses {h.get('id')}: CIRCULARITY R2: the tested term {tested_term!r} is built from " f"inputs of the model that produced {c['outcome']} ({c['source']}, {c['kind']}: {fams}); " "its coefficient partly recovers that model, not the world. Test a direct outcome (the " "ACS measure, vital records), or add a written circularity_justification (20+ " diff --git a/census/manifest/methods.py b/census/manifest/methods.py new file mode 100644 index 0000000..5b977c5 --- /dev/null +++ b/census/manifest/methods.py @@ -0,0 +1,550 @@ +# SPDX-License-Identifier: AGPL-3.0-or-later +"""Single registry of manifest analysis execution, validation and reporting metadata. + +Schema imports this module; schema-dependent helpers therefore import lazily. +""" +from dataclasses import dataclass +from typing import Any, Callable, Dict, List, Mapping, Optional, Tuple +import numpy as np +import pandas as pd + +@dataclass(frozen=True) +class MethodSpec: + name: str + required: Tuple[str, ...] + statistics: Tuple[str, ...] + run: Callable[..., Any] + replicate: bool = False + external_outcome: bool = False + # check(stage, context) preserves the legacy validation error order. + # Stages: count/external, before_spatial, spatial, after_spatial. + check: Optional[Callable[..., Any]] = None + text: Optional[Tuple[str, List[str]]] = None + default_target: str = "population" + +def _run_fit_replicate(a, args, frame, level, replicates, correction, ctx): + from .. import correlation, models, multitest, sae, trends + from .schema import _formula_names + held_external = ctx["held_external"] + method = a["method"] + fixed_vars = list(held_external) + if a.get("replicate_scope") == "covariates_only": # pre-registered: the outcome is held fixed + fixed_vars += [v for v in _formula_names(a["formula"].split("~")[0]) if v not in fixed_vars] + if fixed_vars: + args["fixed_variables"] = fixed_vars + return models.fit_replicate(a["formula"], frame, replicates=replicates, confidence_level=level, **args) + +def _run_fit_eiv(a, args, frame, level, replicates, correction, ctx): + from .. import correlation, models, multitest, sae, trends + from .schema import _formula_names + held_external = ctx["held_external"] + method = a["method"] + if held_external: + args["fixed_variables"] = held_external + return models.fit_eiv(a["formula"], frame, replicates=replicates, confidence_level=level, **args) + +def _run_regression_calibration(a, args, frame, level, replicates, correction, ctx): + from .. import correlation, models, multitest, sae, trends + from .schema import _formula_names + held_external = ctx["held_external"] + method = a["method"] + return getattr(models, method)(a["formula"], frame, confidence_level=level, **args) + +def _run_simex(a, args, frame, level, replicates, correction, ctx): + from .. import correlation, models, multitest, sae, trends + from .schema import _formula_names + held_external = ctx["held_external"] + method = a["method"] + return getattr(models, method)(a["formula"], frame, confidence_level=level, **args) + +def _run_fay_herriot(a, args, frame, level, replicates, correction, ctx): + from .. import correlation, models, multitest, sae, trends + from .schema import _formula_names + held_external = ctx["held_external"] + method = a["method"] + y = a["formula"].split("~")[0].strip() + args.setdefault("se_col", f"{y}_se") + return getattr(sae, method)(a["formula"], frame, confidence_level=level, **args) + +def _run_fh_regression(a, args, frame, level, replicates, correction, ctx): + from .. import correlation, models, multitest, sae, trends + from .schema import _formula_names + held_external = ctx["held_external"] + method = a["method"] + y = a["formula"].split("~")[0].strip() + args.setdefault("se_col", f"{y}_se") + return getattr(sae, method)(a["formula"], frame, confidence_level=level, **args) + +def _run_correlation_matrix(a, args, frame, level, replicates, correction, ctx): + from .. import correlation, models, multitest, sae, trends + from .schema import _formula_names + held_external = ctx["held_external"] + method = a["method"] + return correlation.correlation_matrix(frame, args.pop("variables"), confidence_level=level, + replicates=replicates, correction=correction, **args) + +def _run_linear_trend_test(a, args, frame, level, replicates, correction, ctx): + from .. import correlation, models, multitest, sae, trends + from .schema import _formula_names + held_external = ctx["held_external"] + method = a["method"] + v, g = args.pop("variable"), str(args.pop("geoid")) + sub = frame[frame["GEOID"] == g].sort_values("Year") + return trends.linear_trend_test(sub["Year"].tolist(), sub[v].tolist(), ses=sub[f"{v}_se"].tolist(), + confidence_level=level, **args) + +def _run_batch_z_test(a, args, frame, level, replicates, correction, ctx): + from .. import correlation, models, multitest, sae, trends + from .schema import _formula_names + held_external = ctx["held_external"] + method = a["method"] + v1, v2 = args.pop("variable_1"), args.pop("variable_2") + df = frame[["GEOID"]].copy() + for v, tag in ((v1, "1"), (v2, "2")): + df[f"e{tag}"] = frame[v].to_numpy(float) + df[f"m{tag}"] = frame[f"{v}_se"].to_numpy(float) * 1.645 + out = multitest.batch_z_test(df, "e1", "m1", "e2", "m2", confidence_level=level, + correction=correction, **args) + return out + +def _stat_coefficient(h, res): + c = res.coefficients.set_index("term") + if h["term"] not in c.index: + raise KeyError(f"term {h['term']!r} not in model (terms: {list(c.index)})") + row = c.loc[h["term"]] + from ..models import interval_se_column + se_col = interval_se_column(res) # the SE behind p_value and the interval + return {"estimate": float(row["estimate"]), "se": float(row[se_col]), "se_basis": se_col, + "p_two_sided": float(row["p_value"]), + "ci_lower": float(row["ci_lower"]), "ci_upper": float(row["ci_upper"])} + +def _stat_correlation(h, res): + a, b = h["variables"] + return {"estimate": float(res.matrix.loc[a, b]), "se": None, "p_two_sided": float(res.p_values.loc[a, b]), + "ci_lower": float(res.ci_lower.loc[a, b]), "ci_upper": float(res.ci_upper.loc[a, b])} + +def _stat_trend_slope(h, res): + return {"estimate": res["slope"], "se": res["se_slope"], "p_two_sided": res["p_value"], + "ci_lower": res["ci_lower"], "ci_upper": res["ci_upper"]} + +def _stat_z_difference(h, res): + row = res[res["GEOID"] == str(h["geoid"])] + if row.empty: + raise KeyError(f"GEOID {h['geoid']} not in the test frame") + r = row.iloc[0] + diff = float(r.get("difference", r["e1"] - r["e2"])) + return {"estimate": diff, "se": None, "p_two_sided": float(r["p_value"]), "ci_lower": None, "ci_upper": None} + +def _check_spatial(stage, ctx): + if stage != "spatial": + return + a, manifest, errors, method = (ctx[k] for k in ("a", "manifest", "errors", "method")) + spatial = (a.get("args") or {}).get("spatial_se") + if spatial is not None: + valid_spatial = (isinstance(spatial, Mapping) and set(spatial) == {"cutoff_km"} + and type(spatial["cutoff_km"]) in (int, float) + and np.isfinite(spatial["cutoff_km"]) and spatial["cutoff_km"] > 0) + if not valid_spatial: + errors.append("spatial_se must be an object with exactly cutoff_km (finite number > 0)") + from ..spatial_hac import SPATIAL_HAC_VALIDATED + if not SPATIAL_HAC_VALIDATED and not manifest.get("exploratory", False): + errors.append("spatial_se failed its validation (docs/audits/spatial_hac_validation.md); exploratory only") + if (a.get("args") or {}).get("cluster"): + errors.append("spatial_se and cluster are mutually exclusive") + if method not in ("fit_eiv", "fit_replicate") or (method == "fit_replicate" and + (a.get("args") or {}).get("model", "ols") not in ("binomial", "poisson")): + errors.append("spatial_se is supported for fit_eiv and fit_replicate binomial/poisson") + +def _check_fit_replicate(stage, ctx): + if stage == "hypothesis": + return + from .schema import FORMULA_ID_COLUMNS, LEGACY_X2_MANIFESTS, manifest_hash + from .replicates import census_vre_source + i, a, manifest, names, ext_names, errors = (ctx[k] for k in ("i", "a", "manifest", "names", "ext_names", "errors")) + aargs = a.get("args") or {} + outcome_name = str(a.get("formula", "")).split("~")[0].strip() + if stage == "count": + return (outcome_name if aargs.get("model") == "poisson" + and outcome_name.endswith("_numerator") and outcome_name[:-10] in ext_names else None) + if stage == "external" and aargs.get("target", "population") == "these_areas": + errors.append(f"analyses[{i}] ({a.get('id')}): an external outcome is held fixed in the refits, so " + "target 'these_areas' (replicate SE alone) is refused; use 'population' or " + "'conservative'") + count_outcome = ctx.get("count_outcome") + if stage == "before_spatial": + allowed = {"model", "weights", "exposure", "trials", "cv_threshold", "max_failed_replicates", + "target", "scale", "spatial_se"} + unknown = sorted(set(aargs) - allowed) + if unknown: + errors.append(f"fit_replicate does not take args {unknown}") + glm = aargs.get("model", "ols") + if aargs.get("scale") is not None and (aargs["scale"] not in ("X2", "robust") + or glm not in ("binomial", "poisson")): + errors.append("scale must be 'robust' or 'X2' for binomial or poisson models only") + if aargs.get("scale") == "robust" and aargs.get("spatial_se") is not None: + errors.append("scale 'robust' and spatial_se are alternatives; choose one") + parent = outcome_name if outcome_name in ext_names else count_outcome[:-10] if count_outcome else None + if parent is not None: + if glm not in ("ols", "binomial", "poisson"): + errors.append(f"{glm} not validated for external outcomes (PLAN_ROUND_22 out of scope)") + if glm == "binomial" and aargs.get("trials") != parent + "_denominator": + errors.append(f"external binomial outcome requires trials '{parent}_denominator'; effective trials refused") + if glm == "poisson" and (count_outcome is None or aargs.get("exposure") != parent + "_denominator"): + errors.append(f"external poisson needs {parent}_numerator outcome and exposure '{parent}_denominator'") + if glm in ("binomial", "poisson") and aargs.get("scale") is None and aargs.get("spatial_se") is None: + errors.append("overdispersion would be ignored: set scale 'robust'") + elif (glm in ("binomial", "poisson") and aargs.get("scale") == "X2" + and manifest_hash(manifest) not in LEGACY_X2_MANIFESTS): + errors.append("external count outcome with scale 'X2' refused: Pearson X2 SEs under-cover when " + "areas differ beyond binomial/Poisson chance (validated coverage 0.69-0.79 at " + "nominal 0.90, docs/audits/robust_se_validation.md); set scale 'robust' (HC3)") + for role in ("weights", "exposure", "trials"): + column = aargs.get(role) + valid_count = parent and ((glm == "binomial" and role == "trials") or + (glm == "poisson" and role == "exposure")) and column == parent+"_denominator" + if column is not None and column not in names and column not in FORMULA_ID_COLUMNS and not valid_count: + errors.append(f"fit_replicate {role} {column!r} is not a manifest variable in this role") + _check_spatial(stage, ctx) + +def _check_fit_eiv(stage, ctx): + if stage == "hypothesis": + return + from .schema import FORMULA_ID_COLUMNS, LEGACY_X2_MANIFESTS, manifest_hash + from .replicates import census_vre_source + i, a, manifest, names, ext_names, errors = (ctx[k] for k in ("i", "a", "manifest", "names", "ext_names", "errors")) + aargs = a.get("args") or {} + outcome_name = str(a.get("formula", "")).split("~")[0].strip() + if stage == "after_spatial": + eargs = a.get("args") or {} + unknown = sorted(set(eargs) - {"model", "weights", "cv_threshold", "target", "cluster", "cluster_se", "spatial_se"}) + if unknown: + errors.append(f"analyses[{i}] ({a.get('id')}): fit_eiv does not take args {unknown} (allowed: " + "model, weights, cv_threshold, cluster, cluster_se)") + if "cluster_se" in eargs and (eargs["cluster_se"] not in ("cr1", "jackknife") or "cluster" not in eargs): + errors.append(f"analyses[{i}] ({a.get('id')}): fit_eiv args.cluster_se must be 'cr1' or 'jackknife' " + "and needs args.cluster") + if "target" in eargs: + errors.append(f"analyses[{i}] ({a.get('id')}): fit_eiv takes no 'target': its intervals use the " + "sandwich se_total (a measurement-error correction is a claim about the population " + "relationship, not about these areas)") + if "cluster" in eargs: + cl = eargs["cluster"] + from ..models import CLUSTER_PREFIX_RE + mm = CLUSTER_PREFIX_RE.match(str(cl)) if isinstance(cl, str) else None + if not isinstance(cl, str) or not (cl in names or cl in FORMULA_ID_COLUMNS + or (mm and 1 <= int(mm.group(1)) <= 11)): + errors.append(f"analyses[{i}] ({a.get('id')}): fit_eiv args.cluster must be a manifest variable " + "or GEOID.str[:N] (e.g. GEOID.str[:5] for counties of tracts)") + if str(eargs.get("model", "ols")).lower() not in ("ols", "wls"): + errors.append(f"analyses[{i}] ({a.get('id')}): fit_eiv supports model 'ols' or 'wls'") + if str(eargs.get("model", "ols")).lower() == "wls" and not eargs.get("weights"): + errors.append(f"analyses[{i}] ({a.get('id')}): fit_eiv with model 'wls' needs args.weights") + if not census_vre_source(manifest) and a.get("replicates") != "required": + errors.append(f"analyses[{i}] ({a.get('id')}): fit_eiv needs replicates: declare " + "replicates.source 'census_vre' or \"replicates\": \"required\"") + _check_spatial(stage, ctx) + +def _run_inequality(a, args, frame, level, replicates, correction, ctx): + from ..inequality import inequality_replicate + external = set(ctx["external"]) + fixed = [a["outcome"]] if a["outcome"] in external else [] + if fixed and a["weights"] == a["outcome"]+"_denominator": + fixed.append(a["weights"]) + args.pop("replicate_policy", None) # library always enforces strict completeness + # Manifests support complete_case only; filter before aligning replicates. + used = frame.dropna(subset=[a[k] for k in ("outcome", "rank_by", "weights")]) + result = inequality_replicate(used, a["outcome"], a["rank_by"], a["weights"], + replicates=replicates, fixed=fixed, level=level, **args) + dropped = frame.loc[~frame.index.isin(used.index), "GEOID"].astype(str).tolist() + result.diagnostics["complete_case"] = {"input_rows": len(frame), "used_rows": len(used), + "dropped_geoids": dropped} + if dropped: + result.notes.append(f"Complete-case analysis excluded {len(dropped)} area row(s) with missing outcome, rank or weights.") + return result + + +def _run_smr(a, args, frame, level, replicates, correction, ctx): + from ..standardize import indirect_standardize + from ..models import _replicate_matrices + from ..results import precheck_replicates, stochastic_variables, IncompleteReplicatesError + bands = a["population_bands"] + input_rows = len(frame) + frame = frame.dropna(subset=[a["observed"]]+bands) + mats, held = _replicate_matrices(frame, replicates, bands) if replicates is not None else ({}, {}) + precheck_replicates("strict", rows_held_fixed=held, + variables_without_replicates=[v for v in stochastic_variables(frame, bands) if v not in mats], + context="smr") + reps = (np.stack([mats.get(v, np.broadcast_to(frame[v].to_numpy(float)[:, None], (len(frame), 80))) + for v in bands], axis=1) if mats else None) + idx = pd.Index(frame["GEOID"].astype(str)) + observed = pd.Series(frame[a["observed"]].to_numpy(float), index=idx) + populations = frame[bands].set_axis(idx) + try: + result = indirect_standardize(observed, populations, a["reference_rates"], reps, level) + if len(frame) != input_rows: + note = f"Complete-case analysis excluded {input_rows-len(frame)} area row(s) with missing observed counts or populations." + result["notes"] = result["notes"].map(lambda notes: notes+[note]) + return result + except ValueError as exc: + if reps is not None and "replicate" in str(exc): + raise IncompleteReplicatesError(f"smr: {exc}") from exc + raise + + +def _stat_inequality(h, res): + result = _stat_coefficient({"term": h["statistic"]}, res) + if not all(np.isfinite(result[k]) for k in ("estimate", "se", "p_two_sided", "ci_lower", "ci_upper")): + raise ValueError(f"{h['statistic']} or its uncertainty is undefined; see analysis notes") + return result + + +def _stat_smr(h, res): + row = res[res["GEOID"] == str(h["geoid"])] + if row.empty: + raise KeyError(f"GEOID {h['geoid']} not in the SMR frame") + r = row.iloc[0] + if not np.isfinite(r["log_smr"]) or not np.isfinite(r["se_log_smr"]): + raise ValueError("log SMR undefined at O=0; only the fixed-E exact interval is available") + return {"estimate": float(r["log_smr"]), "se": float(r["se_log_smr"]), "se_basis": "se_log_smr", + "p_two_sided": float(r["p_value"]), "ci_lower": float(np.log(r["smr_lower"])), + "ci_upper": float(np.log(r["smr_upper"]))} + + +def _check_inequality(stage, ctx): + if stage == "spatial": + return _check_spatial(stage, ctx) + if stage == "hypothesis": + h, errors = ctx["h"], ctx["errors"] + if "term" in h: + errors.append(f"hypotheses[{ctx['i']}] ({h.get('analysis')}): inequality names a statistic only, no term") + return + if stage != "before_spatial": + return + from ..inequality import INEQUALITY_VALIDATED + a, errors, names, external = (ctx[k] for k in ("a", "errors", "names", "ext_names")) + prefix = f"analyses[{ctx['i']}] ({a.get('id')}): " + args = a.get("args") or {} + if not INEQUALITY_VALIDATED and not ctx["manifest"].get("exploratory", False): + errors.append(prefix+"inequality failed validation; exploratory only (docs/audits/inequality_validation.md)") + if type(args.get("higher_is_more_deprived")) is not bool: + errors.append(prefix+"higher_is_more_deprived must be an explicit boolean") + if args.get("target", "conservative") not in ("population", "these_areas", "conservative"): + errors.append(prefix+"target must be population, these_areas or conservative") + unknown = sorted(set(args)-{"higher_is_more_deprived", "target"}) + if unknown: + errors.append(prefix+f"inequality does not take args {unknown}") + outcome = a.get("outcome") + for role in ("outcome", "rank_by", "weights"): + v = a.get(role) + allowed_external = (role == "outcome" and v in external) or ( + role == "weights" and outcome in external and v == str(outcome)+"_denominator") + if not isinstance(v, str) or (v not in names and not allowed_external): + errors.append(prefix+f"{role} {v!r} is not a manifest variable in this role") + if outcome in external and args.get("target") == "these_areas": + errors.append(prefix+"external outcome is held fixed; target these_areas omits its uncertainty") + + +def _check_smr(stage, ctx): + if stage == "spatial": + return _check_spatial(stage, ctx) + if stage == "hypothesis": + h, errors = ctx["h"], ctx["errors"] + g = h.get("geoid") + if not isinstance(g, str) or not g or (g != "all" and not (g.isascii() and g.isdigit())): + errors.append(f"hypotheses[{ctx['i']}] ({h.get('analysis')}): smr requires geoid (one GEOID or 'all')") + if "term" in h: + errors.append(f"hypotheses[{ctx['i']}] ({h.get('analysis')}): smr uses geoid, not term") + return + if stage != "before_spatial": + return + from ..standardize import SMR_VALIDATED + a, errors, names, external = (ctx[k] for k in ("a", "errors", "names", "ext_names")) + prefix = f"analyses[{ctx['i']}] ({a.get('id')}): " + args = a.get("args") or {} + if not SMR_VALIDATED and not ctx["manifest"].get("exploratory", False): + errors.append(prefix+"smr failed validation; exploratory only (docs/audits/smr_validation.md)") + if set(args)-{"target"}: + errors.append(prefix+f"smr does not take args {sorted(set(args)-{'target'})}") + if args.get("target", "conservative") != "conservative": + errors.append(prefix+"smr target must be conservative (combined Poisson and expected-count uncertainty)") + observed = a.get("observed") + allowed_count = isinstance(observed, str) and observed.endswith("_numerator") and observed[:-10] in external + if not isinstance(observed, str) or (observed not in names and not allowed_count): + errors.append(prefix+f"observed {observed!r} must be a manifest variable or external _numerator") + bands, rates = a.get("population_bands"), a.get("reference_rates") + if not isinstance(bands, list) or not bands or not all(isinstance(v,str) and v in names for v in bands): + errors.append(prefix+"population_bands must be a non-empty list of manifest variables") + elif len(set(bands)) != len(bands): + errors.append(prefix+"population_bands must not repeat a band") + if (not isinstance(rates, list) or not rates or not isinstance(bands, list) or len(rates) != len(bands) + or not all(type(r) in (int,float) and np.isfinite(r) and r>0 for r in rates)): + errors.append(prefix+"reference_rates must be finite positive numbers, one per population band") + source = a.get("reference_source") + if not isinstance(source, str) or not source.strip(): + errors.append(prefix+"reference_source must be a non-empty citation string") + + +METHODS: Dict[str, MethodSpec] = { + 'fit_replicate': MethodSpec( + 'fit_replicate', ('formula',), ('coefficient',), _run_fit_replicate, + replicate=True, + external_outcome=True, + check=_check_fit_replicate, + text=( + 'Coefficients were estimated with {model} regression (statsmodels) on the full-sample estimates; ' + 'sampling standard errors came from refitting on each of {rep_source}, Var = 4/80 sum (b_r - ' + 'b)^2, combined with the model standard error as se_total = sqrt(se_model^2 + se_sampling^2). ' + 'Reported standard errors, intervals and p-values use the {target_se} ({target} target: ' + '{estimand}), with normal-theory intervals. Without replicates only model standard errors are ' + 'reported.', + ['vre', 'design', 'statsmodels'], + ), + ), + 'fit_eiv': MethodSpec( + 'fit_eiv', ('formula',), ('coefficient',), _run_fit_eiv, + replicate=True, + external_outcome=True, + check=_check_fit_eiv, + text=( + 'Coefficients were estimated by moment-corrected least squares ({model}; Fuller 1987) with ' + 'per-unit sampling-error covariances of every design column and of the outcome, including the ' + 'outcome-covariate error covariance, estimated from {rep_source} (4/80 sum over replicates). ' + 'Standard errors are {se_basis} (se_total, used for normal-theory intervals and p-values); naive ' + 'coefficients are reported alongside. Coefficients are {estimand}.', + ['fuller', 'vre', 'design'], + ), + ), + 'regression_calibration': MethodSpec( + 'regression_calibration', ('formula', 'error_covariates'), ('coefficient',), _run_regression_calibration, + replicate=False, + external_outcome=False, + check=_check_spatial, + text=( + 'Covariates measured with known ACS sampling error ({cov}) were corrected by regression ' + 'calibration with heteroscedastic known error variances; standard errors (se_total, used for ' + 'normal-theory intervals and p-values) by row bootstrap re-estimating the calibration. ' + 'Coefficients are {estimand}.', + ['carroll'], + ), + ), + 'simex': MethodSpec( + 'simex', ('formula', 'error_covariates'), ('coefficient',), _run_simex, + replicate=False, + external_outcome=False, + check=_check_spatial, + text=( + 'Covariates measured with known ACS sampling error ({cov}) were corrected by ' + 'simulation-extrapolation (SIMEX, quadratic extrapolation) with the measurement-error jackknife ' + 'variance (se_total, used for normal-theory intervals and p-values). Coefficients are {estimand}.', + ['simex', 'carroll'], + ), + ), + 'fh_regression': MethodSpec( + 'fh_regression', ('formula',), ('coefficient',), _run_fh_regression, + replicate=False, + external_outcome=True, + check=_check_spatial, + text=( + "The outcome's known sampling variance was modelled with an area-level Fay-Herriot model (GLS " + 'coefficients, REML variance component).', + ['fh'], + ), + ), + 'fay_herriot': MethodSpec( + 'fay_herriot', ('formula',), ('coefficient',), _run_fay_herriot, + replicate=False, + external_outcome=False, + check=_check_spatial, + text=( + 'Small-area estimates are Fay-Herriot EBLUPs (REML) with Prasad-Rao/Datta-Lahiri MSE.', + ['fh'], + ), + ), + 'correlation_matrix': MethodSpec( + 'correlation_matrix', ('variables',), ('correlation',), _run_correlation_matrix, + replicate=True, + external_outcome=False, + check=_check_spatial, + text=( + 'Correlations with Fisher-z intervals{rep}.', + ['fisher', 'vre'], + ), + ), + 'linear_trend_test': MethodSpec( + 'linear_trend_test', ('variable', 'geoid'), ('trend_slope',), _run_linear_trend_test, + replicate=False, + external_outcome=False, + check=_check_spatial, + text=( + 'Linear trends across non-overlapping 5-year periods were tested by GLS with known sampling ' + 'variances.', + ['testing'], + ), + ), + 'batch_z_test': MethodSpec( + 'batch_z_test', ('variable_1', 'variable_2'), ('z_difference',), _run_batch_z_test, + replicate=False, + external_outcome=False, + check=_check_spatial, + text=( + 'Differences were tested with the Census two-sample z test, SE = MOE/1.645.', + ['testing'], + ), + ), + 'inequality': MethodSpec( + "inequality", ("outcome", "rank_by", "weights"), ("sii", "log_rii", "concentration_index"), _run_inequality, + replicate=True, + external_outcome=True, + check=_check_inequality, + default_target="conservative", + text=( + 'Area inequality uses positive population weights and weighted ridits from least (0) to most (1) ' + 'deprived, with tied blocks assigned their weighted mid-rank; {rank_direction}. SII is the WLS ' + 'slope, log RII is log((a+b)/a), and concentration is 2 cov_w(y,r)/mean_w(y). Positive indices ' + 'mean higher outcomes among more deprived areas; concentration has the reverse sign of the usual ' + 'income-ranked convention. Independent-area HC3 SEs (delta method for log RII; ' + 'convenient-regression approximation for concentration) combine with 80 ACS replicate refits, ' + 'recomputing ridits, by quadrature. Target: {target}. External outcomes and their denominator ' + 'weights stay fixed. Median rank CV <=15% is the validated range; above it estimates may be ' + 'attenuated by rank error. These are area associations, not individual inequality or causal ' + 'effects.', + ['pamuk', 'kakwani', 'vre', 'statsmodels'], + ), + ), + 'smr': MethodSpec( + "smr", ("observed", "population_bands", "reference_rates", "reference_source"), ("log_smr",), _run_smr, + replicate=True, + external_outcome=True, + check=_check_smr, + default_target="conservative", + text=( + 'Indirect SMR is observed/expected, expected=sum_b population_b*reference_rate_b; pooling sums ' + 'counts and expected counts first. Fixed reference rates: {reference_source}. Exact Garwood ' + 'limits condition on expected counts. Tests use log SMR against zero, with combined variance 1/O ' + '+ 4/80 sum(log E_r-log E)^2 and exponentiated normal intervals (target: {target}). Without band ' + 'replicates the expected-count component is zero, explicitly noted; at O=0 only the exact fixed-E' + ' interval is defined. External observed counts stay fixed in band refits. Reference-rate ' + 'uncertainty and spatial dependence are not included.', + ['garwood', 'breslow', 'vre'], + ), + ), +} + +STATISTIC_EXTRACTORS = { + 'coefficient': _stat_coefficient, + 'correlation': _stat_correlation, + 'trend_slope': _stat_trend_slope, + 'z_difference': _stat_z_difference, + "sii": _stat_inequality, + "log_rii": _stat_inequality, + "concentration_index": _stat_inequality, + "log_smr": _stat_smr, +} + +ANALYSIS_METHODS = {name: (spec.required, spec.statistics) for name, spec in METHODS.items()} +STATISTICS = tuple(dict.fromkeys(stat for spec in METHODS.values() for stat in spec.statistics)) +# Preserve the historical public tuple order; future entries follow registry order. +REPLICATE_METHODS = tuple(sorted((name for name, spec in METHODS.items() if spec.replicate), + key=lambda name: {"fit_replicate": 0, "correlation_matrix": 1, "fit_eiv": 2}.get(name, 3))) +EXTERNAL_OUTCOME_METHODS = tuple(sorted((name for name, spec in METHODS.items() if spec.external_outcome), + key=lambda name: {"fit_eiv": 0, "fit_replicate": 1, "fh_regression": 2}.get(name, 3))) diff --git a/census/manifest/methods_text.py b/census/manifest/methods_text.py index 84858b7..fba5759 100644 --- a/census/manifest/methods_text.py +++ b/census/manifest/methods_text.py @@ -34,34 +34,14 @@ # Methods text # --------------------------------------------------------------------------- +from .methods import METHODS + +# Retain the legacy introspection order while deriving every value from its spec. +_LEGACY_TEXT_ORDER = ("fit_replicate", "regression_calibration", "fit_eiv", "simex", "fh_regression", + "fay_herriot", "correlation_matrix", "linear_trend_test", "batch_z_test") _METHOD_TEXT = { - "fit_replicate": ("Coefficients were estimated with {model} regression (statsmodels) on the full-sample " - "estimates; sampling standard errors came from refitting on each of {rep_source}, Var = 4/80 sum (b_r - b)^2, combined with the model standard error as " - "se_total = sqrt(se_model^2 + se_sampling^2). Reported standard errors, intervals and p-values use " - "the {target_se} ({target} target: {estimand}), with normal-theory intervals. Without " - "replicates only model standard errors are reported.", ["vre", "design", "statsmodels"]), - "regression_calibration": ("Covariates measured with known ACS sampling error ({cov}) were corrected by " - "regression calibration with heteroscedastic known error variances; standard " - "errors (se_total, used for normal-theory intervals and p-values) by row bootstrap " - "re-estimating the calibration. Coefficients are {estimand}.", ["carroll"]), - "fit_eiv": ("Coefficients were estimated by moment-corrected least squares ({model}; Fuller 1987) with " - "per-unit sampling-error covariances of every design column and of the outcome, including the " - "outcome-covariate error covariance, estimated from {rep_source} (4/80 sum over replicates). " - "Standard errors are {se_basis} (se_total, used for " - "normal-theory intervals and p-values); naive coefficients are reported alongside. Coefficients " - "are {estimand}.", ["fuller", "vre", "design"]), - "simex": ("Covariates measured with known ACS sampling error ({cov}) were corrected by " - "simulation-extrapolation (SIMEX, quadratic extrapolation) with the measurement-error jackknife " - "variance (se_total, used for normal-theory intervals and p-values). Coefficients are " - "{estimand}.", ["simex", "carroll"]), - "fh_regression": ("The outcome's known sampling variance was modelled with an area-level Fay-Herriot model " - "(GLS coefficients, REML variance component).", ["fh"]), - "fay_herriot": ("Small-area estimates are Fay-Herriot EBLUPs (REML) with Prasad-Rao/Datta-Lahiri MSE.", - ["fh"]), - "correlation_matrix": ("Correlations with Fisher-z intervals{rep}.", ["fisher", "vre"]), - "linear_trend_test": ("Linear trends across non-overlapping 5-year periods were tested by GLS with known " - "sampling variances.", ["testing"]), - "batch_z_test": ("Differences were tested with the Census two-sample z test, SE = MOE/1.645.", ["testing"]), + name: METHODS[name].text for name in dict.fromkeys((*_LEGACY_TEXT_ORDER, *METHODS)) + if METHODS[name].text is not None } @@ -320,14 +300,16 @@ def _methods_analyses(m, rep_info, level, rep_short, rep_source, lines, cites): a = _normalize_analysis(a) text, refs = _METHOD_TEXT[a["method"]] args = a.get("args") or {} - tgt = args.get("target", "population") + tgt = args.get("target", METHODS[a["method"]].default_target) # the same interpretation contract as the fitted result (census.results.interval_description) probe = ModelResult(model_type="", formula=str(a.get("formula") or ""), n_obs=0, coefficients=pd.DataFrame(columns=["term"]), method=a["method"], uncertainty_sources=[], confidence_level=level, - diagnostics={"interval_target": tgt} if a["method"] == "fit_replicate" else {}) + diagnostics={"interval_target": tgt} if a["method"] in ("fit_replicate", "inequality") else {}) info = interval_description(probe) - text = text.format(model=args.get("model", "ols").upper(), target=tgt, estimand=info["estimand"], + text = text.format(reference_source=a.get("reference_source", ""), + rank_direction=("higher rank values mean more deprived" if args.get("higher_is_more_deprived") + else "lower rank values mean more deprived"), model=args.get("model", "ols").upper(), target=tgt, estimand=info["estimand"], target_se={"se_model": "model SE (se_model)", "se_sampling": "replicate sampling SE " "(se_sampling)", "se_total": "combined SE (se_total)"}[info["basis_column"]], cov=", ".join(a.get("error_covariates") or args.get("error_covariates") or []), diff --git a/census/manifest/run.py b/census/manifest/run.py index 1543502..efc124d 100644 --- a/census/manifest/run.py +++ b/census/manifest/run.py @@ -136,70 +136,19 @@ def _run_analysis(a: Mapping[str, Any], frame: pd.DataFrame, level: float, repli if args.get("spatial_se") is not None: args["coords"] = coords held_external = [v for v in _formula_names(str(a.get("formula", "")).split("~")[0]) if v in set(external) or (v.endswith("_numerator") and v[:-10] in set(external))] - if method == "fit_replicate": - fixed_vars = list(held_external) - if a.get("replicate_scope") == "covariates_only": # pre-registered: the outcome is held fixed - fixed_vars += [v for v in _formula_names(a["formula"].split("~")[0]) if v not in fixed_vars] - if fixed_vars: - args["fixed_variables"] = fixed_vars - return models.fit_replicate(a["formula"], frame, replicates=replicates, confidence_level=level, **args) - if method == "fit_eiv": - if held_external: - args["fixed_variables"] = held_external - return models.fit_eiv(a["formula"], frame, replicates=replicates, confidence_level=level, **args) - if method in ("regression_calibration", "simex"): - return getattr(models, method)(a["formula"], frame, confidence_level=level, **args) - if method in ("fay_herriot", "fh_regression"): - y = a["formula"].split("~")[0].strip() - args.setdefault("se_col", f"{y}_se") - return getattr(sae, method)(a["formula"], frame, confidence_level=level, **args) - if method == "correlation_matrix": - return correlation.correlation_matrix(frame, args.pop("variables"), confidence_level=level, - replicates=replicates, correction=correction, **args) - if method == "linear_trend_test": - v, g = args.pop("variable"), str(args.pop("geoid")) - sub = frame[frame["GEOID"] == g].sort_values("Year") - return trends.linear_trend_test(sub["Year"].tolist(), sub[v].tolist(), ses=sub[f"{v}_se"].tolist(), - confidence_level=level, **args) - if method == "batch_z_test": - v1, v2 = args.pop("variable_1"), args.pop("variable_2") - df = frame[["GEOID"]].copy() - for v, tag in ((v1, "1"), (v2, "2")): - df[f"e{tag}"] = frame[v].to_numpy(float) - df[f"m{tag}"] = frame[f"{v}_se"].to_numpy(float) * 1.645 - out = multitest.batch_z_test(df, "e1", "m1", "e2", "m2", confidence_level=level, - correction=correction, **args) - return out - raise ValueError(f"unsupported method {method}") + from .methods import METHODS + if method not in METHODS: + raise ValueError(f"unsupported method {method}") + return METHODS[method].run(a, args, frame, level, replicates, correction, + {"held_external": held_external, "coords": coords, "external": external}) def _statistic(h: Mapping[str, Any], res: Any) -> Dict[str, Any]: + from .methods import STATISTIC_EXTRACTORS stat = h["statistic"] - if stat == "coefficient": - c = res.coefficients.set_index("term") - if h["term"] not in c.index: - raise KeyError(f"term {h['term']!r} not in model (terms: {list(c.index)})") - row = c.loc[h["term"]] - from ..models import interval_se_column - se_col = interval_se_column(res) # the SE behind p_value and the interval - return {"estimate": float(row["estimate"]), "se": float(row[se_col]), "se_basis": se_col, - "p_two_sided": float(row["p_value"]), - "ci_lower": float(row["ci_lower"]), "ci_upper": float(row["ci_upper"])} - if stat == "correlation": - a, b = h["variables"] - return {"estimate": float(res.matrix.loc[a, b]), "se": None, "p_two_sided": float(res.p_values.loc[a, b]), - "ci_lower": float(res.ci_lower.loc[a, b]), "ci_upper": float(res.ci_upper.loc[a, b])} - if stat == "trend_slope": - return {"estimate": res["slope"], "se": res["se_slope"], "p_two_sided": res["p_value"], - "ci_lower": res["ci_lower"], "ci_upper": res["ci_upper"]} - if stat == "z_difference": - row = res[res["GEOID"] == str(h["geoid"])] - if row.empty: - raise KeyError(f"GEOID {h['geoid']} not in the test frame") - r = row.iloc[0] - diff = float(r.get("difference", r["e1"] - r["e2"])) - return {"estimate": diff, "se": None, "p_two_sided": float(r["p_value"]), "ci_lower": None, "ci_upper": None} - raise ValueError(stat) + if stat not in STATISTIC_EXTRACTORS: + raise ValueError(stat) + return STATISTIC_EXTRACTORS[stat](h, res) def _tag(result: Any, h: str, analysis_id: str, status: str = STATUS_CONFIRMATORY) -> Any: @@ -656,6 +605,9 @@ def run_manifest( for source_key, target in (("estimate", "odds_ratio"), ("ci_lower", "odds_ratio_lower"), ("ci_upper", "odds_ratio_upper")): row[target] = float(np.exp(row[source_key])) if row.get(source_key) is not None else None + if hyp["statistic"] in ("log_rii", "log_smr"): + for source_key, target in (("estimate", "ratio"), ("ci_lower", "ratio_lower"), ("ci_upper", "ratio_upper")): + row[target] = float(np.exp(row[source_key])) if row.get(source_key) is not None else None rows.append(row) hyp_df = pd.DataFrame(rows) hyp_df["p_adjusted"] = np.nan @@ -799,7 +751,7 @@ def _write_bundle(folder: Path, m, h, results, hyp_df, run_info, citation) -> Di figures_[f"shrinkage_{aid}"] = (figs.shrinkage_plot, {"result": r, "manifest_hash": h, "data_source": fig_source}) cols = ["id", "statistic", "direction", "estimate", "p_value", "p_adjusted", "significant", "status"] - cols += [c for c in ("odds_ratio", "odds_ratio_lower", "odds_ratio_upper") if c in hyp_df] + cols += [c for c in ("odds_ratio", "odds_ratio_lower", "odds_ratio_upper", "ratio", "ratio_lower", "ratio_upper") if c in hyp_df] if m.get("exploratory"): cols.append("analysis_status") show = hyp_df.reindex(columns=cols) diff --git a/census/manifest/schema.py b/census/manifest/schema.py index 10b209f..5961420 100644 --- a/census/manifest/schema.py +++ b/census/manifest/schema.py @@ -34,19 +34,8 @@ DATA_SOURCE_KEYS = ("synthetic", "description", "generator") STATUS_EXPLORATORY = "exploratory" -# method -> (required args, statistics it supports) -ANALYSIS_METHODS: Dict[str, Tuple[Tuple[str, ...], Tuple[str, ...]]] = { - "fit_replicate": (("formula",), ("coefficient",)), - "fit_eiv": (("formula",), ("coefficient",)), - "regression_calibration": (("formula", "error_covariates"), ("coefficient",)), - "simex": (("formula", "error_covariates"), ("coefficient",)), - "fh_regression": (("formula",), ("coefficient",)), - "fay_herriot": (("formula",), ("coefficient",)), - "correlation_matrix": (("variables",), ("correlation",)), - "linear_trend_test": (("variable", "geoid"), ("trend_slope",)), - "batch_z_test": (("variable_1", "variable_2"), ("z_difference",)), -} -STATISTICS = ("coefficient", "correlation", "trend_slope", "z_difference") +from .methods import METHODS, ANALYSIS_METHODS, STATISTICS, REPLICATE_METHODS + EXCLUSION_RULES = ("cv_screen", "geoid_exclude") ADJUSTMENT_TYPES = ("rpp",) RPP_ADJUSTMENT_KEYS = ("type", "variables", "rpp_year", "areas", "rpp_years", "dollar_year", "price_index") @@ -62,6 +51,10 @@ FORMULA_FUNCTIONS = {"C", "I", "np", "log", "exp", "sqrt", "center", "standardize", "scale", "Q"} REFERENCES = { + "pamuk": "Pamuk, E.R. (1985). Social class inequality in mortality from 1921 to 1972 in England and Wales. Population Studies 39:17-31.", + "kakwani": "Kakwani, N., Wagstaff, A. & van Doorslaer, E. (1997). Socioeconomic inequalities in health: measurement, computation, and statistical inference. Journal of Econometrics 77:87-103.", + "garwood": "Garwood, F. (1936). Fiducial limits for the Poisson distribution. Biometrika 28:437-442.", + "breslow": "Breslow, N.E. & Day, N.E. (1987). Statistical Methods in Cancer Research, Vol. II. IARC Scientific Publications 82.", "acs": "U.S. Census Bureau (2020). Understanding and Using American Community Survey Data: " "What All Data Users Need to Know (ACS General Handbook), ch. 7-8.", "vre": "U.S. Census Bureau. Variance Replicate Estimate Tables Documentation. " @@ -216,13 +209,18 @@ def tables_and_refs(manifest: Mapping[str, Any]) -> Dict[str, List[str]]: _SPEC_KEYS = ("formula", "error_covariates", "variables", "variable", "geoid", "variable_1", "variable_2") +def _spec_keys(a): + spec = METHODS.get(a.get("method")) + return tuple(dict.fromkeys(_SPEC_KEYS + (spec.required if spec else ()))) + + def _normalize_analysis(a: Mapping[str, Any]) -> Dict[str, Any]: """Return the analysis with spec keys (formula, variables, ...) lifted from ``args`` to the top level, so either placement works the same way. Duplicates (a key both top-level and in args) are a validation error.""" out = dict(a) args = dict(a.get("args") or {}) - for k in _SPEC_KEYS: + for k in _spec_keys(a): if k in args and k not in out: out[k] = args.pop(k) elif k in args: @@ -233,7 +231,7 @@ def _normalize_analysis(a: Mapping[str, Any]) -> Dict[str, Any]: def _duplicate_spec_keys(a: Mapping[str, Any]) -> List[str]: args = a.get("args") or {} - return [k for k in _SPEC_KEYS if k in a and k in args] + return [k for k in _spec_keys(a) if k in a and k in args] def _formula_names(formula: str) -> List[str]: @@ -253,4 +251,3 @@ def _formula_names(formula: str) -> List[str]: # Manifest methods that use replicates; each runs with replicate_policy="strict". # (Spatial, multilevel, DiD and PUMS analyses are not manifest methods; if added # they must join this set so confirmatory runs refuse incomplete replicates.) -REPLICATE_METHODS = ("fit_replicate", "correlation_matrix", "fit_eiv") diff --git a/census/manifest/validate.py b/census/manifest/validate.py index c01d471..c7376d4 100644 --- a/census/manifest/validate.py +++ b/census/manifest/validate.py @@ -5,6 +5,7 @@ from datetime import datetime from typing import Any, Dict, List, Mapping import numpy as np +from .methods import METHODS from .adjustments import ( _check_adjustments, ) @@ -315,8 +316,7 @@ def _check_analysis(i, a, manifest, names, ext_names, ids, folded_ids, used.append(a[key]) aargs = a.get("args") or {} outcome_name = str(a.get("formula", "")).split("~")[0].strip() - count_outcome = (outcome_name if method == "fit_replicate" and aargs.get("model") == "poisson" - and outcome_name.endswith("_numerator") and outcome_name[:-10] in ext_names else None) + count_outcome = METHODS[method].check("count", locals()) if count_outcome and count_outcome in _formula_names(str(a.get("formula", "")).split("~", 1)[1]): errors.append("external count columns are allowed as the Poisson outcome only") for u in used: @@ -333,86 +333,12 @@ def _check_analysis(i, a, manifest, names, ext_names, ids, folded_ids, elif [u for u in ext_used if u not in outcome] or len(outcome) != 1: errors.append(f"analyses[{i}] ({a.get('id')}): external variables {ext_used} may only be the outcome " "(alone): as covariates their model-based error would go uncorrected") - elif method == "fit_replicate" and (a.get("args") or {}).get("target", "population") == "these_areas": - errors.append(f"analyses[{i}] ({a.get('id')}): an external outcome is held fixed in the refits, so " - "target 'these_areas' (replicate SE alone) is refused; use 'population' or " - "'conservative'") - if method == "fit_replicate": - allowed = {"model", "weights", "exposure", "trials", "cv_threshold", "max_failed_replicates", - "target", "scale", "spatial_se"} - unknown = sorted(set(aargs) - allowed) - if unknown: - errors.append(f"fit_replicate does not take args {unknown}") - glm = aargs.get("model", "ols") - if aargs.get("scale") is not None and (aargs["scale"] not in ("X2", "robust") - or glm not in ("binomial", "poisson")): - errors.append("scale must be 'robust' or 'X2' for binomial or poisson models only") - if aargs.get("scale") == "robust" and aargs.get("spatial_se") is not None: - errors.append("scale 'robust' and spatial_se are alternatives; choose one") - parent = outcome_name if outcome_name in ext_names else count_outcome[:-10] if count_outcome else None - if parent is not None: - if glm not in ("ols", "binomial", "poisson"): - errors.append(f"{glm} not validated for external outcomes (PLAN_ROUND_22 out of scope)") - if glm == "binomial" and aargs.get("trials") != parent + "_denominator": - errors.append(f"external binomial outcome requires trials '{parent}_denominator'; effective trials refused") - if glm == "poisson" and (count_outcome is None or aargs.get("exposure") != parent + "_denominator"): - errors.append(f"external poisson needs {parent}_numerator outcome and exposure '{parent}_denominator'") - if glm in ("binomial", "poisson") and aargs.get("scale") is None and aargs.get("spatial_se") is None: - errors.append("overdispersion would be ignored: set scale 'robust'") - elif (glm in ("binomial", "poisson") and aargs.get("scale") == "X2" - and manifest_hash(manifest) not in LEGACY_X2_MANIFESTS): - errors.append("external count outcome with scale 'X2' refused: Pearson X2 SEs under-cover when " - "areas differ beyond binomial/Poisson chance (validated coverage 0.69-0.79 at " - "nominal 0.90, docs/audits/robust_se_validation.md); set scale 'robust' (HC3)") - for role in ("weights", "exposure", "trials"): - column = aargs.get(role) - valid_count = parent and ((glm == "binomial" and role == "trials") or - (glm == "poisson" and role == "exposure")) and column == parent+"_denominator" - if column is not None and column not in names and column not in FORMULA_ID_COLUMNS and not valid_count: - errors.append(f"fit_replicate {role} {column!r} is not a manifest variable in this role") + else: + METHODS[method].check("external", locals()) + METHODS[method].check("before_spatial", locals()) spatial = (a.get("args") or {}).get("spatial_se") - if spatial is not None: - valid_spatial = (isinstance(spatial, Mapping) and set(spatial) == {"cutoff_km"} - and type(spatial["cutoff_km"]) in (int, float) - and np.isfinite(spatial["cutoff_km"]) and spatial["cutoff_km"] > 0) - if not valid_spatial: - errors.append("spatial_se must be an object with exactly cutoff_km (finite number > 0)") - from ..spatial_hac import SPATIAL_HAC_VALIDATED - if not SPATIAL_HAC_VALIDATED and not manifest.get("exploratory", False): - errors.append("spatial_se failed its validation (docs/audits/spatial_hac_validation.md); exploratory only") - if (a.get("args") or {}).get("cluster"): - errors.append("spatial_se and cluster are mutually exclusive") - if method not in ("fit_eiv", "fit_replicate") or (method == "fit_replicate" and - (a.get("args") or {}).get("model", "ols") not in ("binomial", "poisson")): - errors.append("spatial_se is supported for fit_eiv and fit_replicate binomial/poisson") - if method == "fit_eiv": - eargs = a.get("args") or {} - unknown = sorted(set(eargs) - {"model", "weights", "cv_threshold", "target", "cluster", "cluster_se", "spatial_se"}) - if unknown: - errors.append(f"analyses[{i}] ({a.get('id')}): fit_eiv does not take args {unknown} (allowed: " - "model, weights, cv_threshold, cluster, cluster_se)") - if "cluster_se" in eargs and (eargs["cluster_se"] not in ("cr1", "jackknife") or "cluster" not in eargs): - errors.append(f"analyses[{i}] ({a.get('id')}): fit_eiv args.cluster_se must be 'cr1' or 'jackknife' " - "and needs args.cluster") - if "target" in eargs: - errors.append(f"analyses[{i}] ({a.get('id')}): fit_eiv takes no 'target': its intervals use the " - "sandwich se_total (a measurement-error correction is a claim about the population " - "relationship, not about these areas)") - if "cluster" in eargs: - cl = eargs["cluster"] - from ..models import CLUSTER_PREFIX_RE - mm = CLUSTER_PREFIX_RE.match(str(cl)) if isinstance(cl, str) else None - if not isinstance(cl, str) or not (cl in names or cl in FORMULA_ID_COLUMNS - or (mm and 1 <= int(mm.group(1)) <= 11)): - errors.append(f"analyses[{i}] ({a.get('id')}): fit_eiv args.cluster must be a manifest variable " - "or GEOID.str[:N] (e.g. GEOID.str[:5] for counties of tracts)") - if str(eargs.get("model", "ols")).lower() not in ("ols", "wls"): - errors.append(f"analyses[{i}] ({a.get('id')}): fit_eiv supports model 'ols' or 'wls'") - if str(eargs.get("model", "ols")).lower() == "wls" and not eargs.get("weights"): - errors.append(f"analyses[{i}] ({a.get('id')}): fit_eiv with model 'wls' needs args.weights") - if not census_vre_source(manifest) and a.get("replicates") != "required": - errors.append(f"analyses[{i}] ({a.get('id')}): fit_eiv needs replicates: declare " - "replicates.source 'census_vre' or \"replicates\": \"required\"") + METHODS[method].check("spatial", locals()) + METHODS[method].check("after_spatial", locals()) scope = a.get("replicate_scope", "all") if scope not in REPLICATE_SCOPES: errors.append(f"analyses[{i}] ({a.get('id')}): replicate_scope must be one of {REPLICATE_SCOPES}") @@ -475,6 +401,8 @@ def _check_manifest_hypotheses(manifest, ids, errors): for k in need: if k not in h: errors.append(f"hypotheses[{i}]: {k!r} is required for statistic {h.get('statistic')!r}") + METHODS[method].check("hypothesis", {**locals(), "a": next( + _normalize_analysis(a) for a in manifest["analyses"] if a.get("id") == h.get("analysis"))}) just = h.get("circularity_justification") if just is not None and not isinstance(just, str): errors.append(f"hypotheses[{i}]: circularity_justification must be text") diff --git a/census/mcp_profiles.py b/census/mcp_profiles.py index 0e9c15e..eba85b9 100644 --- a/census/mcp_profiles.py +++ b/census/mcp_profiles.py @@ -78,7 +78,7 @@ "get_recipe": ( "Return numbered steps for a standard analysis. Call this first when unsure what to do.\n" "name: compare_counties | tract_trend | regression_sampling_error | small_area | preregistered | " - "decennial_counts; " + "decennial_counts | external_health_outcome | health_inequality; " "omit to list recipes.\n" "Example: get_recipe(name=\"compare_counties\")" ), @@ -200,6 +200,22 @@ 6. On failure read error_code and fix; do not guess numbers. 7. Health outcomes: use the ACS's own measure (disability: B18101); use CDC PLACES only for measures the ACS lacks, such as diabetes (get_recipe first). +8. Area inequality and standardized ratios: get_recipe('health_inequality'), then + existing manifest tools. Inequality statistics sii/log_rii/concentration_index + have no term; declare higher_is_more_deprived and positive weights. Ridits run + least to most deprived; positive indices mean higher outcomes among more deprived + areas, reversing the usual income-ranked concentration sign. Default conservative + target combines HC3 and ACS replicates; median rank CV <=15% is validated, above + which warn about attenuation. External outcome/denominator weights stay fixed; + R2 tests rank_by. HC3 assumes independent areas. SMR requires observed integer + counts, distinct population_bands, positive reference_rates in events per person + and a hashed reference_source citation; log_smr hypotheses use geoid or all. + Only conservative combined Poisson/replicate log intervals drive SMR tests; + fixed-E Garwood intervals omit expected uncertainty. O=0 has only exact limits; + absent replicates set expected uncertainty to zero, noted. Rates are fixed. + SMR refuses negative band replicates under its pre-specified validation domain. + Log statistics test zero and report exponentiated ratio columns. R1 applies + to external outcomes/counts; reused MI/NY LBW associations remain exploratory. """ REVIEW_INSTRUCTIONS = """\ diff --git a/census/pipeline.py b/census/pipeline.py index 1e28bd8..31bd2b7 100644 --- a/census/pipeline.py +++ b/census/pipeline.py @@ -103,7 +103,7 @@ from census.stats import add_reliability_columns, clean_census_sentinels, parse_variable_code from census.tables import load_variable_metadata -PACKAGE_VERSION = "0.21.1" +PACKAGE_VERSION = "0.22.0" CONCORDANCE_MODES = ("fail", "strict", "code") MAX_ISSUES_IN_RESULT = 100 diff --git a/census/recipes.py b/census/recipes.py index 9523ba0..c1e139a 100644 --- a/census/recipes.py +++ b/census/recipes.py @@ -9,6 +9,7 @@ RECIPE_DIR = os.path.join(ROOT, "docs", "recipes") RECIPES: Dict[str, str] = { + "health_inequality": "Area health inequality indices and indirectly standardized ratios with ACS uncertainty", "compare_counties": "Compare a rate between two counties in one release", "tract_trend": "Trend in a tract or county rate using non-overlapping releases", "regression_sampling_error": "Regression whose SEs include ACS sampling error", @@ -21,7 +22,7 @@ RecipeName = Literal[ "compare_counties", "tract_trend", "regression_sampling_error", "small_area", "preregistered", - "decennial_counts", "external_health_outcome", + "decennial_counts", "external_health_outcome", "health_inequality", ] @@ -30,7 +31,7 @@ def get_recipe(name: Optional[RecipeName] = None) -> Dict[str, Any]: Args: name: compare_counties, tract_trend, regression_sampling_error, - small_area, preregistered, decennial_counts, or external_health_outcome. + small_area, preregistered, decennial_counts, external_health_outcome, or health_inequality. Example: get_recipe(name="compare_counties") """ diff --git a/census/standardize.py b/census/standardize.py index 8accd07..1a39b94 100644 --- a/census/standardize.py +++ b/census/standardize.py @@ -166,3 +166,74 @@ def age_adjusted_interval(cases: Sequence[float], populations: Sequence[float], vu, yu = v + wm**2, y + wm upper = vu / (2 * yu) * float(chi2.ppf(1 - alpha / 2, 2 * yu**2 / vu)) return {"rate": y, "lower": lower, "upper": upper, "confidence_level": confidence_level} + + +# Set only from the pre-specified simulation report. +SMR_VALIDATED = True # docs/audits/smr_validation.md (680707e criterion) + + +def indirect_standardize(observed, band_populations, reference_rates, band_replicates=None, level=0.90): + """Indirect SMR and fixed-expected Garwood / combined log-Wald intervals. + + ``observed`` is a GEOID-indexed Series of nonnegative integer counts; + ``band_populations`` a DataFrame with exactly the same ordered index and + columns in reference-rate order (areas x bands). No implicit joins/drops. + Replicates are an array (areas, bands, 80) in that order. Positive finite + expected counts are required in every refit. This validated implementation + requires nonnegative band replicates; negative ACS band perturbations are + refused even when their total expected count would remain positive. The returned final GEOID + ``all`` pools counts and expected counts BEFORE computing uncertainty. + Rates are fixed; their uncertainty and between-area dependence are not + covered. At O=0 only the exact fixed-E interval is defined. + """ + import pandas as pd + from scipy.stats import chi2, norm + if not isinstance(observed, pd.Series) or not isinstance(band_populations, pd.DataFrame): + raise ValueError("observed must be a GEOID-indexed Series; band_populations a DataFrame") + if (not observed.index.equals(band_populations.index) or observed.index.has_duplicates + or not len(observed) or observed.index.isna().any() + or "all" in observed.index.astype(str)): + raise ValueError("area indexes must match exactly, be unique/nonmissing, and exclude reserved GEOID 'all'") + if observed.index.astype(str).has_duplicates: + raise ValueError("GEOIDs collide after string conversion") + if not 0 < level < 1: + raise ValueError("level must be between zero and one") + o, pop, rates = observed.to_numpy(float), band_populations.to_numpy(float), np.asarray(reference_rates, float) + if rates.ndim != 1 or rates.size != pop.shape[1] or not rates.size or not np.isfinite(rates).all() or (rates <= 0).any(): + raise ValueError("reference_rates must be finite positive numbers, one per population band") + if not np.isfinite(o).all() or (o < 0).any() or (o != np.floor(o)).any(): + raise ValueError("observed counts must be finite nonnegative integers") + if not np.isfinite(pop).all() or (pop < 0).any(): + raise ValueError("band populations must be finite and nonnegative") + e = pop@rates + if (e <= 0).any(): + raise ValueError("expected counts must be positive") + notes = ["Reference rates fixed; exact Garwood interval conditions on expected counts."] + o, e = np.r_[o, o.sum()], np.r_[e, e.sum()] + ve = np.zeros(len(o)) + if band_replicates is not None: + rep = np.asarray(band_replicates, float) + if rep.shape != (*pop.shape, 80) or not np.isfinite(rep).all() or (rep < 0).any(): + raise ValueError("band_replicates must be finite nonnegative (areas, bands, 80)") + er = np.einsum("abr,b->ar", rep, rates) + er = np.vstack((er, er.sum(axis=0))) + if (er <= 0).any(): + raise ValueError("replicate expected counts must be positive") + ve = .05*np.sum((np.log(er)-np.log(e[:, None]))**2, axis=1) + else: + notes.append("No band replicates: expected-count uncertainty set to zero.") + alpha = 1-level + lower = np.zeros(len(o)); positive = o > 0 + lower[positive] = chi2.ppf(alpha/2, 2*o[positive])/(2*e[positive]) + upper = chi2.ppf(1-alpha/2, 2*o+2)/(2*e) + smr = o/e + log, se, lo, hi, p = (np.full(len(o), np.nan) for _ in range(5)) + log[positive] = np.log(smr[positive]) + se[positive] = np.sqrt(1/o[positive]+ve[positive]) + z = norm.ppf(1-alpha/2) + lo[positive], hi[positive] = np.exp(log[positive]-z*se[positive]), np.exp(log[positive]+z*se[positive]) + p[positive] = 2*norm.sf(np.abs(log[positive]/se[positive])) + rownotes = [notes+([] if pos else ["O=0: combined log interval and Wald test undefined."]) for pos in positive] + return pd.DataFrame(dict(GEOID=list(observed.index.astype(str))+["all"], observed=o, expected=e, + smr=smr, log_smr=log, se_log_smr=se, smr_lower=lo, smr_upper=hi, + smr_exact_lower=lower, smr_exact_upper=upper, p_value=p, notes=rownotes)) diff --git a/docs/agent_protocol.md b/docs/agent_protocol.md index f4aa488..a783840 100644 --- a/docs/agent_protocol.md +++ b/docs/agent_protocol.md @@ -237,3 +237,30 @@ TIGER representative points. The pre-set Round 22 simulation **failed** its coverage criterion (`docs/audits/spatial_hac_validation.md`): `spatial_se` is available only in exploratory manifests, direct fits and `run_variant`, never confirmatory manifests. `cluster` and `spatial_se` are mutually exclusive. + +## Inequality and standardized ratios + +Use manifest `inequality` for area-level SII, log RII and concentration. Declare +`outcome`, `rank_by`, positive `weights`, and the boolean +`args.higher_is_more_deprived`; hypotheses name only `sii`, `log_rii` or +`concentration_index` (no term). Ridits run least to most deprived: positive +indices mean higher outcomes among more deprived areas. Concentration therefore +has the reverse sign of the usual income-ranked convention. External outcomes +and their `_denominator` weights stay fixed in refits. Default target is +`conservative` (HC3 plus ACS replicate uncertainty); median rank CV <=15% is the +validated range. Above it, warn about attenuation from rank error. Area HC3 +assumes independent areas. Circularity R2 tests the ranking variable. + +Manifest `smr` requires integer `observed` counts, distinct `population_bands`, +finite positive `reference_rates` in events per person, and a non-empty +`reference_source` citation. Rates and source are hashed. Hypotheses use +`log_smr`, with one `geoid` or `"all"` for pooled counts/expected counts. Log +statistics test zero; ratio columns exponentiate estimates and limits. SMR tests +use the combined Poisson/replicate log interval (`target: conservative` only), +while Garwood limits condition on expected counts. Zero observed counts have +only an exact fixed-E interval; rate uncertainty is excluded. No band replicates +means zero expected-count uncertainty, explicitly noted. Negative ACS band +replicates are refused under the pre-specified validation domain, even when +total expected counts would be positive. See recipe +`health_inequality` and the two fixed simulation audits. Reused MI/NY LBW +associations are exploratory. diff --git a/docs/audits/inequality_validation.json b/docs/audits/inequality_validation.json new file mode 100644 index 0000000..3e43823 --- /dev/null +++ b/docs/audits/inequality_validation.json @@ -0,0 +1,224 @@ +{ + "criterion": "Conservative nominal 90% coverage >= 0.86 for all three statistics in all four cells with rank CV <= 0.15. CV 0.30 is informational.", + "passed": true, + "cells": [ + { + "n": 145, + "rank_cv": 0.05, + "draws": 1000, + "seed": 20261116, + "invalid_draws": 0, + "statistics": { + "sii": { + "truth": 0.03, + "conservative_coverage": 0.905, + "population_coverage": 0.905, + "bias": -7.50038372105688e-05, + "empirical_sd": 0.004494378146362438, + "mean_total_se_over_sd": 1.0261672950554321, + "mean_model_se_over_sd": 1.0188994078597906 + }, + "log_rii": { + "truth": 0.3566749439387324, + "conservative_coverage": 0.9, + "population_coverage": 0.898, + "bias": -0.000720659874120233, + "empirical_sd": 0.05420389752253155, + "mean_total_se_over_sd": 1.024465078766145, + "mean_model_se_over_sd": 1.0170831547032084 + }, + "concentration_index": { + "truth": 0.05881109100544023, + "conservative_coverage": 0.903, + "population_coverage": 0.899, + "bias": -0.0001581508290767336, + "empirical_sd": 0.008744204555380693, + "mean_total_se_over_sd": 1.033987762749107, + "mean_model_se_over_sd": 1.0266659073638067 + } + } + }, + { + "n": 145, + "rank_cv": 0.15, + "draws": 1000, + "seed": 20261117, + "invalid_draws": 0, + "statistics": { + "sii": { + "truth": 0.03, + "conservative_coverage": 0.887, + "population_coverage": 0.851, + "bias": -0.0016816672687974994, + "empirical_sd": 0.004783563621796163, + "mean_total_se_over_sd": 1.0718423445757357, + "mean_model_se_over_sd": 0.988442834219536 + }, + "log_rii": { + "truth": 0.3566749439387324, + "conservative_coverage": 0.888, + "population_coverage": 0.858, + "bias": -0.019990987898913348, + "empirical_sd": 0.05716417498987063, + "mean_total_se_over_sd": 1.0778106778788292, + "mean_model_se_over_sd": 0.9927838803057799 + }, + "concentration_index": { + "truth": 0.058796193374334504, + "conservative_coverage": 0.891, + "population_coverage": 0.862, + "bias": -0.0032763632438349635, + "empirical_sd": 0.009247980139218473, + "mean_total_se_over_sd": 1.0871960567375862, + "mean_model_se_over_sd": 1.0026360801966303 + } + } + }, + { + "n": 145, + "rank_cv": 0.3, + "draws": 1000, + "seed": 20261118, + "invalid_draws": 0, + "statistics": { + "sii": { + "truth": 0.03, + "conservative_coverage": 0.823, + "population_coverage": 0.673, + "bias": -0.0061687772488054834, + "empirical_sd": 0.005101943552726339, + "mean_total_se_over_sd": 1.3240520504773057, + "mean_model_se_over_sd": 1.006663812997526 + }, + "log_rii": { + "truth": 0.3566749439387324, + "conservative_coverage": 0.821, + "population_coverage": 0.672, + "bias": -0.07432138425813428, + "empirical_sd": 0.06105404774007655, + "mean_total_se_over_sd": 1.3301580587862418, + "mean_model_se_over_sd": 1.0143563255222237 + }, + "concentration_index": { + "truth": 0.05878017738420337, + "conservative_coverage": 0.824, + "population_coverage": 0.672, + "bias": -0.012108223091622565, + "empirical_sd": 0.009962394070136655, + "mean_total_se_over_sd": 1.3278286570040922, + "mean_model_se_over_sd": 1.0095998749421176 + } + } + }, + { + "n": 1000, + "rank_cv": 0.05, + "draws": 1000, + "seed": 20261119, + "invalid_draws": 0, + "statistics": { + "sii": { + "truth": 0.03, + "conservative_coverage": 0.902, + "population_coverage": 0.898, + "bias": -0.00032107246945090387, + "empirical_sd": 0.0029831303503507637, + "mean_total_se_over_sd": 1.0005601188829774, + "mean_model_se_over_sd": 0.9933781912509618 + }, + "log_rii": { + "truth": 0.3566749439387324, + "conservative_coverage": 0.9, + "population_coverage": 0.898, + "bias": -0.0036958702024200707, + "empirical_sd": 0.03626691950642228, + "mean_total_se_over_sd": 0.9952431640991851, + "mean_model_se_over_sd": 0.9880404562078594 + }, + "concentration_index": { + "truth": 0.05882278284752297, + "conservative_coverage": 0.902, + "population_coverage": 0.898, + "bias": -0.0006155306762707854, + "empirical_sd": 0.0058577195594591265, + "mean_total_se_over_sd": 0.9993151226503405, + "mean_model_se_over_sd": 0.9921414519503555 + } + } + }, + { + "n": 1000, + "rank_cv": 0.15, + "draws": 1000, + "seed": 20261120, + "invalid_draws": 0, + "statistics": { + "sii": { + "truth": 0.03, + "conservative_coverage": 0.876, + "population_coverage": 0.848, + "bias": -0.0016854225603098703, + "empirical_sd": 0.003166153999902975, + "mean_total_se_over_sd": 1.069383499077787, + "mean_model_se_over_sd": 0.9785080692989536 + }, + "log_rii": { + "truth": 0.3566749439387324, + "conservative_coverage": 0.876, + "population_coverage": 0.843, + "bias": -0.020357488834119253, + "empirical_sd": 0.038110535661435546, + "mean_total_se_over_sd": 1.0729427044328128, + "mean_model_se_over_sd": 0.981471140490761 + }, + "concentration_index": { + "truth": 0.058822898770418255, + "conservative_coverage": 0.877, + "population_coverage": 0.848, + "bias": -0.0033125946981267113, + "empirical_sd": 0.006173349596271994, + "mean_total_se_over_sd": 1.0753442494014946, + "mean_model_se_over_sd": 0.9839711841159384 + } + } + }, + { + "n": 1000, + "rank_cv": 0.3, + "draws": 1000, + "seed": 20261121, + "invalid_draws": 0, + "statistics": { + "sii": { + "truth": 0.03, + "conservative_coverage": 0.644, + "population_coverage": 0.425, + "bias": -0.005861341451890566, + "empirical_sd": 0.003186406963665026, + "mean_total_se_over_sd": 1.364479142478532, + "mean_model_se_over_sd": 1.0125173984044469 + }, + "log_rii": { + "truth": 0.3566749439387324, + "conservative_coverage": 0.638, + "population_coverage": 0.421, + "bias": -0.07058433336612135, + "empirical_sd": 0.03842576885170834, + "mean_total_se_over_sd": 1.3602293853685432, + "mean_model_se_over_sd": 1.0110109062294084 + }, + "concentration_index": { + "truth": 0.05882123359629973, + "conservative_coverage": 0.641, + "population_coverage": 0.421, + "bias": -0.01148101733493627, + "empirical_sd": 0.0062720637032942835, + "mean_total_se_over_sd": 1.3593587246682728, + "mean_model_se_over_sd": 1.0087305294364852 + } + } + } + ], + "criterion_commit": "60330e3", + "script_sha256": "58f63eca0529d75f0ccf55cb6058ba4bdb0e8e345d20b23491032ba392194b37" +} diff --git a/docs/audits/inequality_validation.md b/docs/audits/inequality_validation.md new file mode 100644 index 0000000..bde07fc --- /dev/null +++ b/docs/audits/inequality_validation.md @@ -0,0 +1,34 @@ +# Area-inequality validation + +Conservative nominal 90% coverage >= 0.86 for all three statistics in all four cells with rank CV <= 0.15. CV 0.30 is informational. + +Criterion commit: `60330e3`. + +Result: **PASS**. + +Invalid/undefined draws count as non-coverage. No seeds, designs or criteria were changed. + +| n | Rank CV | Statistic | Conservative coverage | Population coverage | Bias | Mean total SE / SD | Mean model SE / SD | +|---:|---:|:---|---:|---:|---:|---:|---:| +| 145 | 5% | sii | 0.905 | 0.905 | -7.50038e-05 | 1.026 | 1.019 | +| 145 | 5% | log_rii | 0.900 | 0.898 | -0.00072066 | 1.024 | 1.017 | +| 145 | 5% | concentration_index | 0.903 | 0.899 | -0.000158151 | 1.034 | 1.027 | +| 145 | 15% | sii | 0.887 | 0.851 | -0.00168167 | 1.072 | 0.988 | +| 145 | 15% | log_rii | 0.888 | 0.858 | -0.019991 | 1.078 | 0.993 | +| 145 | 15% | concentration_index | 0.891 | 0.862 | -0.00327636 | 1.087 | 1.003 | +| 145 | 30% | sii | 0.823 | 0.673 | -0.00616878 | 1.324 | 1.007 | +| 145 | 30% | log_rii | 0.821 | 0.672 | -0.0743214 | 1.330 | 1.014 | +| 145 | 30% | concentration_index | 0.824 | 0.672 | -0.0121082 | 1.328 | 1.010 | +| 1000 | 5% | sii | 0.902 | 0.898 | -0.000321072 | 1.001 | 0.993 | +| 1000 | 5% | log_rii | 0.900 | 0.898 | -0.00369587 | 0.995 | 0.988 | +| 1000 | 5% | concentration_index | 0.902 | 0.898 | -0.000615531 | 0.999 | 0.992 | +| 1000 | 15% | sii | 0.876 | 0.848 | -0.00168542 | 1.069 | 0.979 | +| 1000 | 15% | log_rii | 0.876 | 0.843 | -0.0203575 | 1.073 | 0.981 | +| 1000 | 15% | concentration_index | 0.877 | 0.848 | -0.00331259 | 1.075 | 0.984 | +| 1000 | 30% | sii | 0.644 | 0.425 | -0.00586134 | 1.364 | 1.013 | +| 1000 | 30% | log_rii | 0.638 | 0.421 | -0.0705843 | 1.360 | 1.011 | +| 1000 | 30% | concentration_index | 0.641 | 0.421 | -0.011481 | 1.359 | 1.009 | + +Design: 1,000 draws per cell, fixed area weights and true ranks, independent areas; all invalid-draw counts are in the JSON. +Validated range, if passed: median rank CV <=15%. CV 30% is informational; HC3/conservative intervals do not correct rank-error attenuation. +Positive indices mean higher outcomes among more deprived areas. Concentration has the reverse sign of the usual income-ranked convention. diff --git a/docs/audits/inequality_validation_criterion.md b/docs/audits/inequality_validation_criterion.md new file mode 100644 index 0000000..d75d7c8 --- /dev/null +++ b/docs/audits/inequality_validation_criterion.md @@ -0,0 +1,22 @@ +# Inequality validation: criterion fixed before running + +Acceptance: conservative nominal 90% coverage >=0.86 for SII, log RII and +concentration in all four n=145/1000 x rank CV=0.05/0.15 cells. CV=0.30 is +informational. Exactly 1,000 draws per cell, seeds 20261116 + cell index, +ordered (145,.05), (145,.15), (145,.30), (1000,.05), (1000,.15), (1000,.30). + +The full design is PLAN_ROUND_24 section 5 and scripts/inequality_validation.py. +Weights and true x are drawn once per cell; every draw redraws u, binomial +outcomes and rank errors, then the 80 make_replicates values. Truths are .03, +log(.10/.07) and the weighted concentration of .07+.03*true_ridit. + +Operational choices fixed here: binomial denominators are w/50 rounded to the +nearest integer (NumPy rint), minimum one; regression weights remain the original +continuous w. Probabilities outside [0,1], undefined refits and invalid draws +count as non-coverage, without clipping, redrawing or reducing the denominator. +Report population coverage, bias and mean SE / empirical SD for all cells. +The rank CV diagnostic uses 100*SE/abs(observed rank), not nominal design CV. + +Smoke runs may use at most 20 draws per cell and do not write audit files. +Full runs require the exact script and criterion in a named, pushed commit. +No seed, design, threshold or criterion may change after the full run. diff --git a/docs/audits/smr_validation.json b/docs/audits/smr_validation.json new file mode 100644 index 0000000..6ded346 --- /dev/null +++ b/docs/audits/smr_validation.json @@ -0,0 +1,68 @@ +{ + "criterion": "Combined nominal 90% interval coverage >= 0.86 in all six cells. O=0 and invalid draws count as non-coverage.", + "passed": true, + "cells": [ + { + "expected": 10, + "band_cv": 0.05, + "draws": 2000, + "seed": 20261117, + "combined_coverage": 0.9095, + "exact_coverage": 0.9255, + "zero_counts": 0, + "invalid_draws": 0 + }, + { + "expected": 10, + "band_cv": 0.2, + "draws": 2000, + "seed": 20261118, + "combined_coverage": 0.9035, + "exact_coverage": 0.881, + "zero_counts": 0, + "invalid_draws": 0 + }, + { + "expected": 50, + "band_cv": 0.05, + "draws": 2000, + "seed": 20261119, + "combined_coverage": 0.893, + "exact_coverage": 0.8905, + "zero_counts": 0, + "invalid_draws": 0 + }, + { + "expected": 50, + "band_cv": 0.2, + "draws": 2000, + "seed": 20261120, + "combined_coverage": 0.906, + "exact_coverage": 0.735, + "zero_counts": 0, + "invalid_draws": 1 + }, + { + "expected": 500, + "band_cv": 0.05, + "draws": 2000, + "seed": 20261121, + "combined_coverage": 0.897, + "exact_coverage": 0.7765, + "zero_counts": 0, + "invalid_draws": 0 + }, + { + "expected": 500, + "band_cv": 0.2, + "draws": 2000, + "seed": 20261122, + "combined_coverage": 0.905, + "exact_coverage": 0.335, + "zero_counts": 0, + "invalid_draws": 0 + } + ], + "criterion_commit": "680707e", + "script_sha256": "3e5d91ba54b48e14c34cbe527290df5b9a44582a06c1776d62a9dee8b1f8efa6" +} diff --git a/docs/audits/smr_validation.md b/docs/audits/smr_validation.md new file mode 100644 index 0000000..d78b709 --- /dev/null +++ b/docs/audits/smr_validation.md @@ -0,0 +1,22 @@ +# Indirect SMR validation + +Combined nominal 90% interval coverage >= 0.86 in all six cells. O=0 and invalid draws count as non-coverage. + +Criterion commit: `680707e`. + +Result: **PASS**. + +| Expected | Band CV | Combined coverage | Fixed-E exact coverage | O=0 | Invalid | +|---:|---:|---:|---:|---:|---:| +| 10 | 5% | 0.909 | 0.925 | 0 | 0 | +| 10 | 20% | 0.903 | 0.881 | 0 | 0 | +| 50 | 5% | 0.893 | 0.890 | 0 | 0 | +| 50 | 20% | 0.906 | 0.735 | 0 | 1 | +| 500 | 5% | 0.897 | 0.776 | 0 | 0 | +| 500 | 20% | 0.905 | 0.335 | 0 | 0 | + +2,000 draws per cell; true SMR 1.2, five equally populated bands scaled to the target expected count. +Gaussian population errors are not clipped/redrawn. Nonpositive/invalid populations or expected counts count as non-coverage. +Reference-rate uncertainty, correlated band errors and spatial dependence were not simulated; rates are treated as fixed. + +Implementation domain: negative ACS band replicate values are refused, even when their expected total would be positive. They were classified as invalid in the pre-specified run; extending that domain requires separate validation. Final-library replay reproduced every committed cell exactly. diff --git a/docs/audits/smr_validation_criterion.md b/docs/audits/smr_validation_criterion.md new file mode 100644 index 0000000..00879bd --- /dev/null +++ b/docs/audits/smr_validation_criterion.md @@ -0,0 +1,18 @@ +# Indirect SMR validation: criterion fixed before running + +Acceptance: combined nominal 90% coverage of true SMR 1.2 >=0.86 in all six +E=10/50/500 x band CV=.05/.20 cells. Exactly 2,000 draws per cell, seed +20261117 + index, ordered (10,.05), (10,.20), (50,.05), (50,.20), (500,.05), +(500,.20). O=0 counts as non-coverage. Also report fixed-E Garwood coverage. + +Five reference rates (.0005,.001,.003,.01,.04), with equal band populations +scaled by E/sum(rates). Each draw: Poisson(1.2 E) observed count, independent +Gaussian relative band-population errors, and 80 make_replicates values with +SE=CV*true_populations. Reference rates are fixed and uncertainty-free. +Nonpositive/invalid populations or expected counts count as non-coverage, +without clipping, redrawing or changing the denominator. No spatial dependence +or between-band correlation is simulated. + +Smoke runs may use at most 20 draws per cell, without writing audits. Full +runs require the exact script and criterion in a named, pushed commit. No seed, +design, threshold or criterion may change after the full run. diff --git a/docs/recipes/external_health_outcome.md b/docs/recipes/external_health_outcome.md index 865c103..47a3867 100644 --- a/docs/recipes/external_health_outcome.md +++ b/docs/recipes/external_health_outcome.md @@ -12,7 +12,7 @@ Goal: a pre-registered study whose outcome comes from outside the ACS, without c - USALEEP: `{"source": "usaleep", "release": 2018, "measure": "E0"}`. It is on 2010 tracts: use ACS 2011-2019 releases, or add `"apportion_to": 2020` (population-weighted, an approximation). Fully predicted tracts (flag 2) are dropped unless `"include_predicted": true`. - County Health Rankings: `{"source": "chr", "release": 2025, "measure": "v127"}` at county level. Non-commercial research use; cite CHR&R. - WONDER: export the county table yourself from wonder.cdc.gov (the engine never automates the form), then `{"source": "wonder", "measure": "age_adjusted_rate", "file": "path/to/export.txt", "sha256": ""}`. Tick the age-adjusted rate standard error. Suppressed counts stay censored (1-9), never zero. -3. Use the external variable only as the outcome of `fit_eiv` (held fixed; ACS covariates corrected for sampling error), `fit_replicate` or `fh_regression`. For tract PLACES outcomes, cluster by county: `"args": {"cluster": "GEOID.str[:5]"}` (PLACES shares county random effects). +3. Use the external variable only as the outcome of `fit_eiv` (held fixed; ACS covariates corrected for sampling error), `fit_replicate` or `fh_regression`; for area inequality or indirect ratios, follow the `health_inequality` recipe (`inequality` / `smr`). For tract PLACES outcomes, cluster by county: `"args": {"cluster": "GEOID.str[:5]"}` (PLACES shares county random effects). 4. `create_manifest`: errors **R1** (same construct as an ACS measure) and **R2** (tested term from the model's inputs) block. Read the notes: they list overlapping controls and the rule. Commit the manifest alone, then `run_manifest`. 5. In the report, state the source's model (methods.md does), its uncertainty type, the join counts in `run.json` (`external`), and that within-county PLACES variation is composition only. diff --git a/docs/recipes/health_inequality.md b/docs/recipes/health_inequality.md new file mode 100644 index 0000000..9e33f79 --- /dev/null +++ b/docs/recipes/health_inequality.md @@ -0,0 +1,61 @@ +# Area health inequality and indirectly standardized ratios + +1. Choose an area-level question. Register a manifest before inspecting associations; + use `exploratory: true` when reusing an outcome association already examined. + These indices describe area associations, not individual inequality or causal effects. +2. For `inequality`, declare `outcome`, `rank_by`, positive population `weights`, and + `args.higher_is_more_deprived` explicitly. Rank areas from least to most deprived; + tied values share their weighted mid-ridit. External outcomes are allowed, with + `weights: "_denominator"` for their published denominator (for example live births). + + ```json + {"id":"ineq","method":"inequality","outcome":"lbw","rank_by":"pov_rate", + "weights":"lbw_denominator","args":{"higher_is_more_deprived":true,"target":"conservative"}} + ``` + +3. Hypotheses name `sii`, `log_rii`, or `concentration_index`, without a `term`. + SII is the WLS slope on ridit; log RII is log((intercept+slope)/intercept), + undefined if either fitted endpoint is nonpositive. Positive indices mean higher + outcomes among more deprived areas. **Concentration has the reverse sign of the + usual income-ranked convention.** Outcome proportions give SII in proportion units; + multiply SII and both limits by 100 for percentage points. Exponentiate log RII + and its limits for the relative index (`ratio` columns do this). +4. Declare ACS VRE replicates for noisy ranks, weights or outcomes. Every refit + recomputes weighted ridits; declared external outcomes and external denominators + stay fixed. Missing stochastic-variable/area replicates are refused. Population + intervals use HC3; `these_areas` uses sampling SE only; default `conservative` + combines both by quadrature. External outcomes cannot use `these_areas`. + Median rank CV <=15% is the validated range. Above it, estimates may be + attenuated by rank error; wider intervals do not correct that bias. HC3 assumes + independent areas and does not cover spatial residual dependence. +5. For `smr`, use observed **integer counts**, `population_bands` (distinct manifest + variables), positive `reference_rates` in **events per person**, in the same band + order, and a non-empty `reference_source` citation. Convert rates per 100,000 + to per person before registration. Source and rates are hashed with the manifest. + + ```json + {"id":"smr","method":"smr","observed":"deaths_numerator", + "population_bands":["pop_young","pop_old"],"reference_rates":[0.001,0.02], + "reference_source":"Named standard, year, age bands; annual deaths per person", + "args":{"target":"conservative"}} + ``` + +6. Test `statistic: "log_smr"` with `geoid` equal to one GEOID or `"all"` (pool + counts and expected counts first). The null is log SMR=0; ratio columns show + SMR and its combined limits. Exact Garwood limits condition on fixed expected + counts. Tests use Poisson variance plus ACS expected-count replicate variance; + only `target: "conservative"` is supported. Without replicates, expected-count + uncertainty is zero and explicitly noted. At zero observed counts, the combined + test is undefined; report the exact fixed-E interval only. Reference-rate + uncertainty is not included. Negative ACS band replicates are refused under + the pre-specified validation domain, even if their expected total is positive. + No age-specific real death-count example is bundled; + WONDER exports remain manual. +7. Circularity R1 applies to external outcomes/counts. R2 treats `rank_by` as the + tested term for inequality; model-input overlap requires the existing written + justification. Read the methods, diagnostics and join counts, and report + unadjusted interval confidence separately from multiplicity-adjusted tests. + +Reproducible validation: `docs/audits/inequality_validation.md` and +`docs/audits/smr_validation.md`. Exploratory MI/NY example: +`examples/manifests/r24_mi_ny_lbw_inequality.json`. diff --git a/docs/studies/r24_lbw_inequality_example.md b/docs/studies/r24_lbw_inequality_example.md new file mode 100644 index 0000000..279fd75 --- /dev/null +++ b/docs/studies/r24_lbw_inequality_example.md @@ -0,0 +1,43 @@ +# MI/NY county low birth weight inequality (exploratory) + +Author self-review, not independent. This is exploratory because Round 22 already +examined poverty and low birth weight; it is not a new confirmatory test. +Manifest: `examples/manifests/r24_mi_ny_lbw_inequality.json`, SHA-256 +`8d1008320c941ab468bf6fcff5319a9d7f43ffc050eec05807d7b0b16b25ad15`. + +CHR&R 2025 v037 low birth weight proportions (NCHS natality 2017–2023) were +ranked by ACS 2018–2022 county poverty percentage (B17001). Weights are live +births, not county counts. The run used existing Census/CHR/VRE caches with +network requests blocked. Of 145 MI/NY counties extracted, 144 had complete +outcome/rank/weight data; one missing CHR outcome/denominator was excluded +(GEOID `26083`), recorded in the analysis diagnostics. +Missing values were not treated as zero. + +| Index | Estimate | Unadjusted 90% conservative interval | +|:---|---:|:---| +| SII (percentage points) | 1.892 | [-0.103, 3.887] | +| log RII | 0.2244 | [-0.0037, 0.4526] | +| RII (exponentiated) | 1.2516 | [0.9963, 1.5723] | +| Concentration index | 0.0371 | [-0.0020, 0.0763] | + +Ridits run from least to most deprived, sharing the weighted mid-rank for ties. +Positive indices mean higher LBW among more deprived areas. **Concentration has +the reverse sign of the usual income-ranked convention.** SII is a fitted +ridit-endpoint difference, not an observed richest/poorest county contrast. +Median poverty-rank CV was **6.63%**, within the <=15% range of the fixed +simulation validation. HC3 model SEs and 80 ACS poverty replicate refits were +combined by quadrature; birth outcomes and birth denominators stayed fixed. +All three intervals include their null (0 for indices, 1 for RII). These +unadjusted intervals are not simultaneous intervals; the manifest separately +applies Holm adjustment to its three hypothesis p-values. + +This is a joint MI/NY birth-weighted area gradient without state adjustment or +individual-level interpretation. It differs from the state-adjusted Round 22 +regressions. Independent-area HC3 does not account for spatial dependence; +rank error can attenuate gradients, and a conservative interval does not +correct that bias. Reference-rate uncertainty is irrelevant to this example: +no SMR was fitted. No age-specific death counts were fabricated. + +Reproduction: run the manifest with the existing caches and +`CENSUS_HEALTH_OFFLINE=1`; outputs are release artifacts, not committed datasets. +Outputs: `r24_study_outputs.tar.gz`, attached (after a key scan) to the release that ships this round. diff --git a/examples/manifests/r24_mi_ny_lbw_inequality.json b/examples/manifests/r24_mi_ny_lbw_inequality.json new file mode 100644 index 0000000..8907c19 --- /dev/null +++ b/examples/manifests/r24_mi_ny_lbw_inequality.json @@ -0,0 +1,101 @@ +{ + "schema_version": 1, + "releases": [ + 2022 + ], + "geography": { + "level": "county", + "states": [ + "26", + "36" + ] + }, + "missing": "complete_case", + "confidence_level": 0.9, + "correction": { + "family": "all", + "method": "holm" + }, + "external": [ + { + "measure": "v037", + "name": "lbw", + "release": 2025, + "source": "chr" + } + ], + "replicates": { + "source": "census_vre" + }, + "title": "Exploratory MI/NY county low birth weight inequality by poverty rank", + "authors": [ + "human:cafeTechne" + ], + "created": "2026-11-15T00:00:00Z", + "exploratory": true, + "variables": [ + { + "name": "pov_total", + "ref": "2022:B17001_001" + }, + { + "name": "pov_below", + "ref": "2022:B17001_002" + } + ], + "derivations": [ + { + "as_percent": true, + "den": "pov_total", + "name": "pov_rate", + "num": "pov_below", + "op": "proportion" + } + ], + "exclusions": [ + { + "rule": "cv_screen", + "variables": [ + "pov_rate" + ], + "threshold": 15, + "action": "flag" + } + ], + "analyses": [ + { + "id": "ineq", + "method": "inequality", + "outcome": "lbw", + "rank_by": "pov_rate", + "weights": "lbw_denominator", + "args": { + "higher_is_more_deprived": true, + "target": "conservative" + } + } + ], + "hypotheses": [ + { + "id": "H1", + "analysis": "ineq", + "statistic": "sii", + "direction": "two-sided", + "statement": "Exploratory area association; prior Round 22 poverty/LBW association already examined." + }, + { + "id": "H2", + "analysis": "ineq", + "statistic": "log_rii", + "direction": "two-sided", + "statement": "Exploratory area association; prior Round 22 poverty/LBW association already examined." + }, + { + "id": "H3", + "analysis": "ineq", + "statistic": "concentration_index", + "direction": "two-sided", + "statement": "Exploratory area association; prior Round 22 poverty/LBW association already examined." + } + ] +} diff --git a/mcp_server.py b/mcp_server.py index 4dca1ff..e9fa072 100644 --- a/mcp_server.py +++ b/mcp_server.py @@ -125,7 +125,27 @@ are direct death records. create_manifest refuses violations (CIRCULARITY R1/R2). Use exploratory:true to label an exploratory manifest; deviations remain recorded. Spatial_se (Bartlett Conley) failed validation: exploratory only, including run_variant. -Count outcomes: binomial trials=_denominator and scale:"X2"; Poisson counts use exposure. +Count outcomes: binomial trials=_denominator and scale:"robust"; Poisson counts use exposure +and scale:"robust". New external-count manifests refuse scale:"X2" (failed coverage validation). + +Inequality and standardized ratios (get_recipe('health_inequality'); existing manifest tools): +method inequality requires outcome/rank_by/positive weights and args.higher_is_more_deprived +(boolean); hypotheses name sii/log_rii/concentration_index only, without term. Ridits run +least to most deprived; positive indices mean higher outcomes among more deprived areas. +Concentration has the reverse sign of the usual income-ranked convention. Default target +conservative combines HC3 and ACS replicate SEs; median rank CV <=15% is validated, above +which warn about rank-error attenuation. External outcomes and their denominator weights +stay fixed; target these_areas is refused for external outcomes. R2 tests rank_by. +method smr requires observed integer counts, distinct population_bands, positive reference_rates +(events per person, in band order), and reference_source (hashed citation). Hypothesis log_smr +uses geoid (one GEOID or all); pooled ratios sum observed and expected counts first. +Log statistics test zero; ratio columns exponentiate estimates and limits. SMR target is +conservative only: tests combine Poisson and expected-count replicate log variance; exact +Garwood limits condition on expected counts. No replicates means zero expected uncertainty, +noted; O=0 has only the exact fixed-E interval. Reference-rate uncertainty is excluded. +SMR refuses negative ACS band replicates under its fixed validation domain, even with positive E. +Area HC3 assumes independence. R1 applies to external outcomes/counts. Reused LBW associations +are exploratory. No new tools are needed. Inflation: default index is R-CPI-U-RS (the Census Bureau's ACS index); 'c-cpi-u' diff --git a/scripts/inequality_validation.py b/scripts/inequality_validation.py new file mode 100644 index 0000000..43b3f4c --- /dev/null +++ b/scripts/inequality_validation.py @@ -0,0 +1,98 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: AGPL-3.0-or-later +"""Pre-specified Round 24 area-inequality validation (see adjacent criterion). + +Full run: python scripts/inequality_validation.py --criterion-commit SHA +Smoke only: --smoke-draws 1..20 (never writes the audit). +""" +import argparse +import hashlib +import json +import subprocess +from pathlib import Path +import sys + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) +import numpy as np +import pandas as pd +from scipy.special import expit +from scipy.stats import norm +from census.inequality import ridits, inequality_replicate, TERMS +from census.simulation import make_replicates +from census.variance import REPLICATE_COLUMNS + +CRITERION = "Conservative nominal 90% coverage >= 0.86 for all three statistics in all four cells with rank CV <= 0.15. CV 0.30 is informational." + + +def run(draws): + cells=[]; z=norm.ppf(.95) + for ci,(n,cv) in enumerate((n,cv) for n in (145,1000) for cv in (.05,.15,.30)): + rng=np.random.default_rng(20261116+ci) + w=rng.lognormal(np.log(20000 if n==145 else 4000),1,n) + x=expit(rng.normal(-1.6,.6,n)); r=ridits(x,w,True) + base=.07+.03*r; rm=np.average(r,weights=w); ym=np.average(base,weights=w) + truth=np.array([.03,np.log(.10/.07),2*np.average((base-ym)*(r-rm),weights=w)/ym]) + denom=np.maximum(1,np.rint(w/50)).astype(int) + f=pd.DataFrame({'GEOID':[str(i) for i in range(n)],'y':base,'x':x,'w':w,'x_se':cv*x}) + rep=pd.DataFrame(np.zeros((n,80)),columns=REPLICATE_COLUMNS) + rep.insert(0,'variable','x'); rep.insert(0,'GEOID',f.GEOID) + est=np.full((draws,3),np.nan); se=np.full_like(est,np.nan); sm=np.full_like(est,np.nan) + failures=0 + for d in range(draws): + probability=base+rng.normal(0,.008,n) + if ((probability<0)|(probability>1)).any(): + failures+=1; continue + f['y']=rng.binomial(denom,probability)/denom + f['x']=x+rng.normal(0,cv*x,n) + rep.loc[:,REPLICATE_COLUMNS]=make_replicates(f.x.to_numpy()[:,None],(cv*x)[:,None],rng)[:,0,:] + try: + result=inequality_replicate(f,'y','x','w',True,rep,fixed=('y','w')) + c=result.coefficients.set_index('term').loc[list(TERMS)] + est[d]=c.estimate; se[d]=c.se_total; sm[d]=c.se_model + except (ValueError,RuntimeError): failures+=1 + stats={} + for j,term in enumerate(TERMS): + good=np.isfinite(est[:,j]); sd=float(np.std(est[good,j],ddof=1)) if good.sum()>1 else float('nan') + coverage=lambda s:float(np.mean(np.isfinite(s[:,j]) & (np.abs(est[:,j]-truth[j])<=z*s[:,j]))) + stats[term]={'truth':float(truth[j]),'conservative_coverage':coverage(se),'population_coverage':coverage(sm), + 'bias':float(np.nanmean(est[:,j])-truth[j]),'empirical_sd':sd, + 'mean_total_se_over_sd':float(np.nanmean(se[:,j])/sd), + 'mean_model_se_over_sd':float(np.nanmean(sm[:,j])/sd)} + cell={'n':n,'rank_cv':cv,'draws':draws,'seed':20261116+ci,'invalid_draws':failures,'statistics':stats} + cells.append(cell) + print(f"n={n} CV={cv}: "+', '.join(f"{k}={v['conservative_coverage']:.3f}" for k,v in stats.items()),flush=True) + passed=all(s['conservative_coverage']>=.86 for c in cells if c['rank_cv']<=.15 for s in c['statistics'].values()) + return {'criterion':CRITERION,'passed':passed,'cells':cells} + + +def main(): + parser=argparse.ArgumentParser(description=__doc__) + parser.add_argument('--smoke-draws',type=int); parser.add_argument('--criterion-commit') + args=parser.parse_args() + if args.smoke_draws is not None: + if not 1<=args.smoke_draws<=20: parser.error('smoke draws must be 1..20') + run(args.smoke_draws); return + if not args.criterion_commit: parser.error('full run requires --criterion-commit') + # Verify the exact script and criterion were in the named, already-pushed commit. + for rel in ('scripts/inequality_validation.py','docs/audits/inequality_validation_criterion.md'): + committed=subprocess.check_output(['git','show',args.criterion_commit+':'+rel],cwd=ROOT) + if committed!=(ROOT/rel).read_bytes(): raise SystemExit('criterion/script differs from pre-run commit') + report=run(1000); report['criterion_commit']=args.criterion_commit + report['script_sha256']=hashlib.sha256(Path(__file__).read_bytes()).hexdigest() + out=ROOT/'docs/audits/inequality_validation' + out.with_suffix('.json').write_text(json.dumps(report,indent=2,allow_nan=False)+'\n') + lines=['# Area-inequality validation','',CRITERION,'',f"Criterion commit: `{args.criterion_commit}`.", + '',f"Result: **{'PASS' if report['passed'] else 'FAIL — exploratory only'}**.", + '', 'Invalid/undefined draws count as non-coverage. No seeds, designs or criteria were changed.', + '', '| n | Rank CV | Statistic | Conservative coverage | Population coverage | Bias | Mean total SE / SD | Mean model SE / SD |', + '|---:|---:|:---|---:|---:|---:|---:|---:|'] + for c in report['cells']: + for term,s in c['statistics'].items(): + lines.append(f"| {c['n']} | {c['rank_cv']:.0%} | {term} | {s['conservative_coverage']:.3f} | {s['population_coverage']:.3f} | {s['bias']:.6g} | {s['mean_total_se_over_sd']:.3f} | {s['mean_model_se_over_sd']:.3f} |") + lines+=['','Design: 1,000 draws per cell, fixed area weights and true ranks, independent areas; all invalid-draw counts are in the JSON.', + 'Validated range, if passed: median rank CV <=15%. CV 30% is informational; HC3/conservative intervals do not correct rank-error attenuation.', + 'Positive indices mean higher outcomes among more deprived areas. Concentration has the reverse sign of the usual income-ranked convention.'] + out.with_suffix('.md').write_text('\n'.join(lines)+'\n') + +if __name__=='__main__': main() diff --git a/scripts/smr_validation.py b/scripts/smr_validation.py new file mode 100644 index 0000000..07abe6d --- /dev/null +++ b/scripts/smr_validation.py @@ -0,0 +1,68 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: AGPL-3.0-or-later +"""Pre-specified Round 24 SMR validation; --smoke-draws 1..20 never writes audits.""" +import argparse +import hashlib +import json +import subprocess +from pathlib import Path +import sys +ROOT=Path(__file__).resolve().parents[1] +sys.path.insert(0,str(ROOT)) +import numpy as np +import pandas as pd +from census.simulation import make_replicates +from census.standardize import indirect_standardize + +CRITERION='Combined nominal 90% interval coverage >= 0.86 in all six cells. O=0 and invalid draws count as non-coverage.' + + +def run(draws): + cells=[]; rates=np.array([.0005,.001,.003,.01,.04]) + for ci,(expected,cv) in enumerate((e,cv) for e in (10,50,500) for cv in (.05,.20)): + rng=np.random.default_rng(20261117+ci) + pop=np.repeat(expected/rates.sum(),5) # Equal populations, then scaled to E. + obs=pd.Series([0],index=['synthetic']) + frame=pd.DataFrame([pop],index=obs.index) + combined=exact=invalid=zeros=0 + for _ in range(draws): + o=int(rng.poisson(1.2*expected)); obs.iloc[0]=o; zeros+=int(o==0) + observed_pop=pop*(1+rng.normal(0,cv,5)) + reps=make_replicates(observed_pop[None,:],(cv*pop)[None,:],rng) + frame.iloc[0,:]=observed_pop + try: + row=indirect_standardize(obs,frame,rates,reps).iloc[0] + combined+=int(o>0 and row.smr_lower<=1.2<=row.smr_upper) + exact+=int(row.smr_exact_lower<=1.2<=row.smr_exact_upper) + except ValueError: invalid+=1 + c={'expected':expected,'band_cv':cv,'draws':draws,'seed':20261117+ci, + 'combined_coverage':combined/draws,'exact_coverage':exact/draws,'zero_counts':zeros,'invalid_draws':invalid} + cells.append(c); print(f"E={expected} CV={cv}: combined={combined/draws:.3f}, exact={exact/draws:.3f}",flush=True) + return {'criterion':CRITERION,'passed':all(c['combined_coverage']>=.86 for c in cells),'cells':cells} + + +def main(): + parser=argparse.ArgumentParser(description=__doc__); parser.add_argument('--smoke-draws',type=int); parser.add_argument('--criterion-commit') + args=parser.parse_args() + if args.smoke_draws is not None: + if not 1<=args.smoke_draws<=20: parser.error('smoke draws must be 1..20') + run(args.smoke_draws); return + if not args.criterion_commit: parser.error('full run requires --criterion-commit') + for rel in ('scripts/smr_validation.py','docs/audits/smr_validation_criterion.md'): + committed=subprocess.check_output(['git','show',args.criterion_commit+':'+rel],cwd=ROOT) + if committed!=(ROOT/rel).read_bytes(): raise SystemExit('criterion/script differs from pre-run commit') + report=run(2000); report['criterion_commit']=args.criterion_commit + report['script_sha256']=hashlib.sha256(Path(__file__).read_bytes()).hexdigest() + out=ROOT/'docs/audits/smr_validation'; out.with_suffix('.json').write_text(json.dumps(report,indent=2)+'\n') + lines=['# Indirect SMR validation','',CRITERION,'',f"Criterion commit: `{args.criterion_commit}`.",'', + f"Result: **{'PASS' if report['passed'] else 'FAIL — exploratory only'}**.",'', + '| Expected | Band CV | Combined coverage | Fixed-E exact coverage | O=0 | Invalid |', + '|---:|---:|---:|---:|---:|---:|'] + for c in report['cells']: + lines.append(f"| {c['expected']} | {c['band_cv']:.0%} | {c['combined_coverage']:.3f} | {c['exact_coverage']:.3f} | {c['zero_counts']} | {c['invalid_draws']} |") + lines+=['','2,000 draws per cell; true SMR 1.2, five equally populated bands scaled to the target expected count.', + 'Gaussian population errors are not clipped/redrawn. Nonpositive/invalid populations or expected counts count as non-coverage.', + 'Reference-rate uncertainty, correlated band errors and spatial dependence were not simulated; rates are treated as fixed.'] + out.with_suffix('.md').write_text('\n'.join(lines)+'\n') + +if __name__=='__main__': main() diff --git a/tests/test_round24.py b/tests/test_round24.py new file mode 100644 index 0000000..bc6f14d --- /dev/null +++ b/tests/test_round24.py @@ -0,0 +1,402 @@ +# SPDX-License-Identifier: AGPL-3.0-or-later +"""Independent numerical and integration checks for Round 24 (offline).""" +import copy +import numpy as np +import pandas as pd +import pytest +import statsmodels.api as sm +from scipy.stats import chi2 + +from census.inequality import (ridits, slope_index, concentration_index, inequality_replicate, + _indices, TERMS) +from census.standardize import indirect_standardize +from census.variance import REPLICATE_COLUMNS +from census.results import IncompleteReplicatesError + + +def sample(): + w = np.array([1, 2, 3, 4, 5, 6.]) + x = np.array([3, 1, 2, 2, 5, 4.]) + y = np.array([.08, .05, .11, .07, .12, .09]) + return y, x, w + + +def test_ridits_hand_ties_and_direction(): + _, x, w = sample() + expected = np.array([9.5, 1, 5.5, 5.5, 18.5, 13.])/21 + np.testing.assert_allclose(ridits(x, w, True), expected) + np.testing.assert_allclose(ridits(x, w, False), 1-expected) + np.testing.assert_allclose(ridits(np.ones(6), w, True), .5) + + +@pytest.mark.parametrize("bad", [None, 0, "true"]) +def test_direction_is_required_boolean(bad): + with pytest.raises(ValueError): + ridits([1, 2], [1, 1], bad) + + +@pytest.mark.parametrize("weights", [[1, 0], [1, -1], [1, np.nan]]) +def test_bad_weights(weights): + with pytest.raises(ValueError): + ridits([1, 2], weights, True) + + +def test_indices_and_hc3_independently(): + y, x, w = sample(); r = ridits(x, w, True) + X = sm.add_constant(r) + fit = sm.WLS(y, X, weights=w).fit(cov_type="HC3") + a, b = fit.params + g = np.array([1/(a+b)-1/a, 1/(a+b)]) + s = slope_index(y, x, w, True) + assert s["sii"] == pytest.approx(b) + assert s["log_rii"] == pytest.approx(np.log((a+b)/a)) + assert s["se_sii"] == pytest.approx(fit.bse[1]) + assert s["se_log_rii"] == pytest.approx(np.sqrt(g@fit.cov_params()@g)) + rm, ym = np.average(r, weights=w), np.average(y, weights=w) + vr = np.average((r-rm)**2, weights=w) + truth = 2*np.average((y-ym)*(r-rm), weights=w)/ym + conv = sm.WLS(2*vr*y/ym, X, weights=w).fit(cov_type="HC3") + c = concentration_index(y, x, w, True) + assert c["concentration_index"] == pytest.approx(truth) + assert c["se_concentration_index"] == pytest.approx(conv.bse[1]) + # Direct normal-equation solution is independent of the centered implementation. + np.testing.assert_allclose(np.linalg.solve(X.T@(w[:, None]*X), X.T@(w*y)), [s['intercept'], s['sii']]) + reverse = slope_index(y, x, w, False) + assert reverse['sii'] == pytest.approx(-s['sii']) + assert reverse['log_rii'] == pytest.approx(-s['log_rii']) + assert concentration_index(y,x,w,False)['concentration_index'] == pytest.approx(-truth) + assert "reverse sign" in c['notes'][0] + + +def test_log_rii_nonpositive_endpoint(): + s = slope_index([-1, 0, 1, 2], [1, 2, 3, 4], [1]*4, True) + assert np.isnan(s['log_rii']) + assert any('undefined' in n for n in s['notes']) + + +def frames(): + y, x, w = sample() + f = pd.DataFrame({'GEOID':list('abcdef'), 'y':y, 'x':x, 'w':w, 'x_se':.1*x}) + rep = pd.DataFrame(np.repeat(x[:,None],80,axis=1), columns=REPLICATE_COLUMNS) + rep.insert(0,'variable','x'); rep.insert(0,'GEOID',f.GEOID) + return f,rep + + +def test_replicate_refits_recompute_ranks_and_fixed(): + f, rep = frames() + # Half the replicates reverse the ordering completely. + rep.loc[:, REPLICATE_COLUMNS[:40]] = -rep[REPLICATE_COLUMNS[:40]].to_numpy() + result = inequality_replicate(f,'y','x','w',True,rep,fixed=('y','w')) + b = slope_index(f.y,f.x,f.w,True)['sii'] + assert result.coefficients.iloc[0].se_sampling == pytest.approx(np.sqrt(.05*40*(2*b)**2)) + assert result.diagnostics['rank_cv_median'] == pytest.approx(10) + assert result.diagnostics['interval_target'] == 'conservative' + + +@pytest.mark.parametrize('kind',['missing_variable','missing_row','nonfinite','bad_weight']) +def test_strict_replicate_policy(kind): + f,rep = frames() + if kind=='missing_variable': rep=None + if kind=='missing_row': rep=rep.iloc[1:] + if kind=='nonfinite': rep.loc[0,REPLICATE_COLUMNS[3]]=np.nan + if kind=='bad_weight': + bad=rep.copy(); bad['variable']='w'; bad.loc[:,REPLICATE_COLUMNS]=-1. + rep=pd.concat([rep,bad],ignore_index=True) + with pytest.raises(IncompleteReplicatesError): + inequality_replicate(f,'y','x','w',True,rep) + + +def test_rank_cv_warning_and_population_target(): + f,rep=frames(); f['x_se']=.3*f.x + r=inequality_replicate(f,'y','x','w',True,rep,target='population') + assert any('attenuated' in n for n in r.notes) + assert r.to_dict()['interval_se_column']=='se_model' + + +def test_garwood_zero_and_pooled(): + obs=pd.Series([0,10],index=['001','002']); pop=pd.DataFrame([[100.,200.],[300.,400.]],index=obs.index) + rates=np.array([.01,.02]); r=indirect_standardize(obs,pop,rates) + np.testing.assert_allclose(r.expected,[5,11,16]); np.testing.assert_allclose(r.smr,[0,10/11,10/16]) + assert r.iloc[0].smr_exact_lower==0 + assert np.isnan(r.iloc[0].smr_lower) and np.isnan(r.iloc[0].p_value) + assert r.iloc[1].smr_exact_lower==pytest.approx(chi2.ppf(.05,20)/(2*11)) + assert r.iloc[1].smr_exact_upper==pytest.approx(chi2.ppf(.95,22)/(2*11)) + assert r.iloc[0].smr_exact_upper==pytest.approx(chi2.ppf(.95,2)/(2*5)) + assert any('No band replicates' in n for n in r.iloc[1].notes) + + +def test_smr_replicate_variance_pools_before_refit(): + obs=pd.Series([10,20],index=['a','b']); pop=pd.DataFrame([[100.,200.],[300.,400.]],index=obs.index) + rep=np.repeat(pop.to_numpy()[:,:,None],80,axis=2) + rep[0,:,0:40]*=1.1; rep[1,:,0:40]*=.9 + r=indirect_standardize(obs,pop,[.01,.02],rep) + er=np.einsum('abr,b->ar',rep,[.01,.02]).sum(axis=0) + expected=np.sqrt(1/30+.05*np.sum((np.log(er)-np.log(16))**2)) + assert r.iloc[-1].se_log_smr==pytest.approx(expected) + + +@pytest.mark.parametrize('bad',['alignment','counts','rates','replicates','reserved','collision']) +def test_smr_rejects_silent_data_loss(bad): + obs=pd.Series([1,2],index=['a','b']); pop=pd.DataFrame([[10.,20.],[30.,40.]],index=obs.index) + rates=[.01,.02]; rep=None + if bad=='alignment': pop=pop.iloc[::-1] + if bad=='counts': obs.iloc[0]=-1 + if bad=='rates': rates=[0,.02] + if bad=='replicates': rep=np.zeros((2,2,80)) + if bad=='reserved': obs.index=pop.index=['all','b'] + if bad=='collision': obs.index=pop.index=[1,'1'] + with pytest.raises(ValueError): indirect_standardize(obs,pop,rates,rep) + + +def manifest(method='inequality', external=False): + from census.manifest import load_manifest + from pathlib import Path + m=load_manifest(Path(__file__).resolve().parents[1]/'examples/manifests/income_population_tracts.json') + m['authors']=['human:example-researcher'] + m['exploratory']=True + if method=='inequality': + a=dict(id='ineq',method=method,outcome='income',rank_by='pop',weights='pop', + args={'higher_is_more_deprived':True}) + hyps=[dict(id=t,analysis='ineq',statistic=t,direction='two-sided') for t in TERMS] + if external: + a.update(outcome='lbw',weights='lbw_denominator') + else: + a=dict(id='smr',method=method,observed='income',population_bands=['pop'],reference_rates=[.01], + reference_source='Synthetic fixed rate per person; no rate uncertainty') + hyps=[dict(id='h',analysis='smr',statistic='log_smr',geoid='all',direction='two-sided')] + if external: a['observed']='lbw_numerator' + m['analyses']=[a];m['hypotheses']=hyps + if external: + m['releases']=[2024] + m['variables']=[{'name':'income','ref':'2024:B19013_001'},{'name':'pop','ref':'2024:B01003_001'}] + m['geography']={'level':'county','states':['26','36']} + m['external']=[{'name':'lbw','source':'chr','release':2025,'measure':'v037'}] + return m + + +@pytest.mark.parametrize('method',['inequality','smr']) +def test_registered_metadata_and_valid_manifest(method): + from census.manifest import validate_manifest, methods_section + from census.manifest.methods import METHODS + from census.health_sources import EXTERNAL_OUTCOME_METHODS + m=manifest(method) + assert validate_manifest(m,resolve=False)==[] + assert METHODS[method].external_outcome and method in EXTERNAL_OUTCOME_METHODS + text=methods_section(m) + assert 'Target: conservative' in text if method=='inequality' else 'Synthetic fixed rate' in text + + +@pytest.mark.parametrize('method',['inequality','smr']) +def test_failed_validation_refuses_confirmatory(method,monkeypatch): + from census.manifest import validate_manifest + import census.inequality as iq + import census.standardize as st + monkeypatch.setattr(iq if method=='inequality' else st, + 'INEQUALITY_VALIDATED' if method=='inequality' else 'SMR_VALIDATED',False) + m=manifest(method); m.pop('exploratory') + assert any('failed validation; exploratory only' in e for e in validate_manifest(m,resolve=False)) + m['exploratory']=True + assert not validate_manifest(m,resolve=False) + + +@pytest.mark.parametrize('mutate,fragment',[ + (lambda a,h:a['args'].pop('higher_is_more_deprived'),'explicit boolean'), + (lambda a,h:a['args'].update(higher_is_more_deprived='true'),'explicit boolean'), + (lambda a,h:a['args'].update(target='invalid'),'target must'), + (lambda a,h:a['args'].update(scale='robust'),'does not take'), + (lambda a,h:a.update(outcome='missing'),'outcome'), + (lambda a,h:a.update(rank_by='missing'),'rank_by'), + (lambda a,h:a.update(weights='missing'),'weights'), + (lambda a,h:h.update(term='pop'),'no term'), +]) +def test_inequality_manifest_rules(mutate,fragment): + from census.manifest import validate_manifest + m=manifest(); mutate(m['analyses'][0],m['hypotheses'][0]) + errors=validate_manifest(m,resolve=False) + assert any(fragment in e and ('ineq' in e or 'analyses[0]' in e) for e in errors),errors + + +@pytest.mark.parametrize('mutate,fragment',[ + (lambda a,h:a.update(observed='missing'),'observed'), + (lambda a,h:a.update(population_bands=[]),'population_bands'), + (lambda a,h:a.update(population_bands=['missing']),'population_bands'), + (lambda a,h:a.update(population_bands=['pop','pop'],reference_rates=[.01,.02]),'repeat'), + (lambda a,h:a.update(reference_rates=[0]),'reference_rates'), + (lambda a,h:a.update(reference_rates=[float('nan')]),'reference_rates'), + (lambda a,h:a.update(reference_rates=[True]),'reference_rates'), + (lambda a,h:a.update(reference_rates=[.01,.02]),'reference_rates'), + (lambda a,h:a.update(reference_source=' '),'reference_source'), + (lambda a,h:a.update(args={'scale':'X2'}),'does not take'), + (lambda a,h:a.update(args={'target':'these_areas'}),'target must'), + (lambda a,h:h.pop('geoid'),'requires geoid'), + (lambda a,h:h.update(geoid=['36061']),'requires geoid'), + (lambda a,h:h.update(term='pop'),'not term'), +]) +def test_smr_manifest_rules(mutate,fragment): + from census.manifest import validate_manifest + m=manifest('smr'); mutate(m['analyses'][0],m['hypotheses'][0]) + errors=validate_manifest(m,resolve=False) + assert any(fragment in e and ('smr' in e or 'analyses[0]' in e) for e in errors),errors + + +@pytest.mark.parametrize('method',['inequality','smr']) +def test_reference_values_and_direction_are_hashed(method): + from census.manifest import manifest_hash + m=manifest(method); before=manifest_hash(m) + if method=='inequality': m['analyses'][0]['args']['higher_is_more_deprived']=False + else: m['analyses'][0]['reference_source']='Different standard' + assert manifest_hash(m)!=before + if method=='smr': + m=manifest(method);m['analyses'][0]['reference_rates']=[.02] + assert manifest_hash(m)!=before + + +@pytest.mark.parametrize('method',['inequality','smr']) +def test_external_roles(method): + from census.manifest import validate_manifest + m=manifest(method,True) + assert not validate_manifest(m,resolve=False) + a=m['analyses'][0] + if method=='inequality': + a['rank_by']='lbw';fragment='rank_by' + else: + a['population_bands']=['lbw'];fragment='population_bands' + assert any(fragment in e for e in validate_manifest(m,resolve=False)) + + +def test_external_inequality_fixed_target(): + from census.manifest import validate_manifest + m=manifest('inequality',True);m['analyses'][0]['args']['target']='these_areas' + assert any('omits its uncertainty' in e for e in validate_manifest(m,resolve=False)) + + +@pytest.mark.parametrize('placement',['top','args']) +def test_circularity_rank_is_tested_term(placement): + from census.manifest import validate_manifest + m=manifest('inequality',True) + m['external']=[dict(name='diabetes',source='places',release=2024,measure='DIABETES')] + m['variables']=[dict(name='rank',ref='2024:B17001_002'),dict(name='pop',ref='2024:B01003_001')] + a=m['analyses'][0];a.update(outcome='diabetes',rank_by='rank',weights='pop') + if placement=='args': + for k in ('outcome','rank_by','weights'): a['args'][k]=a.pop(k) + errors=validate_manifest(m,resolve=False) + assert any('CIRCULARITY R2' in e and "'rank'" in e for e in errors),errors + for h in m['hypotheses']: h['circularity_justification']='Sensitivity check of published model input association only.' + assert not any('CIRCULARITY R2' in e for e in validate_manifest(m,resolve=False)) + + +@pytest.mark.parametrize('method',['inequality','smr']) +def test_circularity_r1_applies_to_nonformula_outcome(method): + from census.manifest import validate_manifest + m=manifest(method,True) + m['external']=[dict(name='disability',source='places',release=2024,measure='DISABILITY')] + m['analyses'][0]['outcome' if method=='inequality' else 'observed']='disability' if method=='inequality' else 'disability_numerator' + if method=='inequality':m['analyses'][0]['weights']='pop' + assert any('CIRCULARITY R1' in e for e in validate_manifest(m,resolve=False)) + + +@pytest.mark.parametrize('method',['inequality','smr']) +def test_full_offline_manifest(method,tmp_path): + from census.manifest import run_manifest, STATUS_EXPLORATORY + from tests.test_manifest import FakeClient, synthetic_payload + m=manifest(method) + m['data_source']={'synthetic':True,'description':'Deterministic offline fixture'} + # Replicates match the API fixture estimates and years; no real network calls. + parts=[] + for v,table in [('income','B19013'),('pop','B01003')]: + rows=synthetic_payload(table) + vals=np.array([float(row[2]) for row in rows[1:]]) + ids=['36061'+row[-1] for row in rows[1:]] + part=pd.DataFrame(np.repeat(vals[:,None],80,axis=1),columns=REPLICATE_COLUMNS) + part['GEOID']=ids;part['Year']=2019;part['variable']=v;parts.append(part) + client=FakeClient();client.synthetic=True + out=run_manifest(m,tmp_path,client=client,use_cache=False,replicates=pd.concat(parts,ignore_index=True)) + assert out['success'],out.get('deviations',out.get('errors')) + assert out['analysis_status']==STATUS_EXPLORATORY + assert (out['hypotheses'].status=='tested').all() + h=out['hypotheses'].set_index('statistic') + logstat='log_rii' if method=='inequality' else 'log_smr' + assert h.loc[logstat,'ratio']==pytest.approx(np.exp(h.loc[logstat,'estimate'])) + assert any('ratio' in p.read_text() for p in (tmp_path/'bundle').rglob('*.csv')) + + +def test_external_quantities_held_fixed_in_runner(): + from census.manifest.run import _run_analysis + f, rep=frames();f=f.rename(columns={'y':'lbw','w':'lbw_denominator'}) + external_rep=rep.copy();external_rep['variable']='lbw' + external_rep.loc[:,REPLICATE_COLUMNS]=1. + reps=pd.concat([rep,external_rep],ignore_index=True) + a=dict(id='i',method='inequality',outcome='lbw',rank_by='x',weights='lbw_denominator', + args={'higher_is_more_deprived':True}) + r=_run_analysis(a,f,.90,reps,'holm',external=['lbw']) + np.testing.assert_allclose(r.coefficients.se_sampling,0,atol=1e-14) + assert set(r.diagnostics['fixed_variables'])=={'lbw','lbw_denominator'} + + +def test_undefined_log_statistic_is_not_tested(): + from census.manifest.run import _statistic + obs=pd.Series([0],index=['36061']);pop=pd.DataFrame([[100]],index=obs.index) + result=indirect_standardize(obs,pop,[.01]) + with pytest.raises(ValueError,match='undefined'): + _statistic({'statistic':'log_smr','geoid':'all'},result) + + +def test_zero_mean_concentration_is_nan(): + c=concentration_index([-1,0,1], [1,2,3], [1,1,1], True) + assert np.isnan(c['concentration_index']) + assert any('undefined' in n for n in c['notes']) + + +def test_replica_alignment_with_single_year_column(): + f,rep=frames();f['Year']=2024 + assert inequality_replicate(f,'y','x','w',True,rep).n_obs==6 + + +def test_manifest_complete_case_exclusions_are_recorded(): + from census.manifest.run import _run_analysis + f,rep=frames();f.loc[0,'y']=np.nan + a=dict(id='i',method='inequality',outcome='y',rank_by='x',weights='w',args={'higher_is_more_deprived':True}) + r=_run_analysis(a,f,.9,rep,'holm') + assert r.n_obs==5 and r.diagnostics['complete_case']['dropped_geoids']==['a'] + assert any('excluded 1' in n for n in r.notes) + f['observed']=[np.nan,10,20,30,40,50] + b=dict(id='s',method='smr',observed='observed',population_bands=['w'],reference_rates=[.1],reference_source='test') + s=_run_analysis(b,f,.9,None,'holm') + assert set(s.GEOID)==set('bcdef')|{'all'} + assert any('excluded 1' in n for n in s.iloc[-1].notes) + + +def test_public_exports_and_recipe(): + import census + from census.recipes import get_recipe + for name in ('ridits','slope_index','concentration_index','inequality_replicate','indirect_standardize'): + assert callable(getattr(census,name)) and name in census.__all__ + recipe=get_recipe('health_inequality') + assert recipe['success'] and 'reverse sign' in recipe['steps'] and 'reference_source' in recipe['steps'] + + +def test_circularity_cannot_be_bypassed_by_unused_formula(): + from census.manifest import validate_manifest + m=manifest('inequality',True) + m['external']=[dict(name='diabetes',source='places',release=2024,measure='DIABETES')] + m['variables'].append(dict(name='rank',ref='2024:B17001_002')) + a=m['analyses'][0];a.update(outcome='diabetes',rank_by='rank',weights='pop',formula='income ~ pop') + assert any('CIRCULARITY R2' in e and "'rank'" in e for e in validate_manifest(m,resolve=False)) + + +def test_negative_replicate_bands_refused_under_preregistered_domain(): + obs=pd.Series([10],index=['36061']);pop=pd.DataFrame([[100.,200.]],index=obs.index) + rep=np.repeat(pop.to_numpy()[:,:,None],80,axis=2);rep[:,0,:]=-5. + # ACS perturbations can be negative; this implementation's fixed validation + # explicitly classified such band inputs as invalid, even with E_r > 0. + with pytest.raises(ValueError,match='nonnegative'): + indirect_standardize(obs,pop,[.01,.02],rep) + + +def test_rank_cv_uses_replicates_when_se_column_is_absent(): + f,rep=frames();f=f.drop(columns='x_se') + # A constant scale perturbation preserves ranks but has substantial known CV. + rep.loc[:,REPLICATE_COLUMNS]=.8*rep[REPLICATE_COLUMNS].to_numpy() + result=inequality_replicate(f,'y','x','w',True,rep) + assert result.diagnostics['rank_cv_median']==pytest.approx(40.) + assert any('attenuated' in n for n in result.notes)