From 054009d385ed8bd89d41374da0419e942a65ca2a Mon Sep 17 00:00:00 2001 From: dickbird <20980982+cafeTechne@users.noreply.github.com> Date: Sun, 15 Nov 2026 20:06:11 -0500 Subject: [PATCH 01/13] Round 24 plan: analysis-method registry, inequality indices (SII, RII, concentration index), indirectly standardized ratios, pre-set validations; archive PLAN_ROUND_23 (kept at v0.21.1) --- CLAUDE.md | 4 +- PLAN_ROUND_23.md | 173 ---------------------------------- PLAN_ROUND_24.md | 235 +++++++++++++++++++++++++++++++++++++++++++++++ STATUS.md | 8 +- 4 files changed, 243 insertions(+), 177 deletions(-) delete mode 100644 PLAN_ROUND_23.md create mode 100644 PLAN_ROUND_24.md diff --git a/CLAUDE.md b/CLAUDE.md index 29210e0..cf8343d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,8 +1,8 @@ # Agent guide Start with `STATUS.md` (current state, validated methods, rules, backlog). Then read only the current -`PLAN_ROUND_.md` if you are implementing a round. Older plans are archived at tag `v0.21.0` -(`git show v0.21.0:PLAN_ROUND_.md`); read one only when a docstring citation is not enough. +`PLAN_ROUND_.md` if you are implementing a round. Older plans are archived at their release tags +(`git show v0.21.0:PLAN_ROUND_.md` for rounds 1–22; `v0.21.1` for 23); read one only when a docstring citation is not enough. ## Commands diff --git a/PLAN_ROUND_23.md b/PLAN_ROUND_23.md deleted file mode 100644 index 13befd6..0000000 --- a/PLAN_ROUND_23.md +++ /dev/null @@ -1,173 +0,0 @@ -# Round 23 plan: split `census/manifest.py` into a package - -**Goal.** `census/manifest.py` (3,010 lines, 180 KB) is the engine's most-edited file. Every change makes an -agent read parts of it. Split it into a package of focused modules with **no behaviour change**, so later -rounds read one module of a few hundred lines. - -**Non-goals.** No new features, no bug fixes, no renamed functions, no changed messages, no reformatting of -moved code, no new dependencies, no changes outside the files named here. If you find a bug, write it in -section 7 (status table) and leave it alone. - -## 0. Ground rules - -1. Branch `phase23/split-manifest` from `phase23/plan`. Push it. Do not merge, tag or release. -2. **No AI attribution anywhere.** Commits and the PR body must not contain `Co-Authored-By`, - `Generated with`, or any similar line, even if your tooling suggests one. -3. Never print, log or commit `.env` or any API key or token. -4. Use `.venv/bin/python` (there is no `python` on PATH). Set `OPENBLAS_NUM_THREADS=1` for the suite. -5. Do not run GitHub CI. Commits may carry `[skip ci]`. -6. Read only what you need: `census/manifest.py`, the files named in sections 3–4, `pyproject.toml`, - `tests/test_packaging.py`. Do not read `docs/studies/`, `docs/audits/`, `studies_output/` or old plans. -7. If something in this plan is impossible as written, stop, record it in section 7, and choose the option - that changes the least. - -## 1. Baseline (before touching any code) - -On a clean checkout of `phase23/plan`: - -```bash -CENSUS_HEALTH_OFFLINE=1 .venv/bin/python scripts/refactor_golden.py /tmp/golden_before.json -.venv/bin/python -c "import census.manifest as m; print('\n'.join(sorted(n for n in vars(m) if not n.startswith('__'))))" > /tmp/names_before.txt -OPENBLAS_NUM_THREADS=1 .venv/bin/python -m pytest -q 2>&1 | tail -1 > /tmp/suite_before.txt -``` - -Expected suite line: `1634 passed, 37 skipped` (warnings and subtests may vary). Record the actual line in -section 7. `refactor_golden.py` must report `478 snapshots from 11 manifests`. - -## 2. Target layout - -Convert the module into a package. Start with `git mv census/manifest.py census/manifest/__init__.py`, then -cut code out of `__init__.py` into the submodules below. Move each definition **verbatim**: same body, -same comments, same docstring. Only import lines may change. - -| Module | Contents (exact names) | -| :--- | :--- | -| `schema.py` | All module-level constants from `SCHEMA_VERSION` through `REFERENCES` (including `ANALYSIS_METHODS`, `BLOCKING_DEVIATIONS`, the `STATUS_*` names, `FORMULA_ID_COLUMNS`, `FORMULA_FUNCTIONS`, `_STATE_PREFIXED_LEVELS`), `LEGACY_X2_MANIFESTS`, `CONFIRMATORY_REPLICATE_POLICY`, `REPLICATE_METHODS`, `_SPEC_KEYS`; and `canonical_json`, `manifest_hash`, `load_manifest`, `_table_of`, `_dataset_for`, `manifest_states`, `manifest_entities`, `_is_int`, `manifest_releases`, `release_overlap_message`, `_normalize_analysis`, `_duplicate_spec_keys`, `_formula_names`, `_variable_tables`, `tables_and_refs`, `_derivation_refs` | -| `external.py` | `external_names`, `external_circularity`, `_ext_source_text`, `_has_equivalent`, `circularity_errors` | -| `adjustments.py` | `default_rpp_areas`, `latest_rpp_year`, `_check_adjustments`, `_check_rpp_years`, `_check_dollar_year`, `_FILL_REASONS`, `_fill_adjustment_defaults`, `_scale_rows`, `_apply_adjustments` | -| `derivations.py` | `_age_standardized_errors`, `_fill_derivation_defaults` | -| `replicates.py` | `_vre_level`, `census_vre_source`, `_check_replicates`, `vre_unavailable`, `_declared_covariates_only`, `_replicate_overlap`, `_has_replicate_rows`, `_replicate_rows`, `_derive_replicates`, `_census_vre_replicates`, `_canonical_column`, `replicates_digest`, `read_replicates`, `replicates_sha256` | -| `datasource.py` | `effective_data_source`, `observed_data_source`, `data_source_record`, `resolve_data_source` | -| `validate.py` | `_check_geography`, `_check_variables`, `validate_manifest` (and the section 5 helpers) | -| `create.py` | `create_manifest` | -| `methods_text.py` | `_METHOD_TEXT`, `_age_standardized_text`, `_multi_release_adjustment_text`, `methods_section` (and the section 5 helpers) | -| `run.py` | `label_exploratory`, `analysis_status`, `_codebook_columns`, `_wide_from_panel`, `_one_sided`, `_run_analysis`, `_statistic`, `_tag`, `run_variant`, `run_manifest`, `_write_analysed_data`, `_write_bundle` | -| `__init__.py` | The original module docstring (lines 1–174, unchanged) and re-exports only (section 3). No logic. | - -Any definition not listed: put it in the module whose functions use it, and list it in section 7. - -**Import rules (to avoid cycles).** Submodules import each other with explicit relative imports -(`from .schema import manifest_hash`). A submodule never imports from `census.manifest` or from `.` (the -package `__init__`). Allowed direction: `schema` ← everything; `external`, `adjustments`, `derivations`, -`replicates`, `datasource` ← `validate`, `create`, `methods_text`, `run`; `validate` ← `create`, `run`; -`methods_text` ← `run`. Keep a function-local import inside a function body as a function-local import. - -**Packaging.** `pyproject.toml` lists `packages = ["census"]` explicitly, so the new subpackage would be -left out of installs. Change it to `packages = ["census", "census.manifest"]`. Nothing else in -`pyproject.toml` changes. - -## 3. Facade: `census/manifest/__init__.py` - -Re-export **every** name in `/tmp/names_before.txt` that was defined in the old module (functions, classes, -constants, including `_private` ones). Names that were only imported into the old module (`np`, `pd`, `json`, -`Path`, `Any`, `integrity_errors`, `IncompleteReplicatesError`, etc.) should also stay reachable: re-export -them too, so that `vars(census.manifest)` is a superset of the baseline. Write the re-exports as explicit -`from .schema import (a, b, ...)` lists, grouped by submodule, alphabetical within a group. No `import *`. - -## 4. Tests: only these lines may change - -Moving a function changes where `mock.patch` must point: a patch on `census.manifest.X` no longer affects -a call to `X` made inside a submodule. These are the only patches on this module: - -| File | Current target | -| :--- | :--- | -| `tests/test_review_fixes_r15.py` | `mock.patch("census.manifest.latest_rpp_year", ...)` | -| `tests/test_review_fixes_r12.py` | `mock.patch("census.manifest.replicates_digest", ...)` | -| `tests/test_metadata_integrity.py`, `tests/test_review_fixes_r14.py` | `patch("census.manifest.run_manifest", ...)` | -| `tests/test_manifest_releases.py` | `mock.patch("census.manifest.validate_manifest", ...)` | -| `tests/test_audit_decisions.py` | `patch.object(manifest, "_run_analysis", ...)` | -| `tests/test_review_fixes_r14.py` | `patch.object(M, "default_rpp_areas", ...)` | -| `tests/test_tract_harmonization.py` | `patch.object(mf, "_write_analysed_data", ...)` | - -For each one: run its test first. If it still passes unchanged (the caller looks the name up through -`census.manifest`, as `mcp_server.py` and `census/workflows*.py` do), leave it. If it fails, re-point the -patch at the submodule where the name is **looked up at call time** (for example -`census.manifest.run._run_analysis`). If a name is called from two submodules and the test needs both, -patch both. Change nothing else in any test file, and record each change in section 7. - -No other file outside `census/manifest/`, `pyproject.toml`, those test lines and the docs in section 6 may -change. In particular, `mcp_server.py` and other `census/*.py` modules keep importing from -`census.manifest` exactly as now. - -## 5. Splitting the three long functions (golden-covered only) - -After sections 2–4 pass section 6's checks, commit. Then split **only** these two functions, which the golden -snapshot covers byte for byte: - -- `validate_manifest` (about 300 lines): extract its consecutive blocks into private helpers in - `validate.py`, called in the same order with the same `errors` list, so the output order is unchanged. - Natural cuts: top-level keys/schema; variables; derivations; the per-analysis loop body - (`_check_analysis(i, a, ...)`); hypotheses; exclusions/missing/correction. Target: the - `validate_manifest` body is at most 80 lines. -- `methods_section` (about 320 lines): extract the paragraph builders (data, derivations, adjustments, - replicates, per-analysis text, references) into private helpers in `methods_text.py`, joined in the same - order. Target: the `methods_section` body is at most 80 lines. - -Do **not** split `run_manifest` this round. The golden snapshot does not cover it byte for byte; record its -natural stages in section 7 for a later round. - -If a split cannot keep the golden snapshot identical, revert that function to the verbatim move and record -it in section 7. - -## 6. Acceptance (all required) - -```bash -CENSUS_HEALTH_OFFLINE=1 .venv/bin/python scripts/refactor_golden.py /tmp/golden_after.json -cmp /tmp/golden_before.json /tmp/golden_after.json # must print nothing -.venv/bin/python -c "import census.manifest as m; print('\n'.join(sorted(n for n in vars(m) if not n.startswith('__'))))" > /tmp/names_after.txt -comm -23 /tmp/names_before.txt /tmp/names_after.txt # must print nothing (superset) -OPENBLAS_NUM_THREADS=1 .venv/bin/python -m pytest -q 2>&1 | tail -1 # same passed/skipped counts as baseline -.venv/bin/python -c "import mcp_server" # server module imports -wc -l census/manifest/*.py # every module <= 900 lines; __init__.py has no defs -grep -n "^def \|^class " census/manifest/__init__.py # must print nothing -``` - -Also run each check after the section 5 commit. Then update the docs: -`grep -rn "census/manifest.py\|manifest\.py" docs README.md CLAUDE.md STATUS.md CONTRIBUTING.md`, and change -each path reference to the package or submodule (no other wording changes). Add a three-line CHANGELOG -entry under a new `## Unreleased` heading: "Internal: `census.manifest` split into a package (no behaviour -change; golden snapshot identical)". - -## 7. Status table (fill in; keep it short) - -| Item | Status | Notes | -| :--- | :--- | :--- | -| Baseline suite line | PASS | 1634 passed, 37 skipped, 4 warnings, 48 subtests passed in 292.07s (0:04:52). Initial sandbox-only socket/MCP failures were rerun outside that restriction before code changed. | -| Golden snapshot count | PASS | 478 snapshots from 11 manifests; baseline namespace 122 names. | -| Unlisted definitions placed | None | All 68 original functions and 31 constants were listed; 23 imported compatibility names re-exported from schema. | -| Packaging change | PASS; least-change exception (0.7) | pyproject packages now ["census", "census.manifest"]. Updated only the identical expected list in tests/test_packaging.py; its old hard-coded assertion conflicted with required packaging. | -| Test patch targets changed | Three listed failures; one omitted target (0.7) | r15 latest_rpp_year -> adjustments; r12 replicates_digest -> run; r14 default_rpp_areas -> adjustments. Full suite also exposed tests/test_manifest_multistate.py::_run_analysis (omitted from section 4): retarget only its patch to run._run_analysis. Assertions unchanged. | -| Section 5: validate_manifest | PASS | 15 body lines; six private helpers; identical ordered errors and golden snapshots; checkpoint b full suite 1634 passed / 37 skipped. | -| Section 5: methods_section | PASS | 30 body lines; nine paragraph helpers; identical rendered bytes; post-commit checkpoint c full suite 1634 passed / 37 skipped. | -| `run_manifest` stages (for later) | Not split | Plan/extract; harmonize/aggregate/frame; fetch/derive replicates; external join; exclusions/adjustments; fit; save analysed data; corrected hypotheses; bundle. | -| Bugs noticed (not fixed) | Fault-injection coverage caveat | test_analysed_data_failure_does_not_lose_the_run passes without verifying its facade mock was called; after the move that mock is ineffective. Left unchanged per section 4. No production bug fix included. | -| Module line counts | PASS | __init__ 318; schema 256; external 113; adjustments 341; derivations 79; replicates 331; datasource 74; validate 498; create 129; methods_text 480; run 824. Facade has only its unchanged docstring and explicit imports. | -| Final suite line | PASS | 1634 passed, 37 skipped, 4 warnings, 48 subtests passed in 300.36s (0:05:00). Same pass/skip counts as baseline and a/b/c; golden byte-identical, no lost names, MCP imports, caps satisfied. | - -## 8. Deliverables - -1. Commits on `phase23/split-manifest`, in this order: (a) package conversion and verbatim moves, - packaging, test targets; (b) the `validate_manifest` split; (c) the `methods_section` split; (d) docs, - CHANGELOG, status table. Each commit passes section 6. -2. Push the branch. If you cannot open a PR, say so. The owner's agent opens it. -3. PR body (if you open it): what moved where (one table), the section 6 outputs pasted, the status table, - no attribution. - -Notes: Section 0.7 exceptions are the single packaging assertion and the omitted -`tests/test_manifest_multistate.py` spy target. Three listed failing targets were -retargeted; every listed test that passed unchanged was left unchanged. No test -assertions were weakened. Documentation was scanned for path tokens only, without -reviewing study/audit content; only `STATUS.md` needed a source-path replacement. -Actual `test_manifest.py` filenames remain valid and were not renamed. Checkpoints -a/b/c retained all 478 golden snapshots (11 manifests), all 122 baseline attributes -(the facade now has 132), MCP import and the <=900-line module cap. diff --git a/PLAN_ROUND_24.md b/PLAN_ROUND_24.md new file mode 100644 index 0000000..b45532b --- /dev/null +++ b/PLAN_ROUND_24.md @@ -0,0 +1,235 @@ +# Round 24 plan: analysis-method registry, health-inequality indices, standardized ratios + +**Goal.** (A) Define each manifest analysis method in one place, so a new method is one registry entry +instead of edits in five files. (B) Use the registry to add two validated public-health methods: area-level +inequality indices (slope index, relative index, concentration index) and indirectly standardized ratios +(SMR) with ACS-based expected counts. (C) Show them on an exploratory MI/NY example. Version 0.22.0. + +**Non-goals.** No `run_manifest` split (next round). No new MCP tools (63 stays 63). No new dependencies. +No individual-level (microdata) inequality measures. No Erreygers/Wagstaff corrections. No change to any +existing method's numbers or messages. + +## 0. Ground rules + +1. Branch `phase24/inequality` from `phase24/plan`. Push it. Do not merge, tag or release. (The owner's + agent merges Round 23 in parallel; `phase24/plan` sits on exactly that history, so no rebase is needed.) +2. **No AI attribution anywhere.** Commits and the PR body must not contain `Co-Authored-By`, + `Generated with`, or any similar line, even if your tooling suggests one. +3. Never print, log or commit `.env` or any API key or token. +4. Read `CLAUDE.md` and `STATUS.md` first. Use `.venv/bin/python`; set `OPENBLAS_NUM_THREADS=1` for the suite. +5. Do not run GitHub CI. Commits may carry `[skip ci]`. +6. **Validation first.** Each validation's script and acceptance criterion are committed and pushed + **before** its first full run. Do not change seeds, designs or criteria after a run. A method that fails + its validation ships as exploratory only (manifests refuse it unless `exploratory: true`), exactly as + `spatial_se` did in Round 22. Smoke runs of at most 20 draws are allowed before the commit, to debug. +7. If something here is impossible as written, record it in section 9 and choose the option that changes + least. + +## 1. Baseline + +```bash +CENSUS_HEALTH_OFFLINE=1 .venv/bin/python scripts/refactor_golden.py /tmp/golden_before.json # 478 snapshots +OPENBLAS_NUM_THREADS=1 .venv/bin/python -m pytest -q 2>&1 | tail -1 # expect 1634 passed, 37 skipped +``` + +## 2. Part A: method registry (no behaviour change) + +Create `census/manifest/methods.py`: + +```python +@dataclass(frozen=True) +class MethodSpec: + name: str + required: Tuple[str, ...] # today's ANALYSIS_METHODS[name][0] + statistics: Tuple[str, ...] # today's ANALYSIS_METHODS[name][1] + run: Callable[..., Any] # (a, args, frame, level, replicates, correction, ctx) -> result + replicate: bool = False # member of REPLICATE_METHODS + external_outcome: bool = False # member of health_sources.EXTERNAL_OUTCOME_METHODS + check: Optional[Callable[..., None]] = None # method-specific validation, appends to errors + text: Optional[Tuple[str, List[str]]] = None # today's _METHOD_TEXT entry, if any + +METHODS: Dict[str, MethodSpec] = {...} # same insertion order as today's ANALYSIS_METHODS +STATISTIC_EXTRACTORS: Dict[str, Callable[[Mapping, Any], Dict[str, Any]]] = {...} +``` + +Then: + +1. `schema.ANALYSIS_METHODS`, `schema.STATISTICS`, `schema.REPLICATE_METHODS` and + `health_sources.EXTERNAL_OUTCOME_METHODS` keep their names, types, values and order, but are **derived** + from `METHODS`. Avoid an import cycle: if `schema` cannot import `methods`, define the derived constants + in `methods.py` and re-export them from `schema` and the facade, so every existing import still works. +2. `run._run_analysis`: keep the shared preamble (argument normalisation, replicate policy, coords, + `held_external`) and replace the `if method == ...` chain with `METHODS[method].run(...)`. Move each + branch body verbatim into a small runner function in `methods.py`. `ctx` carries `held_external` and + `coords`. +3. `run._statistic`: dispatch through `STATISTIC_EXTRACTORS`. Move each branch verbatim. +4. `validate._check_analysis`: the method-specific blocks (today's `fit_replicate` argument block, the + `spatial_se` block, and any other `if method == ...` block) move verbatim into `check` callbacks, called + at the same point so the error order is unchanged. +5. `methods_text._METHOD_TEXT` becomes a view built from `MethodSpec.text`, with the same keys and values. + +**Acceptance A** (commit when all pass): golden snapshot byte-identical to `/tmp/golden_before.json`; the +suite has the same counts; `grep -n 'method == "' census/manifest/run.py census/manifest/validate.py` +prints only lines you list and justify in section 9 (shared, not method-specific, logic). + +## 3. Part B1: `census/inequality.py` (library) + +Conventions (state them in the module docstring): + +- Units are areas with population weights `w_i > 0`. The ranking variable `x` is an area-level + socioeconomic measure. Areas are ordered from **least deprived (ridit 0) to most deprived (ridit 1)**. + The caller says which way `x` points (`higher_is_more_deprived`, required, no default). +- **Ridit:** `r_i = (W_below_i + w_i / 2) / W`, where `W_below_i` is the weight of areas strictly less + deprived. Tied `x` values share the mean ridit of their block. +- Positive SII, log RII and concentration index all mean the outcome is **higher among more deprived + areas**. The concentration index sign is therefore the reverse of the usual income-ranked convention; + say so in every output note. + +Functions (NumPy/pandas/statsmodels only): + +| Function | Definition | Model SE | +| :--- | :--- | :--- | +| `ridits(x, w, higher_is_more_deprived)` | above | none | +| `slope_index(y, x, w, hi)` | WLS of `y` on `[1, r]`, weights `w`: `SII = b`, `log RII = log((a + b) / a)` (undefined, NaN with a note, if `a <= 0` or `a + b <= 0`) | HC3 (`cov_type="HC3"`); log RII by delta method, gradient `(1/(a+b) - 1/a, 1/(a+b))` | +| `concentration_index(y, x, w, hi)` | `C = 2 cov_w(y, r) / mean_w(y)`; equivalently the slope of the weighted convenient regression `2 var_w(r) y / mean_w(y) = alpha + C r` | HC3 slope SE of that regression (an approximation; the validation tests it) | +| `inequality_replicate(frame, outcome, rank_by, weights, hi, replicates, fixed=(), level=0.90, target="conservative")` | Full-sample SII, log RII and C, then the same three on each of the 80 replicates, recomputing ridits each time. The rank variable, the weights and the outcome use their replicates when present; names in `fixed` (external outcomes) stay at their estimates. `se_sampling = sqrt(4/80 sum (t_r - t)^2)`, `se_total = sqrt(se_model^2 + se_sampling^2)`. | Returns a `ModelResult` whose `coefficients` has terms `sii`, `log_rii`, `concentration_index` and the usual SE and interval columns for `target`. | + +`inequality_replicate` uses the strict replicate policy of `census.results`, as `fit_replicate` does. It +records the median CV of `rank_by` in `diagnostics["rank_cv_median"]`. When that is above the validated range +(section 5), it adds a note, and the methods text says the estimates may be attenuated by rank error. + +## 4. Part B2: standardized ratios in `census/standardize.py` + +Add `indirect_standardize(observed, band_populations, reference_rates, band_replicates=None, level=0.90)`: + +- `E_i = sum_b pop_ib * ref_b`, `SMR_i = O_i / E_i`. A pooled ratio over a set of areas is + `sum O / sum E`. +- **Exact interval with E fixed** (Garwood): lower `chi2.ppf(alpha/2, 2O) / (2E)` (0 when `O = 0`), upper + `chi2.ppf(1 - alpha/2, 2O + 2) / (2E)`. Report as `smr_exact_lower`/`upper`. +- **Combined interval** (used for tests): on the log scale, + `var(log SMR) = 1/O + s_E^2`, where `s_E^2 = 4/80 sum (log E_r - log E)^2` from the band replicates + (0 without replicates, with a note). Wald interval, exponentiated. Not defined when `O = 0` (NaN, note). +- Returns a DataFrame with GEOID, observed, expected, smr, `log_smr`, `se_log_smr`, combined and exact + limits, `p_value` (two-sided Wald on log scale), and the notes. + +## 5. Part B3: validations (criteria committed before running) + +**`scripts/inequality_validation.py` → `docs/audits/inequality_validation.md`/`.json`.** +- Cells: n ∈ {145, 1000} × rank CV ∈ {0.05, 0.15, 0.30}. 1,000 draws per cell. Seed `20261116 + cell index`. +- Fixed per cell, drawn once: + - weights: lognormal, median 20,000 (n = 145) or 4,000 (n = 1000), log-SD 1.0; + - true deprivation `x = expit(N(-1.6, 0.6))`; + - true ridits from `x`. +- Each draw: + - outcome proportion `y = 0.07 + 0.03 r + u`, with `u ~ N(0, 0.008^2)`; + - observed `y` from a binomial with denominators `w / 50`, held fixed (an external outcome); + - observed `x_hat = x + N(0, (cv x)^2)`, with replicates from `census.simulation.make_replicates(x_hat, cv * x, rng)`. +- Truths: + - `SII = 0.03`, `log RII = log(0.10 / 0.07)`; + - `C = 2 cov_w(0.07 + 0.03 r, r) / mean_w(0.07 + 0.03 r)` with the true ridits. +- **Acceptance:** coverage of nominal 90% intervals ≥ 0.86 for all three statistics, at target + `conservative`, in the four cells with CV ≤ 0.15. Report target `population` coverage, bias and mean + SE / empirical SD for every cell. The CV 0.30 cells are information only. They define the "validated + range" note (median rank CV ≤ 15%). + +**`scripts/smr_validation.py` → `docs/audits/smr_validation.md`/`.json`.** +- Cells: expected deaths `E_true` ∈ {10, 50, 500} × band CV ∈ {0.05, 0.20}. 2,000 draws. Seed + `20261117 + cell index`. Five bands with reference rates `(0.0005, 0.001, 0.003, 0.01, 0.04)`, band + populations scaled so that `E_true` is hit. True SMR 1.2. +- Each draw: + - `O ~ Poisson(1.2 E_true)`; + - observed band populations `pop_b (1 + N(0, cv^2))`, with replicates from `make_replicates`. +- **Acceptance:** combined-interval coverage of the true SMR ≥ 0.86 in all six cells. A draw with + `O = 0` counts as non-coverage. Report exact-interval coverage too. + +If a validation fails, its method is exploratory only (manifests refuse it unless `exploratory: true`). +Record this in the audit and in section 9. + +## 6. Part B4: manifest methods (registry entries) + +| Method | Required keys | Args | Statistics (hypothesis `statistic`) | +| :--- | :--- | :--- | :--- | +| `inequality` | `outcome`, `rank_by`, `weights` | `higher_is_more_deprived` (bool, required), `target` (default `conservative`) | `sii`, `log_rii`, `concentration_index` | +| `smr` | `observed`, `population_bands`, `reference_rates`, `reference_source` | `target` (no other args) | `log_smr` | + +Rules (implement as the methods' `check` callbacks; error texts are yours, but each must name the analysis): + +- `outcome`, `rank_by`, `weights` and each `population_bands` entry must be manifest variables, except + that `weights` may be `_denominator` when the outcome is external. `outcome` + may also be an external variable, and `observed` may be an external `_numerator`. Set + `external_outcome=True` for both methods. External outcomes are held fixed in the replicate refits. +- `reference_rates`: a list of finite numbers > 0, the same length as `population_bands`. + `reference_source`: a non-empty string (where the rates come from). Both are part of the hash. +- Hypotheses on `inequality` name a `statistic` only (no `term`). Hypotheses on `smr` give `geoid`: one + GEOID, or `"all"` for the pooled ratio. +- **Circularity R2:** for `inequality`, `rank_by` is the tested term, so extend `circularity_errors` + (`census/manifest/external.py`) to apply R2 to it. R1 applies unchanged to `outcome`/`observed`. +- Log-scale statistics (`log_rii`, `log_smr`) are tested against 0, so `_one_sided` needs no change. Add + ratio columns to the hypothesis table, following the `odds_ratio` pattern in `run.py`: `ratio`, + `ratio_lower`, `ratio_upper`. +- Methods text: one paragraph each (ridit definition and direction, HC3 plus replicates, target; SMR + definition, reference source, exact and combined intervals). Add references: Pamuk 1985 (Population + Studies 39:17–31); Kakwani, Wagstaff & van Doorslaer 1997 (J Econometrics 77:87–103); Garwood 1936 + (Biometrika 28:437–442); Breslow & Day 1987 (IARC Sci Publ 82). + +## 7. Part C: docs and exploratory example + +- `docs/recipes/health_inequality.md` and a `health_inequality` entry in `census/recipes.py` (the recipe + count test may need its expected list extended; that is the only allowed test edit outside new tests). +- `docs/agent_protocol.md`: a short "Inequality and standardized ratios" section (statistics, direction + convention, validated range, SMR reference-rate requirement). +- Library exports in `census/__init__.py`: `ridits`, `slope_index`, `concentration_index`, + `inequality_replicate`, `indirect_standardize`. +- Example `examples/manifests/r24_mi_ny_lbw_inequality.json`: + - `exploratory: true`; MI/NY counties; + - outcome CHR v037 low birth weight (`lbw`); `rank_by: pov_rate`, `higher_is_more_deprived: true`; + - weights: `lbw_denominator` (live births; allow `weights: "_denominator"` when the outcome is + external, as the binomial `trials` role does). + + It is exploratory because Round 22 already related poverty to this outcome. Run it with the cached CHR + data. Write `docs/studies/r24_lbw_inequality_example.md` (one page: the three indices, intervals, + median rank CV, the sign convention). Release outputs go in a tarball, not the repo. +- No SMR example with real data, because no age-specific death counts are cached (WONDER exports are + manual). Use a synthetic end-to-end manifest test instead. +- Tests in `tests/test_round24.py`: + - ridits: ties, weights, direction; + - SII/RII/C against hand calculations on a 6-area example; + - HC3 matches statsmodels; + - exact Garwood limits against `scipy.stats.chi2`; + - pooled SMR; + - manifest validation: each rule in section 6; + - end-to-end offline runs of both methods; + - refusal when a failed method is used without `exploratory`. + +## 8. Final acceptance + +- Part A golden identical (section 2). After Part B, regenerate the golden snapshot and confirm by `diff` + that only messages listing supported methods or statistics changed. Paste that diff summary in section 9. +- Full suite: baseline counts plus the new tests, 0 failures. +- Both audits committed, with the criterion commit preceding the run commit. +- CHANGELOG `0.22.0` entry; version metadata 0.22.0 wherever the existing release checklist sets it + (`CHANGELOG.md` "Releasing"). +- `STATUS.md`: add both methods to the validation table with their results, and remove backlog item 5. + +## 9. Status table (fill in; keep it short) + +| Item | Status | Notes | +| :--- | :--- | :--- | +| Baseline | | | +| A: registry, golden identical | | | +| A: remaining `method ==` lines | | | +| B1/B2 library | | | +| Inequality validation (criterion commit, result) | | | +| SMR validation (criterion commit, result) | | | +| B4 manifest methods, R2 extension | | | +| Golden diff after B | | | +| C docs, example run | | | +| Final suite | | | +| Exceptions (0.7) | | | + +## 10. Deliverables + +Commits in order: (1) Part A; (2) B1/B2 library and unit tests; (3) the inequality validation script and +criterion; (4) its run and audit; (5) the SMR validation script and criterion; (6) its run and audit; (7) B4 +manifest methods and tests; (8) Part C; (9) CHANGELOG, STATUS, version, status table. Push +`phase24/inequality`. If you cannot open a PR, say so and leave a PR body at `/tmp/round24-pr-body.md`. diff --git a/STATUS.md b/STATUS.md index 1aebf6b..5e0f790 100644 --- a/STATUS.md +++ b/STATUS.md @@ -18,10 +18,14 @@ standing rules, and what is next. Read this instead of old plans. Updated each r | `examples/manifests/` | Registered manifests (their bytes are fixed once registered) | | `PLAN_ROUND_.md` | Only the current round's plan. Older plans are archived (below) | -**Archived plans.** `PLAN_ROUND_1`–`22` were removed from the tree after Round 22. Code and docs still cite -them as labels (for example "PLAN_ROUND_16 A2"). To read one: `git show v0.21.0:PLAN_ROUND_16.md`, or +**Archived plans.** `PLAN_ROUND_1`–`22` are at tag `v0.21.0`, `PLAN_ROUND_23` at `v0.21.1`. Code and docs still cite +them as labels (for example "PLAN_ROUND_16 A2"). To read one: `git show v0.21.0:PLAN_ROUND_16.md` (or the later tag), or `https://github.com/cafeTechne/Census-Research-Engine/blob/v0.21.0/PLAN_ROUND_16.md`. +**Round 23 (v0.21.1, internal):** `census/manifest.py` became the `census/manifest/` package with no +behaviour change (478 golden snapshots identical; `scripts/refactor_golden.py`). `run_manifest` stages for a +later split are listed in `PLAN_ROUND_23.md` section 7 (at `v0.21.1`). + ## Methods: validation status Validated means a fixed, pre-set simulation or reference check passed. Failed methods stay exploratory only; From 5d6be08c8b0f89e8b8668f09b8840065a8b04531 Mon Sep 17 00:00:00 2001 From: dickbird <20980982+cafeTechne@users.noreply.github.com> Date: Sun, 15 Nov 2026 20:07:10 -0500 Subject: [PATCH 02/13] Round 24 plan: note the Round 23 merge on the plan branch --- PLAN_ROUND_24.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/PLAN_ROUND_24.md b/PLAN_ROUND_24.md index b45532b..e5dc4ae 100644 --- a/PLAN_ROUND_24.md +++ b/PLAN_ROUND_24.md @@ -11,8 +11,9 @@ existing method's numbers or messages. ## 0. Ground rules -1. Branch `phase24/inequality` from `phase24/plan`. Push it. Do not merge, tag or release. (The owner's - agent merges Round 23 in parallel; `phase24/plan` sits on exactly that history, so no rebase is needed.) +1. Branch `phase24/inequality` from `phase24/plan`. Push it. Do not merge, tag or release. (Round 23 is + merged and released as v0.21.1; `phase24/plan` already contains that merge (commit `60bf712`). If you + branched before it, run `git merge origin/phase24/plan` once. Never rebase or force-push.) 2. **No AI attribution anywhere.** Commits and the PR body must not contain `Co-Authored-By`, `Generated with`, or any similar line, even if your tooling suggests one. 3. Never print, log or commit `.env` or any API key or token. From ca8604d0de219a41f5a7187dc39a0f5bf0e3c02c Mon Sep 17 00:00:00 2001 From: dickbird <20980982+cafeTechne@users.noreply.github.com> Date: Sun, 15 Nov 2026 20:28:06 -0500 Subject: [PATCH 03/13] refactor: centralize manifest method registry [skip ci] --- PLAN_ROUND_24.md | 8 +- census/health_sources.py | 2 +- census/manifest/methods.py | 267 ++++++++++++++++++++++++++++++++ census/manifest/methods_text.py | 32 +--- census/manifest/run.py | 69 ++------- census/manifest/schema.py | 17 +- census/manifest/validate.py | 88 +---------- 7 files changed, 294 insertions(+), 189 deletions(-) create mode 100644 census/manifest/methods.py diff --git a/PLAN_ROUND_24.md b/PLAN_ROUND_24.md index e5dc4ae..2fec668 100644 --- a/PLAN_ROUND_24.md +++ b/PLAN_ROUND_24.md @@ -216,9 +216,9 @@ Rules (implement as the methods' `check` callbacks; error texts are yours, but e | Item | Status | Notes | | :--- | :--- | :--- | -| Baseline | | | -| A: registry, golden identical | | | -| A: remaining `method ==` lines | | | +| Baseline | Pass | 478 snapshots; 1634 passed, 37 skipped, 48 subtests. | +| A: registry, golden identical | Pass | 478 snapshots byte-identical; isolated full suite unchanged (1634/37). | +| A: remaining `method ==` lines | Pass | None in run.py/validate.py. Historical public tuple/text order is retained by registry projections. | | B1/B2 library | | | | Inequality validation (criterion commit, result) | | | | SMR validation (criterion commit, result) | | | @@ -226,7 +226,7 @@ Rules (implement as the methods' `check` callbacks; error texts are yours, but e | Golden diff after B | | | | C docs, example run | | | | Final suite | | | -| Exceptions (0.7) | | | +| Exceptions (0.7) | Recorded | Shared checkout switched externally during work; implementation isolated at /tmp/census-round24. Baseline repeated there. | ## 10. Deliverables diff --git a/census/health_sources.py b/census/health_sources.py index 2dd7dae..d7ee59d 100644 --- a/census/health_sources.py +++ b/census/health_sources.py @@ -40,7 +40,7 @@ "wonder": ("file", "sha256")} # where an external variable may appear: only as the outcome of these methods (its error is independent # of the ACS sampling error, so it enters the residual; as a covariate it would go uncorrected) -EXTERNAL_OUTCOME_METHODS = ("fit_eiv", "fit_replicate", "fh_regression") +from .manifest.methods import EXTERNAL_OUTCOME_METHODS JOIN_MISSING_LIMIT = 0.05 KINDS = ("direct", "model_based", "partly_model_based") diff --git a/census/manifest/methods.py b/census/manifest/methods.py new file mode 100644 index 0000000..9e08925 --- /dev/null +++ b/census/manifest/methods.py @@ -0,0 +1,267 @@ +# SPDX-License-Identifier: AGPL-3.0-or-later +"""Single registry of manifest analysis execution, validation and reporting metadata. + +Schema imports this module; schema-dependent helpers therefore import lazily. +""" +from dataclasses import dataclass +from typing import Any, Callable, Dict, List, Mapping, Optional, Tuple +import numpy as np +import pandas as pd + +@dataclass(frozen=True) +class MethodSpec: + name: str + required: Tuple[str, ...] + statistics: Tuple[str, ...] + run: Callable[..., Any] + replicate: bool = False + external_outcome: bool = False + # check(stage, context) preserves the legacy validation error order. + # Stages: count/external, before_spatial, spatial, after_spatial. + check: Optional[Callable[..., Any]] = None + text: Optional[Tuple[str, List[str]]] = None + +def _run_fit_replicate(a, args, frame, level, replicates, correction, ctx): + from .. import correlation, models, multitest, sae, trends + from .schema import _formula_names + held_external = ctx["held_external"] + method = a["method"] + fixed_vars = list(held_external) + if a.get("replicate_scope") == "covariates_only": # pre-registered: the outcome is held fixed + fixed_vars += [v for v in _formula_names(a["formula"].split("~")[0]) if v not in fixed_vars] + if fixed_vars: + args["fixed_variables"] = fixed_vars + return models.fit_replicate(a["formula"], frame, replicates=replicates, confidence_level=level, **args) + +def _run_fit_eiv(a, args, frame, level, replicates, correction, ctx): + from .. import correlation, models, multitest, sae, trends + from .schema import _formula_names + held_external = ctx["held_external"] + method = a["method"] + if held_external: + args["fixed_variables"] = held_external + return models.fit_eiv(a["formula"], frame, replicates=replicates, confidence_level=level, **args) + +def _run_regression_calibration(a, args, frame, level, replicates, correction, ctx): + from .. import correlation, models, multitest, sae, trends + from .schema import _formula_names + held_external = ctx["held_external"] + method = a["method"] + return getattr(models, method)(a["formula"], frame, confidence_level=level, **args) + +def _run_simex(a, args, frame, level, replicates, correction, ctx): + from .. import correlation, models, multitest, sae, trends + from .schema import _formula_names + held_external = ctx["held_external"] + method = a["method"] + return getattr(models, method)(a["formula"], frame, confidence_level=level, **args) + +def _run_fay_herriot(a, args, frame, level, replicates, correction, ctx): + from .. import correlation, models, multitest, sae, trends + from .schema import _formula_names + held_external = ctx["held_external"] + method = a["method"] + y = a["formula"].split("~")[0].strip() + args.setdefault("se_col", f"{y}_se") + return getattr(sae, method)(a["formula"], frame, confidence_level=level, **args) + +def _run_fh_regression(a, args, frame, level, replicates, correction, ctx): + from .. import correlation, models, multitest, sae, trends + from .schema import _formula_names + held_external = ctx["held_external"] + method = a["method"] + y = a["formula"].split("~")[0].strip() + args.setdefault("se_col", f"{y}_se") + return getattr(sae, method)(a["formula"], frame, confidence_level=level, **args) + +def _run_correlation_matrix(a, args, frame, level, replicates, correction, ctx): + from .. import correlation, models, multitest, sae, trends + from .schema import _formula_names + held_external = ctx["held_external"] + method = a["method"] + return correlation.correlation_matrix(frame, args.pop("variables"), confidence_level=level, + replicates=replicates, correction=correction, **args) + +def _run_linear_trend_test(a, args, frame, level, replicates, correction, ctx): + from .. import correlation, models, multitest, sae, trends + from .schema import _formula_names + held_external = ctx["held_external"] + method = a["method"] + v, g = args.pop("variable"), str(args.pop("geoid")) + sub = frame[frame["GEOID"] == g].sort_values("Year") + return trends.linear_trend_test(sub["Year"].tolist(), sub[v].tolist(), ses=sub[f"{v}_se"].tolist(), + confidence_level=level, **args) + +def _run_batch_z_test(a, args, frame, level, replicates, correction, ctx): + from .. import correlation, models, multitest, sae, trends + from .schema import _formula_names + held_external = ctx["held_external"] + method = a["method"] + v1, v2 = args.pop("variable_1"), args.pop("variable_2") + df = frame[["GEOID"]].copy() + for v, tag in ((v1, "1"), (v2, "2")): + df[f"e{tag}"] = frame[v].to_numpy(float) + df[f"m{tag}"] = frame[f"{v}_se"].to_numpy(float) * 1.645 + out = multitest.batch_z_test(df, "e1", "m1", "e2", "m2", confidence_level=level, + correction=correction, **args) + return out + +def _stat_coefficient(h, res): + c = res.coefficients.set_index("term") + if h["term"] not in c.index: + raise KeyError(f"term {h['term']!r} not in model (terms: {list(c.index)})") + row = c.loc[h["term"]] + from ..models import interval_se_column + se_col = interval_se_column(res) # the SE behind p_value and the interval + return {"estimate": float(row["estimate"]), "se": float(row[se_col]), "se_basis": se_col, + "p_two_sided": float(row["p_value"]), + "ci_lower": float(row["ci_lower"]), "ci_upper": float(row["ci_upper"])} + +def _stat_correlation(h, res): + a, b = h["variables"] + return {"estimate": float(res.matrix.loc[a, b]), "se": None, "p_two_sided": float(res.p_values.loc[a, b]), + "ci_lower": float(res.ci_lower.loc[a, b]), "ci_upper": float(res.ci_upper.loc[a, b])} + +def _stat_trend_slope(h, res): + return {"estimate": res["slope"], "se": res["se_slope"], "p_two_sided": res["p_value"], + "ci_lower": res["ci_lower"], "ci_upper": res["ci_upper"]} + +def _stat_z_difference(h, res): + row = res[res["GEOID"] == str(h["geoid"])] + if row.empty: + raise KeyError(f"GEOID {h['geoid']} not in the test frame") + r = row.iloc[0] + diff = float(r.get("difference", r["e1"] - r["e2"])) + return {"estimate": diff, "se": None, "p_two_sided": float(r["p_value"]), "ci_lower": None, "ci_upper": None} + +def _check_spatial(stage, ctx): + if stage != "spatial": + return + a, manifest, errors, method = (ctx[k] for k in ("a", "manifest", "errors", "method")) + spatial = (a.get("args") or {}).get("spatial_se") + if spatial is not None: + valid_spatial = (isinstance(spatial, Mapping) and set(spatial) == {"cutoff_km"} + and type(spatial["cutoff_km"]) in (int, float) + and np.isfinite(spatial["cutoff_km"]) and spatial["cutoff_km"] > 0) + if not valid_spatial: + errors.append("spatial_se must be an object with exactly cutoff_km (finite number > 0)") + from ..spatial_hac import SPATIAL_HAC_VALIDATED + if not SPATIAL_HAC_VALIDATED and not manifest.get("exploratory", False): + errors.append("spatial_se failed its validation (docs/audits/spatial_hac_validation.md); exploratory only") + if (a.get("args") or {}).get("cluster"): + errors.append("spatial_se and cluster are mutually exclusive") + if method not in ("fit_eiv", "fit_replicate") or (method == "fit_replicate" and + (a.get("args") or {}).get("model", "ols") not in ("binomial", "poisson")): + errors.append("spatial_se is supported for fit_eiv and fit_replicate binomial/poisson") + +def _check_fit_replicate(stage, ctx): + from .schema import FORMULA_ID_COLUMNS, LEGACY_X2_MANIFESTS, manifest_hash + from .replicates import census_vre_source + i, a, manifest, names, ext_names, errors = (ctx[k] for k in ("i", "a", "manifest", "names", "ext_names", "errors")) + aargs = a.get("args") or {} + outcome_name = str(a.get("formula", "")).split("~")[0].strip() + if stage == "count": + return (outcome_name if aargs.get("model") == "poisson" + and outcome_name.endswith("_numerator") and outcome_name[:-10] in ext_names else None) + if stage == "external" and aargs.get("target", "population") == "these_areas": + errors.append(f"analyses[{i}] ({a.get('id')}): an external outcome is held fixed in the refits, so " + "target 'these_areas' (replicate SE alone) is refused; use 'population' or " + "'conservative'") + count_outcome = ctx.get("count_outcome") + if stage == "before_spatial": + allowed = {"model", "weights", "exposure", "trials", "cv_threshold", "max_failed_replicates", + "target", "scale", "spatial_se"} + unknown = sorted(set(aargs) - allowed) + if unknown: + errors.append(f"fit_replicate does not take args {unknown}") + glm = aargs.get("model", "ols") + if aargs.get("scale") is not None and (aargs["scale"] not in ("X2", "robust") + or glm not in ("binomial", "poisson")): + errors.append("scale must be 'robust' or 'X2' for binomial or poisson models only") + if aargs.get("scale") == "robust" and aargs.get("spatial_se") is not None: + errors.append("scale 'robust' and spatial_se are alternatives; choose one") + parent = outcome_name if outcome_name in ext_names else count_outcome[:-10] if count_outcome else None + if parent is not None: + if glm not in ("ols", "binomial", "poisson"): + errors.append(f"{glm} not validated for external outcomes (PLAN_ROUND_22 out of scope)") + if glm == "binomial" and aargs.get("trials") != parent + "_denominator": + errors.append(f"external binomial outcome requires trials '{parent}_denominator'; effective trials refused") + if glm == "poisson" and (count_outcome is None or aargs.get("exposure") != parent + "_denominator"): + errors.append(f"external poisson needs {parent}_numerator outcome and exposure '{parent}_denominator'") + if glm in ("binomial", "poisson") and aargs.get("scale") is None and aargs.get("spatial_se") is None: + errors.append("overdispersion would be ignored: set scale 'robust'") + elif (glm in ("binomial", "poisson") and aargs.get("scale") == "X2" + and manifest_hash(manifest) not in LEGACY_X2_MANIFESTS): + errors.append("external count outcome with scale 'X2' refused: Pearson X2 SEs under-cover when " + "areas differ beyond binomial/Poisson chance (validated coverage 0.69-0.79 at " + "nominal 0.90, docs/audits/robust_se_validation.md); set scale 'robust' (HC3)") + for role in ("weights", "exposure", "trials"): + column = aargs.get(role) + valid_count = parent and ((glm == "binomial" and role == "trials") or + (glm == "poisson" and role == "exposure")) and column == parent+"_denominator" + if column is not None and column not in names and column not in FORMULA_ID_COLUMNS and not valid_count: + errors.append(f"fit_replicate {role} {column!r} is not a manifest variable in this role") + _check_spatial(stage, ctx) + +def _check_fit_eiv(stage, ctx): + from .schema import FORMULA_ID_COLUMNS, LEGACY_X2_MANIFESTS, manifest_hash + from .replicates import census_vre_source + i, a, manifest, names, ext_names, errors = (ctx[k] for k in ("i", "a", "manifest", "names", "ext_names", "errors")) + aargs = a.get("args") or {} + outcome_name = str(a.get("formula", "")).split("~")[0].strip() + if stage == "after_spatial": + eargs = a.get("args") or {} + unknown = sorted(set(eargs) - {"model", "weights", "cv_threshold", "target", "cluster", "cluster_se", "spatial_se"}) + if unknown: + errors.append(f"analyses[{i}] ({a.get('id')}): fit_eiv does not take args {unknown} (allowed: " + "model, weights, cv_threshold, cluster, cluster_se)") + if "cluster_se" in eargs and (eargs["cluster_se"] not in ("cr1", "jackknife") or "cluster" not in eargs): + errors.append(f"analyses[{i}] ({a.get('id')}): fit_eiv args.cluster_se must be 'cr1' or 'jackknife' " + "and needs args.cluster") + if "target" in eargs: + errors.append(f"analyses[{i}] ({a.get('id')}): fit_eiv takes no 'target': its intervals use the " + "sandwich se_total (a measurement-error correction is a claim about the population " + "relationship, not about these areas)") + if "cluster" in eargs: + cl = eargs["cluster"] + from ..models import CLUSTER_PREFIX_RE + mm = CLUSTER_PREFIX_RE.match(str(cl)) if isinstance(cl, str) else None + if not isinstance(cl, str) or not (cl in names or cl in FORMULA_ID_COLUMNS + or (mm and 1 <= int(mm.group(1)) <= 11)): + errors.append(f"analyses[{i}] ({a.get('id')}): fit_eiv args.cluster must be a manifest variable " + "or GEOID.str[:N] (e.g. GEOID.str[:5] for counties of tracts)") + if str(eargs.get("model", "ols")).lower() not in ("ols", "wls"): + errors.append(f"analyses[{i}] ({a.get('id')}): fit_eiv supports model 'ols' or 'wls'") + if str(eargs.get("model", "ols")).lower() == "wls" and not eargs.get("weights"): + errors.append(f"analyses[{i}] ({a.get('id')}): fit_eiv with model 'wls' needs args.weights") + if not census_vre_source(manifest) and a.get("replicates") != "required": + errors.append(f"analyses[{i}] ({a.get('id')}): fit_eiv needs replicates: declare " + "replicates.source 'census_vre' or \"replicates\": \"required\"") + _check_spatial(stage, ctx) + +METHODS: Dict[str, MethodSpec] = { + 'fit_replicate': MethodSpec('fit_replicate', ('formula',), ('coefficient',), _run_fit_replicate, replicate=True, external_outcome=True, check=_check_fit_replicate, text=('Coefficients were estimated with {model} regression (statsmodels) on the full-sample estimates; sampling standard errors came from refitting on each of {rep_source}, Var = 4/80 sum (b_r - b)^2, combined with the model standard error as se_total = sqrt(se_model^2 + se_sampling^2). Reported standard errors, intervals and p-values use the {target_se} ({target} target: {estimand}), with normal-theory intervals. Without replicates only model standard errors are reported.', ['vre', 'design', 'statsmodels'])), + 'fit_eiv': MethodSpec('fit_eiv', ('formula',), ('coefficient',), _run_fit_eiv, replicate=True, external_outcome=True, check=_check_fit_eiv, text=('Coefficients were estimated by moment-corrected least squares ({model}; Fuller 1987) with per-unit sampling-error covariances of every design column and of the outcome, including the outcome-covariate error covariance, estimated from {rep_source} (4/80 sum over replicates). Standard errors are {se_basis} (se_total, used for normal-theory intervals and p-values); naive coefficients are reported alongside. Coefficients are {estimand}.', ['fuller', 'vre', 'design'])), + 'regression_calibration': MethodSpec('regression_calibration', ('formula', 'error_covariates'), ('coefficient',), _run_regression_calibration, replicate=False, external_outcome=False, check=_check_spatial, text=('Covariates measured with known ACS sampling error ({cov}) were corrected by regression calibration with heteroscedastic known error variances; standard errors (se_total, used for normal-theory intervals and p-values) by row bootstrap re-estimating the calibration. Coefficients are {estimand}.', ['carroll'])), + 'simex': MethodSpec('simex', ('formula', 'error_covariates'), ('coefficient',), _run_simex, replicate=False, external_outcome=False, check=_check_spatial, text=('Covariates measured with known ACS sampling error ({cov}) were corrected by simulation-extrapolation (SIMEX, quadratic extrapolation) with the measurement-error jackknife variance (se_total, used for normal-theory intervals and p-values). Coefficients are {estimand}.', ['simex', 'carroll'])), + 'fh_regression': MethodSpec('fh_regression', ('formula',), ('coefficient',), _run_fh_regression, replicate=False, external_outcome=True, check=_check_spatial, text=("The outcome's known sampling variance was modelled with an area-level Fay-Herriot model (GLS coefficients, REML variance component).", ['fh'])), + 'fay_herriot': MethodSpec('fay_herriot', ('formula',), ('coefficient',), _run_fay_herriot, replicate=False, external_outcome=False, check=_check_spatial, text=('Small-area estimates are Fay-Herriot EBLUPs (REML) with Prasad-Rao/Datta-Lahiri MSE.', ['fh'])), + 'correlation_matrix': MethodSpec('correlation_matrix', ('variables',), ('correlation',), _run_correlation_matrix, replicate=True, external_outcome=False, check=_check_spatial, text=('Correlations with Fisher-z intervals{rep}.', ['fisher', 'vre'])), + 'linear_trend_test': MethodSpec('linear_trend_test', ('variable', 'geoid'), ('trend_slope',), _run_linear_trend_test, replicate=False, external_outcome=False, check=_check_spatial, text=('Linear trends across non-overlapping 5-year periods were tested by GLS with known sampling variances.', ['testing'])), + 'batch_z_test': MethodSpec('batch_z_test', ('variable_1', 'variable_2'), ('z_difference',), _run_batch_z_test, replicate=False, external_outcome=False, check=_check_spatial, text=('Differences were tested with the Census two-sample z test, SE = MOE/1.645.', ['testing'])), +} + +STATISTIC_EXTRACTORS = { + 'coefficient': _stat_coefficient, + 'correlation': _stat_correlation, + 'trend_slope': _stat_trend_slope, + 'z_difference': _stat_z_difference, +} + +ANALYSIS_METHODS = {name: (spec.required, spec.statistics) for name, spec in METHODS.items()} +STATISTICS = tuple(dict.fromkeys(stat for spec in METHODS.values() for stat in spec.statistics)) +# Preserve the historical public tuple order; future entries follow registry order. +REPLICATE_METHODS = tuple(sorted((name for name, spec in METHODS.items() if spec.replicate), + key=lambda name: {"fit_replicate": 0, "correlation_matrix": 1, "fit_eiv": 2}.get(name, 3))) +EXTERNAL_OUTCOME_METHODS = tuple(sorted((name for name, spec in METHODS.items() if spec.external_outcome), + key=lambda name: {"fit_eiv": 0, "fit_replicate": 1, "fh_regression": 2}.get(name, 3))) diff --git a/census/manifest/methods_text.py b/census/manifest/methods_text.py index 84858b7..af75378 100644 --- a/census/manifest/methods_text.py +++ b/census/manifest/methods_text.py @@ -34,35 +34,9 @@ # Methods text # --------------------------------------------------------------------------- -_METHOD_TEXT = { - "fit_replicate": ("Coefficients were estimated with {model} regression (statsmodels) on the full-sample " - "estimates; sampling standard errors came from refitting on each of {rep_source}, Var = 4/80 sum (b_r - b)^2, combined with the model standard error as " - "se_total = sqrt(se_model^2 + se_sampling^2). Reported standard errors, intervals and p-values use " - "the {target_se} ({target} target: {estimand}), with normal-theory intervals. Without " - "replicates only model standard errors are reported.", ["vre", "design", "statsmodels"]), - "regression_calibration": ("Covariates measured with known ACS sampling error ({cov}) were corrected by " - "regression calibration with heteroscedastic known error variances; standard " - "errors (se_total, used for normal-theory intervals and p-values) by row bootstrap " - "re-estimating the calibration. Coefficients are {estimand}.", ["carroll"]), - "fit_eiv": ("Coefficients were estimated by moment-corrected least squares ({model}; Fuller 1987) with " - "per-unit sampling-error covariances of every design column and of the outcome, including the " - "outcome-covariate error covariance, estimated from {rep_source} (4/80 sum over replicates). " - "Standard errors are {se_basis} (se_total, used for " - "normal-theory intervals and p-values); naive coefficients are reported alongside. Coefficients " - "are {estimand}.", ["fuller", "vre", "design"]), - "simex": ("Covariates measured with known ACS sampling error ({cov}) were corrected by " - "simulation-extrapolation (SIMEX, quadratic extrapolation) with the measurement-error jackknife " - "variance (se_total, used for normal-theory intervals and p-values). Coefficients are " - "{estimand}.", ["simex", "carroll"]), - "fh_regression": ("The outcome's known sampling variance was modelled with an area-level Fay-Herriot model " - "(GLS coefficients, REML variance component).", ["fh"]), - "fay_herriot": ("Small-area estimates are Fay-Herriot EBLUPs (REML) with Prasad-Rao/Datta-Lahiri MSE.", - ["fh"]), - "correlation_matrix": ("Correlations with Fisher-z intervals{rep}.", ["fisher", "vre"]), - "linear_trend_test": ("Linear trends across non-overlapping 5-year periods were tested by GLS with known " - "sampling variances.", ["testing"]), - "batch_z_test": ("Differences were tested with the Census two-sample z test, SE = MOE/1.645.", ["testing"]), -} +from .methods import METHODS + +_METHOD_TEXT = {name: spec.text for name, spec in sorted(METHODS.items(), key=lambda item: {'fit_replicate': 0, 'regression_calibration': 1, 'fit_eiv': 2, 'simex': 3, 'fh_regression': 4, 'fay_herriot': 5, 'correlation_matrix': 6, 'linear_trend_test': 7, 'batch_z_test': 8}.get(item[0], len(METHODS))) if spec.text is not None} def _age_standardized_text(d: Mapping[str, Any]) -> str: diff --git a/census/manifest/run.py b/census/manifest/run.py index 1543502..cb5c393 100644 --- a/census/manifest/run.py +++ b/census/manifest/run.py @@ -136,70 +136,19 @@ def _run_analysis(a: Mapping[str, Any], frame: pd.DataFrame, level: float, repli if args.get("spatial_se") is not None: args["coords"] = coords held_external = [v for v in _formula_names(str(a.get("formula", "")).split("~")[0]) if v in set(external) or (v.endswith("_numerator") and v[:-10] in set(external))] - if method == "fit_replicate": - fixed_vars = list(held_external) - if a.get("replicate_scope") == "covariates_only": # pre-registered: the outcome is held fixed - fixed_vars += [v for v in _formula_names(a["formula"].split("~")[0]) if v not in fixed_vars] - if fixed_vars: - args["fixed_variables"] = fixed_vars - return models.fit_replicate(a["formula"], frame, replicates=replicates, confidence_level=level, **args) - if method == "fit_eiv": - if held_external: - args["fixed_variables"] = held_external - return models.fit_eiv(a["formula"], frame, replicates=replicates, confidence_level=level, **args) - if method in ("regression_calibration", "simex"): - return getattr(models, method)(a["formula"], frame, confidence_level=level, **args) - if method in ("fay_herriot", "fh_regression"): - y = a["formula"].split("~")[0].strip() - args.setdefault("se_col", f"{y}_se") - return getattr(sae, method)(a["formula"], frame, confidence_level=level, **args) - if method == "correlation_matrix": - return correlation.correlation_matrix(frame, args.pop("variables"), confidence_level=level, - replicates=replicates, correction=correction, **args) - if method == "linear_trend_test": - v, g = args.pop("variable"), str(args.pop("geoid")) - sub = frame[frame["GEOID"] == g].sort_values("Year") - return trends.linear_trend_test(sub["Year"].tolist(), sub[v].tolist(), ses=sub[f"{v}_se"].tolist(), - confidence_level=level, **args) - if method == "batch_z_test": - v1, v2 = args.pop("variable_1"), args.pop("variable_2") - df = frame[["GEOID"]].copy() - for v, tag in ((v1, "1"), (v2, "2")): - df[f"e{tag}"] = frame[v].to_numpy(float) - df[f"m{tag}"] = frame[f"{v}_se"].to_numpy(float) * 1.645 - out = multitest.batch_z_test(df, "e1", "m1", "e2", "m2", confidence_level=level, - correction=correction, **args) - return out - raise ValueError(f"unsupported method {method}") + from .methods import METHODS + if method not in METHODS: + raise ValueError(f"unsupported method {method}") + return METHODS[method].run(a, args, frame, level, replicates, correction, + {"held_external": held_external, "coords": coords}) def _statistic(h: Mapping[str, Any], res: Any) -> Dict[str, Any]: + from .methods import STATISTIC_EXTRACTORS stat = h["statistic"] - if stat == "coefficient": - c = res.coefficients.set_index("term") - if h["term"] not in c.index: - raise KeyError(f"term {h['term']!r} not in model (terms: {list(c.index)})") - row = c.loc[h["term"]] - from ..models import interval_se_column - se_col = interval_se_column(res) # the SE behind p_value and the interval - return {"estimate": float(row["estimate"]), "se": float(row[se_col]), "se_basis": se_col, - "p_two_sided": float(row["p_value"]), - "ci_lower": float(row["ci_lower"]), "ci_upper": float(row["ci_upper"])} - if stat == "correlation": - a, b = h["variables"] - return {"estimate": float(res.matrix.loc[a, b]), "se": None, "p_two_sided": float(res.p_values.loc[a, b]), - "ci_lower": float(res.ci_lower.loc[a, b]), "ci_upper": float(res.ci_upper.loc[a, b])} - if stat == "trend_slope": - return {"estimate": res["slope"], "se": res["se_slope"], "p_two_sided": res["p_value"], - "ci_lower": res["ci_lower"], "ci_upper": res["ci_upper"]} - if stat == "z_difference": - row = res[res["GEOID"] == str(h["geoid"])] - if row.empty: - raise KeyError(f"GEOID {h['geoid']} not in the test frame") - r = row.iloc[0] - diff = float(r.get("difference", r["e1"] - r["e2"])) - return {"estimate": diff, "se": None, "p_two_sided": float(r["p_value"]), "ci_lower": None, "ci_upper": None} - raise ValueError(stat) + if stat not in STATISTIC_EXTRACTORS: + raise ValueError(stat) + return STATISTIC_EXTRACTORS[stat](h, res) def _tag(result: Any, h: str, analysis_id: str, status: str = STATUS_CONFIRMATORY) -> Any: diff --git a/census/manifest/schema.py b/census/manifest/schema.py index 10b209f..92bd821 100644 --- a/census/manifest/schema.py +++ b/census/manifest/schema.py @@ -34,19 +34,8 @@ DATA_SOURCE_KEYS = ("synthetic", "description", "generator") STATUS_EXPLORATORY = "exploratory" -# method -> (required args, statistics it supports) -ANALYSIS_METHODS: Dict[str, Tuple[Tuple[str, ...], Tuple[str, ...]]] = { - "fit_replicate": (("formula",), ("coefficient",)), - "fit_eiv": (("formula",), ("coefficient",)), - "regression_calibration": (("formula", "error_covariates"), ("coefficient",)), - "simex": (("formula", "error_covariates"), ("coefficient",)), - "fh_regression": (("formula",), ("coefficient",)), - "fay_herriot": (("formula",), ("coefficient",)), - "correlation_matrix": (("variables",), ("correlation",)), - "linear_trend_test": (("variable", "geoid"), ("trend_slope",)), - "batch_z_test": (("variable_1", "variable_2"), ("z_difference",)), -} -STATISTICS = ("coefficient", "correlation", "trend_slope", "z_difference") +from .methods import ANALYSIS_METHODS, STATISTICS, REPLICATE_METHODS + EXCLUSION_RULES = ("cv_screen", "geoid_exclude") ADJUSTMENT_TYPES = ("rpp",) RPP_ADJUSTMENT_KEYS = ("type", "variables", "rpp_year", "areas", "rpp_years", "dollar_year", "price_index") @@ -253,4 +242,4 @@ def _formula_names(formula: str) -> List[str]: # Manifest methods that use replicates; each runs with replicate_policy="strict". # (Spatial, multilevel, DiD and PUMS analyses are not manifest methods; if added # they must join this set so confirmatory runs refuse incomplete replicates.) -REPLICATE_METHODS = ("fit_replicate", "correlation_matrix", "fit_eiv") + diff --git a/census/manifest/validate.py b/census/manifest/validate.py index c01d471..11df819 100644 --- a/census/manifest/validate.py +++ b/census/manifest/validate.py @@ -5,6 +5,7 @@ from datetime import datetime from typing import Any, Dict, List, Mapping import numpy as np +from .methods import METHODS from .adjustments import ( _check_adjustments, ) @@ -315,8 +316,7 @@ def _check_analysis(i, a, manifest, names, ext_names, ids, folded_ids, used.append(a[key]) aargs = a.get("args") or {} outcome_name = str(a.get("formula", "")).split("~")[0].strip() - count_outcome = (outcome_name if method == "fit_replicate" and aargs.get("model") == "poisson" - and outcome_name.endswith("_numerator") and outcome_name[:-10] in ext_names else None) + count_outcome = METHODS[method].check("count", locals()) if count_outcome and count_outcome in _formula_names(str(a.get("formula", "")).split("~", 1)[1]): errors.append("external count columns are allowed as the Poisson outcome only") for u in used: @@ -333,86 +333,12 @@ def _check_analysis(i, a, manifest, names, ext_names, ids, folded_ids, elif [u for u in ext_used if u not in outcome] or len(outcome) != 1: errors.append(f"analyses[{i}] ({a.get('id')}): external variables {ext_used} may only be the outcome " "(alone): as covariates their model-based error would go uncorrected") - elif method == "fit_replicate" and (a.get("args") or {}).get("target", "population") == "these_areas": - errors.append(f"analyses[{i}] ({a.get('id')}): an external outcome is held fixed in the refits, so " - "target 'these_areas' (replicate SE alone) is refused; use 'population' or " - "'conservative'") - if method == "fit_replicate": - allowed = {"model", "weights", "exposure", "trials", "cv_threshold", "max_failed_replicates", - "target", "scale", "spatial_se"} - unknown = sorted(set(aargs) - allowed) - if unknown: - errors.append(f"fit_replicate does not take args {unknown}") - glm = aargs.get("model", "ols") - if aargs.get("scale") is not None and (aargs["scale"] not in ("X2", "robust") - or glm not in ("binomial", "poisson")): - errors.append("scale must be 'robust' or 'X2' for binomial or poisson models only") - if aargs.get("scale") == "robust" and aargs.get("spatial_se") is not None: - errors.append("scale 'robust' and spatial_se are alternatives; choose one") - parent = outcome_name if outcome_name in ext_names else count_outcome[:-10] if count_outcome else None - if parent is not None: - if glm not in ("ols", "binomial", "poisson"): - errors.append(f"{glm} not validated for external outcomes (PLAN_ROUND_22 out of scope)") - if glm == "binomial" and aargs.get("trials") != parent + "_denominator": - errors.append(f"external binomial outcome requires trials '{parent}_denominator'; effective trials refused") - if glm == "poisson" and (count_outcome is None or aargs.get("exposure") != parent + "_denominator"): - errors.append(f"external poisson needs {parent}_numerator outcome and exposure '{parent}_denominator'") - if glm in ("binomial", "poisson") and aargs.get("scale") is None and aargs.get("spatial_se") is None: - errors.append("overdispersion would be ignored: set scale 'robust'") - elif (glm in ("binomial", "poisson") and aargs.get("scale") == "X2" - and manifest_hash(manifest) not in LEGACY_X2_MANIFESTS): - errors.append("external count outcome with scale 'X2' refused: Pearson X2 SEs under-cover when " - "areas differ beyond binomial/Poisson chance (validated coverage 0.69-0.79 at " - "nominal 0.90, docs/audits/robust_se_validation.md); set scale 'robust' (HC3)") - for role in ("weights", "exposure", "trials"): - column = aargs.get(role) - valid_count = parent and ((glm == "binomial" and role == "trials") or - (glm == "poisson" and role == "exposure")) and column == parent+"_denominator" - if column is not None and column not in names and column not in FORMULA_ID_COLUMNS and not valid_count: - errors.append(f"fit_replicate {role} {column!r} is not a manifest variable in this role") + else: + METHODS[method].check("external", locals()) + METHODS[method].check("before_spatial", locals()) spatial = (a.get("args") or {}).get("spatial_se") - if spatial is not None: - valid_spatial = (isinstance(spatial, Mapping) and set(spatial) == {"cutoff_km"} - and type(spatial["cutoff_km"]) in (int, float) - and np.isfinite(spatial["cutoff_km"]) and spatial["cutoff_km"] > 0) - if not valid_spatial: - errors.append("spatial_se must be an object with exactly cutoff_km (finite number > 0)") - from ..spatial_hac import SPATIAL_HAC_VALIDATED - if not SPATIAL_HAC_VALIDATED and not manifest.get("exploratory", False): - errors.append("spatial_se failed its validation (docs/audits/spatial_hac_validation.md); exploratory only") - if (a.get("args") or {}).get("cluster"): - errors.append("spatial_se and cluster are mutually exclusive") - if method not in ("fit_eiv", "fit_replicate") or (method == "fit_replicate" and - (a.get("args") or {}).get("model", "ols") not in ("binomial", "poisson")): - errors.append("spatial_se is supported for fit_eiv and fit_replicate binomial/poisson") - if method == "fit_eiv": - eargs = a.get("args") or {} - unknown = sorted(set(eargs) - {"model", "weights", "cv_threshold", "target", "cluster", "cluster_se", "spatial_se"}) - if unknown: - errors.append(f"analyses[{i}] ({a.get('id')}): fit_eiv does not take args {unknown} (allowed: " - "model, weights, cv_threshold, cluster, cluster_se)") - if "cluster_se" in eargs and (eargs["cluster_se"] not in ("cr1", "jackknife") or "cluster" not in eargs): - errors.append(f"analyses[{i}] ({a.get('id')}): fit_eiv args.cluster_se must be 'cr1' or 'jackknife' " - "and needs args.cluster") - if "target" in eargs: - errors.append(f"analyses[{i}] ({a.get('id')}): fit_eiv takes no 'target': its intervals use the " - "sandwich se_total (a measurement-error correction is a claim about the population " - "relationship, not about these areas)") - if "cluster" in eargs: - cl = eargs["cluster"] - from ..models import CLUSTER_PREFIX_RE - mm = CLUSTER_PREFIX_RE.match(str(cl)) if isinstance(cl, str) else None - if not isinstance(cl, str) or not (cl in names or cl in FORMULA_ID_COLUMNS - or (mm and 1 <= int(mm.group(1)) <= 11)): - errors.append(f"analyses[{i}] ({a.get('id')}): fit_eiv args.cluster must be a manifest variable " - "or GEOID.str[:N] (e.g. GEOID.str[:5] for counties of tracts)") - if str(eargs.get("model", "ols")).lower() not in ("ols", "wls"): - errors.append(f"analyses[{i}] ({a.get('id')}): fit_eiv supports model 'ols' or 'wls'") - if str(eargs.get("model", "ols")).lower() == "wls" and not eargs.get("weights"): - errors.append(f"analyses[{i}] ({a.get('id')}): fit_eiv with model 'wls' needs args.weights") - if not census_vre_source(manifest) and a.get("replicates") != "required": - errors.append(f"analyses[{i}] ({a.get('id')}): fit_eiv needs replicates: declare " - "replicates.source 'census_vre' or \"replicates\": \"required\"") + METHODS[method].check("spatial", locals()) + METHODS[method].check("after_spatial", locals()) scope = a.get("replicate_scope", "all") if scope not in REPLICATE_SCOPES: errors.append(f"analyses[{i}] ({a.get('id')}): replicate_scope must be one of {REPLICATE_SCOPES}") From 590a35470d85e9d017777fa3de08280f905cd42d Mon Sep 17 00:00:00 2001 From: dickbird <20980982+cafeTechne@users.noreply.github.com> Date: Sun, 15 Nov 2026 20:28:57 -0500 Subject: [PATCH 04/13] feat: add weighted inequality and indirect standardization primitives [skip ci] --- census/inequality.py | 192 ++++++++++++++++++++++++++++++++++++++++++ census/standardize.py | 69 +++++++++++++++ tests/test_round24.py | 148 ++++++++++++++++++++++++++++++++ 3 files changed, 409 insertions(+) create mode 100644 census/inequality.py create mode 100644 tests/test_round24.py diff --git a/census/inequality.py b/census/inequality.py new file mode 100644 index 0000000..2bb77f2 --- /dev/null +++ b/census/inequality.py @@ -0,0 +1,192 @@ +# SPDX-License-Identifier: AGPL-3.0-or-later +"""Population-weighted area inequality, not individual-level inequality. + +Areas have positive weights. Required ``higher_is_more_deprived`` declares +whether increasing x means greater deprivation. Weighted ridits run from least +(0) to most (1) deprived; ties share their block's weighted mid-rank. Positive +SII, log RII and C mean higher outcomes among more deprived areas. C therefore +has the reverse sign of the usual income-ranked concentration convention. +HC3 treats areas as independent; rank-error attenuation is not corrected. +""" +import numpy as np +import pandas as pd + +from .results import ModelResult, precheck_replicates + +TERMS = ("sii", "log_rii", "concentration_index") +DIRECTION_NOTE = ("Positive indices mean higher outcomes among more deprived areas; concentration " + "has the reverse sign of the usual income-ranked convention.") +RANK_ERROR_NOTE = "Median rank CV exceeds 15%; estimates may be attenuated by rank error." +# Set only from the pre-specified validation report, never from the example study. +INEQUALITY_VALIDATED = False + + +def _vectors(y, x, w, hi): + if not isinstance(hi, (bool, np.bool_)): + raise ValueError("higher_is_more_deprived must be a boolean") + x, w = np.asarray(x, float), np.asarray(w, float) + if x.ndim != 1 or x.shape != w.shape or not x.size: + raise ValueError("rank and weights must be non-empty vectors of equal length") + if not np.isfinite(x).all() or not np.isfinite(w).all() or (w <= 0).any(): + raise ValueError("rank and weights must be finite; weights must be positive") + if y is not None: + y = np.asarray(y, float) + if y.shape != x.shape or not np.isfinite(y).all(): + raise ValueError("outcome must be a finite vector matching rank and weights") + return y, x, w + + +def _ridits_batch(x, w, hi): + """Rows are refits; sorting and tied-block mid-mass are vectorized.""" + order = np.argsort(x if hi else -x, axis=1, kind="stable") + sx, sw = np.take_along_axis(x, order, 1), np.take_along_axis(w, order, 1) + cum = np.cumsum(sw, axis=1) + starts = np.concatenate((np.ones((len(x), 1), bool), sx[:, 1:] != sx[:, :-1]), axis=1) + ends = np.concatenate((sx[:, 1:] != sx[:, :-1], np.ones((len(x), 1), bool)), axis=1) + below = np.maximum.accumulate(np.where(starts, cum - sw, 0), axis=1) + above = np.minimum.accumulate(np.where(ends, cum, np.inf)[:, ::-1], axis=1)[:, ::-1] + sorted_r = (below + above) / (2 * cum[:, -1:]) + r = np.empty_like(sorted_r) + np.put_along_axis(r, order, sorted_r, 1) + return r + + +def ridits(x, w, higher_is_more_deprived): + """Weighted mid-ranks from least to most deprived; tied blocks share a ridit.""" + _, x, w = _vectors(None, x, w, higher_is_more_deprived) + return _ridits_batch(x[None], w[None], higher_is_more_deprived)[0] + + +def _indices(y, x, w, hi, model_se=False): + """Batch WLS identities; HC3 uses the whitened WLS leverage and sandwich. + + Centering the design permits scalar bread/meat operations identical to + statsmodels WLS(cov_type='HC3'), without 80 Python model fits per draw. + """ + r = _ridits_batch(x, w, hi) + W = w.sum(axis=1) + rm, ym = (w*r).sum(axis=1)/W, (w*y).sum(axis=1)/W + dr = r-rm[:, None] + srr = (w*dr**2).sum(axis=1) + if (srr <= 0).any(): + raise ValueError("inequality needs at least two distinct deprivation ranks") + b = (w*dr*(y-ym[:, None])).sum(axis=1)/srr + a = ym-b*rm + valid = (a > 0) & (a+b > 0) + with np.errstate(divide="ignore", invalid="ignore"): + log_rii = np.where(valid, np.log((a+b)/a), np.nan) + k = 2*srr/W/ym + c = k*b + estimates = np.column_stack((b, log_rii, c)) + if not model_se: + return estimates + if y.shape[1] < 3: + raise ValueError("HC3 inequality requires at least three areas") + h = w*(1/W[:, None] + dr**2/srr[:, None]) + if (h >= 1-1e-12).any(): + raise ValueError("HC3 is undefined for unit leverage") + q2 = (w*(y-a[:, None]-b[:, None]*r)/(1-h))**2 + vb = (q2*dr**2).sum(axis=1)/srr**2 + vm = q2.sum(axis=1)/W**2 + cmb = (q2*dr).sum(axis=1)/(W*srr) + with np.errstate(divide="ignore", invalid="ignore"): + ga = 1/(a+b)-1/a + gb = 1/(a+b)-ga*rm + vl = ga**2*vm+gb**2*vb+2*ga*gb*cmb + se = np.column_stack((np.sqrt(vb), np.where(valid, np.sqrt(np.maximum(0, vl)), np.nan), + np.abs(k)*np.sqrt(vb))) + return estimates, se, a, b + + +def slope_index(y, x, w, higher_is_more_deprived): + """SII and log RII from weighted ridits, with HC3 and delta-method SEs. + + Returns estimates, SEs, intercept/slope and notes. log RII is undefined + when either fitted endpoint is nonpositive; no clipping is applied. + """ + y, x, w = _vectors(y, x, w, higher_is_more_deprived) + e, se, a, b = _indices(y[None], x[None], w[None], higher_is_more_deprived, True) + notes = [DIRECTION_NOTE] + if not np.isfinite(e[0, 1]): + notes.append("log RII undefined: a fitted endpoint is nonpositive.") + return dict(sii=float(e[0, 0]), se_sii=float(se[0, 0]), log_rii=float(e[0, 1]), + se_log_rii=float(se[0, 1]), intercept=float(a[0]), slope=float(b[0]), notes=notes) + + +def concentration_index(y, x, w, higher_is_more_deprived): + """C=2 cov_w(y,r)/mean_w(y), with approximate convenient-regression HC3 SE.""" + y, x, w = _vectors(y, x, w, higher_is_more_deprived) + e, se, _, _ = _indices(y[None], x[None], w[None], higher_is_more_deprived, True) + notes = [DIRECTION_NOTE, "Concentration SE is the convenient-regression HC3 approximation."] + if not np.isfinite(e[0, 2]): + notes.append("Concentration undefined: weighted outcome mean is zero.") + return dict(concentration_index=float(e[0, 2]), se_concentration_index=float(se[0, 2]), notes=notes) + + +def inequality_replicate(frame, outcome, rank_by, weights, higher_is_more_deprived, + replicates, fixed=(), level=0.90, target="conservative"): + """Three indices, refitting all 80 ACS replicates with fresh weighted ridits. + + Frame/long replicate keys are GEOID (+Year), as in fit_replicate. Missing + stochastic variables/rows and invalid refits are refused, never held fixed + silently. ``fixed`` explicitly declares external quantities held constant. + Target population uses HC3, these_areas replicate SE, conservative their + quadrature sum. No replicate source means model-only (reported in notes). + """ + from .models import _replicate_matrices, _table + from .results import stochastic_variables, IncompleteReplicatesError + if target not in ("population", "these_areas", "conservative") or not 0 < level < 1: + raise ValueError("invalid target or confidence level") + variables = list(dict.fromkeys((outcome, rank_by, weights))) + if set(fixed)-set(variables): + raise ValueError("fixed must name outcome, rank_by or weights") + y, x, w = _vectors(frame[outcome], frame[rank_by], frame[weights], higher_is_more_deprived) + estimates, model, _, _ = _indices(y[None], x[None], w[None], higher_is_more_deprived, True) + notes = [DIRECTION_NOTE, "HC3 assumes independent areas; no correction for rank-error attenuation."] + mats, held = {}, {} + active = [v for v in variables if v not in fixed] + if replicates is not None and len(replicates): + keys = ["GEOID"] + (["Year"] if "Year" in frame else []) + if frame.duplicated(keys).any() or replicates.duplicated(keys+["variable"]).any(): + raise ValueError("duplicate area/variable replicate keys") + mats, held = _replicate_matrices(frame, replicates, active) + missing = [v for v in stochastic_variables(frame, active) if v not in mats] + precheck_replicates("strict", rows_held_fixed=held, variables_without_replicates=missing, + context="inequality") + sampling = np.full(3, np.nan) + if mats: + batch = [mats.get(v, np.broadcast_to(frame[v].to_numpy(float)[:, None], (len(frame), 80))).T + for v in (outcome, rank_by, weights)] + if not all(np.isfinite(b).all() for b in batch) or (batch[2] <= 0).any(): + raise IncompleteReplicatesError("inequality: invalid replicate values or nonpositive weights") + try: + rep = _indices(*batch, higher_is_more_deprived) + except ValueError as exc: + raise IncompleteReplicatesError(f"inequality: replicate refit failed: {exc}") from exc + # Undefined full-sample log RII/C remain undefined, rather than failing SII. + defined = np.isfinite(estimates[0]) + if not np.isfinite(rep[:, defined]).all(): + raise IncompleteReplicatesError("inequality: an estimable full-sample statistic has undefined refits") + sampling = np.sqrt(.05*np.sum((rep-estimates)**2, axis=0)) + else: + notes.append("No replicates: sampling SE unavailable; conservative intervals use model SE only.") + if target == "these_areas": + raise IncompleteReplicatesError("inequality: these_areas requires replicates") + if not np.isfinite(estimates[0, 1]): + notes.append("log RII undefined: a fitted endpoint is nonpositive.") + if not np.isfinite(estimates[0, 2]): + notes.append("Concentration undefined: weighted outcome mean is zero.") + cv = np.full(len(frame), np.nan) + if rank_by+"_se" in frame: + np.divide(100*frame[rank_by+"_se"].to_numpy(float), np.abs(x), out=cv, where=x != 0) + else: + cv[:] = 0 + median = float(np.nanmedian(cv)) if np.isfinite(cv).any() else float("nan") + if median > 15: + notes.append(RANK_ERROR_NOTE) + coefs = _table(pd.Series(estimates[0], index=TERMS), pd.Series(model[0], index=TERMS), + pd.Series(sampling, index=TERMS), level, target) + return ModelResult("inequality", f"{outcome} ~ ridit({rank_by})", len(frame), coefs, + "inequality", ["model"]+(["sampling_replicate"] if mats else []), level, + notes=notes, diagnostics={"interval_target": target, "rank_cv_median": median, + "fixed_variables": list(fixed)}) diff --git a/census/standardize.py b/census/standardize.py index 8accd07..8fa2c69 100644 --- a/census/standardize.py +++ b/census/standardize.py @@ -166,3 +166,72 @@ def age_adjusted_interval(cases: Sequence[float], populations: Sequence[float], vu, yu = v + wm**2, y + wm upper = vu / (2 * yu) * float(chi2.ppf(1 - alpha / 2, 2 * yu**2 / vu)) return {"rate": y, "lower": lower, "upper": upper, "confidence_level": confidence_level} + + +# Set only from the pre-specified simulation report. +SMR_VALIDATED = False + + +def indirect_standardize(observed, band_populations, reference_rates, band_replicates=None, level=0.90): + """Indirect SMR and fixed-expected Garwood / combined log-Wald intervals. + + ``observed`` is a GEOID-indexed Series of nonnegative integer counts; + ``band_populations`` a DataFrame with exactly the same ordered index and + columns in reference-rate order (areas x bands). No implicit joins/drops. + Replicates are an array (areas, bands, 80) in that order. Positive finite + expected counts are required in every refit. The returned final GEOID + ``all`` pools counts and expected counts BEFORE computing uncertainty. + Rates are fixed; their uncertainty and between-area dependence are not + covered. At O=0 only the exact fixed-E interval is defined. + """ + import pandas as pd + from scipy.stats import chi2, norm + if not isinstance(observed, pd.Series) or not isinstance(band_populations, pd.DataFrame): + raise ValueError("observed must be a GEOID-indexed Series; band_populations a DataFrame") + if (not observed.index.equals(band_populations.index) or observed.index.has_duplicates + or not len(observed) or observed.index.isna().any() + or "all" in observed.index.astype(str)): + raise ValueError("area indexes must match exactly, be unique/nonmissing, and exclude reserved GEOID 'all'") + if observed.index.astype(str).has_duplicates: + raise ValueError("GEOIDs collide after string conversion") + if not 0 < level < 1: + raise ValueError("level must be between zero and one") + o, pop, rates = observed.to_numpy(float), band_populations.to_numpy(float), np.asarray(reference_rates, float) + if rates.ndim != 1 or rates.size != pop.shape[1] or not rates.size or not np.isfinite(rates).all() or (rates <= 0).any(): + raise ValueError("reference_rates must be finite positive numbers, one per population band") + if not np.isfinite(o).all() or (o < 0).any() or (o != np.floor(o)).any(): + raise ValueError("observed counts must be finite nonnegative integers") + if not np.isfinite(pop).all() or (pop < 0).any(): + raise ValueError("band populations must be finite and nonnegative") + e = pop@rates + if (e <= 0).any(): + raise ValueError("expected counts must be positive") + notes = ["Reference rates fixed; exact Garwood interval conditions on expected counts."] + o, e = np.r_[o, o.sum()], np.r_[e, e.sum()] + ve = np.zeros(len(o)) + if band_replicates is not None: + rep = np.asarray(band_replicates, float) + if rep.shape != (*pop.shape, 80) or not np.isfinite(rep).all() or (rep < 0).any(): + raise ValueError("band_replicates must be finite nonnegative (areas, bands, 80)") + er = np.einsum("abr,b->ar", rep, rates) + er = np.vstack((er, er.sum(axis=0))) + if (er <= 0).any(): + raise ValueError("replicate expected counts must be positive") + ve = .05*np.sum((np.log(er)-np.log(e[:, None]))**2, axis=1) + else: + notes.append("No band replicates: expected-count uncertainty set to zero.") + alpha = 1-level + lower = np.zeros(len(o)); positive = o > 0 + lower[positive] = chi2.ppf(alpha/2, 2*o[positive])/(2*e[positive]) + upper = chi2.ppf(1-alpha/2, 2*o+2)/(2*e) + smr = o/e + log, se, lo, hi, p = (np.full(len(o), np.nan) for _ in range(5)) + log[positive] = np.log(smr[positive]) + se[positive] = np.sqrt(1/o[positive]+ve[positive]) + z = norm.ppf(1-alpha/2) + lo[positive], hi[positive] = np.exp(log[positive]-z*se[positive]), np.exp(log[positive]+z*se[positive]) + p[positive] = 2*norm.sf(np.abs(log[positive]/se[positive])) + rownotes = [notes+([] if pos else ["O=0: combined log interval and Wald test undefined."]) for pos in positive] + return pd.DataFrame(dict(GEOID=list(observed.index.astype(str))+["all"], observed=o, expected=e, + smr=smr, log_smr=log, se_log_smr=se, smr_lower=lo, smr_upper=hi, + smr_exact_lower=lower, smr_exact_upper=upper, p_value=p, notes=rownotes)) diff --git a/tests/test_round24.py b/tests/test_round24.py new file mode 100644 index 0000000..f227339 --- /dev/null +++ b/tests/test_round24.py @@ -0,0 +1,148 @@ +# SPDX-License-Identifier: AGPL-3.0-or-later +"""Independent numerical and integration checks for Round 24 (offline).""" +import copy +import numpy as np +import pandas as pd +import pytest +import statsmodels.api as sm +from scipy.stats import chi2 + +from census.inequality import (ridits, slope_index, concentration_index, inequality_replicate, + _indices, TERMS) +from census.standardize import indirect_standardize +from census.variance import REPLICATE_COLUMNS +from census.results import IncompleteReplicatesError + + +def sample(): + w = np.array([1, 2, 3, 4, 5, 6.]) + x = np.array([3, 1, 2, 2, 5, 4.]) + y = np.array([.08, .05, .11, .07, .12, .09]) + return y, x, w + + +def test_ridits_hand_ties_and_direction(): + _, x, w = sample() + expected = np.array([9.5, 1, 5.5, 5.5, 18.5, 13.])/21 + np.testing.assert_allclose(ridits(x, w, True), expected) + np.testing.assert_allclose(ridits(x, w, False), 1-expected) + np.testing.assert_allclose(ridits(np.ones(6), w, True), .5) + + +@pytest.mark.parametrize("bad", [None, 0, "true"]) +def test_direction_is_required_boolean(bad): + with pytest.raises(ValueError): + ridits([1, 2], [1, 1], bad) + + +@pytest.mark.parametrize("weights", [[1, 0], [1, -1], [1, np.nan]]) +def test_bad_weights(weights): + with pytest.raises(ValueError): + ridits([1, 2], weights, True) + + +def test_indices_and_hc3_independently(): + y, x, w = sample(); r = ridits(x, w, True) + X = sm.add_constant(r) + fit = sm.WLS(y, X, weights=w).fit(cov_type="HC3") + a, b = fit.params + g = np.array([1/(a+b)-1/a, 1/(a+b)]) + s = slope_index(y, x, w, True) + assert s["sii"] == pytest.approx(b) + assert s["log_rii"] == pytest.approx(np.log((a+b)/a)) + assert s["se_sii"] == pytest.approx(fit.bse[1]) + assert s["se_log_rii"] == pytest.approx(np.sqrt(g@fit.cov_params()@g)) + rm, ym = np.average(r, weights=w), np.average(y, weights=w) + vr = np.average((r-rm)**2, weights=w) + truth = 2*np.average((y-ym)*(r-rm), weights=w)/ym + conv = sm.WLS(2*vr*y/ym, X, weights=w).fit(cov_type="HC3") + c = concentration_index(y, x, w, True) + assert c["concentration_index"] == pytest.approx(truth) + assert c["se_concentration_index"] == pytest.approx(conv.bse[1]) + # Direct normal-equation solution is independent of the centered implementation. + np.testing.assert_allclose(np.linalg.solve(X.T@(w[:, None]*X), X.T@(w*y)), [s['intercept'], s['sii']]) + reverse = slope_index(y, x, w, False) + assert reverse['sii'] == pytest.approx(-s['sii']) + assert reverse['log_rii'] == pytest.approx(-s['log_rii']) + assert concentration_index(y,x,w,False)['concentration_index'] == pytest.approx(-truth) + assert "reverse sign" in c['notes'][0] + + +def test_log_rii_nonpositive_endpoint(): + s = slope_index([-1, 0, 1, 2], [1, 2, 3, 4], [1]*4, True) + assert np.isnan(s['log_rii']) + assert any('undefined' in n for n in s['notes']) + + +def frames(): + y, x, w = sample() + f = pd.DataFrame({'GEOID':list('abcdef'), 'y':y, 'x':x, 'w':w, 'x_se':.1*x}) + rep = pd.DataFrame(np.repeat(x[:,None],80,axis=1), columns=REPLICATE_COLUMNS) + rep.insert(0,'variable','x'); rep.insert(0,'GEOID',f.GEOID) + return f,rep + + +def test_replicate_refits_recompute_ranks_and_fixed(): + f, rep = frames() + # Half the replicates reverse the ordering completely. + rep.loc[:, REPLICATE_COLUMNS[:40]] = -rep[REPLICATE_COLUMNS[:40]].to_numpy() + result = inequality_replicate(f,'y','x','w',True,rep,fixed=('y','w')) + b = slope_index(f.y,f.x,f.w,True)['sii'] + assert result.coefficients.iloc[0].se_sampling == pytest.approx(np.sqrt(.05*40*(2*b)**2)) + assert result.diagnostics['rank_cv_median'] == pytest.approx(10) + assert result.diagnostics['interval_target'] == 'conservative' + + +@pytest.mark.parametrize('kind',['missing_variable','missing_row','nonfinite','bad_weight']) +def test_strict_replicate_policy(kind): + f,rep = frames() + if kind=='missing_variable': rep=None + if kind=='missing_row': rep=rep.iloc[1:] + if kind=='nonfinite': rep.loc[0,REPLICATE_COLUMNS[3]]=np.nan + if kind=='bad_weight': + bad=rep.copy(); bad['variable']='w'; bad.loc[:,REPLICATE_COLUMNS]=-1. + rep=pd.concat([rep,bad],ignore_index=True) + with pytest.raises(IncompleteReplicatesError): + inequality_replicate(f,'y','x','w',True,rep) + + +def test_rank_cv_warning_and_population_target(): + f,rep=frames(); f['x_se']=.3*f.x + r=inequality_replicate(f,'y','x','w',True,rep,target='population') + assert any('attenuated' in n for n in r.notes) + assert r.to_dict()['interval_se_column']=='se_model' + + +def test_garwood_zero_and_pooled(): + obs=pd.Series([0,10],index=['001','002']); pop=pd.DataFrame([[100.,200.],[300.,400.]],index=obs.index) + rates=np.array([.01,.02]); r=indirect_standardize(obs,pop,rates) + np.testing.assert_allclose(r.expected,[5,11,16]); np.testing.assert_allclose(r.smr,[0,10/11,10/16]) + assert r.iloc[0].smr_exact_lower==0 + assert np.isnan(r.iloc[0].smr_lower) and np.isnan(r.iloc[0].p_value) + assert r.iloc[1].smr_exact_lower==pytest.approx(chi2.ppf(.05,20)/(2*11)) + assert r.iloc[1].smr_exact_upper==pytest.approx(chi2.ppf(.95,22)/(2*11)) + assert r.iloc[0].smr_exact_upper==pytest.approx(chi2.ppf(.95,2)/(2*5)) + assert any('No band replicates' in n for n in r.iloc[1].notes) + + +def test_smr_replicate_variance_pools_before_refit(): + obs=pd.Series([10,20],index=['a','b']); pop=pd.DataFrame([[100.,200.],[300.,400.]],index=obs.index) + rep=np.repeat(pop.to_numpy()[:,:,None],80,axis=2) + rep[0,:,0:40]*=1.1; rep[1,:,0:40]*=.9 + r=indirect_standardize(obs,pop,[.01,.02],rep) + er=np.einsum('abr,b->ar',rep,[.01,.02]).sum(axis=0) + expected=np.sqrt(1/30+.05*np.sum((np.log(er)-np.log(16))**2)) + assert r.iloc[-1].se_log_smr==pytest.approx(expected) + + +@pytest.mark.parametrize('bad',['alignment','counts','rates','replicates','reserved','collision']) +def test_smr_rejects_silent_data_loss(bad): + obs=pd.Series([1,2],index=['a','b']); pop=pd.DataFrame([[10.,20.],[30.,40.]],index=obs.index) + rates=[.01,.02]; rep=None + if bad=='alignment': pop=pop.iloc[::-1] + if bad=='counts': obs.iloc[0]=-1 + if bad=='rates': rates=[0,.02] + if bad=='replicates': rep=np.zeros((2,2,80)) + if bad=='reserved': obs.index=pop.index=['all','b'] + if bad=='collision': obs.index=pop.index=[1,'1'] + with pytest.raises(ValueError): indirect_standardize(obs,pop,rates,rep) From 60330e3473cd0d686cf0f0d3ea3a59122b62e2d6 Mon Sep 17 00:00:00 2001 From: dickbird <20980982+cafeTechne@users.noreply.github.com> Date: Sun, 15 Nov 2026 20:29:10 -0500 Subject: [PATCH 05/13] test: preregister inequality coverage validation [skip ci] --- .../audits/inequality_validation_criterion.md | 22 +++++ scripts/inequality_validation.py | 98 +++++++++++++++++++ 2 files changed, 120 insertions(+) create mode 100644 docs/audits/inequality_validation_criterion.md create mode 100644 scripts/inequality_validation.py diff --git a/docs/audits/inequality_validation_criterion.md b/docs/audits/inequality_validation_criterion.md new file mode 100644 index 0000000..d75d7c8 --- /dev/null +++ b/docs/audits/inequality_validation_criterion.md @@ -0,0 +1,22 @@ +# Inequality validation: criterion fixed before running + +Acceptance: conservative nominal 90% coverage >=0.86 for SII, log RII and +concentration in all four n=145/1000 x rank CV=0.05/0.15 cells. CV=0.30 is +informational. Exactly 1,000 draws per cell, seeds 20261116 + cell index, +ordered (145,.05), (145,.15), (145,.30), (1000,.05), (1000,.15), (1000,.30). + +The full design is PLAN_ROUND_24 section 5 and scripts/inequality_validation.py. +Weights and true x are drawn once per cell; every draw redraws u, binomial +outcomes and rank errors, then the 80 make_replicates values. Truths are .03, +log(.10/.07) and the weighted concentration of .07+.03*true_ridit. + +Operational choices fixed here: binomial denominators are w/50 rounded to the +nearest integer (NumPy rint), minimum one; regression weights remain the original +continuous w. Probabilities outside [0,1], undefined refits and invalid draws +count as non-coverage, without clipping, redrawing or reducing the denominator. +Report population coverage, bias and mean SE / empirical SD for all cells. +The rank CV diagnostic uses 100*SE/abs(observed rank), not nominal design CV. + +Smoke runs may use at most 20 draws per cell and do not write audit files. +Full runs require the exact script and criterion in a named, pushed commit. +No seed, design, threshold or criterion may change after the full run. diff --git a/scripts/inequality_validation.py b/scripts/inequality_validation.py new file mode 100644 index 0000000..43b3f4c --- /dev/null +++ b/scripts/inequality_validation.py @@ -0,0 +1,98 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: AGPL-3.0-or-later +"""Pre-specified Round 24 area-inequality validation (see adjacent criterion). + +Full run: python scripts/inequality_validation.py --criterion-commit SHA +Smoke only: --smoke-draws 1..20 (never writes the audit). +""" +import argparse +import hashlib +import json +import subprocess +from pathlib import Path +import sys + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) +import numpy as np +import pandas as pd +from scipy.special import expit +from scipy.stats import norm +from census.inequality import ridits, inequality_replicate, TERMS +from census.simulation import make_replicates +from census.variance import REPLICATE_COLUMNS + +CRITERION = "Conservative nominal 90% coverage >= 0.86 for all three statistics in all four cells with rank CV <= 0.15. CV 0.30 is informational." + + +def run(draws): + cells=[]; z=norm.ppf(.95) + for ci,(n,cv) in enumerate((n,cv) for n in (145,1000) for cv in (.05,.15,.30)): + rng=np.random.default_rng(20261116+ci) + w=rng.lognormal(np.log(20000 if n==145 else 4000),1,n) + x=expit(rng.normal(-1.6,.6,n)); r=ridits(x,w,True) + base=.07+.03*r; rm=np.average(r,weights=w); ym=np.average(base,weights=w) + truth=np.array([.03,np.log(.10/.07),2*np.average((base-ym)*(r-rm),weights=w)/ym]) + denom=np.maximum(1,np.rint(w/50)).astype(int) + f=pd.DataFrame({'GEOID':[str(i) for i in range(n)],'y':base,'x':x,'w':w,'x_se':cv*x}) + rep=pd.DataFrame(np.zeros((n,80)),columns=REPLICATE_COLUMNS) + rep.insert(0,'variable','x'); rep.insert(0,'GEOID',f.GEOID) + est=np.full((draws,3),np.nan); se=np.full_like(est,np.nan); sm=np.full_like(est,np.nan) + failures=0 + for d in range(draws): + probability=base+rng.normal(0,.008,n) + if ((probability<0)|(probability>1)).any(): + failures+=1; continue + f['y']=rng.binomial(denom,probability)/denom + f['x']=x+rng.normal(0,cv*x,n) + rep.loc[:,REPLICATE_COLUMNS]=make_replicates(f.x.to_numpy()[:,None],(cv*x)[:,None],rng)[:,0,:] + try: + result=inequality_replicate(f,'y','x','w',True,rep,fixed=('y','w')) + c=result.coefficients.set_index('term').loc[list(TERMS)] + est[d]=c.estimate; se[d]=c.se_total; sm[d]=c.se_model + except (ValueError,RuntimeError): failures+=1 + stats={} + for j,term in enumerate(TERMS): + good=np.isfinite(est[:,j]); sd=float(np.std(est[good,j],ddof=1)) if good.sum()>1 else float('nan') + coverage=lambda s:float(np.mean(np.isfinite(s[:,j]) & (np.abs(est[:,j]-truth[j])<=z*s[:,j]))) + stats[term]={'truth':float(truth[j]),'conservative_coverage':coverage(se),'population_coverage':coverage(sm), + 'bias':float(np.nanmean(est[:,j])-truth[j]),'empirical_sd':sd, + 'mean_total_se_over_sd':float(np.nanmean(se[:,j])/sd), + 'mean_model_se_over_sd':float(np.nanmean(sm[:,j])/sd)} + cell={'n':n,'rank_cv':cv,'draws':draws,'seed':20261116+ci,'invalid_draws':failures,'statistics':stats} + cells.append(cell) + print(f"n={n} CV={cv}: "+', '.join(f"{k}={v['conservative_coverage']:.3f}" for k,v in stats.items()),flush=True) + passed=all(s['conservative_coverage']>=.86 for c in cells if c['rank_cv']<=.15 for s in c['statistics'].values()) + return {'criterion':CRITERION,'passed':passed,'cells':cells} + + +def main(): + parser=argparse.ArgumentParser(description=__doc__) + parser.add_argument('--smoke-draws',type=int); parser.add_argument('--criterion-commit') + args=parser.parse_args() + if args.smoke_draws is not None: + if not 1<=args.smoke_draws<=20: parser.error('smoke draws must be 1..20') + run(args.smoke_draws); return + if not args.criterion_commit: parser.error('full run requires --criterion-commit') + # Verify the exact script and criterion were in the named, already-pushed commit. + for rel in ('scripts/inequality_validation.py','docs/audits/inequality_validation_criterion.md'): + committed=subprocess.check_output(['git','show',args.criterion_commit+':'+rel],cwd=ROOT) + if committed!=(ROOT/rel).read_bytes(): raise SystemExit('criterion/script differs from pre-run commit') + report=run(1000); report['criterion_commit']=args.criterion_commit + report['script_sha256']=hashlib.sha256(Path(__file__).read_bytes()).hexdigest() + out=ROOT/'docs/audits/inequality_validation' + out.with_suffix('.json').write_text(json.dumps(report,indent=2,allow_nan=False)+'\n') + lines=['# Area-inequality validation','',CRITERION,'',f"Criterion commit: `{args.criterion_commit}`.", + '',f"Result: **{'PASS' if report['passed'] else 'FAIL — exploratory only'}**.", + '', 'Invalid/undefined draws count as non-coverage. No seeds, designs or criteria were changed.', + '', '| n | Rank CV | Statistic | Conservative coverage | Population coverage | Bias | Mean total SE / SD | Mean model SE / SD |', + '|---:|---:|:---|---:|---:|---:|---:|---:|'] + for c in report['cells']: + for term,s in c['statistics'].items(): + lines.append(f"| {c['n']} | {c['rank_cv']:.0%} | {term} | {s['conservative_coverage']:.3f} | {s['population_coverage']:.3f} | {s['bias']:.6g} | {s['mean_total_se_over_sd']:.3f} | {s['mean_model_se_over_sd']:.3f} |") + lines+=['','Design: 1,000 draws per cell, fixed area weights and true ranks, independent areas; all invalid-draw counts are in the JSON.', + 'Validated range, if passed: median rank CV <=15%. CV 30% is informational; HC3/conservative intervals do not correct rank-error attenuation.', + 'Positive indices mean higher outcomes among more deprived areas. Concentration has the reverse sign of the usual income-ranked convention.'] + out.with_suffix('.md').write_text('\n'.join(lines)+'\n') + +if __name__=='__main__': main() From 222bb3cff3f3d64d8358321b42a02729077af86c Mon Sep 17 00:00:00 2001 From: dickbird <20980982+cafeTechne@users.noreply.github.com> Date: Sun, 15 Nov 2026 20:31:11 -0500 Subject: [PATCH 06/13] test: record prespecified inequality coverage results [skip ci] --- docs/audits/inequality_validation.json | 224 +++++++++++++++++++++++++ docs/audits/inequality_validation.md | 34 ++++ 2 files changed, 258 insertions(+) create mode 100644 docs/audits/inequality_validation.json create mode 100644 docs/audits/inequality_validation.md diff --git a/docs/audits/inequality_validation.json b/docs/audits/inequality_validation.json new file mode 100644 index 0000000..3e43823 --- /dev/null +++ b/docs/audits/inequality_validation.json @@ -0,0 +1,224 @@ +{ + "criterion": "Conservative nominal 90% coverage >= 0.86 for all three statistics in all four cells with rank CV <= 0.15. CV 0.30 is informational.", + "passed": true, + "cells": [ + { + "n": 145, + "rank_cv": 0.05, + "draws": 1000, + "seed": 20261116, + "invalid_draws": 0, + "statistics": { + "sii": { + "truth": 0.03, + "conservative_coverage": 0.905, + "population_coverage": 0.905, + "bias": -7.50038372105688e-05, + "empirical_sd": 0.004494378146362438, + "mean_total_se_over_sd": 1.0261672950554321, + "mean_model_se_over_sd": 1.0188994078597906 + }, + "log_rii": { + "truth": 0.3566749439387324, + "conservative_coverage": 0.9, + "population_coverage": 0.898, + "bias": -0.000720659874120233, + "empirical_sd": 0.05420389752253155, + "mean_total_se_over_sd": 1.024465078766145, + "mean_model_se_over_sd": 1.0170831547032084 + }, + "concentration_index": { + "truth": 0.05881109100544023, + "conservative_coverage": 0.903, + "population_coverage": 0.899, + "bias": -0.0001581508290767336, + "empirical_sd": 0.008744204555380693, + "mean_total_se_over_sd": 1.033987762749107, + "mean_model_se_over_sd": 1.0266659073638067 + } + } + }, + { + "n": 145, + "rank_cv": 0.15, + "draws": 1000, + "seed": 20261117, + "invalid_draws": 0, + "statistics": { + "sii": { + "truth": 0.03, + "conservative_coverage": 0.887, + "population_coverage": 0.851, + "bias": -0.0016816672687974994, + "empirical_sd": 0.004783563621796163, + "mean_total_se_over_sd": 1.0718423445757357, + "mean_model_se_over_sd": 0.988442834219536 + }, + "log_rii": { + "truth": 0.3566749439387324, + "conservative_coverage": 0.888, + "population_coverage": 0.858, + "bias": -0.019990987898913348, + "empirical_sd": 0.05716417498987063, + "mean_total_se_over_sd": 1.0778106778788292, + "mean_model_se_over_sd": 0.9927838803057799 + }, + "concentration_index": { + "truth": 0.058796193374334504, + "conservative_coverage": 0.891, + "population_coverage": 0.862, + "bias": -0.0032763632438349635, + "empirical_sd": 0.009247980139218473, + "mean_total_se_over_sd": 1.0871960567375862, + "mean_model_se_over_sd": 1.0026360801966303 + } + } + }, + { + "n": 145, + "rank_cv": 0.3, + "draws": 1000, + "seed": 20261118, + "invalid_draws": 0, + "statistics": { + "sii": { + "truth": 0.03, + "conservative_coverage": 0.823, + "population_coverage": 0.673, + "bias": -0.0061687772488054834, + "empirical_sd": 0.005101943552726339, + "mean_total_se_over_sd": 1.3240520504773057, + "mean_model_se_over_sd": 1.006663812997526 + }, + "log_rii": { + "truth": 0.3566749439387324, + "conservative_coverage": 0.821, + "population_coverage": 0.672, + "bias": -0.07432138425813428, + "empirical_sd": 0.06105404774007655, + "mean_total_se_over_sd": 1.3301580587862418, + "mean_model_se_over_sd": 1.0143563255222237 + }, + "concentration_index": { + "truth": 0.05878017738420337, + "conservative_coverage": 0.824, + "population_coverage": 0.672, + "bias": -0.012108223091622565, + "empirical_sd": 0.009962394070136655, + "mean_total_se_over_sd": 1.3278286570040922, + "mean_model_se_over_sd": 1.0095998749421176 + } + } + }, + { + "n": 1000, + "rank_cv": 0.05, + "draws": 1000, + "seed": 20261119, + "invalid_draws": 0, + "statistics": { + "sii": { + "truth": 0.03, + "conservative_coverage": 0.902, + "population_coverage": 0.898, + "bias": -0.00032107246945090387, + "empirical_sd": 0.0029831303503507637, + "mean_total_se_over_sd": 1.0005601188829774, + "mean_model_se_over_sd": 0.9933781912509618 + }, + "log_rii": { + "truth": 0.3566749439387324, + "conservative_coverage": 0.9, + "population_coverage": 0.898, + "bias": -0.0036958702024200707, + "empirical_sd": 0.03626691950642228, + "mean_total_se_over_sd": 0.9952431640991851, + "mean_model_se_over_sd": 0.9880404562078594 + }, + "concentration_index": { + "truth": 0.05882278284752297, + "conservative_coverage": 0.902, + "population_coverage": 0.898, + "bias": -0.0006155306762707854, + "empirical_sd": 0.0058577195594591265, + "mean_total_se_over_sd": 0.9993151226503405, + "mean_model_se_over_sd": 0.9921414519503555 + } + } + }, + { + "n": 1000, + "rank_cv": 0.15, + "draws": 1000, + "seed": 20261120, + "invalid_draws": 0, + "statistics": { + "sii": { + "truth": 0.03, + "conservative_coverage": 0.876, + "population_coverage": 0.848, + "bias": -0.0016854225603098703, + "empirical_sd": 0.003166153999902975, + "mean_total_se_over_sd": 1.069383499077787, + "mean_model_se_over_sd": 0.9785080692989536 + }, + "log_rii": { + "truth": 0.3566749439387324, + "conservative_coverage": 0.876, + "population_coverage": 0.843, + "bias": -0.020357488834119253, + "empirical_sd": 0.038110535661435546, + "mean_total_se_over_sd": 1.0729427044328128, + "mean_model_se_over_sd": 0.981471140490761 + }, + "concentration_index": { + "truth": 0.058822898770418255, + "conservative_coverage": 0.877, + "population_coverage": 0.848, + "bias": -0.0033125946981267113, + "empirical_sd": 0.006173349596271994, + "mean_total_se_over_sd": 1.0753442494014946, + "mean_model_se_over_sd": 0.9839711841159384 + } + } + }, + { + "n": 1000, + "rank_cv": 0.3, + "draws": 1000, + "seed": 20261121, + "invalid_draws": 0, + "statistics": { + "sii": { + "truth": 0.03, + "conservative_coverage": 0.644, + "population_coverage": 0.425, + "bias": -0.005861341451890566, + "empirical_sd": 0.003186406963665026, + "mean_total_se_over_sd": 1.364479142478532, + "mean_model_se_over_sd": 1.0125173984044469 + }, + "log_rii": { + "truth": 0.3566749439387324, + "conservative_coverage": 0.638, + "population_coverage": 0.421, + "bias": -0.07058433336612135, + "empirical_sd": 0.03842576885170834, + "mean_total_se_over_sd": 1.3602293853685432, + "mean_model_se_over_sd": 1.0110109062294084 + }, + "concentration_index": { + "truth": 0.05882123359629973, + "conservative_coverage": 0.641, + "population_coverage": 0.421, + "bias": -0.01148101733493627, + "empirical_sd": 0.0062720637032942835, + "mean_total_se_over_sd": 1.3593587246682728, + "mean_model_se_over_sd": 1.0087305294364852 + } + } + } + ], + "criterion_commit": "60330e3", + "script_sha256": "58f63eca0529d75f0ccf55cb6058ba4bdb0e8e345d20b23491032ba392194b37" +} diff --git a/docs/audits/inequality_validation.md b/docs/audits/inequality_validation.md new file mode 100644 index 0000000..bde07fc --- /dev/null +++ b/docs/audits/inequality_validation.md @@ -0,0 +1,34 @@ +# Area-inequality validation + +Conservative nominal 90% coverage >= 0.86 for all three statistics in all four cells with rank CV <= 0.15. CV 0.30 is informational. + +Criterion commit: `60330e3`. + +Result: **PASS**. + +Invalid/undefined draws count as non-coverage. No seeds, designs or criteria were changed. + +| n | Rank CV | Statistic | Conservative coverage | Population coverage | Bias | Mean total SE / SD | Mean model SE / SD | +|---:|---:|:---|---:|---:|---:|---:|---:| +| 145 | 5% | sii | 0.905 | 0.905 | -7.50038e-05 | 1.026 | 1.019 | +| 145 | 5% | log_rii | 0.900 | 0.898 | -0.00072066 | 1.024 | 1.017 | +| 145 | 5% | concentration_index | 0.903 | 0.899 | -0.000158151 | 1.034 | 1.027 | +| 145 | 15% | sii | 0.887 | 0.851 | -0.00168167 | 1.072 | 0.988 | +| 145 | 15% | log_rii | 0.888 | 0.858 | -0.019991 | 1.078 | 0.993 | +| 145 | 15% | concentration_index | 0.891 | 0.862 | -0.00327636 | 1.087 | 1.003 | +| 145 | 30% | sii | 0.823 | 0.673 | -0.00616878 | 1.324 | 1.007 | +| 145 | 30% | log_rii | 0.821 | 0.672 | -0.0743214 | 1.330 | 1.014 | +| 145 | 30% | concentration_index | 0.824 | 0.672 | -0.0121082 | 1.328 | 1.010 | +| 1000 | 5% | sii | 0.902 | 0.898 | -0.000321072 | 1.001 | 0.993 | +| 1000 | 5% | log_rii | 0.900 | 0.898 | -0.00369587 | 0.995 | 0.988 | +| 1000 | 5% | concentration_index | 0.902 | 0.898 | -0.000615531 | 0.999 | 0.992 | +| 1000 | 15% | sii | 0.876 | 0.848 | -0.00168542 | 1.069 | 0.979 | +| 1000 | 15% | log_rii | 0.876 | 0.843 | -0.0203575 | 1.073 | 0.981 | +| 1000 | 15% | concentration_index | 0.877 | 0.848 | -0.00331259 | 1.075 | 0.984 | +| 1000 | 30% | sii | 0.644 | 0.425 | -0.00586134 | 1.364 | 1.013 | +| 1000 | 30% | log_rii | 0.638 | 0.421 | -0.0705843 | 1.360 | 1.011 | +| 1000 | 30% | concentration_index | 0.641 | 0.421 | -0.011481 | 1.359 | 1.009 | + +Design: 1,000 draws per cell, fixed area weights and true ranks, independent areas; all invalid-draw counts are in the JSON. +Validated range, if passed: median rank CV <=15%. CV 30% is informational; HC3/conservative intervals do not correct rank-error attenuation. +Positive indices mean higher outcomes among more deprived areas. Concentration has the reverse sign of the usual income-ranked convention. From 680707efe48a48fb746f39562b385d6da048b74c Mon Sep 17 00:00:00 2001 From: dickbird <20980982+cafeTechne@users.noreply.github.com> Date: Sun, 15 Nov 2026 20:31:24 -0500 Subject: [PATCH 07/13] test: preregister indirect SMR coverage validation [skip ci] --- docs/audits/smr_validation_criterion.md | 18 +++++++ scripts/smr_validation.py | 68 +++++++++++++++++++++++++ 2 files changed, 86 insertions(+) create mode 100644 docs/audits/smr_validation_criterion.md create mode 100644 scripts/smr_validation.py diff --git a/docs/audits/smr_validation_criterion.md b/docs/audits/smr_validation_criterion.md new file mode 100644 index 0000000..00879bd --- /dev/null +++ b/docs/audits/smr_validation_criterion.md @@ -0,0 +1,18 @@ +# Indirect SMR validation: criterion fixed before running + +Acceptance: combined nominal 90% coverage of true SMR 1.2 >=0.86 in all six +E=10/50/500 x band CV=.05/.20 cells. Exactly 2,000 draws per cell, seed +20261117 + index, ordered (10,.05), (10,.20), (50,.05), (50,.20), (500,.05), +(500,.20). O=0 counts as non-coverage. Also report fixed-E Garwood coverage. + +Five reference rates (.0005,.001,.003,.01,.04), with equal band populations +scaled by E/sum(rates). Each draw: Poisson(1.2 E) observed count, independent +Gaussian relative band-population errors, and 80 make_replicates values with +SE=CV*true_populations. Reference rates are fixed and uncertainty-free. +Nonpositive/invalid populations or expected counts count as non-coverage, +without clipping, redrawing or changing the denominator. No spatial dependence +or between-band correlation is simulated. + +Smoke runs may use at most 20 draws per cell, without writing audits. Full +runs require the exact script and criterion in a named, pushed commit. No seed, +design, threshold or criterion may change after the full run. diff --git a/scripts/smr_validation.py b/scripts/smr_validation.py new file mode 100644 index 0000000..07abe6d --- /dev/null +++ b/scripts/smr_validation.py @@ -0,0 +1,68 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: AGPL-3.0-or-later +"""Pre-specified Round 24 SMR validation; --smoke-draws 1..20 never writes audits.""" +import argparse +import hashlib +import json +import subprocess +from pathlib import Path +import sys +ROOT=Path(__file__).resolve().parents[1] +sys.path.insert(0,str(ROOT)) +import numpy as np +import pandas as pd +from census.simulation import make_replicates +from census.standardize import indirect_standardize + +CRITERION='Combined nominal 90% interval coverage >= 0.86 in all six cells. O=0 and invalid draws count as non-coverage.' + + +def run(draws): + cells=[]; rates=np.array([.0005,.001,.003,.01,.04]) + for ci,(expected,cv) in enumerate((e,cv) for e in (10,50,500) for cv in (.05,.20)): + rng=np.random.default_rng(20261117+ci) + pop=np.repeat(expected/rates.sum(),5) # Equal populations, then scaled to E. + obs=pd.Series([0],index=['synthetic']) + frame=pd.DataFrame([pop],index=obs.index) + combined=exact=invalid=zeros=0 + for _ in range(draws): + o=int(rng.poisson(1.2*expected)); obs.iloc[0]=o; zeros+=int(o==0) + observed_pop=pop*(1+rng.normal(0,cv,5)) + reps=make_replicates(observed_pop[None,:],(cv*pop)[None,:],rng) + frame.iloc[0,:]=observed_pop + try: + row=indirect_standardize(obs,frame,rates,reps).iloc[0] + combined+=int(o>0 and row.smr_lower<=1.2<=row.smr_upper) + exact+=int(row.smr_exact_lower<=1.2<=row.smr_exact_upper) + except ValueError: invalid+=1 + c={'expected':expected,'band_cv':cv,'draws':draws,'seed':20261117+ci, + 'combined_coverage':combined/draws,'exact_coverage':exact/draws,'zero_counts':zeros,'invalid_draws':invalid} + cells.append(c); print(f"E={expected} CV={cv}: combined={combined/draws:.3f}, exact={exact/draws:.3f}",flush=True) + return {'criterion':CRITERION,'passed':all(c['combined_coverage']>=.86 for c in cells),'cells':cells} + + +def main(): + parser=argparse.ArgumentParser(description=__doc__); parser.add_argument('--smoke-draws',type=int); parser.add_argument('--criterion-commit') + args=parser.parse_args() + if args.smoke_draws is not None: + if not 1<=args.smoke_draws<=20: parser.error('smoke draws must be 1..20') + run(args.smoke_draws); return + if not args.criterion_commit: parser.error('full run requires --criterion-commit') + for rel in ('scripts/smr_validation.py','docs/audits/smr_validation_criterion.md'): + committed=subprocess.check_output(['git','show',args.criterion_commit+':'+rel],cwd=ROOT) + if committed!=(ROOT/rel).read_bytes(): raise SystemExit('criterion/script differs from pre-run commit') + report=run(2000); report['criterion_commit']=args.criterion_commit + report['script_sha256']=hashlib.sha256(Path(__file__).read_bytes()).hexdigest() + out=ROOT/'docs/audits/smr_validation'; out.with_suffix('.json').write_text(json.dumps(report,indent=2)+'\n') + lines=['# Indirect SMR validation','',CRITERION,'',f"Criterion commit: `{args.criterion_commit}`.",'', + f"Result: **{'PASS' if report['passed'] else 'FAIL — exploratory only'}**.",'', + '| Expected | Band CV | Combined coverage | Fixed-E exact coverage | O=0 | Invalid |', + '|---:|---:|---:|---:|---:|---:|'] + for c in report['cells']: + lines.append(f"| {c['expected']} | {c['band_cv']:.0%} | {c['combined_coverage']:.3f} | {c['exact_coverage']:.3f} | {c['zero_counts']} | {c['invalid_draws']} |") + lines+=['','2,000 draws per cell; true SMR 1.2, five equally populated bands scaled to the target expected count.', + 'Gaussian population errors are not clipped/redrawn. Nonpositive/invalid populations or expected counts count as non-coverage.', + 'Reference-rate uncertainty, correlated band errors and spatial dependence were not simulated; rates are treated as fixed.'] + out.with_suffix('.md').write_text('\n'.join(lines)+'\n') + +if __name__=='__main__': main() From a638270a3b415a59a007acaaf0a94bd3d6b0b964 Mon Sep 17 00:00:00 2001 From: dickbird <20980982+cafeTechne@users.noreply.github.com> Date: Sun, 15 Nov 2026 20:33:09 -0500 Subject: [PATCH 08/13] test: record prespecified indirect SMR coverage results [skip ci] --- docs/audits/smr_validation.json | 68 +++++++++++++++++++++++++++++++++ docs/audits/smr_validation.md | 20 ++++++++++ 2 files changed, 88 insertions(+) create mode 100644 docs/audits/smr_validation.json create mode 100644 docs/audits/smr_validation.md diff --git a/docs/audits/smr_validation.json b/docs/audits/smr_validation.json new file mode 100644 index 0000000..6ded346 --- /dev/null +++ b/docs/audits/smr_validation.json @@ -0,0 +1,68 @@ +{ + "criterion": "Combined nominal 90% interval coverage >= 0.86 in all six cells. O=0 and invalid draws count as non-coverage.", + "passed": true, + "cells": [ + { + "expected": 10, + "band_cv": 0.05, + "draws": 2000, + "seed": 20261117, + "combined_coverage": 0.9095, + "exact_coverage": 0.9255, + "zero_counts": 0, + "invalid_draws": 0 + }, + { + "expected": 10, + "band_cv": 0.2, + "draws": 2000, + "seed": 20261118, + "combined_coverage": 0.9035, + "exact_coverage": 0.881, + "zero_counts": 0, + "invalid_draws": 0 + }, + { + "expected": 50, + "band_cv": 0.05, + "draws": 2000, + "seed": 20261119, + "combined_coverage": 0.893, + "exact_coverage": 0.8905, + "zero_counts": 0, + "invalid_draws": 0 + }, + { + "expected": 50, + "band_cv": 0.2, + "draws": 2000, + "seed": 20261120, + "combined_coverage": 0.906, + "exact_coverage": 0.735, + "zero_counts": 0, + "invalid_draws": 1 + }, + { + "expected": 500, + "band_cv": 0.05, + "draws": 2000, + "seed": 20261121, + "combined_coverage": 0.897, + "exact_coverage": 0.7765, + "zero_counts": 0, + "invalid_draws": 0 + }, + { + "expected": 500, + "band_cv": 0.2, + "draws": 2000, + "seed": 20261122, + "combined_coverage": 0.905, + "exact_coverage": 0.335, + "zero_counts": 0, + "invalid_draws": 0 + } + ], + "criterion_commit": "680707e", + "script_sha256": "3e5d91ba54b48e14c34cbe527290df5b9a44582a06c1776d62a9dee8b1f8efa6" +} diff --git a/docs/audits/smr_validation.md b/docs/audits/smr_validation.md new file mode 100644 index 0000000..f283e05 --- /dev/null +++ b/docs/audits/smr_validation.md @@ -0,0 +1,20 @@ +# Indirect SMR validation + +Combined nominal 90% interval coverage >= 0.86 in all six cells. O=0 and invalid draws count as non-coverage. + +Criterion commit: `680707e`. + +Result: **PASS**. + +| Expected | Band CV | Combined coverage | Fixed-E exact coverage | O=0 | Invalid | +|---:|---:|---:|---:|---:|---:| +| 10 | 5% | 0.909 | 0.925 | 0 | 0 | +| 10 | 20% | 0.903 | 0.881 | 0 | 0 | +| 50 | 5% | 0.893 | 0.890 | 0 | 0 | +| 50 | 20% | 0.906 | 0.735 | 0 | 1 | +| 500 | 5% | 0.897 | 0.776 | 0 | 0 | +| 500 | 20% | 0.905 | 0.335 | 0 | 0 | + +2,000 draws per cell; true SMR 1.2, five equally populated bands scaled to the target expected count. +Gaussian population errors are not clipped/redrawn. Nonpositive/invalid populations or expected counts count as non-coverage. +Reference-rate uncertainty, correlated band errors and spatial dependence were not simulated; rates are treated as fixed. From c5d358b7641525984dc64e362a58fd01224bc405 Mon Sep 17 00:00:00 2001 From: dickbird <20980982+cafeTechne@users.noreply.github.com> Date: Sun, 15 Nov 2026 20:40:03 -0500 Subject: [PATCH 09/13] feat: register inequality and SMR manifest analyses [skip ci] --- census/inequality.py | 12 +- census/manifest/external.py | 27 ++++- census/manifest/methods.py | 158 +++++++++++++++++++++++++ census/manifest/methods_text.py | 8 +- census/manifest/run.py | 7 +- census/manifest/schema.py | 16 ++- census/manifest/validate.py | 2 + census/standardize.py | 2 +- tests/test_round24.py | 204 ++++++++++++++++++++++++++++++++ 9 files changed, 415 insertions(+), 21 deletions(-) diff --git a/census/inequality.py b/census/inequality.py index 2bb77f2..c80560b 100644 --- a/census/inequality.py +++ b/census/inequality.py @@ -18,7 +18,7 @@ "has the reverse sign of the usual income-ranked convention.") RANK_ERROR_NOTE = "Median rank CV exceeds 15%; estimates may be attenuated by rank error." # Set only from the pre-specified validation report, never from the example study. -INEQUALITY_VALIDATED = False +INEQUALITY_VALIDATED = True # docs/audits/inequality_validation.md (60330e3 criterion) def _vectors(y, x, w, hi): @@ -75,7 +75,7 @@ def _indices(y, x, w, hi, model_se=False): valid = (a > 0) & (a+b > 0) with np.errstate(divide="ignore", invalid="ignore"): log_rii = np.where(valid, np.log((a+b)/a), np.nan) - k = 2*srr/W/ym + k = np.where(ym != 0, 2*srr/W/ym, np.nan) c = k*b estimates = np.column_stack((b, log_rii, c)) if not model_se: @@ -146,7 +146,7 @@ def inequality_replicate(frame, outcome, rank_by, weights, higher_is_more_depriv mats, held = {}, {} active = [v for v in variables if v not in fixed] if replicates is not None and len(replicates): - keys = ["GEOID"] + (["Year"] if "Year" in frame else []) + keys = ["GEOID"] + (["Year"] if "Year" in frame and "Year" in replicates else []) if frame.duplicated(keys).any() or replicates.duplicated(keys+["variable"]).any(): raise ValueError("duplicate area/variable replicate keys") mats, held = _replicate_matrices(frame, replicates, active) @@ -178,10 +178,12 @@ def inequality_replicate(frame, outcome, rank_by, weights, higher_is_more_depriv notes.append("Concentration undefined: weighted outcome mean is zero.") cv = np.full(len(frame), np.nan) if rank_by+"_se" in frame: - np.divide(100*frame[rank_by+"_se"].to_numpy(float), np.abs(x), out=cv, where=x != 0) + rank_se = frame[rank_by+"_se"].to_numpy(float) + np.divide(100*rank_se, np.abs(x), out=cv, where=x != 0) + cv[(x == 0) & (rank_se > 0)] = np.inf else: cv[:] = 0 - median = float(np.nanmedian(cv)) if np.isfinite(cv).any() else float("nan") + median = float(np.nanmedian(cv)) if (~np.isnan(cv)).any() else float("nan") if median > 15: notes.append(RANK_ERROR_NOTE) coefs = _table(pd.Series(estimates[0], index=TERMS), pd.Series(model[0], index=TERMS), diff --git a/census/manifest/external.py b/census/manifest/external.py index 9364b8a..fcac714 100644 --- a/census/manifest/external.py +++ b/census/manifest/external.py @@ -4,6 +4,7 @@ from typing import Any, Dict, List, Mapping from .schema import ( _formula_names, + _normalize_analysis, _variable_tables, ) @@ -26,9 +27,18 @@ def external_circularity(manifest: Mapping[str, Any]) -> List[Dict[str, Any]]: tables = _variable_tables(manifest) out = [] for a in manifest.get("analyses") or []: - if not isinstance(a, Mapping) or "~" not in str(a.get("formula", "")): + if not isinstance(a, Mapping): + continue + if "~" in str(a.get("formula", "")): + lhs, rhs = str(a["formula"]).split("~", 1) + elif a.get("method") in ("inequality", "smr"): + a = _normalize_analysis(a) + lhs = str(a.get("outcome", a.get("observed", ""))) + bands = a.get("population_bands") + rhs = str(a.get("rank_by", " ".join(v for v in bands if isinstance(v, str)) + if isinstance(bands, list) else "")) + else: continue - lhs, rhs = str(a["formula"]).split("~", 1) ys = [y if y in ext else y[:-10] for y in _formula_names(lhs) if y in ext or (y.endswith("_numerator") and y[:-10] in ext)] if not ys: @@ -75,6 +85,9 @@ def circularity_errors(manifest: Mapping[str, Any]) -> List[str]: hyps = [h for h in manifest.get("hypotheses") or [] if isinstance(h, Mapping)] tables = _variable_tables(manifest) for aid, recs in by_analysis.items(): + analysis = next(a for a in manifest["analyses"] if a.get("id") == aid) + if analysis.get("method") in ("inequality", "smr"): + analysis = _normalize_analysis(analysis) tested = [h for h in hyps if h.get("analysis") == aid] for c in recs: if c["has_equivalent"]: @@ -89,23 +102,25 @@ def circularity_errors(manifest: Mapping[str, Any]) -> List[str]: f"and only it may be tested). {CIRCULARITY_RULE}") continue for h in tested: - term = [n for n in _formula_names(str(h.get("term", ""))) if n in tables] or [str(h.get("term"))] + tested_term = analysis.get("rank_by") if analysis.get("method") == "inequality" else h.get("term") + term = [n for n in _formula_names(str(tested_term if tested_term is not None else "")) if n in tables] or [str(tested_term)] if not all(t in c["equivalent"] for t in term): errors.append( f"hypotheses {h.get('id')}: CIRCULARITY R1: analysis {aid} compares the external " f"{c['outcome']} with its ACS equivalent {sorted(c['equivalent'])}; only that " - f"comparison may be tested, not {h.get('term')!r} (a determinant study uses the ACS " + f"comparison may be tested, not {tested_term!r} (a determinant study uses the ACS " "measure as its outcome)") if c["kind"] == "direct": continue for h in tested: - term = [n for n in _formula_names(str(h.get("term", ""))) if n in c["overlap"]] + tested_term = analysis.get("rank_by") if analysis.get("method") == "inequality" else h.get("term") + term = [n for n in _formula_names(str(tested_term if tested_term is not None else "")) if n in c["overlap"]] just = h.get("circularity_justification") if term and not (isinstance(just, str) and len(just.strip()) >= 20): fams = "; ".join(f"{t}: " + ", ".join(f"{lab} {'/'.join(tabs)}" for lab, tabs in c["overlap"][t].items()) for t in term) errors.append( - f"hypotheses {h.get('id')}: CIRCULARITY R2: the tested term {h.get('term')!r} is built from " + f"hypotheses {h.get('id')}: CIRCULARITY R2: the tested term {tested_term!r} is built from " f"inputs of the model that produced {c['outcome']} ({c['source']}, {c['kind']}: {fams}); " "its coefficient partly recovers that model, not the world. Test a direct outcome (the " "ACS measure, vital records), or add a written circularity_justification (20+ " diff --git a/census/manifest/methods.py b/census/manifest/methods.py index 9e08925..db163d2 100644 --- a/census/manifest/methods.py +++ b/census/manifest/methods.py @@ -20,6 +20,7 @@ class MethodSpec: # Stages: count/external, before_spatial, spatial, after_spatial. check: Optional[Callable[..., Any]] = None text: Optional[Tuple[str, List[str]]] = None + default_target: str = "population" def _run_fit_replicate(a, args, frame, level, replicates, correction, ctx): from .. import correlation, models, multitest, sae, trends @@ -155,6 +156,8 @@ def _check_spatial(stage, ctx): errors.append("spatial_se is supported for fit_eiv and fit_replicate binomial/poisson") def _check_fit_replicate(stage, ctx): + if stage == "hypothesis": + return from .schema import FORMULA_ID_COLUMNS, LEGACY_X2_MANIFESTS, manifest_hash from .replicates import census_vre_source i, a, manifest, names, ext_names, errors = (ctx[k] for k in ("i", "a", "manifest", "names", "ext_names", "errors")) @@ -204,6 +207,8 @@ def _check_fit_replicate(stage, ctx): _check_spatial(stage, ctx) def _check_fit_eiv(stage, ctx): + if stage == "hypothesis": + return from .schema import FORMULA_ID_COLUMNS, LEGACY_X2_MANIFESTS, manifest_hash from .replicates import census_vre_source i, a, manifest, names, ext_names, errors = (ctx[k] for k in ("i", "a", "manifest", "names", "ext_names", "errors")) @@ -239,6 +244,132 @@ def _check_fit_eiv(stage, ctx): "replicates.source 'census_vre' or \"replicates\": \"required\"") _check_spatial(stage, ctx) +def _run_inequality(a, args, frame, level, replicates, correction, ctx): + from ..inequality import inequality_replicate + external = set(ctx["external"]) + fixed = [a["outcome"]] if a["outcome"] in external else [] + if fixed and a["weights"] == a["outcome"]+"_denominator": + fixed.append(a["weights"]) + args.pop("replicate_policy", None) # library always enforces strict completeness + return inequality_replicate(frame, a["outcome"], a["rank_by"], a["weights"], + replicates=replicates, fixed=fixed, level=level, **args) + + +def _run_smr(a, args, frame, level, replicates, correction, ctx): + from ..standardize import indirect_standardize + from ..models import _replicate_matrices + from ..results import precheck_replicates, stochastic_variables, IncompleteReplicatesError + bands = a["population_bands"] + mats, held = _replicate_matrices(frame, replicates, bands) if replicates is not None else ({}, {}) + precheck_replicates("strict", rows_held_fixed=held, + variables_without_replicates=[v for v in stochastic_variables(frame, bands) if v not in mats], + context="smr") + reps = (np.stack([mats.get(v, np.broadcast_to(frame[v].to_numpy(float)[:, None], (len(frame), 80))) + for v in bands], axis=1) if mats else None) + idx = pd.Index(frame["GEOID"].astype(str)) + observed = pd.Series(frame[a["observed"]].to_numpy(float), index=idx) + populations = frame[bands].set_axis(idx) + try: + return indirect_standardize(observed, populations, a["reference_rates"], reps, level) + except ValueError as exc: + if reps is not None and "replicate" in str(exc): + raise IncompleteReplicatesError(f"smr: {exc}") from exc + raise + + +def _stat_inequality(h, res): + result = _stat_coefficient({"term": h["statistic"]}, res) + if not all(np.isfinite(result[k]) for k in ("estimate", "se", "p_two_sided", "ci_lower", "ci_upper")): + raise ValueError(f"{h['statistic']} or its uncertainty is undefined; see analysis notes") + return result + + +def _stat_smr(h, res): + row = res[res["GEOID"] == str(h["geoid"])] + if row.empty: + raise KeyError(f"GEOID {h['geoid']} not in the SMR frame") + r = row.iloc[0] + if not np.isfinite(r["log_smr"]) or not np.isfinite(r["se_log_smr"]): + raise ValueError("log SMR undefined at O=0; only the fixed-E exact interval is available") + return {"estimate": float(r["log_smr"]), "se": float(r["se_log_smr"]), "se_basis": "se_log_smr", + "p_two_sided": float(r["p_value"]), "ci_lower": float(np.log(r["smr_lower"])), + "ci_upper": float(np.log(r["smr_upper"]))} + + +def _check_inequality(stage, ctx): + if stage == "spatial": + return _check_spatial(stage, ctx) + if stage == "hypothesis": + h, errors = ctx["h"], ctx["errors"] + if "term" in h: + errors.append(f"hypotheses[{ctx['i']}] ({h.get('analysis')}): inequality names a statistic only, no term") + return + if stage != "before_spatial": + return + from ..inequality import INEQUALITY_VALIDATED + a, errors, names, external = (ctx[k] for k in ("a", "errors", "names", "ext_names")) + prefix = f"analyses[{ctx['i']}] ({a.get('id')}): " + args = a.get("args") or {} + if not INEQUALITY_VALIDATED and not ctx["manifest"].get("exploratory", False): + errors.append(prefix+"inequality failed validation; exploratory only (docs/audits/inequality_validation.md)") + if type(args.get("higher_is_more_deprived")) is not bool: + errors.append(prefix+"higher_is_more_deprived must be an explicit boolean") + if args.get("target", "conservative") not in ("population", "these_areas", "conservative"): + errors.append(prefix+"target must be population, these_areas or conservative") + unknown = sorted(set(args)-{"higher_is_more_deprived", "target"}) + if unknown: + errors.append(prefix+f"inequality does not take args {unknown}") + outcome = a.get("outcome") + for role in ("outcome", "rank_by", "weights"): + v = a.get(role) + allowed_external = (role == "outcome" and v in external) or ( + role == "weights" and outcome in external and v == str(outcome)+"_denominator") + if not isinstance(v, str) or (v not in names and not allowed_external): + errors.append(prefix+f"{role} {v!r} is not a manifest variable in this role") + if outcome in external and args.get("target") == "these_areas": + errors.append(prefix+"external outcome is held fixed; target these_areas omits its uncertainty") + + +def _check_smr(stage, ctx): + if stage == "spatial": + return _check_spatial(stage, ctx) + if stage == "hypothesis": + h, errors = ctx["h"], ctx["errors"] + g = h.get("geoid") + if not isinstance(g, str) or not g or (g != "all" and not (g.isascii() and g.isdigit())): + errors.append(f"hypotheses[{ctx['i']}] ({h.get('analysis')}): smr requires geoid (one GEOID or 'all')") + if "term" in h: + errors.append(f"hypotheses[{ctx['i']}] ({h.get('analysis')}): smr uses geoid, not term") + return + if stage != "before_spatial": + return + from ..standardize import SMR_VALIDATED + a, errors, names, external = (ctx[k] for k in ("a", "errors", "names", "ext_names")) + prefix = f"analyses[{ctx['i']}] ({a.get('id')}): " + args = a.get("args") or {} + if not SMR_VALIDATED and not ctx["manifest"].get("exploratory", False): + errors.append(prefix+"smr failed validation; exploratory only (docs/audits/smr_validation.md)") + if set(args)-{"target"}: + errors.append(prefix+f"smr does not take args {sorted(set(args)-{'target'})}") + if args.get("target", "conservative") != "conservative": + errors.append(prefix+"smr target must be conservative (combined Poisson and expected-count uncertainty)") + observed = a.get("observed") + allowed_count = isinstance(observed, str) and observed.endswith("_numerator") and observed[:-10] in external + if not isinstance(observed, str) or (observed not in names and not allowed_count): + errors.append(prefix+f"observed {observed!r} must be a manifest variable or external _numerator") + bands, rates = a.get("population_bands"), a.get("reference_rates") + if not isinstance(bands, list) or not bands or not all(isinstance(v,str) and v in names for v in bands): + errors.append(prefix+"population_bands must be a non-empty list of manifest variables") + elif len(set(bands)) != len(bands): + errors.append(prefix+"population_bands must not repeat a band") + if (not isinstance(rates, list) or not rates or not isinstance(bands, list) or len(rates) != len(bands) + or not all(type(r) in (int,float) and np.isfinite(r) and r>0 for r in rates)): + errors.append(prefix+"reference_rates must be finite positive numbers, one per population band") + source = a.get("reference_source") + if not isinstance(source, str) or not source.strip(): + errors.append(prefix+"reference_source must be a non-empty citation string") + + METHODS: Dict[str, MethodSpec] = { 'fit_replicate': MethodSpec('fit_replicate', ('formula',), ('coefficient',), _run_fit_replicate, replicate=True, external_outcome=True, check=_check_fit_replicate, text=('Coefficients were estimated with {model} regression (statsmodels) on the full-sample estimates; sampling standard errors came from refitting on each of {rep_source}, Var = 4/80 sum (b_r - b)^2, combined with the model standard error as se_total = sqrt(se_model^2 + se_sampling^2). Reported standard errors, intervals and p-values use the {target_se} ({target} target: {estimand}), with normal-theory intervals. Without replicates only model standard errors are reported.', ['vre', 'design', 'statsmodels'])), 'fit_eiv': MethodSpec('fit_eiv', ('formula',), ('coefficient',), _run_fit_eiv, replicate=True, external_outcome=True, check=_check_fit_eiv, text=('Coefficients were estimated by moment-corrected least squares ({model}; Fuller 1987) with per-unit sampling-error covariances of every design column and of the outcome, including the outcome-covariate error covariance, estimated from {rep_source} (4/80 sum over replicates). Standard errors are {se_basis} (se_total, used for normal-theory intervals and p-values); naive coefficients are reported alongside. Coefficients are {estimand}.', ['fuller', 'vre', 'design'])), @@ -249,6 +380,29 @@ def _check_fit_eiv(stage, ctx): 'correlation_matrix': MethodSpec('correlation_matrix', ('variables',), ('correlation',), _run_correlation_matrix, replicate=True, external_outcome=False, check=_check_spatial, text=('Correlations with Fisher-z intervals{rep}.', ['fisher', 'vre'])), 'linear_trend_test': MethodSpec('linear_trend_test', ('variable', 'geoid'), ('trend_slope',), _run_linear_trend_test, replicate=False, external_outcome=False, check=_check_spatial, text=('Linear trends across non-overlapping 5-year periods were tested by GLS with known sampling variances.', ['testing'])), 'batch_z_test': MethodSpec('batch_z_test', ('variable_1', 'variable_2'), ('z_difference',), _run_batch_z_test, replicate=False, external_outcome=False, check=_check_spatial, text=('Differences were tested with the Census two-sample z test, SE = MOE/1.645.', ['testing'])), + "inequality": MethodSpec("inequality", ("outcome", "rank_by", "weights"), + ("sii", "log_rii", "concentration_index"), _run_inequality, replicate=True, external_outcome=True, + check=_check_inequality, default_target="conservative", text=( + "Area inequality uses positive population weights and weighted ridits from least (0) to most (1) " + "deprived, with tied blocks assigned their weighted mid-rank; {rank_direction}. SII is the WLS " + "slope, log RII is log((a+b)/a), and concentration is 2 cov_w(y,r)/mean_w(y). Positive indices " + "mean higher outcomes among more deprived areas; concentration has the reverse sign of the " + "usual income-ranked convention. Independent-area HC3 SEs (delta method for log RII; convenient-" + "regression approximation for concentration) combine with 80 ACS replicate refits, recomputing " + "ridits, by quadrature. Target: {target}. External outcomes and their denominator weights stay " + "fixed. Median rank CV <=15% is the validated range; above it estimates may be attenuated by " + "rank error. These are area associations, not individual inequality or causal effects.", + ["pamuk", "kakwani", "vre", "statsmodels"])), + "smr": MethodSpec("smr", ("observed", "population_bands", "reference_rates", "reference_source"), + ("log_smr",), _run_smr, replicate=True, external_outcome=True, check=_check_smr, + default_target="conservative", text=( + "Indirect SMR is observed/expected, expected=sum_b population_b*reference_rate_b; pooling " + "sums counts and expected counts first. Fixed reference rates: {reference_source}. Exact " + "Garwood limits condition on expected counts. Tests use log SMR against zero, with combined " + "variance 1/O + 4/80 sum(log E_r-log E)^2 and exponentiated normal intervals (target: {target}). " + "Without band replicates the expected-count component is zero, explicitly noted; at O=0 only " + "the exact fixed-E interval is defined. External observed counts stay fixed in band refits. " + "Reference-rate uncertainty and spatial dependence are not included.", ["garwood", "breslow", "vre"])), } STATISTIC_EXTRACTORS = { @@ -256,6 +410,10 @@ def _check_fit_eiv(stage, ctx): 'correlation': _stat_correlation, 'trend_slope': _stat_trend_slope, 'z_difference': _stat_z_difference, + "sii": _stat_inequality, + "log_rii": _stat_inequality, + "concentration_index": _stat_inequality, + "log_smr": _stat_smr, } ANALYSIS_METHODS = {name: (spec.required, spec.statistics) for name, spec in METHODS.items()} diff --git a/census/manifest/methods_text.py b/census/manifest/methods_text.py index af75378..70dd5df 100644 --- a/census/manifest/methods_text.py +++ b/census/manifest/methods_text.py @@ -294,14 +294,16 @@ def _methods_analyses(m, rep_info, level, rep_short, rep_source, lines, cites): a = _normalize_analysis(a) text, refs = _METHOD_TEXT[a["method"]] args = a.get("args") or {} - tgt = args.get("target", "population") + tgt = args.get("target", METHODS[a["method"]].default_target) # the same interpretation contract as the fitted result (census.results.interval_description) probe = ModelResult(model_type="", formula=str(a.get("formula") or ""), n_obs=0, coefficients=pd.DataFrame(columns=["term"]), method=a["method"], uncertainty_sources=[], confidence_level=level, - diagnostics={"interval_target": tgt} if a["method"] == "fit_replicate" else {}) + diagnostics={"interval_target": tgt} if a["method"] in ("fit_replicate", "inequality") else {}) info = interval_description(probe) - text = text.format(model=args.get("model", "ols").upper(), target=tgt, estimand=info["estimand"], + text = text.format(reference_source=a.get("reference_source", ""), + rank_direction=("higher rank values mean more deprived" if args.get("higher_is_more_deprived") + else "lower rank values mean more deprived"), model=args.get("model", "ols").upper(), target=tgt, estimand=info["estimand"], target_se={"se_model": "model SE (se_model)", "se_sampling": "replicate sampling SE " "(se_sampling)", "se_total": "combined SE (se_total)"}[info["basis_column"]], cov=", ".join(a.get("error_covariates") or args.get("error_covariates") or []), diff --git a/census/manifest/run.py b/census/manifest/run.py index cb5c393..efc124d 100644 --- a/census/manifest/run.py +++ b/census/manifest/run.py @@ -140,7 +140,7 @@ def _run_analysis(a: Mapping[str, Any], frame: pd.DataFrame, level: float, repli if method not in METHODS: raise ValueError(f"unsupported method {method}") return METHODS[method].run(a, args, frame, level, replicates, correction, - {"held_external": held_external, "coords": coords}) + {"held_external": held_external, "coords": coords, "external": external}) def _statistic(h: Mapping[str, Any], res: Any) -> Dict[str, Any]: @@ -605,6 +605,9 @@ def run_manifest( for source_key, target in (("estimate", "odds_ratio"), ("ci_lower", "odds_ratio_lower"), ("ci_upper", "odds_ratio_upper")): row[target] = float(np.exp(row[source_key])) if row.get(source_key) is not None else None + if hyp["statistic"] in ("log_rii", "log_smr"): + for source_key, target in (("estimate", "ratio"), ("ci_lower", "ratio_lower"), ("ci_upper", "ratio_upper")): + row[target] = float(np.exp(row[source_key])) if row.get(source_key) is not None else None rows.append(row) hyp_df = pd.DataFrame(rows) hyp_df["p_adjusted"] = np.nan @@ -748,7 +751,7 @@ def _write_bundle(folder: Path, m, h, results, hyp_df, run_info, citation) -> Di figures_[f"shrinkage_{aid}"] = (figs.shrinkage_plot, {"result": r, "manifest_hash": h, "data_source": fig_source}) cols = ["id", "statistic", "direction", "estimate", "p_value", "p_adjusted", "significant", "status"] - cols += [c for c in ("odds_ratio", "odds_ratio_lower", "odds_ratio_upper") if c in hyp_df] + cols += [c for c in ("odds_ratio", "odds_ratio_lower", "odds_ratio_upper", "ratio", "ratio_lower", "ratio_upper") if c in hyp_df] if m.get("exploratory"): cols.append("analysis_status") show = hyp_df.reindex(columns=cols) diff --git a/census/manifest/schema.py b/census/manifest/schema.py index 92bd821..5961420 100644 --- a/census/manifest/schema.py +++ b/census/manifest/schema.py @@ -34,7 +34,7 @@ DATA_SOURCE_KEYS = ("synthetic", "description", "generator") STATUS_EXPLORATORY = "exploratory" -from .methods import ANALYSIS_METHODS, STATISTICS, REPLICATE_METHODS +from .methods import METHODS, ANALYSIS_METHODS, STATISTICS, REPLICATE_METHODS EXCLUSION_RULES = ("cv_screen", "geoid_exclude") ADJUSTMENT_TYPES = ("rpp",) @@ -51,6 +51,10 @@ FORMULA_FUNCTIONS = {"C", "I", "np", "log", "exp", "sqrt", "center", "standardize", "scale", "Q"} REFERENCES = { + "pamuk": "Pamuk, E.R. (1985). Social class inequality in mortality from 1921 to 1972 in England and Wales. Population Studies 39:17-31.", + "kakwani": "Kakwani, N., Wagstaff, A. & van Doorslaer, E. (1997). Socioeconomic inequalities in health: measurement, computation, and statistical inference. Journal of Econometrics 77:87-103.", + "garwood": "Garwood, F. (1936). Fiducial limits for the Poisson distribution. Biometrika 28:437-442.", + "breslow": "Breslow, N.E. & Day, N.E. (1987). Statistical Methods in Cancer Research, Vol. II. IARC Scientific Publications 82.", "acs": "U.S. Census Bureau (2020). Understanding and Using American Community Survey Data: " "What All Data Users Need to Know (ACS General Handbook), ch. 7-8.", "vre": "U.S. Census Bureau. Variance Replicate Estimate Tables Documentation. " @@ -205,13 +209,18 @@ def tables_and_refs(manifest: Mapping[str, Any]) -> Dict[str, List[str]]: _SPEC_KEYS = ("formula", "error_covariates", "variables", "variable", "geoid", "variable_1", "variable_2") +def _spec_keys(a): + spec = METHODS.get(a.get("method")) + return tuple(dict.fromkeys(_SPEC_KEYS + (spec.required if spec else ()))) + + def _normalize_analysis(a: Mapping[str, Any]) -> Dict[str, Any]: """Return the analysis with spec keys (formula, variables, ...) lifted from ``args`` to the top level, so either placement works the same way. Duplicates (a key both top-level and in args) are a validation error.""" out = dict(a) args = dict(a.get("args") or {}) - for k in _SPEC_KEYS: + for k in _spec_keys(a): if k in args and k not in out: out[k] = args.pop(k) elif k in args: @@ -222,7 +231,7 @@ def _normalize_analysis(a: Mapping[str, Any]) -> Dict[str, Any]: def _duplicate_spec_keys(a: Mapping[str, Any]) -> List[str]: args = a.get("args") or {} - return [k for k in _SPEC_KEYS if k in a and k in args] + return [k for k in _spec_keys(a) if k in a and k in args] def _formula_names(formula: str) -> List[str]: @@ -242,4 +251,3 @@ def _formula_names(formula: str) -> List[str]: # Manifest methods that use replicates; each runs with replicate_policy="strict". # (Spatial, multilevel, DiD and PUMS analyses are not manifest methods; if added # they must join this set so confirmatory runs refuse incomplete replicates.) - diff --git a/census/manifest/validate.py b/census/manifest/validate.py index 11df819..c7376d4 100644 --- a/census/manifest/validate.py +++ b/census/manifest/validate.py @@ -401,6 +401,8 @@ def _check_manifest_hypotheses(manifest, ids, errors): for k in need: if k not in h: errors.append(f"hypotheses[{i}]: {k!r} is required for statistic {h.get('statistic')!r}") + METHODS[method].check("hypothesis", {**locals(), "a": next( + _normalize_analysis(a) for a in manifest["analyses"] if a.get("id") == h.get("analysis"))}) just = h.get("circularity_justification") if just is not None and not isinstance(just, str): errors.append(f"hypotheses[{i}]: circularity_justification must be text") diff --git a/census/standardize.py b/census/standardize.py index 8fa2c69..eeaca06 100644 --- a/census/standardize.py +++ b/census/standardize.py @@ -169,7 +169,7 @@ def age_adjusted_interval(cases: Sequence[float], populations: Sequence[float], # Set only from the pre-specified simulation report. -SMR_VALIDATED = False +SMR_VALIDATED = True # docs/audits/smr_validation.md (680707e criterion) def indirect_standardize(observed, band_populations, reference_rates, band_replicates=None, level=0.90): diff --git a/tests/test_round24.py b/tests/test_round24.py index f227339..9ed9832 100644 --- a/tests/test_round24.py +++ b/tests/test_round24.py @@ -146,3 +146,207 @@ def test_smr_rejects_silent_data_loss(bad): if bad=='reserved': obs.index=pop.index=['all','b'] if bad=='collision': obs.index=pop.index=[1,'1'] with pytest.raises(ValueError): indirect_standardize(obs,pop,rates,rep) + + +def manifest(method='inequality', external=False): + from census.manifest import load_manifest + from pathlib import Path + m=load_manifest(Path(__file__).resolve().parents[1]/'examples/manifests/income_population_tracts.json') + m['authors']=['human:example-researcher'] + m['exploratory']=True + if method=='inequality': + a=dict(id='ineq',method=method,outcome='income',rank_by='pop',weights='pop', + args={'higher_is_more_deprived':True}) + hyps=[dict(id=t,analysis='ineq',statistic=t,direction='two-sided') for t in TERMS] + if external: + a.update(outcome='lbw',weights='lbw_denominator') + else: + a=dict(id='smr',method=method,observed='income',population_bands=['pop'],reference_rates=[.01], + reference_source='Synthetic fixed rate per person; no rate uncertainty') + hyps=[dict(id='h',analysis='smr',statistic='log_smr',geoid='all',direction='two-sided')] + if external: a['observed']='lbw_numerator' + m['analyses']=[a];m['hypotheses']=hyps + if external: + m['releases']=[2024] + m['variables']=[{'name':'income','ref':'2024:B19013_001'},{'name':'pop','ref':'2024:B01003_001'}] + m['geography']={'level':'county','states':['26','36']} + m['external']=[{'name':'lbw','source':'chr','release':2025,'measure':'v037'}] + return m + + +@pytest.mark.parametrize('method',['inequality','smr']) +def test_registered_metadata_and_valid_manifest(method): + from census.manifest import validate_manifest, methods_section + from census.manifest.methods import METHODS + from census.health_sources import EXTERNAL_OUTCOME_METHODS + m=manifest(method) + assert validate_manifest(m,resolve=False)==[] + assert METHODS[method].external_outcome and method in EXTERNAL_OUTCOME_METHODS + text=methods_section(m) + assert 'Target: conservative' in text if method=='inequality' else 'Synthetic fixed rate' in text + + +@pytest.mark.parametrize('method',['inequality','smr']) +def test_failed_validation_refuses_confirmatory(method,monkeypatch): + from census.manifest import validate_manifest + import census.inequality as iq + import census.standardize as st + monkeypatch.setattr(iq if method=='inequality' else st, + 'INEQUALITY_VALIDATED' if method=='inequality' else 'SMR_VALIDATED',False) + m=manifest(method); m.pop('exploratory') + assert any('failed validation; exploratory only' in e for e in validate_manifest(m,resolve=False)) + m['exploratory']=True + assert not validate_manifest(m,resolve=False) + + +@pytest.mark.parametrize('mutate,fragment',[ + (lambda a,h:a['args'].pop('higher_is_more_deprived'),'explicit boolean'), + (lambda a,h:a['args'].update(higher_is_more_deprived='true'),'explicit boolean'), + (lambda a,h:a['args'].update(target='invalid'),'target must'), + (lambda a,h:a['args'].update(scale='robust'),'does not take'), + (lambda a,h:a.update(outcome='missing'),'outcome'), + (lambda a,h:a.update(rank_by='missing'),'rank_by'), + (lambda a,h:a.update(weights='missing'),'weights'), + (lambda a,h:h.update(term='pop'),'no term'), +]) +def test_inequality_manifest_rules(mutate,fragment): + from census.manifest import validate_manifest + m=manifest(); mutate(m['analyses'][0],m['hypotheses'][0]) + errors=validate_manifest(m,resolve=False) + assert any(fragment in e and ('ineq' in e or 'analyses[0]' in e) for e in errors),errors + + +@pytest.mark.parametrize('mutate,fragment',[ + (lambda a,h:a.update(observed='missing'),'observed'), + (lambda a,h:a.update(population_bands=[]),'population_bands'), + (lambda a,h:a.update(population_bands=['missing']),'population_bands'), + (lambda a,h:a.update(population_bands=['pop','pop'],reference_rates=[.01,.02]),'repeat'), + (lambda a,h:a.update(reference_rates=[0]),'reference_rates'), + (lambda a,h:a.update(reference_rates=[float('nan')]),'reference_rates'), + (lambda a,h:a.update(reference_rates=[True]),'reference_rates'), + (lambda a,h:a.update(reference_rates=[.01,.02]),'reference_rates'), + (lambda a,h:a.update(reference_source=' '),'reference_source'), + (lambda a,h:a.update(args={'scale':'X2'}),'does not take'), + (lambda a,h:a.update(args={'target':'these_areas'}),'target must'), + (lambda a,h:h.pop('geoid'),'requires geoid'), + (lambda a,h:h.update(geoid=['36061']),'requires geoid'), + (lambda a,h:h.update(term='pop'),'not term'), +]) +def test_smr_manifest_rules(mutate,fragment): + from census.manifest import validate_manifest + m=manifest('smr'); mutate(m['analyses'][0],m['hypotheses'][0]) + errors=validate_manifest(m,resolve=False) + assert any(fragment in e and ('smr' in e or 'analyses[0]' in e) for e in errors),errors + + +@pytest.mark.parametrize('method',['inequality','smr']) +def test_reference_values_and_direction_are_hashed(method): + from census.manifest import manifest_hash + m=manifest(method); before=manifest_hash(m) + if method=='inequality': m['analyses'][0]['args']['higher_is_more_deprived']=False + else: m['analyses'][0]['reference_source']='Different standard' + assert manifest_hash(m)!=before + if method=='smr': + m=manifest(method);m['analyses'][0]['reference_rates']=[.02] + assert manifest_hash(m)!=before + + +@pytest.mark.parametrize('method',['inequality','smr']) +def test_external_roles(method): + from census.manifest import validate_manifest + m=manifest(method,True) + assert not validate_manifest(m,resolve=False) + a=m['analyses'][0] + if method=='inequality': + a['rank_by']='lbw';fragment='rank_by' + else: + a['population_bands']=['lbw'];fragment='population_bands' + assert any(fragment in e for e in validate_manifest(m,resolve=False)) + + +def test_external_inequality_fixed_target(): + from census.manifest import validate_manifest + m=manifest('inequality',True);m['analyses'][0]['args']['target']='these_areas' + assert any('omits its uncertainty' in e for e in validate_manifest(m,resolve=False)) + + +@pytest.mark.parametrize('placement',['top','args']) +def test_circularity_rank_is_tested_term(placement): + from census.manifest import validate_manifest + m=manifest('inequality',True) + m['external']=[dict(name='diabetes',source='places',release=2024,measure='DIABETES')] + m['variables']=[dict(name='rank',ref='2024:B17001_002'),dict(name='pop',ref='2024:B01003_001')] + a=m['analyses'][0];a.update(outcome='diabetes',rank_by='rank',weights='pop') + if placement=='args': + for k in ('outcome','rank_by','weights'): a['args'][k]=a.pop(k) + errors=validate_manifest(m,resolve=False) + assert any('CIRCULARITY R2' in e and "'rank'" in e for e in errors),errors + for h in m['hypotheses']: h['circularity_justification']='Sensitivity check of published model input association only.' + assert not any('CIRCULARITY R2' in e for e in validate_manifest(m,resolve=False)) + + +@pytest.mark.parametrize('method',['inequality','smr']) +def test_circularity_r1_applies_to_nonformula_outcome(method): + from census.manifest import validate_manifest + m=manifest(method,True) + m['external']=[dict(name='disability',source='places',release=2024,measure='DISABILITY')] + m['analyses'][0]['outcome' if method=='inequality' else 'observed']='disability' if method=='inequality' else 'disability_numerator' + if method=='inequality':m['analyses'][0]['weights']='pop' + assert any('CIRCULARITY R1' in e for e in validate_manifest(m,resolve=False)) + + +@pytest.mark.parametrize('method',['inequality','smr']) +def test_full_offline_manifest(method,tmp_path): + from census.manifest import run_manifest, STATUS_EXPLORATORY + from tests.test_manifest import FakeClient, synthetic_payload + m=manifest(method) + m['data_source']={'synthetic':True,'description':'Deterministic offline fixture'} + # Replicates match the API fixture estimates and years; no real network calls. + parts=[] + for v,table in [('income','B19013'),('pop','B01003')]: + rows=synthetic_payload(table) + vals=np.array([float(row[2]) for row in rows[1:]]) + ids=['36061'+row[-1] for row in rows[1:]] + part=pd.DataFrame(np.repeat(vals[:,None],80,axis=1),columns=REPLICATE_COLUMNS) + part['GEOID']=ids;part['Year']=2019;part['variable']=v;parts.append(part) + client=FakeClient();client.synthetic=True + out=run_manifest(m,tmp_path,client=client,use_cache=False,replicates=pd.concat(parts,ignore_index=True)) + assert out['success'],out.get('deviations',out.get('errors')) + assert out['analysis_status']==STATUS_EXPLORATORY + assert (out['hypotheses'].status=='tested').all() + h=out['hypotheses'].set_index('statistic') + logstat='log_rii' if method=='inequality' else 'log_smr' + assert h.loc[logstat,'ratio']==pytest.approx(np.exp(h.loc[logstat,'estimate'])) + assert any('ratio' in p.read_text() for p in (tmp_path/'bundle').rglob('*.csv')) + + +def test_external_quantities_held_fixed_in_runner(): + from census.manifest.run import _run_analysis + f, rep=frames();f=f.rename(columns={'y':'lbw','w':'lbw_denominator'}) + external_rep=rep.copy();external_rep['variable']='lbw' + external_rep.loc[:,REPLICATE_COLUMNS]=1. + reps=pd.concat([rep,external_rep],ignore_index=True) + a=dict(id='i',method='inequality',outcome='lbw',rank_by='x',weights='lbw_denominator', + args={'higher_is_more_deprived':True}) + r=_run_analysis(a,f,.90,reps,'holm',external=['lbw']) + np.testing.assert_allclose(r.coefficients.se_sampling,0,atol=1e-14) + assert set(r.diagnostics['fixed_variables'])=={'lbw','lbw_denominator'} + + +def test_undefined_log_statistic_is_not_tested(): + from census.manifest.run import _statistic + obs=pd.Series([0],index=['36061']);pop=pd.DataFrame([[100]],index=obs.index) + result=indirect_standardize(obs,pop,[.01]) + with pytest.raises(ValueError,match='undefined'): + _statistic({'statistic':'log_smr','geoid':'all'},result) + + +def test_zero_mean_concentration_is_nan(): + c=concentration_index([-1,0,1], [1,2,3], [1,1,1], True) + assert np.isnan(c['concentration_index']) + assert any('undefined' in n for n in c['notes']) + + +def test_replica_alignment_with_single_year_column(): + f,rep=frames();f['Year']=2024 + assert inequality_replicate(f,'y','x','w',True,rep).n_obs==6 From 7c6c73338c9b650dd04010f13892477d4e5cdc27 Mon Sep 17 00:00:00 2001 From: dickbird <20980982+cafeTechne@users.noreply.github.com> Date: Sun, 15 Nov 2026 20:50:11 -0500 Subject: [PATCH 10/13] docs: add inequality workflow and cached exploratory county study [skip ci] --- census/__init__.py | 4 + census/manifest/methods.py | 193 +++++++++++++++--- census/manifest/methods_text.py | 8 +- census/mcp_profiles.py | 17 +- census/recipes.py | 5 +- docs/agent_protocol.md | 25 +++ docs/recipes/external_health_outcome.md | 2 +- docs/recipes/health_inequality.md | 59 ++++++ docs/studies/r24_lbw_inequality_example.md | 43 ++++ .../manifests/r24_mi_ny_lbw_inequality.json | 101 +++++++++ mcp_server.py | 21 +- tests/test_round24.py | 23 +++ 12 files changed, 461 insertions(+), 40 deletions(-) create mode 100644 docs/recipes/health_inequality.md create mode 100644 docs/studies/r24_lbw_inequality_example.md create mode 100644 examples/manifests/r24_mi_ny_lbw_inequality.json diff --git a/census/__init__.py b/census/__init__.py index 3f8daca..b022d5d 100644 --- a/census/__init__.py +++ b/census/__init__.py @@ -199,7 +199,11 @@ from census.web_maps import web_choropleth from census.api import PRODUCTS, dataset_for_product, dataset_product +from census.inequality import ridits, slope_index, concentration_index, inequality_replicate +from census.standardize import indirect_standardize + __all__ = [ + "ridits", "slope_index", "concentration_index", "inequality_replicate", "indirect_standardize", # api "CensusAPIClient", "fetch_census_data", diff --git a/census/manifest/methods.py b/census/manifest/methods.py index db163d2..5b977c5 100644 --- a/census/manifest/methods.py +++ b/census/manifest/methods.py @@ -251,8 +251,16 @@ def _run_inequality(a, args, frame, level, replicates, correction, ctx): if fixed and a["weights"] == a["outcome"]+"_denominator": fixed.append(a["weights"]) args.pop("replicate_policy", None) # library always enforces strict completeness - return inequality_replicate(frame, a["outcome"], a["rank_by"], a["weights"], + # Manifests support complete_case only; filter before aligning replicates. + used = frame.dropna(subset=[a[k] for k in ("outcome", "rank_by", "weights")]) + result = inequality_replicate(used, a["outcome"], a["rank_by"], a["weights"], replicates=replicates, fixed=fixed, level=level, **args) + dropped = frame.loc[~frame.index.isin(used.index), "GEOID"].astype(str).tolist() + result.diagnostics["complete_case"] = {"input_rows": len(frame), "used_rows": len(used), + "dropped_geoids": dropped} + if dropped: + result.notes.append(f"Complete-case analysis excluded {len(dropped)} area row(s) with missing outcome, rank or weights.") + return result def _run_smr(a, args, frame, level, replicates, correction, ctx): @@ -260,6 +268,8 @@ def _run_smr(a, args, frame, level, replicates, correction, ctx): from ..models import _replicate_matrices from ..results import precheck_replicates, stochastic_variables, IncompleteReplicatesError bands = a["population_bands"] + input_rows = len(frame) + frame = frame.dropna(subset=[a["observed"]]+bands) mats, held = _replicate_matrices(frame, replicates, bands) if replicates is not None else ({}, {}) precheck_replicates("strict", rows_held_fixed=held, variables_without_replicates=[v for v in stochastic_variables(frame, bands) if v not in mats], @@ -270,7 +280,11 @@ def _run_smr(a, args, frame, level, replicates, correction, ctx): observed = pd.Series(frame[a["observed"]].to_numpy(float), index=idx) populations = frame[bands].set_axis(idx) try: - return indirect_standardize(observed, populations, a["reference_rates"], reps, level) + result = indirect_standardize(observed, populations, a["reference_rates"], reps, level) + if len(frame) != input_rows: + note = f"Complete-case analysis excluded {input_rows-len(frame)} area row(s) with missing observed counts or populations." + result["notes"] = result["notes"].map(lambda notes: notes+[note]) + return result except ValueError as exc: if reps is not None and "replicate" in str(exc): raise IncompleteReplicatesError(f"smr: {exc}") from exc @@ -371,38 +385,149 @@ def _check_smr(stage, ctx): METHODS: Dict[str, MethodSpec] = { - 'fit_replicate': MethodSpec('fit_replicate', ('formula',), ('coefficient',), _run_fit_replicate, replicate=True, external_outcome=True, check=_check_fit_replicate, text=('Coefficients were estimated with {model} regression (statsmodels) on the full-sample estimates; sampling standard errors came from refitting on each of {rep_source}, Var = 4/80 sum (b_r - b)^2, combined with the model standard error as se_total = sqrt(se_model^2 + se_sampling^2). Reported standard errors, intervals and p-values use the {target_se} ({target} target: {estimand}), with normal-theory intervals. Without replicates only model standard errors are reported.', ['vre', 'design', 'statsmodels'])), - 'fit_eiv': MethodSpec('fit_eiv', ('formula',), ('coefficient',), _run_fit_eiv, replicate=True, external_outcome=True, check=_check_fit_eiv, text=('Coefficients were estimated by moment-corrected least squares ({model}; Fuller 1987) with per-unit sampling-error covariances of every design column and of the outcome, including the outcome-covariate error covariance, estimated from {rep_source} (4/80 sum over replicates). Standard errors are {se_basis} (se_total, used for normal-theory intervals and p-values); naive coefficients are reported alongside. Coefficients are {estimand}.', ['fuller', 'vre', 'design'])), - 'regression_calibration': MethodSpec('regression_calibration', ('formula', 'error_covariates'), ('coefficient',), _run_regression_calibration, replicate=False, external_outcome=False, check=_check_spatial, text=('Covariates measured with known ACS sampling error ({cov}) were corrected by regression calibration with heteroscedastic known error variances; standard errors (se_total, used for normal-theory intervals and p-values) by row bootstrap re-estimating the calibration. Coefficients are {estimand}.', ['carroll'])), - 'simex': MethodSpec('simex', ('formula', 'error_covariates'), ('coefficient',), _run_simex, replicate=False, external_outcome=False, check=_check_spatial, text=('Covariates measured with known ACS sampling error ({cov}) were corrected by simulation-extrapolation (SIMEX, quadratic extrapolation) with the measurement-error jackknife variance (se_total, used for normal-theory intervals and p-values). Coefficients are {estimand}.', ['simex', 'carroll'])), - 'fh_regression': MethodSpec('fh_regression', ('formula',), ('coefficient',), _run_fh_regression, replicate=False, external_outcome=True, check=_check_spatial, text=("The outcome's known sampling variance was modelled with an area-level Fay-Herriot model (GLS coefficients, REML variance component).", ['fh'])), - 'fay_herriot': MethodSpec('fay_herriot', ('formula',), ('coefficient',), _run_fay_herriot, replicate=False, external_outcome=False, check=_check_spatial, text=('Small-area estimates are Fay-Herriot EBLUPs (REML) with Prasad-Rao/Datta-Lahiri MSE.', ['fh'])), - 'correlation_matrix': MethodSpec('correlation_matrix', ('variables',), ('correlation',), _run_correlation_matrix, replicate=True, external_outcome=False, check=_check_spatial, text=('Correlations with Fisher-z intervals{rep}.', ['fisher', 'vre'])), - 'linear_trend_test': MethodSpec('linear_trend_test', ('variable', 'geoid'), ('trend_slope',), _run_linear_trend_test, replicate=False, external_outcome=False, check=_check_spatial, text=('Linear trends across non-overlapping 5-year periods were tested by GLS with known sampling variances.', ['testing'])), - 'batch_z_test': MethodSpec('batch_z_test', ('variable_1', 'variable_2'), ('z_difference',), _run_batch_z_test, replicate=False, external_outcome=False, check=_check_spatial, text=('Differences were tested with the Census two-sample z test, SE = MOE/1.645.', ['testing'])), - "inequality": MethodSpec("inequality", ("outcome", "rank_by", "weights"), - ("sii", "log_rii", "concentration_index"), _run_inequality, replicate=True, external_outcome=True, - check=_check_inequality, default_target="conservative", text=( - "Area inequality uses positive population weights and weighted ridits from least (0) to most (1) " - "deprived, with tied blocks assigned their weighted mid-rank; {rank_direction}. SII is the WLS " - "slope, log RII is log((a+b)/a), and concentration is 2 cov_w(y,r)/mean_w(y). Positive indices " - "mean higher outcomes among more deprived areas; concentration has the reverse sign of the " - "usual income-ranked convention. Independent-area HC3 SEs (delta method for log RII; convenient-" - "regression approximation for concentration) combine with 80 ACS replicate refits, recomputing " - "ridits, by quadrature. Target: {target}. External outcomes and their denominator weights stay " - "fixed. Median rank CV <=15% is the validated range; above it estimates may be attenuated by " - "rank error. These are area associations, not individual inequality or causal effects.", - ["pamuk", "kakwani", "vre", "statsmodels"])), - "smr": MethodSpec("smr", ("observed", "population_bands", "reference_rates", "reference_source"), - ("log_smr",), _run_smr, replicate=True, external_outcome=True, check=_check_smr, - default_target="conservative", text=( - "Indirect SMR is observed/expected, expected=sum_b population_b*reference_rate_b; pooling " - "sums counts and expected counts first. Fixed reference rates: {reference_source}. Exact " - "Garwood limits condition on expected counts. Tests use log SMR against zero, with combined " - "variance 1/O + 4/80 sum(log E_r-log E)^2 and exponentiated normal intervals (target: {target}). " - "Without band replicates the expected-count component is zero, explicitly noted; at O=0 only " - "the exact fixed-E interval is defined. External observed counts stay fixed in band refits. " - "Reference-rate uncertainty and spatial dependence are not included.", ["garwood", "breslow", "vre"])), + 'fit_replicate': MethodSpec( + 'fit_replicate', ('formula',), ('coefficient',), _run_fit_replicate, + replicate=True, + external_outcome=True, + check=_check_fit_replicate, + text=( + 'Coefficients were estimated with {model} regression (statsmodels) on the full-sample estimates; ' + 'sampling standard errors came from refitting on each of {rep_source}, Var = 4/80 sum (b_r - ' + 'b)^2, combined with the model standard error as se_total = sqrt(se_model^2 + se_sampling^2). ' + 'Reported standard errors, intervals and p-values use the {target_se} ({target} target: ' + '{estimand}), with normal-theory intervals. Without replicates only model standard errors are ' + 'reported.', + ['vre', 'design', 'statsmodels'], + ), + ), + 'fit_eiv': MethodSpec( + 'fit_eiv', ('formula',), ('coefficient',), _run_fit_eiv, + replicate=True, + external_outcome=True, + check=_check_fit_eiv, + text=( + 'Coefficients were estimated by moment-corrected least squares ({model}; Fuller 1987) with ' + 'per-unit sampling-error covariances of every design column and of the outcome, including the ' + 'outcome-covariate error covariance, estimated from {rep_source} (4/80 sum over replicates). ' + 'Standard errors are {se_basis} (se_total, used for normal-theory intervals and p-values); naive ' + 'coefficients are reported alongside. Coefficients are {estimand}.', + ['fuller', 'vre', 'design'], + ), + ), + 'regression_calibration': MethodSpec( + 'regression_calibration', ('formula', 'error_covariates'), ('coefficient',), _run_regression_calibration, + replicate=False, + external_outcome=False, + check=_check_spatial, + text=( + 'Covariates measured with known ACS sampling error ({cov}) were corrected by regression ' + 'calibration with heteroscedastic known error variances; standard errors (se_total, used for ' + 'normal-theory intervals and p-values) by row bootstrap re-estimating the calibration. ' + 'Coefficients are {estimand}.', + ['carroll'], + ), + ), + 'simex': MethodSpec( + 'simex', ('formula', 'error_covariates'), ('coefficient',), _run_simex, + replicate=False, + external_outcome=False, + check=_check_spatial, + text=( + 'Covariates measured with known ACS sampling error ({cov}) were corrected by ' + 'simulation-extrapolation (SIMEX, quadratic extrapolation) with the measurement-error jackknife ' + 'variance (se_total, used for normal-theory intervals and p-values). Coefficients are {estimand}.', + ['simex', 'carroll'], + ), + ), + 'fh_regression': MethodSpec( + 'fh_regression', ('formula',), ('coefficient',), _run_fh_regression, + replicate=False, + external_outcome=True, + check=_check_spatial, + text=( + "The outcome's known sampling variance was modelled with an area-level Fay-Herriot model (GLS " + 'coefficients, REML variance component).', + ['fh'], + ), + ), + 'fay_herriot': MethodSpec( + 'fay_herriot', ('formula',), ('coefficient',), _run_fay_herriot, + replicate=False, + external_outcome=False, + check=_check_spatial, + text=( + 'Small-area estimates are Fay-Herriot EBLUPs (REML) with Prasad-Rao/Datta-Lahiri MSE.', + ['fh'], + ), + ), + 'correlation_matrix': MethodSpec( + 'correlation_matrix', ('variables',), ('correlation',), _run_correlation_matrix, + replicate=True, + external_outcome=False, + check=_check_spatial, + text=( + 'Correlations with Fisher-z intervals{rep}.', + ['fisher', 'vre'], + ), + ), + 'linear_trend_test': MethodSpec( + 'linear_trend_test', ('variable', 'geoid'), ('trend_slope',), _run_linear_trend_test, + replicate=False, + external_outcome=False, + check=_check_spatial, + text=( + 'Linear trends across non-overlapping 5-year periods were tested by GLS with known sampling ' + 'variances.', + ['testing'], + ), + ), + 'batch_z_test': MethodSpec( + 'batch_z_test', ('variable_1', 'variable_2'), ('z_difference',), _run_batch_z_test, + replicate=False, + external_outcome=False, + check=_check_spatial, + text=( + 'Differences were tested with the Census two-sample z test, SE = MOE/1.645.', + ['testing'], + ), + ), + 'inequality': MethodSpec( + "inequality", ("outcome", "rank_by", "weights"), ("sii", "log_rii", "concentration_index"), _run_inequality, + replicate=True, + external_outcome=True, + check=_check_inequality, + default_target="conservative", + text=( + 'Area inequality uses positive population weights and weighted ridits from least (0) to most (1) ' + 'deprived, with tied blocks assigned their weighted mid-rank; {rank_direction}. SII is the WLS ' + 'slope, log RII is log((a+b)/a), and concentration is 2 cov_w(y,r)/mean_w(y). Positive indices ' + 'mean higher outcomes among more deprived areas; concentration has the reverse sign of the usual ' + 'income-ranked convention. Independent-area HC3 SEs (delta method for log RII; ' + 'convenient-regression approximation for concentration) combine with 80 ACS replicate refits, ' + 'recomputing ridits, by quadrature. Target: {target}. External outcomes and their denominator ' + 'weights stay fixed. Median rank CV <=15% is the validated range; above it estimates may be ' + 'attenuated by rank error. These are area associations, not individual inequality or causal ' + 'effects.', + ['pamuk', 'kakwani', 'vre', 'statsmodels'], + ), + ), + 'smr': MethodSpec( + "smr", ("observed", "population_bands", "reference_rates", "reference_source"), ("log_smr",), _run_smr, + replicate=True, + external_outcome=True, + check=_check_smr, + default_target="conservative", + text=( + 'Indirect SMR is observed/expected, expected=sum_b population_b*reference_rate_b; pooling sums ' + 'counts and expected counts first. Fixed reference rates: {reference_source}. Exact Garwood ' + 'limits condition on expected counts. Tests use log SMR against zero, with combined variance 1/O ' + '+ 4/80 sum(log E_r-log E)^2 and exponentiated normal intervals (target: {target}). Without band ' + 'replicates the expected-count component is zero, explicitly noted; at O=0 only the exact fixed-E' + ' interval is defined. External observed counts stay fixed in band refits. Reference-rate ' + 'uncertainty and spatial dependence are not included.', + ['garwood', 'breslow', 'vre'], + ), + ), } STATISTIC_EXTRACTORS = { diff --git a/census/manifest/methods_text.py b/census/manifest/methods_text.py index 70dd5df..fba5759 100644 --- a/census/manifest/methods_text.py +++ b/census/manifest/methods_text.py @@ -36,7 +36,13 @@ from .methods import METHODS -_METHOD_TEXT = {name: spec.text for name, spec in sorted(METHODS.items(), key=lambda item: {'fit_replicate': 0, 'regression_calibration': 1, 'fit_eiv': 2, 'simex': 3, 'fh_regression': 4, 'fay_herriot': 5, 'correlation_matrix': 6, 'linear_trend_test': 7, 'batch_z_test': 8}.get(item[0], len(METHODS))) if spec.text is not None} +# Retain the legacy introspection order while deriving every value from its spec. +_LEGACY_TEXT_ORDER = ("fit_replicate", "regression_calibration", "fit_eiv", "simex", "fh_regression", + "fay_herriot", "correlation_matrix", "linear_trend_test", "batch_z_test") +_METHOD_TEXT = { + name: METHODS[name].text for name in dict.fromkeys((*_LEGACY_TEXT_ORDER, *METHODS)) + if METHODS[name].text is not None +} def _age_standardized_text(d: Mapping[str, Any]) -> str: diff --git a/census/mcp_profiles.py b/census/mcp_profiles.py index 0e9c15e..fdb39d5 100644 --- a/census/mcp_profiles.py +++ b/census/mcp_profiles.py @@ -78,7 +78,7 @@ "get_recipe": ( "Return numbered steps for a standard analysis. Call this first when unsure what to do.\n" "name: compare_counties | tract_trend | regression_sampling_error | small_area | preregistered | " - "decennial_counts; " + "decennial_counts | external_health_outcome | health_inequality; " "omit to list recipes.\n" "Example: get_recipe(name=\"compare_counties\")" ), @@ -200,6 +200,21 @@ 6. On failure read error_code and fix; do not guess numbers. 7. Health outcomes: use the ACS's own measure (disability: B18101); use CDC PLACES only for measures the ACS lacks, such as diabetes (get_recipe first). +8. Area inequality and standardized ratios: get_recipe('health_inequality'), then + existing manifest tools. Inequality statistics sii/log_rii/concentration_index + have no term; declare higher_is_more_deprived and positive weights. Ridits run + least to most deprived; positive indices mean higher outcomes among more deprived + areas, reversing the usual income-ranked concentration sign. Default conservative + target combines HC3 and ACS replicates; median rank CV <=15% is validated, above + which warn about attenuation. External outcome/denominator weights stay fixed; + R2 tests rank_by. HC3 assumes independent areas. SMR requires observed integer + counts, distinct population_bands, positive reference_rates in events per person + and a hashed reference_source citation; log_smr hypotheses use geoid or all. + Only conservative combined Poisson/replicate log intervals drive SMR tests; + fixed-E Garwood intervals omit expected uncertainty. O=0 has only exact limits; + absent replicates set expected uncertainty to zero, noted. Rates are fixed. + Log statistics test zero and report exponentiated ratio columns. R1 applies + to external outcomes/counts; reused MI/NY LBW associations remain exploratory. """ REVIEW_INSTRUCTIONS = """\ diff --git a/census/recipes.py b/census/recipes.py index 9523ba0..c1e139a 100644 --- a/census/recipes.py +++ b/census/recipes.py @@ -9,6 +9,7 @@ RECIPE_DIR = os.path.join(ROOT, "docs", "recipes") RECIPES: Dict[str, str] = { + "health_inequality": "Area health inequality indices and indirectly standardized ratios with ACS uncertainty", "compare_counties": "Compare a rate between two counties in one release", "tract_trend": "Trend in a tract or county rate using non-overlapping releases", "regression_sampling_error": "Regression whose SEs include ACS sampling error", @@ -21,7 +22,7 @@ RecipeName = Literal[ "compare_counties", "tract_trend", "regression_sampling_error", "small_area", "preregistered", - "decennial_counts", "external_health_outcome", + "decennial_counts", "external_health_outcome", "health_inequality", ] @@ -30,7 +31,7 @@ def get_recipe(name: Optional[RecipeName] = None) -> Dict[str, Any]: Args: name: compare_counties, tract_trend, regression_sampling_error, - small_area, preregistered, decennial_counts, or external_health_outcome. + small_area, preregistered, decennial_counts, external_health_outcome, or health_inequality. Example: get_recipe(name="compare_counties") """ diff --git a/docs/agent_protocol.md b/docs/agent_protocol.md index f4aa488..4286305 100644 --- a/docs/agent_protocol.md +++ b/docs/agent_protocol.md @@ -237,3 +237,28 @@ TIGER representative points. The pre-set Round 22 simulation **failed** its coverage criterion (`docs/audits/spatial_hac_validation.md`): `spatial_se` is available only in exploratory manifests, direct fits and `run_variant`, never confirmatory manifests. `cluster` and `spatial_se` are mutually exclusive. + +## Inequality and standardized ratios + +Use manifest `inequality` for area-level SII, log RII and concentration. Declare +`outcome`, `rank_by`, positive `weights`, and the boolean +`args.higher_is_more_deprived`; hypotheses name only `sii`, `log_rii` or +`concentration_index` (no term). Ridits run least to most deprived: positive +indices mean higher outcomes among more deprived areas. Concentration therefore +has the reverse sign of the usual income-ranked convention. External outcomes +and their `_denominator` weights stay fixed in refits. Default target is +`conservative` (HC3 plus ACS replicate uncertainty); median rank CV <=15% is the +validated range. Above it, warn about attenuation from rank error. Area HC3 +assumes independent areas. Circularity R2 tests the ranking variable. + +Manifest `smr` requires integer `observed` counts, distinct `population_bands`, +finite positive `reference_rates` in events per person, and a non-empty +`reference_source` citation. Rates and source are hashed. Hypotheses use +`log_smr`, with one `geoid` or `"all"` for pooled counts/expected counts. Log +statistics test zero; ratio columns exponentiate estimates and limits. SMR tests +use the combined Poisson/replicate log interval (`target: conservative` only), +while Garwood limits condition on expected counts. Zero observed counts have +only an exact fixed-E interval; rate uncertainty is excluded. No band replicates +means zero expected-count uncertainty, explicitly noted. See recipe +`health_inequality` and the two fixed simulation audits. Reused MI/NY LBW +associations are exploratory. diff --git a/docs/recipes/external_health_outcome.md b/docs/recipes/external_health_outcome.md index 865c103..47a3867 100644 --- a/docs/recipes/external_health_outcome.md +++ b/docs/recipes/external_health_outcome.md @@ -12,7 +12,7 @@ Goal: a pre-registered study whose outcome comes from outside the ACS, without c - USALEEP: `{"source": "usaleep", "release": 2018, "measure": "E0"}`. It is on 2010 tracts: use ACS 2011-2019 releases, or add `"apportion_to": 2020` (population-weighted, an approximation). Fully predicted tracts (flag 2) are dropped unless `"include_predicted": true`. - County Health Rankings: `{"source": "chr", "release": 2025, "measure": "v127"}` at county level. Non-commercial research use; cite CHR&R. - WONDER: export the county table yourself from wonder.cdc.gov (the engine never automates the form), then `{"source": "wonder", "measure": "age_adjusted_rate", "file": "path/to/export.txt", "sha256": ""}`. Tick the age-adjusted rate standard error. Suppressed counts stay censored (1-9), never zero. -3. Use the external variable only as the outcome of `fit_eiv` (held fixed; ACS covariates corrected for sampling error), `fit_replicate` or `fh_regression`. For tract PLACES outcomes, cluster by county: `"args": {"cluster": "GEOID.str[:5]"}` (PLACES shares county random effects). +3. Use the external variable only as the outcome of `fit_eiv` (held fixed; ACS covariates corrected for sampling error), `fit_replicate` or `fh_regression`; for area inequality or indirect ratios, follow the `health_inequality` recipe (`inequality` / `smr`). For tract PLACES outcomes, cluster by county: `"args": {"cluster": "GEOID.str[:5]"}` (PLACES shares county random effects). 4. `create_manifest`: errors **R1** (same construct as an ACS measure) and **R2** (tested term from the model's inputs) block. Read the notes: they list overlapping controls and the rule. Commit the manifest alone, then `run_manifest`. 5. In the report, state the source's model (methods.md does), its uncertainty type, the join counts in `run.json` (`external`), and that within-county PLACES variation is composition only. diff --git a/docs/recipes/health_inequality.md b/docs/recipes/health_inequality.md new file mode 100644 index 0000000..571d691 --- /dev/null +++ b/docs/recipes/health_inequality.md @@ -0,0 +1,59 @@ +# Area health inequality and indirectly standardized ratios + +1. Choose an area-level question. Register a manifest before inspecting associations; + use `exploratory: true` when reusing an outcome association already examined. + These indices describe area associations, not individual inequality or causal effects. +2. For `inequality`, declare `outcome`, `rank_by`, positive population `weights`, and + `args.higher_is_more_deprived` explicitly. Rank areas from least to most deprived; + tied values share their weighted mid-ridit. External outcomes are allowed, with + `weights: "_denominator"` for their published denominator (for example live births). + + ```json + {"id":"ineq","method":"inequality","outcome":"lbw","rank_by":"pov_rate", + "weights":"lbw_denominator","args":{"higher_is_more_deprived":true,"target":"conservative"}} + ``` + +3. Hypotheses name `sii`, `log_rii`, or `concentration_index`, without a `term`. + SII is the WLS slope on ridit; log RII is log((intercept+slope)/intercept), + undefined if either fitted endpoint is nonpositive. Positive indices mean higher + outcomes among more deprived areas. **Concentration has the reverse sign of the + usual income-ranked convention.** Outcome proportions give SII in proportion units; + multiply SII and both limits by 100 for percentage points. Exponentiate log RII + and its limits for the relative index (`ratio` columns do this). +4. Declare ACS VRE replicates for noisy ranks, weights or outcomes. Every refit + recomputes weighted ridits; declared external outcomes and external denominators + stay fixed. Missing stochastic-variable/area replicates are refused. Population + intervals use HC3; `these_areas` uses sampling SE only; default `conservative` + combines both by quadrature. External outcomes cannot use `these_areas`. + Median rank CV <=15% is the validated range. Above it, estimates may be + attenuated by rank error; wider intervals do not correct that bias. HC3 assumes + independent areas and does not cover spatial residual dependence. +5. For `smr`, use observed **integer counts**, `population_bands` (distinct manifest + variables), positive `reference_rates` in **events per person**, in the same band + order, and a non-empty `reference_source` citation. Convert rates per 100,000 + to per person before registration. Source and rates are hashed with the manifest. + + ```json + {"id":"smr","method":"smr","observed":"deaths_numerator", + "population_bands":["pop_young","pop_old"],"reference_rates":[0.001,0.02], + "reference_source":"Named standard, year, age bands; annual deaths per person", + "args":{"target":"conservative"}} + ``` + +6. Test `statistic: "log_smr"` with `geoid` equal to one GEOID or `"all"` (pool + counts and expected counts first). The null is log SMR=0; ratio columns show + SMR and its combined limits. Exact Garwood limits condition on fixed expected + counts. Tests use Poisson variance plus ACS expected-count replicate variance; + only `target: "conservative"` is supported. Without replicates, expected-count + uncertainty is zero and explicitly noted. At zero observed counts, the combined + test is undefined; report the exact fixed-E interval only. Reference-rate + uncertainty is not included. No age-specific real death-count example is bundled; + WONDER exports remain manual. +7. Circularity R1 applies to external outcomes/counts. R2 treats `rank_by` as the + tested term for inequality; model-input overlap requires the existing written + justification. Read the methods, diagnostics and join counts, and report + unadjusted interval confidence separately from multiplicity-adjusted tests. + +Reproducible validation: `docs/audits/inequality_validation.md` and +`docs/audits/smr_validation.md`. Exploratory MI/NY example: +`examples/manifests/r24_mi_ny_lbw_inequality.json`. diff --git a/docs/studies/r24_lbw_inequality_example.md b/docs/studies/r24_lbw_inequality_example.md new file mode 100644 index 0000000..d36da20 --- /dev/null +++ b/docs/studies/r24_lbw_inequality_example.md @@ -0,0 +1,43 @@ +# MI/NY county low birth weight inequality (exploratory) + +Author self-review, not independent. This is exploratory because Round 22 already +examined poverty and low birth weight; it is not a new confirmatory test. +Manifest: `examples/manifests/r24_mi_ny_lbw_inequality.json`, SHA-256 +`8d1008320c941ab468bf6fcff5319a9d7f43ffc050eec05807d7b0b16b25ad15`. + +CHR&R 2025 v037 low birth weight proportions (NCHS natality 2017–2023) were +ranked by ACS 2018–2022 county poverty percentage (B17001). Weights are live +births, not county counts. The run used existing Census/CHR/VRE caches with +network requests blocked. Of 145 MI/NY counties extracted, 144 had complete +outcome/rank/weight data; one missing CHR outcome/denominator was excluded +(GEOID `26083`), recorded in the analysis diagnostics. +Missing values were not treated as zero. + +| Index | Estimate | Unadjusted 90% conservative interval | +|:---|---:|:---| +| SII (percentage points) | 1.892 | [-0.103, 3.887] | +| log RII | 0.2244 | [-0.0037, 0.4526] | +| RII (exponentiated) | 1.2516 | [0.9963, 1.5723] | +| Concentration index | 0.0371 | [-0.0020, 0.0763] | + +Ridits run from least to most deprived, sharing the weighted mid-rank for ties. +Positive indices mean higher LBW among more deprived areas. **Concentration has +the reverse sign of the usual income-ranked convention.** SII is a fitted +ridit-endpoint difference, not an observed richest/poorest county contrast. +Median poverty-rank CV was **6.63%**, within the <=15% range of the fixed +simulation validation. HC3 model SEs and 80 ACS poverty replicate refits were +combined by quadrature; birth outcomes and birth denominators stayed fixed. +All three intervals include their null (0 for indices, 1 for RII). These +unadjusted intervals are not simultaneous intervals; the manifest separately +applies Holm adjustment to its three hypothesis p-values. + +This is a joint MI/NY birth-weighted area gradient without state adjustment or +individual-level interpretation. It differs from the state-adjusted Round 22 +regressions. Independent-area HC3 does not account for spatial dependence; +rank error can attenuate gradients, and a conservative interval does not +correct that bias. Reference-rate uncertainty is irrelevant to this example: +no SMR was fitted. No age-specific death counts were fabricated. + +Reproduction: run the manifest with the existing caches and +`CENSUS_HEALTH_OFFLINE=1`; outputs are release artifacts, not committed datasets. +Local output archive: `/tmp/round24-study-outputs.tar.gz` (not uploaded). diff --git a/examples/manifests/r24_mi_ny_lbw_inequality.json b/examples/manifests/r24_mi_ny_lbw_inequality.json new file mode 100644 index 0000000..8907c19 --- /dev/null +++ b/examples/manifests/r24_mi_ny_lbw_inequality.json @@ -0,0 +1,101 @@ +{ + "schema_version": 1, + "releases": [ + 2022 + ], + "geography": { + "level": "county", + "states": [ + "26", + "36" + ] + }, + "missing": "complete_case", + "confidence_level": 0.9, + "correction": { + "family": "all", + "method": "holm" + }, + "external": [ + { + "measure": "v037", + "name": "lbw", + "release": 2025, + "source": "chr" + } + ], + "replicates": { + "source": "census_vre" + }, + "title": "Exploratory MI/NY county low birth weight inequality by poverty rank", + "authors": [ + "human:cafeTechne" + ], + "created": "2026-11-15T00:00:00Z", + "exploratory": true, + "variables": [ + { + "name": "pov_total", + "ref": "2022:B17001_001" + }, + { + "name": "pov_below", + "ref": "2022:B17001_002" + } + ], + "derivations": [ + { + "as_percent": true, + "den": "pov_total", + "name": "pov_rate", + "num": "pov_below", + "op": "proportion" + } + ], + "exclusions": [ + { + "rule": "cv_screen", + "variables": [ + "pov_rate" + ], + "threshold": 15, + "action": "flag" + } + ], + "analyses": [ + { + "id": "ineq", + "method": "inequality", + "outcome": "lbw", + "rank_by": "pov_rate", + "weights": "lbw_denominator", + "args": { + "higher_is_more_deprived": true, + "target": "conservative" + } + } + ], + "hypotheses": [ + { + "id": "H1", + "analysis": "ineq", + "statistic": "sii", + "direction": "two-sided", + "statement": "Exploratory area association; prior Round 22 poverty/LBW association already examined." + }, + { + "id": "H2", + "analysis": "ineq", + "statistic": "log_rii", + "direction": "two-sided", + "statement": "Exploratory area association; prior Round 22 poverty/LBW association already examined." + }, + { + "id": "H3", + "analysis": "ineq", + "statistic": "concentration_index", + "direction": "two-sided", + "statement": "Exploratory area association; prior Round 22 poverty/LBW association already examined." + } + ] +} diff --git a/mcp_server.py b/mcp_server.py index 4dca1ff..afb3b33 100644 --- a/mcp_server.py +++ b/mcp_server.py @@ -125,7 +125,26 @@ are direct death records. create_manifest refuses violations (CIRCULARITY R1/R2). Use exploratory:true to label an exploratory manifest; deviations remain recorded. Spatial_se (Bartlett Conley) failed validation: exploratory only, including run_variant. -Count outcomes: binomial trials=_denominator and scale:"X2"; Poisson counts use exposure. +Count outcomes: binomial trials=_denominator and scale:"robust"; Poisson counts use exposure +and scale:"robust". New external-count manifests refuse scale:"X2" (failed coverage validation). + +Inequality and standardized ratios (get_recipe('health_inequality'); existing manifest tools): +method inequality requires outcome/rank_by/positive weights and args.higher_is_more_deprived +(boolean); hypotheses name sii/log_rii/concentration_index only, without term. Ridits run +least to most deprived; positive indices mean higher outcomes among more deprived areas. +Concentration has the reverse sign of the usual income-ranked convention. Default target +conservative combines HC3 and ACS replicate SEs; median rank CV <=15% is validated, above +which warn about rank-error attenuation. External outcomes and their denominator weights +stay fixed; target these_areas is refused for external outcomes. R2 tests rank_by. +method smr requires observed integer counts, distinct population_bands, positive reference_rates +(events per person, in band order), and reference_source (hashed citation). Hypothesis log_smr +uses geoid (one GEOID or all); pooled ratios sum observed and expected counts first. +Log statistics test zero; ratio columns exponentiate estimates and limits. SMR target is +conservative only: tests combine Poisson and expected-count replicate log variance; exact +Garwood limits condition on expected counts. No replicates means zero expected uncertainty, +noted; O=0 has only the exact fixed-E interval. Reference-rate uncertainty is excluded. +Area HC3 assumes independence. R1 applies to external outcomes/counts. Reused LBW associations +are exploratory. No new tools are needed. Inflation: default index is R-CPI-U-RS (the Census Bureau's ACS index); 'c-cpi-u' diff --git a/tests/test_round24.py b/tests/test_round24.py index 9ed9832..1881731 100644 --- a/tests/test_round24.py +++ b/tests/test_round24.py @@ -350,3 +350,26 @@ def test_zero_mean_concentration_is_nan(): def test_replica_alignment_with_single_year_column(): f,rep=frames();f['Year']=2024 assert inequality_replicate(f,'y','x','w',True,rep).n_obs==6 + + +def test_manifest_complete_case_exclusions_are_recorded(): + from census.manifest.run import _run_analysis + f,rep=frames();f.loc[0,'y']=np.nan + a=dict(id='i',method='inequality',outcome='y',rank_by='x',weights='w',args={'higher_is_more_deprived':True}) + r=_run_analysis(a,f,.9,rep,'holm') + assert r.n_obs==5 and r.diagnostics['complete_case']['dropped_geoids']==['a'] + assert any('excluded 1' in n for n in r.notes) + f['observed']=[np.nan,10,20,30,40,50] + b=dict(id='s',method='smr',observed='observed',population_bands=['w'],reference_rates=[.1],reference_source='test') + s=_run_analysis(b,f,.9,None,'holm') + assert set(s.GEOID)==set('bcdef')|{'all'} + assert any('excluded 1' in n for n in s.iloc[-1].notes) + + +def test_public_exports_and_recipe(): + import census + from census.recipes import get_recipe + for name in ('ridits','slope_index','concentration_index','inequality_replicate','indirect_standardize'): + assert callable(getattr(census,name)) and name in census.__all__ + recipe=get_recipe('health_inequality') + assert recipe['success'] and 'reverse sign' in recipe['steps'] and 'reference_source' in recipe['steps'] From ba10da8d7ddc00b30b5c3c3f36fedb5b81ab30f3 Mon Sep 17 00:00:00 2001 From: dickbird <20980982+cafeTechne@users.noreply.github.com> Date: Sun, 15 Nov 2026 21:13:16 -0500 Subject: [PATCH 11/13] chore: finalize Round 24 validation status and version 0.22.0 [skip ci] --- CHANGELOG.md | 19 +++++++++++++++++++ CITATION.cff | 2 +- PLAN_ROUND_24.md | 28 ++++++++++++++++++---------- STATUS.md | 14 ++++++++++---- census/__init__.py | 2 +- census/inequality.py | 8 +++++--- census/manifest/external.py | 6 +++--- census/mcp_profiles.py | 1 + census/pipeline.py | 2 +- census/standardize.py | 4 +++- docs/agent_protocol.md | 4 +++- docs/audits/smr_validation.md | 2 ++ docs/recipes/health_inequality.md | 4 +++- mcp_server.py | 1 + tests/test_round24.py | 27 +++++++++++++++++++++++++++ 15 files changed, 98 insertions(+), 26 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2fe12d1..a72a18e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,25 @@ each tagged release (`v`) is archived on Zenodo with its own DOI (see [Releasing](#releasing)). Plans and acceptance criteria for each round are in `PLAN_ROUND_.md`. +## 0.22.0 (Round 24, unreleased): method registry and public-health inequality + +- Centralize manifest execution, method-specific validation, statistics and methods text in + `MethodSpec`; preserve all 478 refactor snapshots and the existing public constant ordering. +- Add weighted area ridits, SII, log RII and concentration with independent-area HC3 and strict + ACS replicate uncertainty; concentration ranks from least to most deprived (the reverse of + the usual income-ranked sign). Warn about rank-error attenuation above median CV 15%. +- Add indirect SMR, pooled counts/expected counts, fixed-E Garwood intervals and combined + Poisson/ACS log intervals; require explicit per-person reference rates and a hashed citation. + Negative ACS band replicates are refused under the fixed validation domain, even with positive E. +- Both pre-specified validations passed: inequality conservative coverage 0.876–0.905 in + required CV<=15% cells; SMR combined coverage 0.893–0.909. High-error inequality cells + remain outside the validated range (coverage as low as 0.638 at rank CV 30%). +- Register both manifest methods without adding MCP tools (63). Extend circularity to ranked + external outcomes, add log-scale ratio columns, explicit complete-case exclusion records, + public library exports and the `health_inequality` recipe across agent guidance. +- Cache-only exploratory MI/NY LBW example: 144 complete counties, rank CV 6.63%; all + conservative intervals include their null. Outputs remain a separate local archive. + ## 0.21.1 (Round 23): internal package split Internal: `census.manifest` split into a package (no behaviour change; golden snapshot identical) diff --git a/CITATION.cff b/CITATION.cff index 518a17f..a23ad26 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -12,7 +12,7 @@ authors: - alias: cafeTechne license: AGPL-3.0-or-later repository-code: "https://github.com/cafeTechne/Census-Research-Engine" -version: "0.21.1" +version: "0.22.0" doi: "10.5281/zenodo.23925301" # doi: Zenodo concept DOI, added after the first archived release (see CHANGELOG.md "Releasing"). date-released: "2026-11-15" diff --git a/PLAN_ROUND_24.md b/PLAN_ROUND_24.md index 2fec668..9ad73e7 100644 --- a/PLAN_ROUND_24.md +++ b/PLAN_ROUND_24.md @@ -217,16 +217,24 @@ Rules (implement as the methods' `check` callbacks; error texts are yours, but e | Item | Status | Notes | | :--- | :--- | :--- | | Baseline | Pass | 478 snapshots; 1634 passed, 37 skipped, 48 subtests. | -| A: registry, golden identical | Pass | 478 snapshots byte-identical; isolated full suite unchanged (1634/37). | -| A: remaining `method ==` lines | Pass | None in run.py/validate.py. Historical public tuple/text order is retained by registry projections. | -| B1/B2 library | | | -| Inequality validation (criterion commit, result) | | | -| SMR validation (criterion commit, result) | | | -| B4 manifest methods, R2 extension | | | -| Golden diff after B | | | -| C docs, example run | | | -| Final suite | | | -| Exceptions (0.7) | Recorded | Shared checkout switched externally during work; implementation isolated at /tmp/census-round24. Baseline repeated there. | +| A: registry, golden identical | Pass | 478 byte-identical snapshots; isolated full suite 1634/37. | +| A: remaining `method ==` lines | Pass | None in run.py/validate.py; historical tuple/text ordering preserved. | +| B1/B2 library | Pass | Weighted indices/strict refits; keyed SMR/Garwood/combined intervals. | +| Inequality validation (criterion commit, result) | Pass | 60330e3 pushed pre-run; required coverage 0.876–0.905; CV30% informational. | +| SMR validation (criterion commit, result) | Pass | 680707e pushed pre-run; combined coverage 0.893–0.909. | +| B4 manifest methods, R2 extension | Pass | Registry callbacks, rank R2/R1, ratios, failed-validation refusal tested. | +| Golden diff after B | Pass | Legacy 478: 48 supported-method-list message leaves changed; all else identical. Final: 516 snapshots including the new example. | +| C docs, example run | Pass | HTTP blocked; 144/145 counties, rank CV 6.63%; docs/MCP mirrored (63 tools); scanned local archive. | +| Final suite | Pass | 1703 passed, 37 skipped, 49 subtests (69 new tests); 4 unchanged warnings. Both audit replays exact. 0.22.0 cache-only example reproduces scientific results. | +| Exceptions (0.7) | Recorded | External branch switch required /tmp/census-round24. Registry default_target metadata added; SMR conservative only. Complete-case integration finished in commit 8; final review added three edge-case tests. | + +Operational choices: binomial denominators use `max(1, rint(w/50))`; SMR has equal band +populations before scaling. Invalid draws count as non-coverage. SMR's combined-test requirement +means conservative target only. Library inputs: matching keyed Series/DataFrame (areas x bands), +replicates (areas, bands, 80); output includes pooled `all`. Complete-case exclusions are explicit. +Final review covered unused-formula circularity bypass, CV from replicates when SE is absent, and +negative ACS band replicates (refused per the pre-specified invalid-population policy, +even with positive E). Supporting those perturbations needs separate validation. Both final-library replays exactly reproduced every committed validation cell. Seeds/designs/criteria unchanged. ## 10. Deliverables diff --git a/STATUS.md b/STATUS.md index 5e0f790..c4e0f67 100644 --- a/STATUS.md +++ b/STATUS.md @@ -3,14 +3,14 @@ The single current summary of the Census Research Engine: what it does, which methods are validated, the standing rules, and what is next. Read this instead of old plans. Updated each round when it merges. -**Current release:** v0.21.0 (Round 22), DOI 10.5281/zenodo.24264540. All versions: concept record +**Current release:** v0.21.1 (Round 23, internal package split). Latest study release: v0.21.0 (Round 22), DOI 10.5281/zenodo.24264540. All versions: concept record [zenodo.org/records/23925301](https://zenodo.org/records/23925301). Per-round detail: `CHANGELOG.md`. ## Where things are | Path | What | | :--- | :--- | -| `census/` | Library. `manifest/` package (pre-registered studies: `schema`, `validate`, `create`, `run`, `methods_text`, plus `adjustments`, `replicates`, `external`, `derivations`, `datasource`; import from `census.manifest`, but patch tests at the submodule that looks the name up), `models.py` (`fit_replicate`, `fit_eiv`, robust GLM SEs), `health_sources.py` (PLACES, USALEEP, CHR&R, WONDER exports), `spatial*.py`, `sae*.py`, `simulation*.py` (validation studies) | +| `census/` | Library. `manifest/` package (pre-registered studies: `schema`, `validate`, `create`, `run`, `methods_text`, `methods` (single method registry), plus `adjustments`, `replicates`, `external`, `derivations`, `datasource`; import from `census.manifest`, but patch tests at the submodule that looks the name up), `models.py` (`fit_replicate`, `fit_eiv`, robust GLM SEs), `health_sources.py` (PLACES, USALEEP, CHR&R, WONDER exports), `spatial*.py`, `sae*.py`, `simulation*.py` (validation studies) | | `mcp_server.py` | MCP server (63 tools; profiles in `census/mcp_profiles.py`) | | `docs/agent_protocol.md`, `docs/recipes/` | What agents are told; keep in step with engine rules | | `docs/audits/` | Validation reports (pass/fail, reproducible scripts in `scripts/`) | @@ -38,12 +38,20 @@ manifests refuse them in confirmatory analyses. | `fit_eiv` (errors-in-variables) and its cluster jackknife | Validated (0.87–0.93) | `eiv_validation.md`, `eiv_cluster_validation.md` | | Binomial/Poisson GLM with `scale: "robust"` (HC3) | Validated (0.879–0.897); required for external count outcomes | `robust_se_validation.md` | | Binomial/Poisson with `scale: "X2"` (quasi-likelihood) | Failed under between-area overdispersion (0.689–0.79); refused for new manifests | `robust_se_validation.md` | +| Area SII / log RII / concentration (`inequality`, Round 24 implementation) | Passed fixed 90% conservative-interval validation: 0.876–0.905 for rank CV<=15%; high-error attenuation persists (CV30% information only). Independent areas. | `inequality_validation.md` (criterion `60330e3`) | +| Indirect SMR (`smr`, Round 24 implementation) | Passed combined Poisson/ACS log-interval validation: 0.893–0.909; rates fixed, independent band errors; negative band replicates refused. Exact Garwood limits condition on E. | `smr_validation.md` (criterion `680707e`) | | Conley spatial HAC SEs (`spatial_se`) | **Failed** (0.71–0.87 at 150 km); exploratory only | `spatial_hac_validation.md` | | Spatial error model λ correction (ME-SEM) | Validated for λ > 0 (0.86–0.93); lag model at γ 0.6 under-covers | `spatial_regression_validation.md` | | LISA under sampling error | Characterised, information only | `lisa_validation.md` | | Tract 2010→2020 apportionment: 1:1 and merged | Validated | `rate_apportionment_validation.md` | | Split tracts: default `missing`; opt-in `split: "dominant"` (≥0.95 share) | Dominant passed held-out OH/PA (ratio 0.913); general split failed | `rate_apportionment_retest.md` | +**Round 24 implementation (v0.22.0, awaiting review/merge):** method registry, area inequality and +indirect SMR; no new MCP tools (63). The exploratory cached LBW example uses 144 complete counties +and is documented in `docs/studies/r24_lbw_inequality_example.md`. Full suite: 1703 passed, 37 skipped; +69 new tests. Legacy 478 golden snapshots change only supported-method lists; final total 516. +Both final-library validation replays reproduce every committed cell. No merge, tag or release performed. + ## Standing rules - **Circularity (enforced, R1/R2):** determinant studies use the ACS's own measure (disability: B18101). @@ -73,8 +81,6 @@ manifests refuse them in confirmatory analyses. 2. Measurement-error correction for GLM covariates (none yet; `regression_calibration` is approximate). 3. Replication of the county findings in held-out states (OH/PA). 4. Split `run_manifest` into stages (Round 23 recorded them in `PLAN_ROUND_23.md` section 7). -5. A per-method registry so a new analysis method is defined in one place (today: validation args, - `_run_analysis`, methods text, recipes, agent protocol). ## Plan format (keeps agent context small) diff --git a/census/__init__.py b/census/__init__.py index b022d5d..6d9c568 100644 --- a/census/__init__.py +++ b/census/__init__.py @@ -1,7 +1,7 @@ # SPDX-License-Identifier: AGPL-3.0-or-later """Headless Census ACS data client and processing library.""" -__version__ = "0.21.1" +__version__ = "0.22.0" from census.api import ( DATASETS, diff --git a/census/inequality.py b/census/inequality.py index c80560b..d2cff1e 100644 --- a/census/inequality.py +++ b/census/inequality.py @@ -179,10 +179,12 @@ def inequality_replicate(frame, outcome, rank_by, weights, higher_is_more_depriv cv = np.full(len(frame), np.nan) if rank_by+"_se" in frame: rank_se = frame[rank_by+"_se"].to_numpy(float) - np.divide(100*rank_se, np.abs(x), out=cv, where=x != 0) - cv[(x == 0) & (rank_se > 0)] = np.inf + elif rank_by in mats: + rank_se = np.sqrt(.05*np.sum((mats[rank_by]-x[:, None])**2, axis=1)) else: - cv[:] = 0 + rank_se = np.zeros(len(frame)) # explicitly nonstochastic rank + np.divide(100*rank_se, np.abs(x), out=cv, where=x != 0) + cv[(x == 0) & (rank_se > 0)] = np.inf median = float(np.nanmedian(cv)) if (~np.isnan(cv)).any() else float("nan") if median > 15: notes.append(RANK_ERROR_NOTE) diff --git a/census/manifest/external.py b/census/manifest/external.py index fcac714..dc58f56 100644 --- a/census/manifest/external.py +++ b/census/manifest/external.py @@ -29,14 +29,14 @@ def external_circularity(manifest: Mapping[str, Any]) -> List[Dict[str, Any]]: for a in manifest.get("analyses") or []: if not isinstance(a, Mapping): continue - if "~" in str(a.get("formula", "")): - lhs, rhs = str(a["formula"]).split("~", 1) - elif a.get("method") in ("inequality", "smr"): + if a.get("method") in ("inequality", "smr"): a = _normalize_analysis(a) lhs = str(a.get("outcome", a.get("observed", ""))) bands = a.get("population_bands") rhs = str(a.get("rank_by", " ".join(v for v in bands if isinstance(v, str)) if isinstance(bands, list) else "")) + elif "~" in str(a.get("formula", "")): + lhs, rhs = str(a["formula"]).split("~", 1) else: continue ys = [y if y in ext else y[:-10] for y in _formula_names(lhs) diff --git a/census/mcp_profiles.py b/census/mcp_profiles.py index fdb39d5..eba85b9 100644 --- a/census/mcp_profiles.py +++ b/census/mcp_profiles.py @@ -213,6 +213,7 @@ Only conservative combined Poisson/replicate log intervals drive SMR tests; fixed-E Garwood intervals omit expected uncertainty. O=0 has only exact limits; absent replicates set expected uncertainty to zero, noted. Rates are fixed. + SMR refuses negative band replicates under its pre-specified validation domain. Log statistics test zero and report exponentiated ratio columns. R1 applies to external outcomes/counts; reused MI/NY LBW associations remain exploratory. """ diff --git a/census/pipeline.py b/census/pipeline.py index 1e28bd8..31bd2b7 100644 --- a/census/pipeline.py +++ b/census/pipeline.py @@ -103,7 +103,7 @@ from census.stats import add_reliability_columns, clean_census_sentinels, parse_variable_code from census.tables import load_variable_metadata -PACKAGE_VERSION = "0.21.1" +PACKAGE_VERSION = "0.22.0" CONCORDANCE_MODES = ("fail", "strict", "code") MAX_ISSUES_IN_RESULT = 100 diff --git a/census/standardize.py b/census/standardize.py index eeaca06..1a39b94 100644 --- a/census/standardize.py +++ b/census/standardize.py @@ -179,7 +179,9 @@ def indirect_standardize(observed, band_populations, reference_rates, band_repli ``band_populations`` a DataFrame with exactly the same ordered index and columns in reference-rate order (areas x bands). No implicit joins/drops. Replicates are an array (areas, bands, 80) in that order. Positive finite - expected counts are required in every refit. The returned final GEOID + expected counts are required in every refit. This validated implementation + requires nonnegative band replicates; negative ACS band perturbations are + refused even when their total expected count would remain positive. The returned final GEOID ``all`` pools counts and expected counts BEFORE computing uncertainty. Rates are fixed; their uncertainty and between-area dependence are not covered. At O=0 only the exact fixed-E interval is defined. diff --git a/docs/agent_protocol.md b/docs/agent_protocol.md index 4286305..a783840 100644 --- a/docs/agent_protocol.md +++ b/docs/agent_protocol.md @@ -259,6 +259,8 @@ statistics test zero; ratio columns exponentiate estimates and limits. SMR tests use the combined Poisson/replicate log interval (`target: conservative` only), while Garwood limits condition on expected counts. Zero observed counts have only an exact fixed-E interval; rate uncertainty is excluded. No band replicates -means zero expected-count uncertainty, explicitly noted. See recipe +means zero expected-count uncertainty, explicitly noted. Negative ACS band +replicates are refused under the pre-specified validation domain, even when +total expected counts would be positive. See recipe `health_inequality` and the two fixed simulation audits. Reused MI/NY LBW associations are exploratory. diff --git a/docs/audits/smr_validation.md b/docs/audits/smr_validation.md index f283e05..d78b709 100644 --- a/docs/audits/smr_validation.md +++ b/docs/audits/smr_validation.md @@ -18,3 +18,5 @@ Result: **PASS**. 2,000 draws per cell; true SMR 1.2, five equally populated bands scaled to the target expected count. Gaussian population errors are not clipped/redrawn. Nonpositive/invalid populations or expected counts count as non-coverage. Reference-rate uncertainty, correlated band errors and spatial dependence were not simulated; rates are treated as fixed. + +Implementation domain: negative ACS band replicate values are refused, even when their expected total would be positive. They were classified as invalid in the pre-specified run; extending that domain requires separate validation. Final-library replay reproduced every committed cell exactly. diff --git a/docs/recipes/health_inequality.md b/docs/recipes/health_inequality.md index 571d691..9e33f79 100644 --- a/docs/recipes/health_inequality.md +++ b/docs/recipes/health_inequality.md @@ -47,7 +47,9 @@ only `target: "conservative"` is supported. Without replicates, expected-count uncertainty is zero and explicitly noted. At zero observed counts, the combined test is undefined; report the exact fixed-E interval only. Reference-rate - uncertainty is not included. No age-specific real death-count example is bundled; + uncertainty is not included. Negative ACS band replicates are refused under + the pre-specified validation domain, even if their expected total is positive. + No age-specific real death-count example is bundled; WONDER exports remain manual. 7. Circularity R1 applies to external outcomes/counts. R2 treats `rank_by` as the tested term for inequality; model-input overlap requires the existing written diff --git a/mcp_server.py b/mcp_server.py index afb3b33..e9fa072 100644 --- a/mcp_server.py +++ b/mcp_server.py @@ -143,6 +143,7 @@ conservative only: tests combine Poisson and expected-count replicate log variance; exact Garwood limits condition on expected counts. No replicates means zero expected uncertainty, noted; O=0 has only the exact fixed-E interval. Reference-rate uncertainty is excluded. +SMR refuses negative ACS band replicates under its fixed validation domain, even with positive E. Area HC3 assumes independence. R1 applies to external outcomes/counts. Reused LBW associations are exploratory. No new tools are needed. diff --git a/tests/test_round24.py b/tests/test_round24.py index 1881731..bc6f14d 100644 --- a/tests/test_round24.py +++ b/tests/test_round24.py @@ -373,3 +373,30 @@ def test_public_exports_and_recipe(): assert callable(getattr(census,name)) and name in census.__all__ recipe=get_recipe('health_inequality') assert recipe['success'] and 'reverse sign' in recipe['steps'] and 'reference_source' in recipe['steps'] + + +def test_circularity_cannot_be_bypassed_by_unused_formula(): + from census.manifest import validate_manifest + m=manifest('inequality',True) + m['external']=[dict(name='diabetes',source='places',release=2024,measure='DIABETES')] + m['variables'].append(dict(name='rank',ref='2024:B17001_002')) + a=m['analyses'][0];a.update(outcome='diabetes',rank_by='rank',weights='pop',formula='income ~ pop') + assert any('CIRCULARITY R2' in e and "'rank'" in e for e in validate_manifest(m,resolve=False)) + + +def test_negative_replicate_bands_refused_under_preregistered_domain(): + obs=pd.Series([10],index=['36061']);pop=pd.DataFrame([[100.,200.]],index=obs.index) + rep=np.repeat(pop.to_numpy()[:,:,None],80,axis=2);rep[:,0,:]=-5. + # ACS perturbations can be negative; this implementation's fixed validation + # explicitly classified such band inputs as invalid, even with E_r > 0. + with pytest.raises(ValueError,match='nonnegative'): + indirect_standardize(obs,pop,[.01,.02],rep) + + +def test_rank_cv_uses_replicates_when_se_column_is_absent(): + f,rep=frames();f=f.drop(columns='x_se') + # A constant scale perturbation preserves ranks but has substantial known CV. + rep.loc[:,REPLICATE_COLUMNS]=.8*rep[REPLICATE_COLUMNS].to_numpy() + result=inequality_replicate(f,'y','x','w',True,rep) + assert result.diagnostics['rank_cv_median']==pytest.approx(40.) + assert any('attenuated' in n for n in result.notes) From c3533744afa1d130253bda3be5efad34b5352e9b Mon Sep 17 00:00:00 2001 From: dickbird <20980982+cafeTechne@users.noreply.github.com> Date: Sun, 15 Nov 2026 21:34:15 -0500 Subject: [PATCH 12/13] Review: study report points to the release attachment instead of a local /tmp archive --- docs/studies/r24_lbw_inequality_example.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/studies/r24_lbw_inequality_example.md b/docs/studies/r24_lbw_inequality_example.md index d36da20..279fd75 100644 --- a/docs/studies/r24_lbw_inequality_example.md +++ b/docs/studies/r24_lbw_inequality_example.md @@ -40,4 +40,4 @@ no SMR was fitted. No age-specific death counts were fabricated. Reproduction: run the manifest with the existing caches and `CENSUS_HEALTH_OFFLINE=1`; outputs are release artifacts, not committed datasets. -Local output archive: `/tmp/round24-study-outputs.tar.gz` (not uploaded). +Outputs: `r24_study_outputs.tar.gz`, attached (after a key scan) to the release that ships this round. From 45d3b02260f64eb198019945c6fd6501d1c5f973 Mon Sep 17 00:00:00 2001 From: dickbird <20980982+cafeTechne@users.noreply.github.com> Date: Sun, 15 Nov 2026 21:36:58 -0500 Subject: [PATCH 13/13] Round 24 merge prep: STATUS outcome and study row, archive PLAN_ROUND_24 (kept at v0.22.0), CHANGELOG heading --- CHANGELOG.md | 2 +- CLAUDE.md | 2 +- PLAN_ROUND_24.md | 244 ----------------------------------------------- STATUS.md | 17 ++-- 4 files changed, 11 insertions(+), 254 deletions(-) delete mode 100644 PLAN_ROUND_24.md diff --git a/CHANGELOG.md b/CHANGELOG.md index a72a18e..90cb530 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,7 +5,7 @@ each tagged release (`v`) is archived on Zenodo with its own DOI (see [Releasing](#releasing)). Plans and acceptance criteria for each round are in `PLAN_ROUND_.md`. -## 0.22.0 (Round 24, unreleased): method registry and public-health inequality +## 0.22.0 (Round 24): method registry and public-health inequality - Centralize manifest execution, method-specific validation, statistics and methods text in `MethodSpec`; preserve all 478 refactor snapshots and the existing public constant ordering. diff --git a/CLAUDE.md b/CLAUDE.md index cf8343d..ca958b8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -2,7 +2,7 @@ Start with `STATUS.md` (current state, validated methods, rules, backlog). Then read only the current `PLAN_ROUND_.md` if you are implementing a round. Older plans are archived at their release tags -(`git show v0.21.0:PLAN_ROUND_.md` for rounds 1–22; `v0.21.1` for 23); read one only when a docstring citation is not enough. +(`git show v0.21.0:PLAN_ROUND_.md` for rounds 1–22; `v0.21.1` for 23; `v0.22.0` for 24); read one only when a docstring citation is not enough. ## Commands diff --git a/PLAN_ROUND_24.md b/PLAN_ROUND_24.md deleted file mode 100644 index 9ad73e7..0000000 --- a/PLAN_ROUND_24.md +++ /dev/null @@ -1,244 +0,0 @@ -# Round 24 plan: analysis-method registry, health-inequality indices, standardized ratios - -**Goal.** (A) Define each manifest analysis method in one place, so a new method is one registry entry -instead of edits in five files. (B) Use the registry to add two validated public-health methods: area-level -inequality indices (slope index, relative index, concentration index) and indirectly standardized ratios -(SMR) with ACS-based expected counts. (C) Show them on an exploratory MI/NY example. Version 0.22.0. - -**Non-goals.** No `run_manifest` split (next round). No new MCP tools (63 stays 63). No new dependencies. -No individual-level (microdata) inequality measures. No Erreygers/Wagstaff corrections. No change to any -existing method's numbers or messages. - -## 0. Ground rules - -1. Branch `phase24/inequality` from `phase24/plan`. Push it. Do not merge, tag or release. (Round 23 is - merged and released as v0.21.1; `phase24/plan` already contains that merge (commit `60bf712`). If you - branched before it, run `git merge origin/phase24/plan` once. Never rebase or force-push.) -2. **No AI attribution anywhere.** Commits and the PR body must not contain `Co-Authored-By`, - `Generated with`, or any similar line, even if your tooling suggests one. -3. Never print, log or commit `.env` or any API key or token. -4. Read `CLAUDE.md` and `STATUS.md` first. Use `.venv/bin/python`; set `OPENBLAS_NUM_THREADS=1` for the suite. -5. Do not run GitHub CI. Commits may carry `[skip ci]`. -6. **Validation first.** Each validation's script and acceptance criterion are committed and pushed - **before** its first full run. Do not change seeds, designs or criteria after a run. A method that fails - its validation ships as exploratory only (manifests refuse it unless `exploratory: true`), exactly as - `spatial_se` did in Round 22. Smoke runs of at most 20 draws are allowed before the commit, to debug. -7. If something here is impossible as written, record it in section 9 and choose the option that changes - least. - -## 1. Baseline - -```bash -CENSUS_HEALTH_OFFLINE=1 .venv/bin/python scripts/refactor_golden.py /tmp/golden_before.json # 478 snapshots -OPENBLAS_NUM_THREADS=1 .venv/bin/python -m pytest -q 2>&1 | tail -1 # expect 1634 passed, 37 skipped -``` - -## 2. Part A: method registry (no behaviour change) - -Create `census/manifest/methods.py`: - -```python -@dataclass(frozen=True) -class MethodSpec: - name: str - required: Tuple[str, ...] # today's ANALYSIS_METHODS[name][0] - statistics: Tuple[str, ...] # today's ANALYSIS_METHODS[name][1] - run: Callable[..., Any] # (a, args, frame, level, replicates, correction, ctx) -> result - replicate: bool = False # member of REPLICATE_METHODS - external_outcome: bool = False # member of health_sources.EXTERNAL_OUTCOME_METHODS - check: Optional[Callable[..., None]] = None # method-specific validation, appends to errors - text: Optional[Tuple[str, List[str]]] = None # today's _METHOD_TEXT entry, if any - -METHODS: Dict[str, MethodSpec] = {...} # same insertion order as today's ANALYSIS_METHODS -STATISTIC_EXTRACTORS: Dict[str, Callable[[Mapping, Any], Dict[str, Any]]] = {...} -``` - -Then: - -1. `schema.ANALYSIS_METHODS`, `schema.STATISTICS`, `schema.REPLICATE_METHODS` and - `health_sources.EXTERNAL_OUTCOME_METHODS` keep their names, types, values and order, but are **derived** - from `METHODS`. Avoid an import cycle: if `schema` cannot import `methods`, define the derived constants - in `methods.py` and re-export them from `schema` and the facade, so every existing import still works. -2. `run._run_analysis`: keep the shared preamble (argument normalisation, replicate policy, coords, - `held_external`) and replace the `if method == ...` chain with `METHODS[method].run(...)`. Move each - branch body verbatim into a small runner function in `methods.py`. `ctx` carries `held_external` and - `coords`. -3. `run._statistic`: dispatch through `STATISTIC_EXTRACTORS`. Move each branch verbatim. -4. `validate._check_analysis`: the method-specific blocks (today's `fit_replicate` argument block, the - `spatial_se` block, and any other `if method == ...` block) move verbatim into `check` callbacks, called - at the same point so the error order is unchanged. -5. `methods_text._METHOD_TEXT` becomes a view built from `MethodSpec.text`, with the same keys and values. - -**Acceptance A** (commit when all pass): golden snapshot byte-identical to `/tmp/golden_before.json`; the -suite has the same counts; `grep -n 'method == "' census/manifest/run.py census/manifest/validate.py` -prints only lines you list and justify in section 9 (shared, not method-specific, logic). - -## 3. Part B1: `census/inequality.py` (library) - -Conventions (state them in the module docstring): - -- Units are areas with population weights `w_i > 0`. The ranking variable `x` is an area-level - socioeconomic measure. Areas are ordered from **least deprived (ridit 0) to most deprived (ridit 1)**. - The caller says which way `x` points (`higher_is_more_deprived`, required, no default). -- **Ridit:** `r_i = (W_below_i + w_i / 2) / W`, where `W_below_i` is the weight of areas strictly less - deprived. Tied `x` values share the mean ridit of their block. -- Positive SII, log RII and concentration index all mean the outcome is **higher among more deprived - areas**. The concentration index sign is therefore the reverse of the usual income-ranked convention; - say so in every output note. - -Functions (NumPy/pandas/statsmodels only): - -| Function | Definition | Model SE | -| :--- | :--- | :--- | -| `ridits(x, w, higher_is_more_deprived)` | above | none | -| `slope_index(y, x, w, hi)` | WLS of `y` on `[1, r]`, weights `w`: `SII = b`, `log RII = log((a + b) / a)` (undefined, NaN with a note, if `a <= 0` or `a + b <= 0`) | HC3 (`cov_type="HC3"`); log RII by delta method, gradient `(1/(a+b) - 1/a, 1/(a+b))` | -| `concentration_index(y, x, w, hi)` | `C = 2 cov_w(y, r) / mean_w(y)`; equivalently the slope of the weighted convenient regression `2 var_w(r) y / mean_w(y) = alpha + C r` | HC3 slope SE of that regression (an approximation; the validation tests it) | -| `inequality_replicate(frame, outcome, rank_by, weights, hi, replicates, fixed=(), level=0.90, target="conservative")` | Full-sample SII, log RII and C, then the same three on each of the 80 replicates, recomputing ridits each time. The rank variable, the weights and the outcome use their replicates when present; names in `fixed` (external outcomes) stay at their estimates. `se_sampling = sqrt(4/80 sum (t_r - t)^2)`, `se_total = sqrt(se_model^2 + se_sampling^2)`. | Returns a `ModelResult` whose `coefficients` has terms `sii`, `log_rii`, `concentration_index` and the usual SE and interval columns for `target`. | - -`inequality_replicate` uses the strict replicate policy of `census.results`, as `fit_replicate` does. It -records the median CV of `rank_by` in `diagnostics["rank_cv_median"]`. When that is above the validated range -(section 5), it adds a note, and the methods text says the estimates may be attenuated by rank error. - -## 4. Part B2: standardized ratios in `census/standardize.py` - -Add `indirect_standardize(observed, band_populations, reference_rates, band_replicates=None, level=0.90)`: - -- `E_i = sum_b pop_ib * ref_b`, `SMR_i = O_i / E_i`. A pooled ratio over a set of areas is - `sum O / sum E`. -- **Exact interval with E fixed** (Garwood): lower `chi2.ppf(alpha/2, 2O) / (2E)` (0 when `O = 0`), upper - `chi2.ppf(1 - alpha/2, 2O + 2) / (2E)`. Report as `smr_exact_lower`/`upper`. -- **Combined interval** (used for tests): on the log scale, - `var(log SMR) = 1/O + s_E^2`, where `s_E^2 = 4/80 sum (log E_r - log E)^2` from the band replicates - (0 without replicates, with a note). Wald interval, exponentiated. Not defined when `O = 0` (NaN, note). -- Returns a DataFrame with GEOID, observed, expected, smr, `log_smr`, `se_log_smr`, combined and exact - limits, `p_value` (two-sided Wald on log scale), and the notes. - -## 5. Part B3: validations (criteria committed before running) - -**`scripts/inequality_validation.py` → `docs/audits/inequality_validation.md`/`.json`.** -- Cells: n ∈ {145, 1000} × rank CV ∈ {0.05, 0.15, 0.30}. 1,000 draws per cell. Seed `20261116 + cell index`. -- Fixed per cell, drawn once: - - weights: lognormal, median 20,000 (n = 145) or 4,000 (n = 1000), log-SD 1.0; - - true deprivation `x = expit(N(-1.6, 0.6))`; - - true ridits from `x`. -- Each draw: - - outcome proportion `y = 0.07 + 0.03 r + u`, with `u ~ N(0, 0.008^2)`; - - observed `y` from a binomial with denominators `w / 50`, held fixed (an external outcome); - - observed `x_hat = x + N(0, (cv x)^2)`, with replicates from `census.simulation.make_replicates(x_hat, cv * x, rng)`. -- Truths: - - `SII = 0.03`, `log RII = log(0.10 / 0.07)`; - - `C = 2 cov_w(0.07 + 0.03 r, r) / mean_w(0.07 + 0.03 r)` with the true ridits. -- **Acceptance:** coverage of nominal 90% intervals ≥ 0.86 for all three statistics, at target - `conservative`, in the four cells with CV ≤ 0.15. Report target `population` coverage, bias and mean - SE / empirical SD for every cell. The CV 0.30 cells are information only. They define the "validated - range" note (median rank CV ≤ 15%). - -**`scripts/smr_validation.py` → `docs/audits/smr_validation.md`/`.json`.** -- Cells: expected deaths `E_true` ∈ {10, 50, 500} × band CV ∈ {0.05, 0.20}. 2,000 draws. Seed - `20261117 + cell index`. Five bands with reference rates `(0.0005, 0.001, 0.003, 0.01, 0.04)`, band - populations scaled so that `E_true` is hit. True SMR 1.2. -- Each draw: - - `O ~ Poisson(1.2 E_true)`; - - observed band populations `pop_b (1 + N(0, cv^2))`, with replicates from `make_replicates`. -- **Acceptance:** combined-interval coverage of the true SMR ≥ 0.86 in all six cells. A draw with - `O = 0` counts as non-coverage. Report exact-interval coverage too. - -If a validation fails, its method is exploratory only (manifests refuse it unless `exploratory: true`). -Record this in the audit and in section 9. - -## 6. Part B4: manifest methods (registry entries) - -| Method | Required keys | Args | Statistics (hypothesis `statistic`) | -| :--- | :--- | :--- | :--- | -| `inequality` | `outcome`, `rank_by`, `weights` | `higher_is_more_deprived` (bool, required), `target` (default `conservative`) | `sii`, `log_rii`, `concentration_index` | -| `smr` | `observed`, `population_bands`, `reference_rates`, `reference_source` | `target` (no other args) | `log_smr` | - -Rules (implement as the methods' `check` callbacks; error texts are yours, but each must name the analysis): - -- `outcome`, `rank_by`, `weights` and each `population_bands` entry must be manifest variables, except - that `weights` may be `_denominator` when the outcome is external. `outcome` - may also be an external variable, and `observed` may be an external `_numerator`. Set - `external_outcome=True` for both methods. External outcomes are held fixed in the replicate refits. -- `reference_rates`: a list of finite numbers > 0, the same length as `population_bands`. - `reference_source`: a non-empty string (where the rates come from). Both are part of the hash. -- Hypotheses on `inequality` name a `statistic` only (no `term`). Hypotheses on `smr` give `geoid`: one - GEOID, or `"all"` for the pooled ratio. -- **Circularity R2:** for `inequality`, `rank_by` is the tested term, so extend `circularity_errors` - (`census/manifest/external.py`) to apply R2 to it. R1 applies unchanged to `outcome`/`observed`. -- Log-scale statistics (`log_rii`, `log_smr`) are tested against 0, so `_one_sided` needs no change. Add - ratio columns to the hypothesis table, following the `odds_ratio` pattern in `run.py`: `ratio`, - `ratio_lower`, `ratio_upper`. -- Methods text: one paragraph each (ridit definition and direction, HC3 plus replicates, target; SMR - definition, reference source, exact and combined intervals). Add references: Pamuk 1985 (Population - Studies 39:17–31); Kakwani, Wagstaff & van Doorslaer 1997 (J Econometrics 77:87–103); Garwood 1936 - (Biometrika 28:437–442); Breslow & Day 1987 (IARC Sci Publ 82). - -## 7. Part C: docs and exploratory example - -- `docs/recipes/health_inequality.md` and a `health_inequality` entry in `census/recipes.py` (the recipe - count test may need its expected list extended; that is the only allowed test edit outside new tests). -- `docs/agent_protocol.md`: a short "Inequality and standardized ratios" section (statistics, direction - convention, validated range, SMR reference-rate requirement). -- Library exports in `census/__init__.py`: `ridits`, `slope_index`, `concentration_index`, - `inequality_replicate`, `indirect_standardize`. -- Example `examples/manifests/r24_mi_ny_lbw_inequality.json`: - - `exploratory: true`; MI/NY counties; - - outcome CHR v037 low birth weight (`lbw`); `rank_by: pov_rate`, `higher_is_more_deprived: true`; - - weights: `lbw_denominator` (live births; allow `weights: "_denominator"` when the outcome is - external, as the binomial `trials` role does). - - It is exploratory because Round 22 already related poverty to this outcome. Run it with the cached CHR - data. Write `docs/studies/r24_lbw_inequality_example.md` (one page: the three indices, intervals, - median rank CV, the sign convention). Release outputs go in a tarball, not the repo. -- No SMR example with real data, because no age-specific death counts are cached (WONDER exports are - manual). Use a synthetic end-to-end manifest test instead. -- Tests in `tests/test_round24.py`: - - ridits: ties, weights, direction; - - SII/RII/C against hand calculations on a 6-area example; - - HC3 matches statsmodels; - - exact Garwood limits against `scipy.stats.chi2`; - - pooled SMR; - - manifest validation: each rule in section 6; - - end-to-end offline runs of both methods; - - refusal when a failed method is used without `exploratory`. - -## 8. Final acceptance - -- Part A golden identical (section 2). After Part B, regenerate the golden snapshot and confirm by `diff` - that only messages listing supported methods or statistics changed. Paste that diff summary in section 9. -- Full suite: baseline counts plus the new tests, 0 failures. -- Both audits committed, with the criterion commit preceding the run commit. -- CHANGELOG `0.22.0` entry; version metadata 0.22.0 wherever the existing release checklist sets it - (`CHANGELOG.md` "Releasing"). -- `STATUS.md`: add both methods to the validation table with their results, and remove backlog item 5. - -## 9. Status table (fill in; keep it short) - -| Item | Status | Notes | -| :--- | :--- | :--- | -| Baseline | Pass | 478 snapshots; 1634 passed, 37 skipped, 48 subtests. | -| A: registry, golden identical | Pass | 478 byte-identical snapshots; isolated full suite 1634/37. | -| A: remaining `method ==` lines | Pass | None in run.py/validate.py; historical tuple/text ordering preserved. | -| B1/B2 library | Pass | Weighted indices/strict refits; keyed SMR/Garwood/combined intervals. | -| Inequality validation (criterion commit, result) | Pass | 60330e3 pushed pre-run; required coverage 0.876–0.905; CV30% informational. | -| SMR validation (criterion commit, result) | Pass | 680707e pushed pre-run; combined coverage 0.893–0.909. | -| B4 manifest methods, R2 extension | Pass | Registry callbacks, rank R2/R1, ratios, failed-validation refusal tested. | -| Golden diff after B | Pass | Legacy 478: 48 supported-method-list message leaves changed; all else identical. Final: 516 snapshots including the new example. | -| C docs, example run | Pass | HTTP blocked; 144/145 counties, rank CV 6.63%; docs/MCP mirrored (63 tools); scanned local archive. | -| Final suite | Pass | 1703 passed, 37 skipped, 49 subtests (69 new tests); 4 unchanged warnings. Both audit replays exact. 0.22.0 cache-only example reproduces scientific results. | -| Exceptions (0.7) | Recorded | External branch switch required /tmp/census-round24. Registry default_target metadata added; SMR conservative only. Complete-case integration finished in commit 8; final review added three edge-case tests. | - -Operational choices: binomial denominators use `max(1, rint(w/50))`; SMR has equal band -populations before scaling. Invalid draws count as non-coverage. SMR's combined-test requirement -means conservative target only. Library inputs: matching keyed Series/DataFrame (areas x bands), -replicates (areas, bands, 80); output includes pooled `all`. Complete-case exclusions are explicit. -Final review covered unused-formula circularity bypass, CV from replicates when SE is absent, and -negative ACS band replicates (refused per the pre-specified invalid-population policy, -even with positive E). Supporting those perturbations needs separate validation. Both final-library replays exactly reproduced every committed validation cell. Seeds/designs/criteria unchanged. - -## 10. Deliverables - -Commits in order: (1) Part A; (2) B1/B2 library and unit tests; (3) the inequality validation script and -criterion; (4) its run and audit; (5) the SMR validation script and criterion; (6) its run and audit; (7) B4 -manifest methods and tests; (8) Part C; (9) CHANGELOG, STATUS, version, status table. Push -`phase24/inequality`. If you cannot open a PR, say so and leave a PR body at `/tmp/round24-pr-body.md`. diff --git a/STATUS.md b/STATUS.md index c4e0f67..33b5ce0 100644 --- a/STATUS.md +++ b/STATUS.md @@ -3,7 +3,7 @@ The single current summary of the Census Research Engine: what it does, which methods are validated, the standing rules, and what is next. Read this instead of old plans. Updated each round when it merges. -**Current release:** v0.21.1 (Round 23, internal package split). Latest study release: v0.21.0 (Round 22), DOI 10.5281/zenodo.24264540. All versions: concept record +**Current release:** v0.22.0 (Round 24: method registry, inequality indices, indirect SMR). All versions: concept record [zenodo.org/records/23925301](https://zenodo.org/records/23925301). Per-round detail: `CHANGELOG.md`. ## Where things are @@ -18,7 +18,7 @@ standing rules, and what is next. Read this instead of old plans. Updated each r | `examples/manifests/` | Registered manifests (their bytes are fixed once registered) | | `PLAN_ROUND_.md` | Only the current round's plan. Older plans are archived (below) | -**Archived plans.** `PLAN_ROUND_1`–`22` are at tag `v0.21.0`, `PLAN_ROUND_23` at `v0.21.1`. Code and docs still cite +**Archived plans.** `PLAN_ROUND_1`–`22` are at tag `v0.21.0`, `PLAN_ROUND_23` at `v0.21.1`, `PLAN_ROUND_24` at `v0.22.0`. Code and docs still cite them as labels (for example "PLAN_ROUND_16 A2"). To read one: `git show v0.21.0:PLAN_ROUND_16.md` (or the later tag), or `https://github.com/cafeTechne/Census-Research-Engine/blob/v0.21.0/PLAN_ROUND_16.md`. @@ -46,11 +46,11 @@ manifests refuse them in confirmatory analyses. | Tract 2010→2020 apportionment: 1:1 and merged | Validated | `rate_apportionment_validation.md` | | Split tracts: default `missing`; opt-in `split: "dominant"` (≥0.95 share) | Dominant passed held-out OH/PA (ratio 0.913); general split failed | `rate_apportionment_retest.md` | -**Round 24 implementation (v0.22.0, awaiting review/merge):** method registry, area inequality and -indirect SMR; no new MCP tools (63). The exploratory cached LBW example uses 144 complete counties -and is documented in `docs/studies/r24_lbw_inequality_example.md`. Full suite: 1703 passed, 37 skipped; -69 new tests. Legacy 478 golden snapshots change only supported-method lists; final total 516. -Both final-library validation replays reproduce every committed cell. No merge, tag or release performed. +**Round 24 (v0.22.0):** a new manifest method is one `MethodSpec` entry in `census/manifest/methods.py` +(golden identical for the refactor; 516 snapshots now). New methods `inequality` (SII, log RII, +concentration index; ridits least to most deprived, positive = worse in more deprived areas) and `smr` +(combined interval drives tests; fixed-E Garwood limits reported only). Both assume independent areas. +Exploratory example: `docs/studies/r24_lbw_inequality_example.md`. Suite: 1703 passed, 37 skipped. ## Standing rules @@ -73,6 +73,7 @@ Both final-library validation replays reproduce every committed cell. No merge, | 21 | PLACES disability vs ACS | PLACES inflates the education gradient 1.4× | | 21b | County premature mortality | Poverty, disability supported; vehicle, cost burden not | | 22 | County low birth weight | Poverty supported (holds with robust SEs); cost burden fragile (fails with robust SEs) | +| 24 | LBW inequality by poverty (exploratory) | RII 1.25 (90% CI 1.00–1.57); all intervals include the null | ## Open backlog @@ -80,7 +81,7 @@ Both final-library validation replays reproduce every committed cell. No merge, Candidates: spatial random effects or a spatial block bootstrap, tested on the Round 22 design. 2. Measurement-error correction for GLM covariates (none yet; `regression_calibration` is approximate). 3. Replication of the county findings in held-out states (OH/PA). -4. Split `run_manifest` into stages (Round 23 recorded them in `PLAN_ROUND_23.md` section 7). +4. Split `run_manifest` into stages (listed in `PLAN_ROUND_23.md` section 7, at tag `v0.21.1`). ## Plan format (keeps agent context small)