[anth-volk]

Fixes #3937

Summary

Phase 1 step 1.9: weighted-microdata rules. Stacked on #3936, which removes the unused economy comparison module.

A weighted MicroSeries or MicroDataFrame weights every sum, mean and count, so aggregating a weight column through one returns the sum of squared weights. The Simulation API published exactly that as budget.households until PolicyEngine/policyengine-sim-api#767.

This PR registers the rule that weights aren't values, plus three AST guards for the shapes a source check can recognise.

• No current code is in scope. Since Remove the unused economy comparison module #3936, nothing in policyengine_api imports microdf. So all three guarded rules start enforced, with no findings and no baseline, and they apply to any new code that handles microdata.
• No production code changes. The guards are tests. Everything else is:
  • the registry and its rendered views;
  • the proof fixtures;
  • a new skill doc, docs/engineering/skills/microdata-weights.md, with pointers from AGENTS.md and the skills index;
  • a changelog fragment.
• One rule isn't registered here: the Simulation API's builder-tests-use-weighted-frames. This repository has no output builders.

Rules

Rule	Status	Findings
api.microdata.weights-not-values: weights, values, labels and counts each allow different operations	doc-only	—
api.microdata.weights-not-aggregated-as-values: a weight is never the value of a weighted MicroSeries, the variable of a weighted aggregator, or frame[<weight>] followed by sum, mean, median, quantile, std, count, groupby or cumsum	enforced	0
api.microdata.weight-stripping-confined: raw values (np.array, np.asarray, .values, .to_numpy(), .tolist(), pd.DataFrame, pd.Series) are read from weighted microdata only in a designated stripping module; none is designated yet	enforced	0
api.microdata.unweighted-series-are-labels: a MicroSeries built without weights= is never reduced	enforced	0

The guards live in tests/unit/services/test_microdata_weight_boundaries.py, which make test runs in the Test job.

How the guards recognise a weight: a string or f-string ending in _weight or _weights, a name assigned from one, or a .weights attribute.

What counts as weighted microdata:

• a MicroSeries or MicroDataFrame built with weights=;
• names assigned from one;
• anything indexed, filtered or computed from those;
• in a module that imports microdf, a string-keyed read from a baseline or reform economy-output dict, whose microdata columns sit beside their weight columns.

The guards don't follow data through function calls or containers.

Proof

Each rule has a bad and a good fixture under rules/fixtures/<rule-id>/, and a test_rule_proof_* test asserting exactly which expressions are flagged.

• Aggregation:
  • The bad fixture is the real pre-Update PolicyEngine US to 0.469.0 #767 Simulation API line, households=_sum_output_variable(baseline, "household_weight", entity="household"), plus the same mistake in this repository's MicroSeries idiom: MicroSeries(baseline["household_weight"], weights=baseline["household_weight"]).
  • The good fixture is Update PolicyEngine US to 0.469.0 #767's replacement, _count_output_entities(baseline, entity="household", weight="household_weight"), plus a plain sum of household_weight.
• Stripping: np.array(baseline["household_net_income"]).sum() and income.values.sum() are unweighted sample totals. They're checked against weighted MicroSeries totals.
• Labels: decile.mean() and (age < 18).sum() are checked against grouping and filtering weighted series by those labels.

Testing

• pytest tests/unit: 2,733 passed, including the 6 guard and proof tests.
• python scripts/run_quality_guards.py: migration-contracts, rules-registry and ci-scripts passed.
• python -m scripts.rules guard --base-ref remove-economy-comparison passes, and python -m scripts.rules render leaves no diff.
• python -m unittest discover -s scripts/rules/tests -t .: OK on Python 3.11 and 3.12.
• uv run --frozen --extra dev mypy: no issues.
• ruff check . and ruff format --check . (ruff 0.15.12, from uv.lock) are clean, with no new noqa.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.72%. Comparing base (615ec9e) to head (b9eab24).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #3938       +/-   ##
===========================================
+ Coverage   51.11%   87.72%   +36.60%     
===========================================
  Files         242      258       +16     
  Lines       14441    15864     +1423     
  Branches     2307     2594      +287     
===========================================
+ Hits         7382    13917     +6535     
+ Misses       6436     1280     -5156     
- Partials      623      667       +44     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.