Skip to content

Lock policyengine-core 3.34.2 - #10589

Merged
MaxGhenis merged 2 commits into
mainfrom
lock-core-3342
Nov 2, 2026
Merged

MaxGhenis merged 2 commits into
mainfrom
lock-core-3342

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

This moves uv.lock's policyengine-core from 3.33.2 to 3.34.2, and nothing else. CI's test jobs already run uv lock --upgrade-package policyengine-core before uv sync --locked, so they have run 3.34.2 since its release on 2026-10-31. This brings the environments that install the committed lock (local uv sync, PR and push lint, Rest shard planning, the PR's rc1 and final candidate wheel builds, and the publish job) to the same version.

Core releases spanned (3.33.3 to 3.34.2)

Changes that could touch a PE-US result or test:

  • 3.34.2: the YAML test runner pauses automatic garbage collection while it builds a reform combination's system, which affects test time and memory only.
  • 3.34.1: parameter clones copy metadata dicts without deep-copying their values. interpolate_parameters now reaches every parameter a node propagates interpolation to, but no PE-US parameter file sets interpolation. policyengine_us/tests/test_pinned_tbs_sharing.py exercises interpolation and metadata copying, and passes under 3.34.2 (33 passed in an independent review).
  • 3.34.0: removes Reform.from_api, Reform.api_id and policyengine_core.charts.api. A grep of the whole repository (code, tests, scripts, docs) finds no use of them.
  • 3.33.14: the YAML runner builds test nodes with pytest's public from_parent, for pytest 9.2.
  • 3.33.12: set_input for one of a variable's own periods drops the sums and twelfths already calculated over overlapping periods. Core also compiles each variable source file once per process.
  • 3.33.11 and 3.33.7: dataset loading rejects implicit defaults over a role's capacity, and validates group identity and membership.
  • 3.33.8 and 3.33.10: parameter lookups with empty key arrays, and group reductions in simulations with no persons, return empty or neutral values instead of raising.

Overlaps

Draft #10000 locks policyengine-core 3.34.0 and will conflict on uv.lock once this lands; its author relocks then. #10620 also touches uv.lock and merges cleanly with this PR.

Tests run

Against the locked core (policyengine_core imported at 3.34.2), one process each:

  • policyengine_us/tests/core/test_ssi_state_supplement_medicaid_dependency.py: 23 passed.
  • policyengine_us/tests/policy/contrib/ctc/integration.yaml (two reform objects, ten cases with different parameter settings): 10 passed.

impact: none for released or CI-computed outputs. A wheel doesn't carry uv.lock and requires policyengine-core>=3.33.1, so a fresh PyPI install already resolves 3.34.2, and CI's test jobs already run it. Local uv sync environments move to 3.34.2. No baseline microsim pair compared 3.33.2 with 3.34.2, and an independent review found no PE-US parameter value or metadata that differs between them.

axiom: n/a: engine dependency update with no policy change.

🤖 Generated with Claude Code

uv.lock held policyengine-core 3.33.2, and CI installs from the lock, so
this repository's YAML shards did not get core 3.34.1's cheaper parameter
clones or 3.34.2's faster reform builds. Only policyengine-core moves;
pyproject's >=3.33.1 floor is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Ready for hub review

Head: a56726c70b0cef91d093fed5ef8cfe9326055fb2 (draft)

Steps

  1. Precondition: PyPI serves policyengine-core 3.34.2. uv lock --upgrade-package policyengine-core changed only uv.lock (3 lines): the version, the sdist and the wheel. No other package moved, and pyproject.toml's >=3.33.1 floor is unchanged.
  2. Changelog: changelog.d/lock-core-3342.changed.md, one sentence, the same shape as Stop rebuilding dated parameter values for each pinned system #10550's fragment. I used changed, not fixed, because nothing is broken.
  3. Core changelog 3.33.3 to 3.34.2: read from the 3.34.2 sdist's CHANGELOG.md on PyPI, because the core repository is outside this session's GitHub scope. Grep across the whole repo for from_api, api_id, charts.api, get_api_chart_data, intra_decile_chart and decile_chart: no hits. No PE-US parameter sets interpolation metadata or uses a linear average-rate scale, so 3.34.1's interpolation change and 3.33.5/6/9's scale fixes don't reach PE-US.
  4. Tests against the locked core (policyengine_core imported at 3.34.2): tests/core/test_ssi_state_supplement_medicaid_dependency.py 23 passed (159 s); tests/policy/contrib/ctc/integration.yaml 10 passed (117 s). I picked that file because it builds two reform objects with 26 distinct parameter settings across 10 cases. No peak memory was measured, because /usr/bin/time isn't on this VM.

CI: not run, because the PR is a draft. No microsimulation was run.

Judgment calls

  • The VM's uv 0.11.32 rewrote the lockfile header from revision = 5 to revision = 3. I discarded that, and relocked with uv 0.13.6 in a scratch venv. That keeps revision = 5 and gives the 3-line diff.
  • The API rewrote the body's closing line into the standard "Generated by Claude Code" footer.

Proposed follow-up

  • Fix the broken policyengine_core.charts import in tools/documentation.py. policyengine_us/tools/documentation.py imports LIGHTER_BLUE, LIGHT_BLUE and DARK_BLUE from policyengine_core.charts. Core 3.33.2 doesn't export them either, so this was broken before this PR. The import fails, and with it the three docs notebooks that import the module (acp, lifeline, ctc).

Generated by Claude Code

@MaxGhenis
MaxGhenis marked this pull request as ready for review November 2, 2026 08:27
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Ready for hub review
VERDICT: REQUEST_CHANGES
a56726c
METHODOLOGY_CHOICE: no

The lock change is correct and safe: it moves policyengine-core alone, the hashes match PyPI, and I found no PE-US parameter value or metadata that differs between core 3.33.2 and 3.34.2. The PR's stated reason, repeated in the changelog fragment that will ship in CHANGELOG.md, is wrong, though. CI's test jobs don't install core from the lock. They already resolve the latest core, and have run 3.34.2 since it was published on 2026-10-31. Only that text needs to change.

No **Hub review in progress** comment from an earlier session exists on this PR, so this review starts from the beginning. The head has not moved from a56726c70b.

Findings

P1. The changelog fragment and the PR body say CI gets core 3.34.x only through this lock. CI's test jobs already run 3.34.2.

  • changelog.d/lock-core-3342.changed.md:1: "…whose faster parameter cloning and YAML reform builds reach this repository's CI only through the lock."
  • PR body, first sentence: "…so CI, which installs from the lock, gets 3.34.1's cheaper parameter clones and 3.34.2's faster YAML reform builds."

What the workflows do. Every test job runs uv lock --upgrade-package policyengine-core (step "Resolve latest compatible PolicyEngine Core") before uv sync --locked:

  • .github/workflows/baseline-tests.yaml:62
  • contrib-tests.yaml:54
  • pr.yaml:200 (Quick Feedback) and pr.yaml:280 (Household API Partners)
  • python-tests.yaml:98,176 (Rest shards, Microsimulation)
  • push.yaml:228,262,322,428,515

Only four jobs run uv sync --locked without that step: PR lint (pr.yaml:45), push Lint (push.yaml:61), Plan Rest Python shards (python-tests.yaml:34, push.yaml:361) and Publish (push.yaml:573).

Witness. I read the job logs for a PR run whose uv.lock still holds 3.33.2: run 41488313567, branch idr-cap-before-proration, head f7b65447af, whose lock has version = "3.33.2" for policyengine-core.

Baseline (irs)      job 125424727071: Updated policyengine-core v3.33.2 -> v3.34.2 / + policyengine-core==3.34.2
Quick Feedback      job 125424726666: Updated policyengine-core v3.33.2 -> v3.34.2 / + policyengine-core==3.34.2
PR validation       job 125424379765: + policyengine-core==3.33.2
Plan Rest shards    job 125424726764: + policyengine-core==3.33.2

So the speed and memory savings the fragment credits to this lock already reach every YAML and Python test shard on main. This PR moves the lint, shard-planning and publish environments, and every local uv sync. The assignment rested on the same premise ("PE-US CI installs from the lock"), so the build followed it. A reader of the released changelog would take it as fact.

Replacement fragment (changelog.d/lock-core-3342.changed.md):

Lock policyengine-core 3.34.2, the version CI's test jobs already resolve, so local `uv sync` environments and the CI jobs that install the committed lock (lint, Rest shard planning and the release build) use it too.

Replacement first paragraph of the body:

This moves `uv.lock`'s policyengine-core from 3.33.2 to 3.34.2, and nothing else. CI's test jobs already run `uv lock --upgrade-package policyengine-core` before `uv sync --locked`, so they have run 3.34.2 since it was published on 2026-10-31. This brings the environments that install the committed lock (local `uv sync`, PR and push lint, Rest shard planning and the publish job) to the same version.

P3. Body: the test list leaves out the one PE-US test file that drives 3.34.1's changed code

The body's 3.34.1 line is accurate: no PE-US parameter file sets interpolation. But policyengine_us/tests/test_pinned_tbs_sharing.py sets interpolation metadata and calls interpolate_parameters at lines 315–319 and 579–587, and writes metadata["interpolation"]["completed"] at line 187. Under "Changes that could touch a PE-US result or test", that file is the one to name. I ran it under both cores, and it passes under each (results below). Suggested addition to the 3.34.1 line: "policyengine_us/tests/test_pinned_tbs_sharing.py exercises interpolation and metadata copying, and passes under 3.34.2 (33 passed)."

P3 (follow-up, identical on main). Stale docstring in test_pinned_tbs_sharing.py under core ≥ 3.34.1

  • policyengine_us/tests/test_pinned_tbs_sharing.py:305 says "Core records in a parameter's metadata that it interpolated the parameter."
  • Line 186's comment says "What interpolation does to a parameter's metadata."

Since 3.34.1, core sets Parameter.interpolated = True and no longer writes completed. That's in core's interpolate_parameters.py, where 3.33.2 had parameter.metadata["interpolation"]["completed"] = True and 3.34.2 has parameter.interpolated = True. Main's test shards already run 3.34.2, so this isn't this PR's defect. Suggested replacement for line 305: "Core records on each parameter that it interpolated it, and core before 3.34.1 recorded it in the parameter's interpolation metadata." I'm not proposing a code change.

P3 (follow-up, identical on main). policyengine_us/tools/documentation.py:9-16 import fails under both cores

I confirmed the hand-off's report. LIGHTER_BLUE, LIGHT_BLUE and DARK_BLUE are missing from policyengine_core.charts in both 3.33.2 and 3.34.2:

3.34.2: ImportError cannot import name 'LIGHTER_BLUE' from 'policyengine_core.charts'
3.33.2: ImportError cannot import name 'LIGHTER_BLUE' from 'policyengine_core.charts'
missing names (both): ['LIGHTER_BLUE', 'LIGHT_BLUE', 'DARK_BLUE']

docs/gov/irs/credits/ctc.ipynb, docs/gov/fcc/lifeline.ipynb and docs/gov/fcc/acp.ipynb import it. Docs CI doesn't execute notebooks (docs/_config.yml:6 execute_notebooks: off) and doesn't import policyengine_us (docs.yaml comment, lines 44–48), so no CI job fails. The only user of the names is line 120's color_discrete_sequence. I'd fix it in a separate PR, not this one.

P3. Draft #10000 will conflict on uv.lock once this lands

Draft #10000 locks policyengine-core 3.34.0. git merge-tree pr-10000 a56726c70b reports CONFLICT (content): Merge conflict in uv.lock, while #10000 merges cleanly with current main. Its author will need to relock. #10620 also touches uv.lock (spm-calculator 1.1) and merges cleanly with this PR.

Checked and sound

  • Diff. Two files: uv.lock (+3/−3) and the changelog fragment. pyproject.toml is unchanged and its policyengine-core>=3.33.1 floor stays. uv.lock keeps version = 1 / revision = 5. The only changed lines are policyengine-core's version, sdist and wheels entries. Nothing under .github/ or policyengine_us/tests/policy/baseline/partners/ changes.

  • Hashes. I checked against PyPI's JSON for 3.34.2:

    • sdist 669eceecc8ce1036af977df1d4a495134ccdab916b4118f4700a69af6e7f46a4, 745403 bytes, uploaded 2026-10-31T08:10:05.237Z;
    • wheel 47c258e26f86382eff87711d40ace9359bdc478fe679530de3609d5b351da48b, 307223 bytes, uploaded 2026-10-31T08:10:03.329Z.

    Both match the lock. I also downloaded both sdists, and sha256sum matched.

  • Dependencies. requires_dist and requires_python are identical for 3.33.2 and 3.34.2 on PyPI, so no other package needed to move. That agrees with the unchanged dependency list in the lock.

  • The lock installs. uv sync --locked --extra dev at the head succeeded, and policyengine_core imported at 3.34.2.

  • Removed APIs (3.34.0). grep -rnE "from_api|api_id|charts\.api|get_api_chart_data|intra_decile_chart|decile_chart" over the whole repository found nothing outside uv.lock.

  • Interpolation (3.34.1).

    • No PE-US YAML sets interpolation; the only hits are the Python test file above.
    • policyengine_us/system.py:131 calls interpolate_parameters, which is a no-op for PE-US's tree.
    • The 3.34.1 diff (read from the sdists) changes only how a parameter records that it has been interpolated, not how values are filled in.
  • Scale fixes (3.33.5, 3.33.6, 3.33.9). Every PE-US scale is marginal_rate (214) or single_amount (749). No code calls to_average, to_marginal or LinearAverageRateTaxScale, so these fixes don't reach PE-US.

  • Parameter values at every instant. I loaded CountryTaxBenefitSystem() under 3.34.2 and under 3.33.2, with the same source and dependencies and only core swapped, and dumped every node's metadata and every parameter's full values_list, keyed by name.

    • Result: 156,957 nodes, 124,982 parameters, 1,338,703 dated entries, and 0 names whose values or metadata differ.
    • The order of get_descendants() differs between runs even on the same core version, so an order-dependent hash differed. A per-name comparison is the right test.
    • Parameters are step functions of values_list, and core's Parameter.__call__ path didn't change, so no PE-US parameter value changes at any instant.
  • Shallow metadata copy (3.34.1). Clones now copy each metadata dict one level deep. PE-US's only post-clone metadata writes are top-level (system.py:152-155, update/pop on the owners'-draw cap clone), which the clone's own dict isolates. A grep of non-test code finds no nested metadata writes.

  • Reform construction and caching (3.33.3 to 3.34.2).

    • 3.34.1 shallow-copies metadata in Parameter, ParameterNode, ParameterScale and ParameterAtInstant.clone, which cuts the cost of TaxBenefitSystem.clone().
    • 3.34.2 pauses automatic GC in test_runner._get_tax_benefit_system around a cache-miss reform build, with one gc.collect() while fewer than 3 systems are cached.
    • 3.34.0 only removes from_api/api_id from reform.py.
    • tests/core/test_ssi_state_supplement_medicaid_dependency.py builds baseline Simulations only (calculation order, set_input, all-states poverty). It covers neither reform-system construction nor the YAML runner's cache. It is relevant to 3.33.12's set_input change (its test_ssp_preserves_medicaid_inputs_set_after_initialization). The change that matters for reform memory is covered by any YAML file with reforms:, such as contrib/ctc/integration.yaml.
  • Tests I ran, one file per process, with peak RSS from getrusage(RUSAGE_CHILDREN) on this VM (4 vCPU, 16 GB). These are local figures, not Linux CI results.

    File Core Result Wall (incl. setup) Peak RSS
    tests/core/test_ssi_state_supplement_medicaid_dependency.py 3.34.2 23 passed 231 s 3.2 GB
    tests/test_pinned_tbs_sharing.py 3.34.2 33 passed 708 s 3.3 GB
    tests/test_pinned_tbs_sharing.py 3.33.2 33 passed 800 s 3.6 GB
    tests/policy/contrib/ctc/integration.yaml 3.34.2 10 passed (109 s in pytest) 367 s 9.4 GB
    tests/policy/contrib/ctc/integration.yaml 3.33.2 10 passed (214 s in pytest) 493 s 10.7 GB

    The ctc runs are one sample each. They support the claim that 3.34.x makes reform-heavy YAML cheaper, but CI already gets that saving (P1).

  • CI at the head. Run 41413241170 ("Pull request"): 34 of 34 jobs passed, none cancelled or skipped. In its Baseline (irs) log, core installs as 3.34.2 with no "Updated" line, because the lock already says 3.34.2. The PR is not a draft, so the hand-off's "CI: not run, because the PR is a draft" is out of date.

  • Mutants. The hand-off lists none. I ran two on scratch copies of core 3.34.2, put first on PYTHONPATH. Both survive the PE-US tests I aimed at them. The site-packages copy and the git tree were left untouched, and git status is clean.

    1. clone.metadata = self.metadata: the clone shares the dict outright, in parameter.py, parameter_node.py and parameter_scale.py.
      • The mutant is live. The baseline gov.irs.payroll.social_security.cap gets the label "Current-law OASDI cap preserved for owners' draw" and loses uprating, through system.py:152-155.
      • It survives tests/core/test_per_capita_uprating.py -k trustees (3 passed).
      • No PE-US test that I found checks that cap's metadata. Core's own clone tests guard this, so I don't ask this PR for a test. A PE-US follow-up could assert that system.parameters.gov.irs.payroll.social_security.cap.metadata keeps its uprating after system.py builds the owners'-draw clone.
    2. Revert interpolate_parameters to writing metadata["interpolation"]["completed"] = True.
      • The mutant is live: a core-cloned tree then stops interpolating (copy at 2024 = 0.01, against 0.105 with real 3.34.2).
      • It survives test_pinned_tbs_sharing.py -k "interpolates_for_itself or reads_past_a_view or metadata_reads_as_a_deep_copy" (3 passed), because that file copies trees with PE-US's own spm.copy_parameter_nodes, which deep-copies metadata dicts.
      • For PE-US's real parameters this mutant is equivalent, since none sets interpolation. That supports the body's claim that 3.34.1's interpolation change doesn't reach PE-US.
  • A published release. The wheel packages only policyengine_us ([tool.hatch.build.targets.wheel] packages = ["policyengine_us"]) and requires policyengine-core>=3.33.1. A PyPI install resolves the newest core whatever this lock says. Publish (push.yaml:573-577) installs from the lock, but its make → python -m build and release_build.py don't import core, so the built wheel doesn't depend on the locked core. Landing this before the next release can't change a computed output in a PyPI install. It can change outputs only in local uv sync environments, where I found no parameter difference. Model results in CI are already computed with 3.34.2.

  • Body lines. impact: pending (hub) and axiom: n/a: engine dependency update with no policy change. are present. I checked the changelog summaries for 3.34.2, 3.34.1, 3.34.0, 3.33.14, 3.33.12, 3.33.11/3.33.7 and 3.33.8/3.33.10 against core's CHANGELOG.md in the 3.34.2 sdist, and they are accurate. Baseline outputs: I found no parameter difference, and CI's test shards already compute with 3.34.2. The population check that remains is the hub's baseline microsim pair for 3.33.2 against 3.34.2, mainly for 3.33.7's dataset-loading change, which reorders how flat-file memberships and group values load.

  • Overlap. Open-PR searches for uv.lock, policyengine-core and lock-core found Preserve supplied marital units and infer missing explicit couples #10000 and Require spm-calculator 1.1 for the 2022–2036 SPM forecast #10620 touching uv.lock (see P3 above). No other open PR changes the core lock.

  • Law. This PR changes no policy, so I fetched no legal source.

What I couldn't check

  • Core's GitHub repository is outside this session's scope. I read core's changelog and source diffs from the 3.33.2 and 3.34.2 sdists on PyPI instead.
  • Population data. I ran no microsimulation. Whether 3.33.7 and 3.33.11's dataset-loading changes move any population output needs the hub's impact pair.
  • Timings. The ctc timing and memory comparison is one local sample per core version, not Linux CI.

Generated by Claude Code

….34.2

The test jobs run `uv lock --upgrade-package policyengine-core` before
`uv sync --locked`, so they already run 3.34.2. The lock moves local
environments and the jobs that install it as committed: lint, Rest
shard planning and the publish job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis merged commit 1eb1121 into main Nov 2, 2026
34 checks passed
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

US + core hub merge audit, us#10589 at c8188891ccd287b1f9da5b089111bd584d1339cf (squash): Lock policyengine-core 3.34.2.

  • Review: An independent Opus 5.5 review in a cloud session found the lock change correct and safe at a56726c (it moves policyengine-core alone, and the hashes match PyPI) but asked for changes: the changelog fragment and the body said CI gets core 3.34.x only through the lock, which is wrong, and two P3 notes. The hub rewrote the fragment (c818889, the one new commit) and the body. An in-session Opus 5.5 delta review approved c818889. It checked every workflow claim against the workflow files and CI logs, and the hub applied its two body notes (the candidate wheel builds also install the committed lock; a fuller impact line).
  • Gates checked live at merge:
    • gh pr checks exits 0;
    • the latest Pull request run at the head concluded success;
    • MERGEABLE;
    • not a draft;
    • no reviewer's latest review requests changes;
    • the semantic-overlap guard;
    • head pinned.
  • Impact: None for released or CI-computed outputs. The wheel packages only policyengine_us and requires policyengine-core>=3.33.1 (pyproject.toml lines 61 and 26, rechecked by the hub at the head), so a fresh PyPI install already resolves 3.34.2 (PyPI's latest, rechecked), and the delta review found every test job upgrading core to 3.34.2 before syncing. Local uv sync environments move to 3.34.2. No baseline microsimulation pair compared core 3.33.2 with 3.34.2; the round 1 review found no PE-US parameter value or metadata that differs between them.
  • Order: Preserve supplied marital units and infer missing explicit couples #10000, a draft pinned to a core git revision, gains two uv.lock conflict hunks from this PR and must relock before it merges anyway. Require spm-calculator 1.1 for the 2022–2036 SPM forecast #10620 and the drafts Let farm and partnership plans take the self-employed plan deduction #10131 and Separate SPM measurement scope from ordinary income #9462 merge cleanly with it.
  • Axiom: n/a: engine dependency update with no policy change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants