[anth-volk]

Fixes #4022

Important

This PR must not merge before its PolicyEngine.py bump commit. On the locked PolicyEngine.py 6.7.5, every catalog extraction now stops with "the installed PolicyEngine.py does not publish catalog format 2's fields", including the seed job's catalog publication during a deploy. The bump commit cannot be written yet. It waits for:

1. a PolicyEngine.py release containing Publish the model metadata the Public API can only read through a private method policyengine.py#701;
2. the simulation API supporting that bundle (the "Ensure PolicyEngine bundle is supported by simulation API" check);
3. a decision by a person, not taken here, on moving the UK default dataset to microcosm_uk_2024_25, which newer PolicyEngine.py releases use, and then updated installed-catalog counts.

The PRs stacked above this one wait with it.

Summary

Catalog format 2 stores the metadata the app's v3 workspace reads, taken only from PolicyEngine.py's public records (PolicyEngine/policyengine.py#701), never from a country package's tax-benefit system or a private PolicyEngine.py method.

• Schema. Revision f5c1d9b4685b (on fa7fa1035444) was autogenerated against the v2 configuration with no corrections. It adds nullable columns with no defaults, so catalog versions published earlier keep them empty, and it writes no data:

  • tax_benefit_model_versions: catalog_format, entities, basic_inputs;
  • variables: is_input_variable, hidden_input, module_name, adds_parameter_path, subtracts_parameter_path, enum_values;
  • parameters: economy, household.

  entities and enum_values are typed with CatalogEntity and CatalogEnumValue.

• Extraction.

  • normalize_model_metadata reads entities (key, plural, person flag), basic inputs, variable flags and modules, the parameter path that adds or subtracts names (beside the list PolicyEngine.py resolves, which stays), enum values with their declared labels (an integer label stays an integer), and parameter flags.
  • Bracket schedules arrive as public parameter nodes, with their stored label or none.
  • Extraction first checks that PolicyEngine.py publishes these fields and names each one missing. It then fails with a specific message on:
    • other than one person entity;
    • a basic input or parameter path that is not published;
    • enum labels that do not match the possible values or are not JSON scalars;
    • a flag that is not a boolean.
  • An empty basic-input list is stored as empty.

• Storage and comparison. Publication stages and inserts the new fields. When the same PolicyEngine.py version is published again, a version stored before format 2 (null catalog_format) is compared without the format 2 fields and keeps them empty; a format 2 version must match on them too. Publication never rewrites a stored version.

• UK regions in newer releases. Newer certified UK releases have no constituency or local-authority weight matrices and route those regions to their own dataset through region_datasets.

  • With the matrices (the release 6.7.5 pins), regions keep weight_replacement.
  • Without them, regions get a row_filter on constituency_code_oa or la_code_oa with the ONS code, on the routed dataset. The existing verification checks that the column exists before a new version is published.
  • A release with neither stops publication.

• Parity check. tests/integration/test_v2_catalog_metadata_parity.py (opt-in through RUN_V2_CATALOG_COMPATIBILITY=1) compares every new field with /{country}/metadata for the US and the UK. Enum labels are compared with their types. The v2 integration check runs it in its existing installed-catalog step.

Parity with /{country}/metadata

Measured against PolicyEngine.py #701's branch installed locally (6.10.0 with the new fields). 0 mismatches in every field:

Field	US	UK
Entities (key, plural, person flag, in order)	6	3
Basic inputs	3	5
Variables compared	6,167	1,170
Enum variables (labels and their types)	290	37
Input variables	925	300
Hidden inputs	54	0
Parameter-named adds	237	4
Parameter-named subtracts	2	0
gov parameters (economy and household flags)	100,276	1,551

Every gov bracket schedule is a catalog node, and its stored label matches the label /{country}/metadata shows when one is stored. A local publication of the full installed catalog with format 2 fields took 19.1 s, and republishing with the comparison 11.8 s.

What the PolicyEngine.py bump commit must change

• policyengine[models]==6.7.5 in pyproject.toml and docker/Dockerfile → the release containing Publish the model metadata the Public API can only read through a private method policyengine.py#701, then uv lock.
• REVIEWED_DEFAULT_DATASETS["uk"] in catalog extraction. Newer releases default to microcosm_uk_2024_25; this needs the decision above. With 6.10.0 installed, extraction stops at "uk default dataset does not match the reviewed selection". With that default accepted in memory only, scoping places all 1,015 UK regions:
  • the national region and the four nations on microcosm_uk_2024_25;
  • 650 constituencies and 360 local authorities as row filters on microcosm_uk_2024_25_dense.
• tests/integration/test_v2_catalog_installed.py: the exact counts, which 6.10.0 changes to 7,337 variables, 30,214 parameter nodes, 104,172 parameters and 1,194,360 values, plus the UK dataset expectations.
• tests/unit/services/test_metadata_service.py::test_default_dataset_uses_user_facing_label[uk-Enhanced FRS], which fails with the newer UK default dataset.
• The region data verification of the UK region, constituency_code_oa and la_code_oa columns could not be run locally (no read access to the UK data), so it first runs in the seed job.

Open decision left to a person

Local authority N09000011 is in PolicyEngine.py's newer local-authority list but not in the Public API's list, which keeps 360 local authorities and 1,015 UK regions. Whether to add it is not decided here.

Migration notes

• Route groups: none change. This prepares stored metadata that a later PR serves through the v2 catalog routes.
• API v1 routes: /{country}/metadata and the other v1 routes are unchanged.
• User-visible contract changes: none.
• Migration: one generated Alembic revision on the v2 chain. It merges one at a time with any other open v2 revision.

Environment variables

None read or added. The v2 integration workflow only adds a test file to an existing step, whose RUN_V2_CATALOG_COMPATIBILITY stays a literal "1".

Testing

GitHub Actions runs on this branch are cancelled at the user's request, so the evidence is local. Commits are listed from the bottom of the branch.

Commit	Failing run first	Passing run
c1608c089 columns and revision	column test KeyError: 'catalog_format'; revision chain and new-revision tests (no child of fa7fa1035444); lifecycle test KeyError with the revision removed	70 focused tests; v2 lifecycle 8 passed, 1 expected failure; alembic check clean; 1,105 v2 unit tests
98ce64876 extraction and parity test	new modules failed at import (CATALOG_FORMAT, normalize_model_metadata); workflow test did not find the parity test	21 unit tests; parity 8 of 8 on the US and UK; 1,113 v2 unit tests
41cead6a5 storage and comparison	staging rows without the new columns; stored fields all null; changed format 2 field "DID NOT RAISE"	publication, route and lifecycle integration 31 passed, 1 expected failure; 1,125 v2 unit tests
739bbc1c8 UK scoping for newer releases	routed-dataset regions still got weight_replacement; three cases got the old message	66 scoping and verification tests; 1,103 v2 unit tests; 25 integration tests
7e0352949 naming missing fields	8 cases failed; on real 6.7.5 the error was "variable age_group enum values do not match its possible values"	79 focused tests; 1,110 v2 unit tests
0340aaf58 label types in the parity test	with integer labels turned into floats on purpose, the old comparison passed and the new one found 1 mismatch	parity 8 of 8
d20f7435f round-trip and change detection	with the fields left out of the comparison and labels stored as strings on purpose, all 4 new tests failed	29 integration tests; 74 catalog unit tests
18d014a92 typed JSON columns	ImportError: CatalogEntity	1,111 v2 unit tests; 37 integration tests, 1 expected failure

On the final branch:

• v2 unit, Alembic workflow and contract artifact tests: 1,161 passed;
• publication, route and lifecycle integration tests on a disposable Postgres 17: 37 passed, 1 expected failure;
• parity test with Update PolicyEngine US to 0.434.0 #701's branch: 8 passed;
• ruff, ruff format, mypy, scripts/run_quality_guards.py, python -m scripts.rules validate and guard: clean;
• scripts/export_migration_contracts.py: no changes;
• the full make test set under the locked 6.7.5: 3,508 passed.

Under 6.7.5, the opt-in installed-catalog tests fail with the missing-fields message: test_v2_catalog_installed.py (1 test) and every parity test. That is expected until the bump commit.

There are no inline # noqa suppressions and no changes under rules/.

🤖 Generated with Claude Code