Store the metadata v3 reads in the v2 catalog from PolicyEngine.py's public records (catalog format 2)

.github/workflows/v2-integration-check.yml

@@ -73,8 +73,8 @@ jobs:
         run: uv run python scripts/reconcile_v2_privileges.py apply --create-missing-roles
       - name: Verify the managed v2 roles hold their declared privileges
         run: uv run python scripts/reconcile_v2_privileges.py verify
-      - name: Verify the installed PolicyEngine.py catalog interface
-        run: uv run coverage run --branch -m pytest -q tests/integration/test_v2_catalog_installed.py
+      - name: Verify the installed PolicyEngine.py catalog interface and its parity with v1 metadata
+        run: uv run coverage run --branch -m pytest -q tests/integration/test_v2_catalog_installed.py tests/integration/test_v2_catalog_metadata_parity.py
         env:
           RUN_V2_CATALOG_COMPATIBILITY: "1"
       - name: Test v2 metadata publication and resource routes

changelog.d/v2plan-1f-115-catalog-metadata-fields.added.md

@@ -0,0 +1 @@
+Catalog format 2 stores the metadata the app's v3 workspace reads, taken only from PolicyEngine.py's public records: each model version's entities and basic inputs; each variable's input and hidden-input flags, module, the parameter path its adds or subtracts names, and its enum values' labels; each parameter's economy and household flags. Bracket schedules are stored as parameter nodes with their stored label. A generated revision adds the columns as nullable, so catalog versions published earlier keep them empty and are compared without them when republished. An opt-in parity test, run by the v2 integration check, compares every new field for the US and the UK with /{country}/metadata. UK constituencies and local authorities follow what the certified data release provides: with the weight matrices they keep the weight_replacement filter, and in a release without them that routes each region type to its own dataset they use a row filter on constituency_code_oa or la_code_oa with the ONS code, on that dataset; a release with neither stops publication.

docs/migration/stage-9-v2-metadata.md

@@ -99,6 +99,34 @@ metadata time-period options are stored on the model-version snapshot.
 PolicyEngine Core and country-package versions are derived diagnostic evidence
 and are not separately persisted.
 
+Catalog format 2 adds the metadata the app's v3 workspace reads, taken only
+from PolicyEngine.py's public records, which carry it from the release that
+publishes entities, basic inputs and these flags:
+
+| Record | Fields |
+| --- | --- |
+| Model version | `catalog_format` (2), `entities` (`key`, `plural`, `is_person`, in the package's order), `basic_inputs` (the household builder's default fields) |
+| Variable | `is_input_variable`, `hidden_input`, `module_name`, `adds_parameter_path` and `subtracts_parameter_path` (the parameter whose dated value lists the variables summed; `adds` and `subtracts` keep the list PolicyEngine.py resolved), `enum_values` (`value` and declared `label`, in order) |
+| Parameter | `economy`, `household` |
+
+Bracket schedules are parameter nodes with their stored label, or none when
+the package stores none. An empty basic-input list is stored as empty.
+Extraction stops before reading any record if the installed PolicyEngine.py
+does not publish these fields, as 6.7.5 does not, and names each missing
+field. It also stops if the model has other than one person entity, names a
+basic input or a parameter path that is not published, gives enum labels that
+do not match the variable's possible values or are not JSON scalars, or gives
+a flag that is not a boolean. An opt-in test
+(`tests/integration/test_v2_catalog_metadata_parity.py`, run by the v2
+integration check) compares every format 2 field for the US and the UK with
+what `/{country}/metadata` builds.
+
+Publication never rewrites a stored version. A version published before
+format 2 has a null `catalog_format` and null format 2 fields, and later
+publications of the same PolicyEngine.py version compare it without those
+fields. The first PolicyEngine.py version published after format 2 deploys
+carries them.
+
 Parameter extraction requires the PolicyEngine.py public history contract from
 5.0.4 and later: values are ordered from oldest to newest, bounded `end_date`
 values are inclusive and fall one day before the following `start_date`, and
@@ -120,9 +148,12 @@ dataset from the certified data release and its filter:
 | US states and congressional districts | the release's state and district dataset | `row_filter` on `state_fips` or `congressional_district_geoid` |
 | UK national | `enhanced_frs_2024_25` | none |
 | UK nations | `enhanced_frs_2024_25` | `row_filter` on `region` |
-| UK constituencies and local authorities | `enhanced_frs_2024_25` | `weight_replacement`: field `parliamentary_constituency_weights` or `local_authority_weights`, value the ONS code |
+| UK constituencies and local authorities, when the release certifies the weight matrices | `enhanced_frs_2024_25` | `weight_replacement`: field `parliamentary_constituency_weights` or `local_authority_weights`, value the ONS code |
+| UK constituencies and local authorities, when the release has no weight matrices and routes the region type to its own dataset | that dataset (for example `microcosm_uk_2024_25_dense`) | `row_filter` on `constituency_code_oa` or `la_code_oa`, value the ONS code |
 
 A weight matrix must come from the same release as the dataset it reweights.
+A release with neither the weight matrix nor a dataset for the region type
+stops publication.
 If the release names no dataset for a sub-national type, its regions use the
 default dataset and the command emits one summary warning with the affected
 region types and counts. Region types without a rule, such as US places until

migrations/v2/versions/f5c1d9b4685b_store_catalog_format_2_metadata.py

@@ -0,0 +1,80 @@
+"""store catalog format 2 metadata
+
+Revision ID: f5c1d9b4685b
+Revises: fa7fa1035444
+Create Date: 2026-10-22 18:59:48.186978
+Generation: uv run alembic -c alembic-v2.ini revision --autogenerate
+"""
+
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+import sqlmodel
+
+
+revision: str = "f5c1d9b4685b"
+down_revision: Union[str, None] = "fa7fa1035444"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.add_column("parameters", sa.Column("economy", sa.Boolean(), nullable=True))
+    op.add_column("parameters", sa.Column("household", sa.Boolean(), nullable=True))
+    op.add_column(
+        "tax_benefit_model_versions",
+        sa.Column("catalog_format", sa.Integer(), nullable=True),
+    )
+    op.add_column(
+        "tax_benefit_model_versions", sa.Column("entities", sa.JSON(), nullable=True)
+    )
+    op.add_column(
+        "tax_benefit_model_versions",
+        sa.Column("basic_inputs", sa.JSON(), nullable=True),
+    )
+    op.add_column(
+        "variables", sa.Column("is_input_variable", sa.Boolean(), nullable=True)
+    )
+    op.add_column("variables", sa.Column("hidden_input", sa.Boolean(), nullable=True))
+    op.add_column(
+        "variables",
+        sa.Column(
+            "module_name", sqlmodel.sql.sqltypes.AutoString(length=512), nullable=True
+        ),
+    )
+    op.add_column(
+        "variables",
+        sa.Column(
+            "adds_parameter_path",
+            sqlmodel.sql.sqltypes.AutoString(length=512),
+            nullable=True,
+        ),
+    )
+    op.add_column(
+        "variables",
+        sa.Column(
+            "subtracts_parameter_path",
+            sqlmodel.sql.sqltypes.AutoString(length=512),
+            nullable=True,
+        ),
+    )
+    op.add_column("variables", sa.Column("enum_values", sa.JSON(), nullable=True))
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_column("variables", "enum_values")
+    op.drop_column("variables", "subtracts_parameter_path")
+    op.drop_column("variables", "adds_parameter_path")
+    op.drop_column("variables", "module_name")
+    op.drop_column("variables", "hidden_input")
+    op.drop_column("variables", "is_input_variable")
+    op.drop_column("tax_benefit_model_versions", "basic_inputs")
+    op.drop_column("tax_benefit_model_versions", "entities")
+    op.drop_column("tax_benefit_model_versions", "catalog_format")
+    op.drop_column("parameters", "household")
+    op.drop_column("parameters", "economy")
+    # ### end Alembic commands ###