Store the metadata v3 reads from /{country}/metadata in the v2 catalog

changelog.d/apiv2-1e-catalog-fields.added.md

@@ -0,0 +1 @@
+The v2 metadata catalog (format 2) stores the presentation metadata the app shows and PolicyEngine.py's public records omit: model entities and basic inputs; variable value types, definition periods, units, modules, input and hidden-input flags and enum labels; parameter metadata labels, periods and economy and household flags; and bracket schedules as parameter nodes. The routes return them, with null values for catalog versions published before this change, which publication never rewrites.

docs/migration/stage-9-v2-metadata.md

@@ -95,6 +95,26 @@ metadata time-period options are stored on the model-version snapshot.
 PolicyEngine Core and country-package versions are derived diagnostic evidence
 and are not separately persisted.
 
+Catalog format 2 adds the presentation metadata the app's v3 workspace shows,
+which PolicyEngine.py's public records omit: model entities (key, plural,
+label, description, whether it is a person, and roles) and basic inputs on the
+model version; each variable's value type, definition period, unit, module,
+input and hidden-input flags and enum value labels; each parameter's own
+metadata label, period, and economy and household flags; and the bracket
+schedules (Core `ParameterScale` nodes) as parameter nodes. The extractor reads
+them from the country system that PolicyEngine.py built its model from,
+through the model version's `_load_system` hook, by attribute and without
+importing PolicyEngine Core or a country package. That hook is not part of
+PolicyEngine.py's public interface, so a release that removes it stops
+extraction with an error; publishing these fields in PolicyEngine.py's public
+records would remove the dependency.
+
+Publication never rewrites a stored version. A version published before
+format 2 has `catalog_format` null and null format 2 fields, and later
+publications of the same version compare only the fields it was published
+with. The first PolicyEngine.py version published after format 2 deploys
+carries the new fields; until then the routes return them as null.
+
 Parameter extraction requires the PolicyEngine.py public history contract from
 5.0.4 and later: values are ordered from oldest to newest, bounded `end_date`
 values are inclusive and fall one day before the following `start_date`, and
@@ -207,9 +227,10 @@ workspace that load the whole model at start-up; the paged routes would need
 about 2,550 requests of 500 rows for the US catalog (100,276 `gov` parameters
 and 1,174,586 values). Each API instance builds the serialized document once
 per country and catalog version and keeps the four most recently used. Measured
-locally against the 6.7.5 catalog, the US document takes 5.8 seconds to build
-and is 60.7 MB (12.1 MB gzipped); the UK document takes 1.0 second and is
-1.3 MB. The response carries an ETag naming the country, catalog version and
+locally against the 6.7.5 catalog in format 2, the US document takes 6.8
+seconds to build and is 70.1 MB (12.5 MB gzipped), with the building process
+peaking at 544 MB resident; the UK document takes 1.0 second and is 1.7 MB.
+The v1 `GET /us/metadata` document for the same models is 74.7 MB. The response carries an ETag naming the country, catalog version and
 document format, and `Cache-Control: public, max-age=86400` when the request
 names `policyengine_version` (`max-age=300` otherwise, because the running
 version changes on deploy). The release workflow checks the UK document,

migrations/v2/versions/72fab86fa97d_store_catalog_presentation_metadata.py

@@ -0,0 +1,99 @@
+"""store catalog presentation metadata
+
+Revision ID: 72fab86fa97d
+Revises: fbb7488ebe8a
+Create Date: 2026-10-20 02:03:28.633166
+Generation: uv run alembic -c alembic-v2.ini revision --autogenerate
+"""
+
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+import sqlmodel
+
+
+revision: str = "72fab86fa97d"
+down_revision: Union[str, None] = "fbb7488ebe8a"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.add_column(
+        "parameters",
+        sa.Column(
+            "metadata_label",
+            sqlmodel.sql.sqltypes.AutoString(length=512),
+            nullable=True,
+        ),
+    )
+    op.add_column(
+        "parameters",
+        sa.Column("period", sqlmodel.sql.sqltypes.AutoString(length=32), nullable=True),
+    )
+    op.add_column("parameters", sa.Column("economy", sa.Boolean(), nullable=True))
+    op.add_column("parameters", sa.Column("household", sa.Boolean(), nullable=True))
+    op.add_column(
+        "tax_benefit_model_versions",
+        sa.Column("catalog_format", sa.Integer(), nullable=True),
+    )
+    op.add_column(
+        "tax_benefit_model_versions", sa.Column("entities", sa.JSON(), nullable=True)
+    )
+    op.add_column(
+        "tax_benefit_model_versions",
+        sa.Column("basic_inputs", sa.JSON(), nullable=True),
+    )
+    op.add_column(
+        "variables",
+        sa.Column(
+            "value_type", sqlmodel.sql.sqltypes.AutoString(length=64), nullable=True
+        ),
+    )
+    op.add_column(
+        "variables",
+        sa.Column(
+            "definition_period",
+            sqlmodel.sql.sqltypes.AutoString(length=32),
+            nullable=True,
+        ),
+    )
+    op.add_column(
+        "variables",
+        sa.Column("unit", sqlmodel.sql.sqltypes.AutoString(length=128), nullable=True),
+    )
+    op.add_column(
+        "variables",
+        sa.Column(
+            "module_name", sqlmodel.sql.sqltypes.AutoString(length=512), nullable=True
+        ),
+    )
+    op.add_column(
+        "variables", sa.Column("is_input_variable", sa.Boolean(), nullable=True)
+    )
+    op.add_column("variables", sa.Column("hidden_input", sa.Boolean(), nullable=True))
+    op.add_column(
+        "variables", sa.Column("possible_value_labels", sa.JSON(), nullable=True)
+    )
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_column("variables", "possible_value_labels")
+    op.drop_column("variables", "hidden_input")
+    op.drop_column("variables", "is_input_variable")
+    op.drop_column("variables", "module_name")
+    op.drop_column("variables", "unit")
+    op.drop_column("variables", "definition_period")
+    op.drop_column("variables", "value_type")
+    op.drop_column("tax_benefit_model_versions", "basic_inputs")
+    op.drop_column("tax_benefit_model_versions", "entities")
+    op.drop_column("tax_benefit_model_versions", "catalog_format")
+    op.drop_column("parameters", "household")
+    op.drop_column("parameters", "economy")
+    op.drop_column("parameters", "period")
+    op.drop_column("parameters", "metadata_label")
+    # ### end Alembic commands ###