Serve v2 metadata in parts loaded on demand, cacheable per catalog version

changelog.d/v2plan-1f-116-catalog-routes-on-demand.added.md

@@ -0,0 +1,5 @@
+Every successful v2 metadata response carries a strong ETag (a hash of its body) and a public Cache-Control lifetime: a day when the request names its catalog with policyengine_version, five minutes for the running version, and never past the UTC midnight after the request for current parameter values (current=true). A request whose If-None-Match names the ETag gets 304 with no body. These responses and their 304s carry Vary: Origin, Accept-Encoding, each value once, and browser scripts can read the ETag.
+GET /v2/model-catalog returns only the small parts of one catalog version (model, model version, entities, basic inputs, regions, logical input datasets and economy choices, which are empty lists for a catalog without regions) and names the selected PolicyEngine.py version, so a client can request every other part for that exact version.
+GET /v2/model-catalog/variables returns every variable of a catalog version in one response, with the catalog format 2 fields (input and hidden-input flags, module, the parameter path an adds or subtracts list comes from, enum labels, unit and definition period), and the current-law values by date of each parameter such a list comes from. GET /v2/variables pages carry the same variable fields.
+GET /v2/model-catalog/parameters fetches up to 100 parameters of a catalog version by name in one request, each with its economy and household flags and its current-law values by date, plus the stored labels of their ancestor nodes for breadcrumbs and the requested names the version lacks. /v2/parameters pages and details carry the economy and household flags.
+Each API process keeps the serialized /v2/model-catalog and /v2/model-catalog/variables bodies and their ETags once built (up to eight route, country and version combinations), so only the first successful request per process reads them from the database; requests arriving during that read receive its outcome.

changelog.d/v2plan-1f-116-catalog-routes-on-demand.changed.md

@@ -0,0 +1 @@
+Every v2 metadata route reads its query through the shared query models, so unknown or repeated parameters, a missing or unsupported country_id and a malformed policyengine_version return 422 before any read (unsupported countries and malformed versions returned 400 before, and unknown parameters were ignored). /v2/parameters accepts repeated name keys and /v2/parameter-values repeated parameter_name keys to page only those parameters.

changelog.d/v2plan-1f-116-catalog-routes-on-demand.fixed.md

@@ -0,0 +1 @@
+v2 metadata reads run in autocommit sessions on the shared pool, and each process keeps the catalog versions it has selected. A request that makes one read for a version the process has already selected, such as a page of parameters or one parameter's values, now sends two statements to the database (the pool's liveness check and the read) instead of five; each further read in a request adds one statement.

docs/engineering/migration-contracts.md

@@ -8,7 +8,7 @@ Generated from `policyengine_api/migration_registry.py` and `tests/contract/regi
 | --- | ---: |
 | route group count | 9 |
 | workflow count | 16 |
-| request count | 89 |
+| request count | 92 |
 | db entity count | 6 |
 | sim flow count | 3 |
 
@@ -18,7 +18,7 @@ Generated from `policyengine_api/migration_registry.py` and `tests/contract/regi
 | --- | --- | --- | --- |
 | `health` | `health`, `simulation-gateway-check`, `liveness-check`, `readiness-check` | `none` | `none` |
 | `specification` | `specification` | `none` | `none` |
-| `metadata` | `metadata`, `datasets`, `economy-options`, `parameter-values`, `parameters`, `regions`, `tax-benefit-model-versions`, `tax-benefit-models`, `variables` | `metadata` | `none` |
+| `metadata` | `metadata`, `datasets`, `economy-options`, `model-catalog`, `parameter-values`, `parameters`, `regions`, `tax-benefit-model-versions`, `tax-benefit-models`, `variables` | `metadata` | `none` |
 | `policy` | `policy`, `policies`, `user-policy`, `user-policies` | `policy` | `none` |
 | `household` | `household`, `households`, `user-household`, `user-households`, `calculate`, `calculate-full` | `household` | `household` |
 | `economy` | `economy` | `simulation` | `economy` |
@@ -215,6 +215,9 @@ Generated from `policyengine_api/migration_registry.py` and `tests/contract/regi
 | `GET` | `/v2/regions/by-code/state/ca?country_id=us` | 200 | `metadata` | none | `status`, `message`, `result.policyengine_version`, `result.item` |
 | `GET` | `/v2/tax-benefit-models/by-country/us` | 200 | `metadata` | none | `status`, `message`, `result.policyengine_version`, `result.model`, `result.model_version` |
 | `GET` | `/v2/economy-options?country_id=us` | 200 | `metadata` | none | `status`, `message`, `result.policyengine_version`, `result.current_law_id`, `result.region`, `result.time_period`, `result.datasets` |
+| `GET` | `/v2/model-catalog?country_id=us` | 200 | `metadata` | none | `status`, `message`, `result.policyengine_version`, `result.catalog_format`, `result.entities`, `result.basic_inputs`, `result.regions`, `result.datasets`, `result.economy_options` |
+| `GET` | `/v2/model-catalog/variables?country_id=us` | 200 | `metadata` | none | `status`, `message`, `result.policyengine_version`, `result.catalog_format`, `result.variables`, `result.part_parameter_values` |
+| `GET` | `/v2/model-catalog/parameters?country_id=us&name=gov.irs.income.bracket.rates.1` | 200 | `metadata` | none | `status`, `message`, `result.policyengine_version`, `result.catalog_format`, `result.parameters`, `result.node_labels`, `result.missing_names` |
 
 ### `simulation_submit_poll`
 

docs/engineering/skills/v2-code-organization.md

@@ -21,9 +21,18 @@ user_policies/
 metadata/
   response_models.py
   common.py
+  query_dependencies.py
+  response_cache.py
   *_routes.py
 ```
 
+`common.py` also holds the route class that gives metadata responses their
+caching headers. `query_dependencies.py` defines the typed FastAPI
+dependencies for the shared query models once, for every metadata route
+module to import. `response_cache.py` keeps serialized whole-version
+responses (bodies and ETags) for the process; it is HTTP-level, so it lives
+with the routes.
+
 `errors.py` defines the strict error response and serialization function shared
 by every API v2 resource. Application-wide exception handling distinguishes
 API v2 from non-v2 requests but must not select error behavior by matching
@@ -52,6 +61,13 @@ services/v2/<resource>/
     deletes.py
 ```
 
+A resource may also add a module for process-wide state its service reuses
+across requests, named for what it keeps, such as the metadata package's
+`selected_catalogs.py` (each catalog version a request has selected, which
+never changes once published). Such a module holds plain data, constructs no
+SQL and owns no session; the composition code creates one instance per process
+and injects it into the service.
+
 Only add CRUD connector modules for operations the resource supports. Append a
 specific descriptor when one operation file would become too broad, such as
 `reads_variables.py` or `reads_parameter_tree.py`.

docs/generated/migration_contracts.json

@@ -1,7 +1,7 @@
 {
   "metadata": {
     "db_entity_count": 6,
-    "request_count": 89,
+    "request_count": 92,
     "route_group_count": 9,
     "sim_flow_count": 3,
     "workflow_count": 16
@@ -33,6 +33,7 @@
         "metadata",
         "datasets",
         "economy-options",
+        "model-catalog",
         "parameter-values",
         "parameters",
         "regions",
@@ -1614,6 +1615,55 @@
             "result.time_period",
             "result.datasets"
           ]
+        },
+        {
+          "expected_status": 200,
+          "headers": [],
+          "method": "GET",
+          "path": "/v2/model-catalog?country_id=us",
+          "route_group": "metadata",
+          "stable_response_fields": [
+            "status",
+            "message",
+            "result.policyengine_version",
+            "result.catalog_format",
+            "result.entities",
+            "result.basic_inputs",
+            "result.regions",
+            "result.datasets",
+            "result.economy_options"
+          ]
+        },
+        {
+          "expected_status": 200,
+          "headers": [],
+          "method": "GET",
+          "path": "/v2/model-catalog/variables?country_id=us",
+          "route_group": "metadata",
+          "stable_response_fields": [
+            "status",
+            "message",
+            "result.policyengine_version",
+            "result.catalog_format",
+            "result.variables",
+            "result.part_parameter_values"
+          ]
+        },
+        {
+          "expected_status": 200,
+          "headers": [],
+          "method": "GET",
+          "path": "/v2/model-catalog/parameters?country_id=us&name=gov.irs.income.bracket.rates.1",
+          "route_group": "metadata",
+          "stable_response_fields": [
+            "status",
+            "message",
+            "result.policyengine_version",
+            "result.catalog_format",
+            "result.parameters",
+            "result.node_labels",
+            "result.missing_names"
+          ]
         }
       ]
     },

docs/migration/stage-9-v2-metadata.md

@@ -27,6 +27,127 @@ package-derived `DatasetVersion` rows or simulation/report output datasets and
 does not modify `Simulation`, `Report`, or `ReportRun` rows or their dataset
 references.
 
+## Loading a catalog version on demand
+
+`GET /v2/model-catalog?country_id=<us|uk>[&policyengine_version=<version>]`
+returns only the small parts of one catalog version: the model and model
+version (with the current-law policy id and the years offered), the entities
+(`key`, `plural`, `is_person`), the basic inputs, every region and logical
+input dataset, and the economy choices. A catalog without regions, such as a
+household-only country's, has empty region, dataset and economy-choice lists
+rather than an error. A version published before catalog format 2 has a null
+`catalog_format` and empty entities and basic inputs. The result names the
+selected PolicyEngine.py version, so a client that starts without a version
+learns the running one and requests every other part for that exact version.
+The query uses the shared `CatalogQuery` model: unknown and repeated
+parameters are rejected.
+
+`GET /v2/model-catalog/variables` (same query) returns every variable of the
+version in name order, in one response, with the catalog format 2 fields:
+`is_input_variable`, `hidden_input`, `module_name`, `adds_parameter_path`,
+`subtracts_parameter_path`, `enum_values` (each value's name and its label,
+in the type the country package declares), `unit` (for example
+`currency-USD`) and `definition_period` (`year`, `month` or `eternity`), which
+the household builder and breakdown read (currency formatting, per-period
+labels and household validation). Where a variable's `adds` or `subtracts` list
+comes from a parameter, `part_parameter_values` maps that
+parameter's path to its current-law values in start-date order
+(`start_date`, `end_date`, `value`), so a client picks the list for the year
+it shows instead of the list resolved on 2025-01-01; a path whose parameter
+the version lacks maps to an empty list. `GET /v2/variables` pages carry the
+same variable fields.
+
+`GET /v2/model-catalog/parameters` takes the same query plus one or more
+`name` keys (at most 100, each an exact parameter path) and returns those
+parameters in name order, each with its `economy` and `household` flags and its
+current-law values in start-date order, in one response: a client showing or
+editing a reform loads all of its parameters at once. `node_labels` maps each
+ancestor of a returned parameter that the version stores as a parameter node
+to its stored label (null when it has none), for breadcrumbs; a bracket index
+such as `rates[0]` counts as a child of its schedule `rates`. `missing_names`
+lists requested names the version has no parameter for, so a reform saved
+against an older version still opens. `GET /v2/parameters` pages and details
+carry the same `economy` and `household` flags.
+
+`GET /v2/parameters` also accepts repeated `name` keys (at most 100) to page
+only those exact parameters, and `GET /v2/parameter-values` repeated
+`parameter_name` keys to page only those parameters' values.
+
+Every v2 metadata route reads its query through the shared query models in
+`policyengine_api.query_parameters` (`CatalogQuery`, `CatalogPageQuery` and the
+resource-specific models built on them): unknown parameters, repeated scalar
+parameters, a missing or unsupported `country_id`, a malformed
+`policyengine_version` and out-of-range `offset` or `limit` values are
+rejected with HTTP 422 before any read. Before, the routes other than the
+`/v2/model-catalog` ones ignored unknown parameters, and an unsupported country
+or malformed version returned HTTP 400 from the read. A country given in the
+path (`/v2/tax-benefit-models/by-country/{country_id}`) that is not supported
+still returns HTTP 400.
+
+## Database round trips per request
+
+Metadata reads use sessions in autocommit mode on the shared v2 pool
+(`get_v2_read_session_factory`), which only the metadata reader opens: each
+read is one statement on rows a published catalog version never changes, so no
+transaction is needed, and the database receives no `BEGIN` before the reads
+and no `ROLLBACK` when the session closes. The sessions refuse to flush added
+or changed rows (`ReadOnlySessionError`); a statement passed directly to
+`execute` is not checked, so the metadata connectors keep to `SELECT`. Each process also keeps every catalog version a request has
+selected (`SelectedCatalogStore`), since a published version never changes, so
+later requests for the same country and version skip the selection query;
+failed selections are not kept. Measured with statement logging in a local
+database, a page of 500 parameters or one parameter's values took five
+statements per request before (the pool's liveness check, `BEGIN`, the catalog
+selection, the read and `ROLLBACK`) and two after, once the process has
+selected that version (the liveness check and the read). A route that makes
+several reads, such as `/v2/model-catalog` (regions and datasets), sends one
+statement per read after the liveness check. The pool's liveness check
+(`pool_pre_ping`) is unchanged.
+
+## Response caching
+
+Every successful metadata response carries a strong `ETag`, a hash of its
+exact body, and `Cache-Control: public`. A request that names its catalog with
+`policyengine_version` may be reused for a day (`max-age=86400`): a published
+catalog version never changes, so such a response changes only with the
+route's response format or with the day it is read on, and the `ETag` lets a
+client revalidate after the day. A request without it selects the running
+version, which changes on deploy, and may be reused for five minutes
+(`max-age=300`). A response that depends on the request's UTC day, which is
+`GET /v2/parameter-values?current=true` (the values in effect today), takes
+the request's instant once: the read selects that day's values, and the
+response may be reused only until the UTC midnight after that instant,
+whichever lifetime would otherwise apply (`max-age=0` if its headers are
+written after that midnight). A request whose `If-None-Match` names the current `ETag` (strong or
+weak form, alone or in a list) gets HTTP 304 with no body. Error responses
+carry neither header. Every cacheable response, and every 304, carries
+`Vary: Origin, Accept-Encoding`, each value once: the API reflects every
+origin in `Access-Control-Allow-Origin` and a request without `Origin` gets no
+such header, so a shared cache must keep responses apart by `Origin`, and gzip
+compresses bodies for clients that accept it. A 304 carries the same `Vary` as
+its 200. The CORS and gzip middleware append to `Vary` without checking for
+values already present, so the application's outermost middleware
+(`UniqueVaryMiddleware`) lists each value once. `ETag` is
+listed in `Access-Control-Expose-Headers`, so
+browser scripts on other origins can read it (`Cache-Control` is readable
+without that).
+
+Each API process also keeps the serialized bodies of `/v2/model-catalog` and
+`/v2/model-catalog/variables` once built, each with its `ETag` computed once,
+for up to eight route, country and version combinations (least recently used
+dropped first; the 6.10.0 US variables body is 4,190,824 bytes uncompressed,
+so eight entries hold about 12 MB for both countries' two routes at two
+versions). A response is kept under the exact version it was read from,
+so a request without `policyengine_version` (the process's running version,
+which never changes while it runs) and one naming that version share it. Only
+the first successful request per process reads the database. Requests that
+arrive while that read is in progress wait for it and all receive its outcome,
+the kept response or a copy of its error response, so one failing read is not
+repeated by each of them; a failed read is not kept, so the first request after
+it reads again. Measured locally against the full 6.10.0 US catalog, the
+variables route takes about 160 ms when read and about 25 ms when kept, most of
+it compression.
+
 ## Configuration and credential boundaries
 
 Select the target with `V2_SUPABASE_PROJECT_REF` and
@@ -263,13 +384,24 @@ A successful response has HTTP 200, `status: "ok"`, `message: null`, and a
 typed `result`. Collection results contain `policyengine_version`, `items`,
 `offset`, `limit`, and `has_more`; detail results contain
 `policyengine_version` and `item`. A malformed or noncanonical explicit
-version returns a typed HTTP 400 error. A valid explicit version that has not
+version returns a typed HTTP 422 error. A valid explicit version that has not
 been published for the country returns a typed HTTP 404 error. An absent
 catalog for the candidate's installed default version returns a typed HTTP 503
 service error. None of these outcomes reads or repairs v1. Unsupported
 countries and methods
 return typed client errors.
 
+The same test loads each country's catalog in parts as a client starting
+cold would: `GET /v2/model-catalog` for the running version, then
+`/v2/model-catalog/variables` for that exact version twice (the second request
+may be served by another instance, so its time is not necessarily a kept
+response's) and once more with `If-None-Match`, which must return 304, then a
+page of 500 parameters, one parameter's values (the two routes measured at
+0.8–2.6 s and 0.5–0.65 s in production on 2026-10-21) and that parameter by
+name. It appends each request's time and body size to the GitHub job summary,
+even when an assertion fails, so every staging and production release records
+these route times against its own catalog.
+
 Request `GET /v2/openapi.json` and confirm that the public document contains
 every documented resource path and explicit component schema references for
 success and error responses.

policyengine_api/asgi_factory.py

@@ -53,6 +53,7 @@
     current_observability_id,
     generate_request_id,
 )
+from policyengine_api.vary_headers import UniqueVaryMiddleware
 
 if TYPE_CHECKING:
     from collections.abc import AsyncIterator, Awaitable, Callable, Iterable
@@ -471,7 +472,9 @@ def create_asgi_app(
     app.mount("/", WSGIMiddleware(wsgi_app))
     # The public API already permits every web origin. Use the standard ASGI
     # implementation while preserving the existing reflected-origin response.
-    return CORSMiddleware(
+    # UniqueVaryMiddleware wraps it, because CORS and gzip each append to Vary
+    # values a route may already have named.
+    cors_app = CORSMiddleware(
         app=app,
         allow_origin_regex=".*",
         allow_methods=["DELETE", "GET", "HEAD", "OPTIONS", "PATCH", "POST", "PUT"],
@@ -480,7 +483,10 @@ def create_asgi_app(
             REQUEST_ID_HEADER,
             OBSERVABILITY_ID_HEADER,
             "WWW-Authenticate",
+            # v2 metadata responses carry an ETag a client may revalidate with.
+            "ETag",
         ],
         allow_credentials=False,
         max_age=600,
     )
+    return UniqueVaryMiddleware(cors_app)