[anth-volk]

Fixes #4041

Summary

The v2 metadata routes now serve one catalog version in parts that a browser can load on demand and cache. Before, they served paged collections only, and v3 downloads the whole /{country}/metadata response at start. v3 does not call these routes until it switches to v2.

Routes and their shapes

All responses use the usual {status, message, result} envelope.

Route	Query	result
GET /v2/model-catalog	country_id, optional policyengine_version	policyengine_version, catalog_format, model, model_version (with current_law_id, metadata_time_periods), entities (key, plural, is_person), basic_inputs, regions, datasets, economy_options (region, time_period, datasets)
GET /v2/model-catalog/variables	same	policyengine_version, catalog_format, variables, part_parameter_values
GET /v2/model-catalog/parameters	same, plus repeated name (1 to 100)	policyengine_version, catalog_format, parameters, node_labels, missing_names
GET /v2/parameters	adds repeated name (up to 100)	items gain economy, household
GET /v2/parameter-values	adds repeated parameter_name (up to 100)	unchanged

• /v2/model-catalog: names the selected version, so a client that starts without a version learns the running one and requests every other part for that exact version. A catalog without regions (a household-only country) gets empty regions, datasets and economy_options lists, not an error. A version published before catalog format 2 has a null catalog_format and empty entities and basic_inputs.
• Variables: every variable of the version in one response, with the catalog format 2 fields (is_input_variable, hidden_input, module_name, adds_parameter_path, subtracts_parameter_path, enum_values, unit and definition_period). /v2/variables pages carry the same fields. unit (for example currency-USD) and definition_period (year, month or eternity) are there because v3's household builder and breakdown need them for currency formatting, per-month or per-year labels and household validation; they are null for versions published before catalog format 2. part_parameter_values maps each parameter path a variable's adds or subtracts names to that parameter's current-law values (start_date, end_date, value), so a client picks the list for the year it shows; a path the version lacks maps to an empty list.
• Parameters by name: each parameter carries its summary, economy, household and its current-law values. node_labels maps each ancestor node to its stored label or null; a bracket index such as rates[0] counts as a child of rates. missing_names lists requested names the version lacks, so a reform saved against an older version still opens. Parameter-tree leaves (/v2/parameters/children) now carry the same summary, flags included.

Caching

• ETag and 304: every successful metadata response carries a strong ETag (a hash of its exact body). A request whose If-None-Match names it (strong or weak form, alone or in a list) gets 304 with no body. Browser scripts can read the ETag (Access-Control-Expose-Headers).
• Lifetimes: Cache-Control: public, max-age=86400 when the request names policyengine_version, and max-age=300 for the running version, which changes on deploy.
• Current values: /v2/parameter-values?current=true takes the request time once. The read uses that UTC day, and the response may be reused only until the UTC midnight after it (max-age=0 if the headers are written after that midnight).
• Vary: every cacheable 200 and its 304 carry Vary: Origin, Accept-Encoding, each value once. Starlette's CORS and gzip middleware append to Vary without checking for values already present, so a small UniqueVaryMiddleware now wraps the CORS middleware.
• OpenAPI: declares the 304 and the ETag, Cache-Control and Vary headers. 400 is declared only on /v2/tax-benefit-models/by-country/{country_id}, the one route with the country in its path.
• In-process stores (default dependencies only; tests pass their own or none):
  • MetadataResponseCache keeps the serialized /v2/model-catalog and /v2/model-catalog/variables bodies with their ETags, under the exact version they were read from. A running-version request and an explicit request for that version share one entry.
  • Its bound is eight entries. The 6.10.0 US variables body is 4,190,824 bytes uncompressed, so the store holds about 12 MB for both countries' two routes at two versions, and about 34 MB at most.
  • Requests that arrive while a read for the same key is in progress wait for it and all receive its outcome: the stored response, or their own copy of its error response. A failed read is not kept, so the first request after it reads again.
  • SelectedCatalogStore keeps each catalog version a request has selected. Failed selections are not kept.

Contract change: 422 on every metadata route

Every v2 metadata route now reads its query through the shared query models in policyengine_api.query_parameters (CatalogQuery, CatalogVersionQuery, CatalogPageQuery and resource models built on them), as docs/engineering/skills/api-routes.md requires. Unknown parameters, repeated scalar parameters, a missing or unsupported country_id, a malformed policyengine_version and out-of-range pages return 422 before any read. Before, the older routes ignored unknown parameters and returned 400 for an unsupported country or malformed version. The release live test and the Stage 9 guide now expect 422. Only an unsupported country in the path of /v2/tax-benefit-models/by-country/{country_id} still returns 400.

Measured locally (full 6.10.0 catalog, local Postgres)

Request (US unless noted)	Time	Uncompressed	Compressed
/v2/model-catalog	10 ms; 2 ms once stored	0.258 MB	0.025 MB
/v2/model-catalog/variables	163–195 ms read; 26 ms once stored	4.19 MB	0.589 MB
/v2/model-catalog/variables, UK	27–41 ms	0.820 MB	0.157 MB
/v2/model-catalog/parameters, 20 names	5 ms	0.016 MB	0.003 MB
/v2/parameters, page of 500	7 ms	0.130 MB	0.018 MB
Revalidation (304)	1–3 ms	0	0

The US variables payload is 0.59 MB compressed, above the plan's estimate of about 0.4 MB. Adding unit and definition_period grew it from 3.91 MB to 4.19 MB uncompressed but only from 0.583 MB to 0.589 MB compressed.

Latency: what was and was not shown

• Shown locally: with the production engine options and a warm pool, the database's statement log showed five statements per single-read request on the two routes measured slow in production on 2026-10-21 (a page of 500 parameters, 0.8–2.6 s; one parameter's values, 0.5–0.65 s). They were the pool's liveness check, BEGIN, the catalog selection, the read and ROLLBACK.
• Now two statements: metadata reads run in autocommit sessions on the shared pool, which refuse ORM flushes and are used only by the metadata reader. Each process also keeps the catalog versions it has selected. So such a request sends the liveness check and the read. Each further read in a request adds one statement. The pool's liveness check (pool_pre_ping) is unchanged.
• Not shown: the production cause. It is not measured from here: the database's region relative to Cloud Run, the external connection pooler and connection reuse across requests are not visible. The release live test now times both flagged routes and the new routes on every staging and production deploy, and writes the times to the job summary even when an assertion fails. Staging times are therefore recorded after deploy.

SPM availability

Not served. Nothing stores it yet; a later item records it per catalog version at publication, and /v2/model-catalog serves it then.

Migration notes

• Route group: metadata (adds model-catalog). The contract registry gains the three new routes, and the generated migration contract documents are refreshed.
• What remains on the Flask path: GET /{country}/metadata is unchanged.
• Changelog fragments: .added, .changed and .fixed.
• Inline suppressions added: one, # noqa: E501 on the reads_parameter_nodes import in services/v2/metadata/services.py. It is a dotted module path that cannot be wrapped, and it matches the existing reads_parameter_tree import in the same module.

Environment variables

None new.

• The code reads the existing V2_RUNTIME_DATABASE_URL_SECRET_RESOURCE (GitHub Actions secret → job env: in push.yml → deploy_cloud_run_candidate.sh → Cloud Run, resolved through Google Secret Manager).
• It also reads V2_SUPABASE_PROJECT_REF and V2_SUPABASE_ENVIRONMENT (GitHub Actions variables, same wiring).
• The release live test reads GITHUB_STEP_SUMMARY (set by the GitHub runner in every step) and API_BASE_URL (from the candidate step's output).
• Not verified from here that those secrets and variables are set, though the existing live tests depend on them.

What this waits for

• Store the metadata v3 reads in the v2 catalog from PolicyEngine.py's public records (catalog format 2) #4023 and the PRs below it in the stack (Let saved reforms and households point at a new v2 policy or household #4005, Rename the v2 table user_policies to user_policy_associations #4010, Publish v2 catalog regions from checked-in reference files and the certified data release #4020). Store the metadata v3 reads in the v2 catalog from PolicyEngine.py's public records (catalog format 2) #4023 waits for the .py release containing Publish the model metadata the Public API can only read through a private method policyengine.py#701.
• Review.
• Staging route times, recorded by the release live test after deploy.

Testing

CI on these branches is cancelled at the user's request. The evidence is local, red-green per commit: a failing run before each change, then a passing run.

Latest updates.

• Unit and definition period: the branch was rebased onto Store the metadata v3 reads in the v2 catalog from PolicyEngine.py's public records (catalog format 2) #4023's commits that store each variable's unit and definition_period in catalog format 2. A commit now serves both fields; the review of that commit found no substantiated bugs.
• Final pass: the branch is rebased onto Store the metadata v3 reads in the v2 catalog from PolicyEngine.py's public records (catalog format 2) #4023's final commits (7e0fb9518 and f803106ee), with no conflicts. Four commits follow:
  • The catalog format 2 fields on MetadataVariable and MetadataParameterSummary are required (nullable) in the OpenAPI schema, so a generated client does not type them as possibly missing. Before the change, the new test failed: the schema's required list lacked the format 2 fields. After it, 1,233 v2 unit tests passed.
  • Route-level tests check that /v2/model-catalog/variables (read, then served from the stored body) and /v2/variables pages carry unit and definition_period for the US, a UK variable with its own values, and a version whose catalog_format is null. These tests pin existing behavior.
  • One test helper, taking the model name, now looks up model versions.
  • The Stage 9 guide says consistently that the household builder and breakdown read the two fields.

On the final tip, with the locked .py 6.10.0 and the test schema recreated at head:

• the full make test set without coverage: 3,619 passed;
• Postgres tests for metadata routes, catalog publication and the v2 schema migrations: 49 passed, with the 6 live tests skipped without a deployment;
• ruff lint and format, mypy, quality guards and rules checks: clean;
• the contract export produced no changes;
• no commit adds .coverage or names a private repository.

Three review rounds ran:

• Round 1 (2 bugs):
  • Day-long public caching of current=true values.
  • The latency evidence was wrong: a commit claimed a per-request Secret Manager call, but the loader was already cached. The commit was dropped, and the round trips were measured and reduced instead.
  • Quality findings: consistent 422, cache ETag reuse and key normalization, typing, duplication, CORS ETag, and live test timings.
• Round 2 (2 bugs):
  • The read and its lifetime used different clock readings.
  • Responses lacked Vary: Origin without an Origin header.
  • Quality findings: OpenAPI headers and the 400 description, the build lock released before the result was stored, enforcement of read-only sessions, and layout documentation.
• Round 3 (3 minor bugs):
  • Tree leaves lacked the flags.
  • A failed whole-version read was retried serially by each waiting request.
  • A 304 lacked Accept-Encoding in Vary.
  • Quality findings: 400 declared only on the path route, a pooled-engine autocommit test plus a test of the production combination, and documentation.

🤖 Generated with Claude Code