[anth-volk]

Fixes #4021

Summary

.py 6.10.0 runs UK national economy reports on microcosm_uk_2024_25, a national dataset with no area codes, so a national run no longer returns constituency or local-authority impacts. The simulation API (PolicyEngine/policyengine-sim-api#881) adds a separate run on the local-area dataset that returns only those impacts. This PR lets the Public API request that run, fixes the Northern Ireland region name, and moves the Public API to .py 6.10.0. It supersedes #3991, whose commit is cherry-picked here with -x.

Merge order: merge this PR last. Merge it only after PolicyEngine/policyengine-sim-api#881 is merged and deployed (so the simulation API serves .py 6.10.0 with the local_area_breakdown capability), and after the app-v2 PR that adds the local-area Compute button (v2 and v3) is merged. Until sim-api deploys 6.10.0, the required check "Ensure PolicyEngine bundle is supported by simulation API" fails with PolicyEngine .py bundle 6.10.0 is NOT deployed in the simulation API. That failure is expected, and so is the "CI complete" failure that follows from it; the jobs that depend on the bundle check (Docker, Cloud Run container, Test environment variables, Test) are skipped until then.

#4026 (raise the economy cache schema version from 3 to 4) is merged into master, and master has been merged into this branch, so this PR carries schema version 4: every economy result, including a local-area breakdown, is cached under an e4 cache version.

What changes for callers

• GET /{country_id}/economy/{policy_id}/over/{baseline_policy_id} accepts breakdown=local_areas. The budget-window route does not.
• The value is accepted only for the UK, and not with target=cliff. The region must be exactly uk, one of the UK countries the metadata publishes (country/england, country/scotland, country/wales, country/ni), or country/northern_ireland, the code .py uses for Northern Ireland. Other regions (including other casings, surrounding spaces and unknown countries), other values and repeated values return 400 before any cache read or submission. The 400 message and the published breakdown description list all six accepted regions.
• breakdown is part of the calculation options only when it is set, so a breakdown request has its own options hash and cached result, and the options hash of every request without it is unchanged. Neither hash matches the other's lookup pattern.
• The simulation payload carries a top-level "breakdown": "local_areas" only when it is set. National and budget-window payloads are unchanged.
• The result (kind: "local_areas", constituency_impact, local_authority_impact, and the dataset and version fields) is stored and returned like any other economy result: its fields are unchanged, and the service adds only the policyengine_bundle and resolved_app_name it adds to every economy result.
• If the simulation API refuses a breakdown submission with 400 (gateway: the routed .py release does not declare breakdowns) or 422 (entry: field unknown), the route returns 502 with the simulation API's reason, instead of the generic 500 the annual path gave before. Nothing is stored, so the next request submits again. Other statuses, and every request without a breakdown, still raise as before.
• A breakdown job that fails (for example a timeout or out-of-memory on the local-area dataset), or that completes with a result whose kind is not local_areas, returns 502, and its result is never stored, so a national result is never cached under the breakdown hash. The error is kept for two minutes on that job's own record, found by its execution ID: every request in that time gets the same error, which ends "A new local-area breakdown can be requested 2 minutes after this failure." After that the record expires and the next request (the app's Retry) submits a new job, instead of receiving the stored error for 30 days. No other job's record is read or changed. Failed national jobs are still stored and replayed for 30 days as before.
• After the .py 6.10.0 update, UK national and country results no longer include constituency_impact or local_authority_impact; clients get them with breakdown=local_areas.
• The UK metadata's region list now offers Ards and North Down (local_authority/Ards and North Down, code N09000011), so it lists 361 local authorities, and the four unitary authorities created in April 2023 have their own hex positions instead of (0, 0). The list's codes, names and positions now equal .py 6.10.0's packaged 2023 list.
• A 422 refusal's message names each rejected field and its problem (body.breakdown: Extra inputs are not permitted). Budget-window refusals now use the same format instead of the Python representation of the error list.
• A breakdown submission's telemetry has simulation_kind local_area_breakdown; geography_type and geography_code still name its region. The simulation API's telemetry envelope forbids extra fields, so no new field is added.
• Northern Ireland: the UK metadata names it country/ni, but .py's UK region registry (6.7.5 and 6.10.0) names it country/northern_ireland and has no aliases. The simulation payload now uses country/northern_ireland; the cache scope keeps country/ni.
• With 6.10.0, the UK metadata labels the default dataset "Microcosm" instead of the generic "Certified dataset".
• The v2 catalog gives the 650 UK constituencies and 361 local authorities the dataset .py 6.10.0 routes them to, microcosm_uk_2024_25_dense, and lists that dataset. Before, they were published against the national file, which has no area codes. UK countries still filter the national file.

Commits

Each red-green commit was written test-first: the tests were run and failed for the stated reason, then the code was written and the same tests passed. The commits are:

1. Accept breakdown=local_areas on the annual economy route (red-green). AnnualEconomyQuery.breakdown: Literal["local_areas"] | None; calculation_options() includes it only when set; the served /specification publishes it with its enum. Red: 11 failures, with the parser rejecting breakdown as extra_forbidden, no breakdown attribute, and no parameter in the specification.
2. Validate breakdown=local_areas and send it to the simulation API (red-green). Validation in _build_economic_impact_setup_options, SimulationOptions.breakdown excluded when None, and the 400/422 refusal handling in _handle_create_impact. Red: 21 failures, with unsupported requests not raising, route requests returning 200 instead of 400, no breakdown in the payload, and refusals propagating as HTTPStatusError. Some tests passed before the change; they pin behaviour that must not change (national payloads, lookup patterns, other statuses, the HTTP client's forwarding of non-null fields). One test, for an unknown-value branch the query parser already makes unreachable, was added after the code.
3. Send Northern Ireland to the simulation API as country/northern_ireland (red-green). Red: 2 failures, country/ni returned unchanged by _setup_region and in the built payload.
4. Update PolicyEngine bundle to 6.10.0 (cherry-picked from Update PolicyEngine bundle to 6.10.0 #3991 with -x; not test-first). pyproject.toml, docker/Dockerfile, uv.lock and its changelog fragment, unchanged.
5. Review microcosm_uk_2024_25 as the UK default dataset for .py 6.10.0 (not test-first; follows the bump). REVIEWED_DEFAULT_DATASETS["uk"] = "microcosm_uk_2024_25", the catalog fixtures, and the installed-catalog test's exact counts, updated after review: 7 more variables, 1 more parameter node and parameter, 3 more parameter values (policyengine-uk 2.132.6), and 1,011 more regions (the 650 constituencies and 361 local authorities the 6.10.0 UK registry now lists).
6. Label the UK Microcosm dataset as Microcosm in the metadata (red-green). Red, with 6.10.0 installed: 3 failures, the UK labels being "Certified dataset".

Commits added after the first review:

7. Accept only uk and the published UK country codes for a breakdown (red-green). The check accepted any text after country/, so country/%20, country/NI and country/france reached the simulation API. The region must now be exactly one of uk, the four published UK country codes, or country/northern_ireland. The UK country list moves to constants.UK_COUNTRY_REGIONS, which country.py's metadata and this check both read (the metadata is unchanged), and the Northern Ireland mapping to constants.UK_REGION_REGISTRY_CODES. Red: 9 failures: 7 cases (country/ , country/%20, country/NI, country/france, country/scotland ) raised no ValueError, and the route returned 200 instead of 400 for country/france and a raw region=country/%20. A new test checks that an invalid breakdown makes no reform-impact cache call, worker lookup or submission.
8. Test that a completed breakdown is stored and returned unchanged (tests only; they describe existing behaviour and passed when written). A route-level test submits a breakdown, polls it to completion (_handle_execution_state), reads it back from storage (_handle_completed_impact), and checks the stored record and that a national request does not read it. The test fixture's mocked resolve_app_name now keeps the requested model version, as the real client does.
9. Share one type for the local_areas breakdown value (no behaviour change). LocalAreaBreakdown = Literal["local_areas"] and LOCAL_AREA_BREAKDOWN in services/economic_impact_request.py, used by AnnualEconomyQuery, SimulationOptions, the service and _breakdown_refusal.
10. Name each rejected field in every simulation API 422 message (red-green). The per-field formatting moves into the shared _build_submission_error_message. Red: 2 failures, the builder and a budget-window 422 refusal returning the list's Python representation.
11. Mark local-area breakdown runs in the simulation telemetry (red-green). Red: 2 failures, breakdown submissions for uk and country/scotland reporting simulation_kind national and country.

Commits added after the second review:

12. Give UK area regions the local-area dataset in the v2 catalog (red-green). _select_region_datasets now follows any subnational region's dataset_path, not only US ones, so UK constituencies and local authorities point at microcosm_uk_2024_25_dense, which joins the UK dataset records the same way US regional datasets do. The installed-catalog test checks each UK region type's dataset and now counts 4 datasets. docs/migration/stage-9-v2-metadata.md describes the UK selection. Red: 2 failures, the unit test's UK datasets being only microcosm_uk_2024_25, and the installed catalog having 3 datasets instead of 4.
13. Don't keep a failed or wrong-kind breakdown job for replay (red-green; how the record is handled is replaced by commit 16). For breakdown requests, a failed job, or a completed job whose result is not kind: "local_areas", had its computing record deleted with the existing delete_reform_impact, and the error is returned as 502. No submission claim is held at poll time; it was released when the job was handed off. Red: 4 failures: the retry after a failed breakdown job returned the stored error instead of submitting, and completed breakdown jobs with no kind, kind national or no result were stored and returned as ok. The test that a failed national job is still replayed passed before and after.
14. List every accepted breakdown region in the 400 and the query description (red-green). constants.LOCAL_AREA_BREAKDOWN_REGIONS is one ordered tuple that the check, the message and the description all use. Red: 2 failures, the message and the description not listing country/northern_ireland.
15. Test the Northern Ireland breakdown end to end and the early rejection (tests only; they describe existing behaviour and passed when written). A Northern Ireland breakdown sends country/northern_ireland while the stored record keeps country/ni, and the second request polls the same job. The unsupported-breakdown test now checks that every case is rejected before the cache version is read and without any simulation client call; its SPM capability assertion applies only to the US cases, since a UK request has no SPM selection.

Commits added after the third review:

16. Keep a failed breakdown's error for two minutes on its own record (red-green). Commit 13's delete_reform_impact matched on scope, options hash and status == "computing" and searched only the newest 1,000 records across every country. After 1,000 newer jobs it found nothing, so the failed job was polled and returned 502 for 30 days; and a late poll of a failed job J1 could delete the record of the next job J2 for the same scope and hash. Now only the failed job's own record, found by its execution ID, becomes an error that expires after 120 seconds. ReformImpactCache.set and update take an optional record_ttl_seconds for the record key only (the indexes keep their 30-day lifetime, and an entry whose record has expired is skipped when read), and set_error_reform_impact takes keep_seconds. _handle_execution_state and the new helper annotate the record as CachedReformImpact. Red: 6 failures: no retry note in the message and an error instead of a new job after two minutes; after 1,000 newer records, the failed job polled again 121 seconds later; J2's record deleted by a late poll of J1; and no retry note in the three wrong-kind cases. The national replay test, now also after 121 seconds, passed before and after. A cache test that a briefly kept error expires and leaves the next job's record was added with the code; on the previous code it fails with a TypeError, because keep_seconds did not exist.
17. Note that UK national results no longer carry area impacts (changelog only).

Commit added at the user's request:

18. Add Ards and North Down and the 2023 unitaries' positions to the UK list (red-green). policyengine_api/data/local_authorities_2021.csv gains N09000011,-3.0,16.0,Ards and North Down after N09000010, and Cumberland, Westmorland and Furness, North Yorkshire and Somerset (E06000063 to E06000066) take their x and y from .py 6.10.0's local_authorities_2023.csv. The file name is unchanged. New tests compare every code, name and position with .py's list read from the installed package, and check that every offered local-authority name resolves to a code in .py's installed UK region registry by label, ignoring case, as the simulation API matches it. Red: 6 failures: 360 authorities instead of 361 in the metadata and the file, local_authority/Ards and North Down not offered, 10 Northern Ireland districts instead of 11, the file differing from .py's list, and the four unitaries still at (0, 0). The registry test passed before and after for the 360 existing names.

Merge of master (merged, not rebased, so the commits above keep their SHAs):

19. Merge origin/master into uk-local-area-breakdown. Brings in Raise the economy cache schema version so ×100 decile results stop being served #4026 and the 5.7.0 "Update PolicyEngine API" release commit. The one conflict was in tests/unit/services/test_reform_impacts_service.py, where both sides added a test at the end of the file; both tests are kept. constants.py, pyproject.toml, uv.lock, tests/unit/test_constants.py and the changelog merged without conflict: the schema version is 4, the UK region constants are kept, the package version is 5.7.0, and the lock keeps .py 6.10.0's dependency set.

Testing

Run locally on the merge of master (commit 19), with .py 6.10.0 installed from uv.lock:

• make format (no changes), ruff check and uv run --frozen --extra dev mypy: clean. uv lock --check: the lock matches pyproject.toml.
• tests/unit/test_constants.py, tests/unit/services/test_reform_impacts_service.py, tests/unit/services/test_economy_service.py, tests/unit/services/test_economy_service_request_plumbing.py, tests/unit/services/test_report_output_service.py and tests/unit/runtime_cache/: 374 passed. These are the file with the merge conflict and the tests that read the economy cache version.
• RUN_V2_CATALOG_COMPATIBILITY=1 pytest tests/integration/test_v2_catalog_installed.py: passed.
• python scripts/run_quality_guards.py: migration-contracts, rules-registry and ci-scripts passed.
• The full make test was not run on the merge; CI runs it.

Run locally on commit 18, before the merge of master:

• make test (unit, contract, tests/to_refactor and the budget-window dedupe integration test): 3381 passed on commit 18 (3344 before the first review's commits, 3365 before the second review's, 3373 before the third review's, 3376 before the local-authority commit).
• RUN_V2_CATALOG_COMPATIBILITY=1 pytest tests/integration/test_v2_catalog_installed.py: passed (it failed on Update PolicyEngine bundle to 6.10.0 #3991 with uk default dataset does not match the reviewed selection). It now also checks each UK region type's dataset.
• Focused: tests/unit/test_query_parameters.py, tests/unit/routes/test_economy_spm_query.py, tests/unit/test_openapi_spm.py, tests/unit/services/test_economy_service.py, tests/unit/services/test_economy_service_request_plumbing.py, tests/unit/libs/test_simulation_entrypoint.py, tests/contract/, tests/unit/v2/, tests/unit/test_constants.py, tests/unit/services/test_metadata_service.py: all passed.
• make format, ruff check (whole repository) and uv run --frozen --extra dev mypy: clean. economy_service.py is outside mypy's configured files; checking it directly gives the same 11 existing errors before and after this PR.
• python scripts/run_quality_guards.py: migration-contracts, rules-registry and ci-scripts passed.
• python scripts/export_migration_contracts.py: no change to the generated files.
• uv lock --check: the lock matches pyproject.toml.

Not run locally: the Postgres-backed integration tests (tests/integration/test_v2_catalog_publication.py, tests/integration/test_v2_metadata_routes.py), whose expected UK dataset name and label are updated here, and the container builds. CI runs them.

Environment variables

This PR changes docker/Dockerfile (the policyengine[models] pin only) and the .py pin that the bundle check reads. It adds no environment variables and reads none in new code.

• docker/Dockerfile has no ENV or ARG. It is built by pr.yml (test_container_builds) and push.yml (docker), which log in to ghcr.io with secrets.GITHUB_TOKEN, the token GitHub Actions provides to every job.
• "Ensure PolicyEngine bundle is supported by simulation API" runs in the staging GitHub environment with job-level env: SIM_ENTRYPOINT: ${{ vars.SIM_ENTRYPOINT }} (GitHub Actions environment variable) and SIMULATION_ENTRYPOINT_URL: ${{ secrets.SIMULATION_ENTRYPOINT_URL }} (GitHub Actions environment secret). Both names exist in the staging environment. .github/scripts/simulation_entrypoint_env.sh selects SIMULATION_ENTRYPOINT_URL when SIM_ENTRYPOINT=cloud_run_simulation_entrypoint, which is the value the Update PolicyEngine bundle to 6.10.0 #3991 run logged. That run reached the simulation API and reported only that 6.10.0 is not deployed, so the wiring works. If SIM_ENTRYPOINT were changed to old_gateway_direct, the job would need OLD_SIMULATION_GATEWAY_URL, which its env: block does not pass; that is unchanged by this PR.
• At runtime the breakdown request uses the existing simulation client and its existing variables (SIM_ENTRYPOINT, SIMULATION_ENTRYPOINT_URL or OLD_SIMULATION_GATEWAY_URL, and GATEWAY_AUTH_*), with no change to how they are provided.

Not covered here

• policyengine_bundle.dataset in a breakdown result names the dataset the simulation API's submit response reports, which is the national default; the result's own dataset field names the local-area file. This is unchanged here and is tracked in the cross-repository plan.
• docs/migration/stage-12-worker-contract.md describes the simulation API's Stage 12 adapter, which skips breakdown requests (sim-api#881); it is not updated here.

No inline # noqa suppressions are added.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 98.90110% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 87.79%. Comparing base (a4bd9d3) to head (37bef99).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
policyengine_api/services/economy_service.py	98.66%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #4024       +/-   ##
===========================================
+ Coverage   51.11%   87.79%   +36.67%     
===========================================
  Files         242      258       +16     
  Lines       14441    15950     +1509     
  Branches     2307     2621      +314     
===========================================
+ Hits         7382    14003     +6621     
+ Misses       6436     1280     -5156     
- Partials      623      667       +44     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[PavelMakarchuk]

Thanks Anthony, this looks good. I approve, with a few small notes.

I checked it locally on .py 6.10.0 (policyengine-uk 2.132.6): the make test set passes (3,385 tests), and ruff and mypy pass. The head already includes #4026, so the cache version stays e4… and nothing from 6.7.5 is reused.

• The breakdown has its own options hash, a non-local_areas result is never stored, and a failed breakdown expires after two minutes. This matches what app#1398 expects, including message on the 502 and policyengine_bundle.policyengine_version on cached results.
• The local authority list now matches .py's registry exactly (361 codes and names, no shared map positions), and the constituencies match too (650, 18 in NI).

Notes:

1. The unit "Test" job depends on the bundle check, so CI hasn't run the unit tests yet. After #881's deploy finishes, please re-run the whole workflow, not only the bundle job. Codecov's 11% patch number should also correct itself then.
2. country/ni and country/northern_ireland are both accepted for a breakdown, but they make separate cache entries, so the same NI run can happen twice. The app always sends country/ni, so this is minor. It could be a follow-up to normalise the region before the cache lookup.
3. For the record: this also moves UK household calculations to policyengine-uk 2.132.6 and the UK national default to Microcosm, so UK numbers will shift after it ships.