Request UK local-area breakdowns, fix the Northern Ireland region name, and update .py to 6.10.0

changelog.d/uk-local-area-breakdown-national-results.changed.md

@@ -0,0 +1 @@
+After the .py 6.10.0 update, UK national and country economy results (`region=uk` or `country/*`) no longer include `constituency_impact` or `local_authority_impact`, because .py runs them on a national dataset with no area codes. Clients get those impacts with a separate request that adds `breakdown=local_areas`.

changelog.d/uk-local-area-breakdown.added.md

@@ -0,0 +1 @@
+The annual economy route accepts `breakdown=local_areas`, which requests the UK constituency and local-authority impacts in a run separate from the national result. It is accepted only for the UK, with region `uk` or one UK country (`country/england`, `country/scotland`, `country/wales`, `country/ni`, or `country/northern_ireland`, the code .py uses), and not with `target=cliff`; other requests return 400. The option gets its own options hash and cached result, and is sent to the simulation API only when set, so requests without it are unchanged. If the simulation API refuses a breakdown with 400 or 422, the route returns 502 with the reason instead of a generic 500. A breakdown job that fails, or that completes without `kind: "local_areas"`, returns 502; its error is kept for two minutes instead of 30 days, and after that the next request submits a new job.

changelog.d/uk-local-area-breakdown.changed.md

@@ -0,0 +1 @@
+With .py 6.10.0, the UK default dataset is `microcosm_uk_2024_25`: the v2 catalog extractor accepts it as the reviewed UK default, and the metadata labels it "Microcosm", as it does the US Microcosm dataset, instead of the generic "Certified dataset". When the simulation API refuses a budget-window submission with a request-validation (422) body, the error message now names each field and its problem (for example `body.window_size: Input should be less than or equal to 75`) instead of quoting the raw error list, as breakdown refusals do.

changelog.d/uk-local-area-breakdown.fixed.md

@@ -0,0 +1 @@
+UK economy requests for Northern Ireland (`country/ni` in the Public API metadata) now send `country/northern_ireland`, the name .py's UK region registry uses, to the simulation API. The cached result keeps the `country/ni` region. The v2 catalog gives UK constituency and local-authority regions the local-area dataset .py 6.10.0 routes them to (`microcosm_uk_2024_25_dense`) and lists that dataset, instead of the national file, which has no area codes. The UK metadata now offers Ards and North Down (`N09000011`), the 11th Northern Ireland district, which the local-authority list lacked, so it lists 361 local authorities. The four unitary authorities created in April 2023 (Cumberland, Westmorland and Furness, North Yorkshire and Somerset) now have their own hex positions instead of the shared (0, 0) placeholder. The list's codes, names and positions now match .py 6.10.0's packaged 2023 list.

changelog.d/update-policyengine-bundle-6.10.0.changed.md

@@ -0,0 +1 @@
+Update the PolicyEngine bundle to 6.10.0.

docker/Dockerfile

@@ -1,4 +1,4 @@
 FROM python:3.12
 # Match the API bundle in pyproject.toml; the bundle updater changes both pins.
 # The models extra pins the complete certified runtime package set.
-RUN pip install "policyengine[models]==6.7.5" ipython
+RUN pip install "policyengine[models]==6.10.0" ipython

docs/canonical-spm.md

@@ -319,10 +319,34 @@ HTTP 400. An omitted `spm` inherits the certified default measurement, while a
 misspelled parameter is rejected. `region` is required;
 annual requests require `time_period`, while budget-window requests require
 `start_year` and `window_size` (1–75, ending no later than 2099). Years use four
-digits. Optional fields are `version` (installed country model version) and
-`target` (`general`, or annual-only `cliff`). Omitted `spm` inherits certified
-bundle defaults. The served `/specification` generates these query declarations
-from the same typed models as the Flask parser.
+digits. Optional fields are `version` (installed country model version),
+`target` (`general`, or annual-only `cliff`) and the annual-only `breakdown`.
+Omitted `spm` inherits certified bundle defaults. The served `/specification`
+generates these query declarations from the same typed models as the Flask
+parser.
+
+`breakdown=local_areas` asks the annual route for the UK constituency and
+local-authority impacts only. It is a calculation option, so the request has
+its own options hash and its own cached result. Any other value returns HTTP
+400, and requests without `breakdown` keep their existing options hash. It is
+accepted only for the UK, with `region` exactly `uk` or one UK country
+(`country/england`, `country/scotland`, `country/wales` or `country/ni`, or
+.py's `country/northern_ireland`), and not with `target=cliff`; other
+requests, including other spellings of those regions, return HTTP 400 before
+anything is read from the cache or submitted. The API sends `"breakdown": "local_areas"` to the simulation API,
+which computes the impacts in a run separate from the national result, and
+returns its result (`kind: "local_areas"`) like any other economy result. If
+the simulation API refuses the request with HTTP 400 or 422, because the
+deployed simulation API or the .py release it routes to cannot run breakdowns,
+the route returns HTTP 502 with the simulation API's reason and stores
+nothing, so a later request tries again. A breakdown job that fails (for
+example by running out of time or memory on the local-area dataset), or that
+completes without `kind: "local_areas"`, also returns HTTP 502. Its result is
+not stored. Its error is kept for two minutes on that job's own record, so
+every request in that time gets the same error, which says when a new
+breakdown can be requested; then the record expires and the next request
+submits a new job. Other jobs' records are not changed. A failed national job
+is still stored and returned again for the cache lifetime of 30 days.
 
 Before cache access or submission,
 the API validates the selected worker application's `canonical-spm-v1` capability

docs/migration/stage-9-v2-metadata.md

@@ -106,8 +106,11 @@ congressional-district, or place region uses its PolicyEngine.py regional
 alternative when one is available. Until a later reviewed change removes the
 fallback, a subnational region without such an alternative uses
 `populace_us_2024`. The command emits one summary warning containing only the
-affected region types and counts. UK regions use the
-`enhanced_frs_2024_25` default certified by PolicyEngine.py 5.0.4.
+affected region types and counts. The UK national and country regions use
+`microcosm_uk_2024_25`, the default certified by PolicyEngine.py 6.10.0; a
+country filters that national file. UK constituency and local-authority
+regions use the dataset PolicyEngine.py routes them to,
+`microcosm_uk_2024_25_dense`, because the national file has no area codes.
 
 Review the fallback summary on every new PolicyEngine.py release. A changed
 count may identify an upstream catalog change that needs a reviewed dataset

policyengine_api/constants.py

@@ -290,6 +290,25 @@ def get_report_output_cache_version(country_id: str) -> str:
     ],
 }
 
+# The UK country regions the metadata publishes, as (region name, label).
+UK_COUNTRY_REGIONS = (
+    ("country/england", "England"),
+    ("country/scotland", "Scotland"),
+    ("country/wales", "Wales"),
+    ("country/ni", "Northern Ireland"),
+)
+# UK regions whose metadata name differs from the code .py's UK region
+# registry uses, mapped to that code.
+UK_REGION_REGISTRY_CODES = {"country/ni": "country/northern_ireland"}
+# The regions a UK local-area breakdown can cover, in the order messages and
+# the query description list them: the UK, each UK country the metadata
+# publishes, and .py's own code for a country the metadata names differently.
+LOCAL_AREA_BREAKDOWN_REGIONS = (
+    "uk",
+    *(name for name, _label in UK_COUNTRY_REGIONS),
+    *UK_REGION_REGISTRY_CODES.values(),
+)
+
 # Modal simulation API status values
 MODAL_EXECUTION_STATUS_SUBMITTED = "submitted"
 MODAL_EXECUTION_STATUS_RUNNING = "running"

policyengine_api/country.py

@@ -28,6 +28,7 @@
 from policyengine_core.types import ArrayLike
 
 from policyengine_api.constants import (
+    UK_COUNTRY_REGIONS,
     get_bundle_default_dataset_option,
 )
 from policyengine_api.data.congressional_districts import (
@@ -203,10 +204,10 @@ def build_microsimulation_options(self) -> dict:
             constituency_names = constituency_names.sort_values("name")
             region = [
                 {"name": "uk", "label": "the UK", "type": "national"},
-                {"name": "country/england", "label": "England", "type": "country"},
-                {"name": "country/scotland", "label": "Scotland", "type": "country"},
-                {"name": "country/wales", "label": "Wales", "type": "country"},
-                {"name": "country/ni", "label": "Northern Ireland", "type": "country"},
+                *(
+                    {"name": name, "label": label, "type": "country"}
+                    for name, label in UK_COUNTRY_REGIONS
+                ),
             ]
             region.extend(
                 {

policyengine_api/data/local_authorities_2021.csv

@@ -58,10 +58,10 @@ E06000059,-1.0,0.0,Dorset
 E06000060,5.0,5.0,Buckinghamshire
 E06000061,9.0,9.0,North Northamptonshire
 E06000062,7.0,6.0,West Northamptonshire
-E06000063,0.0,0.0,Cumberland
-E06000064,0.0,0.0,Westmorland and Furness
-E06000065,0.0,0.0,North Yorkshire
-E06000066,0.0,0.0,Somerset
+E06000063,4.0,18.0,Cumberland
+E06000064,4.0,17.0,Westmorland and Furness
+E06000065,8.0,17.0,North Yorkshire
+E06000066,0.0,1.0,Somerset
 E07000008,12.0,8.0,Cambridge
 E07000009,12.0,9.0,East Cambridgeshire
 E07000010,13.0,10.0,Fenland
@@ -305,6 +305,7 @@ N09000007,-5.0,15.0,Lisburn and Castlereagh
 N09000008,-4.0,18.0,Mid and East Antrim
 N09000009,-5.0,17.0,Mid Ulster
 N09000010,-4.0,15.0,"Newry, Mourne and Down"
+N09000011,-3.0,16.0,Ards and North Down
 S12000005,2.0,24.0,Clackmannanshire
 S12000006,4.0,20.0,Dumfries and Galloway
 S12000008,3.0,20.0,East Ayrshire

policyengine_api/data/v2/catalog/extraction.py

@@ -39,7 +39,7 @@
 )
 REVIEWED_DEFAULT_DATASETS = {
     "us": "populace_us_2024",
-    "uk": "enhanced_frs_2024_25",
+    "uk": "microcosm_uk_2024_25",
 }
 CURRENT_LAW_IDS = {"us": 2, "uk": 1}
 METADATA_TIME_PERIODS = {
@@ -645,8 +645,12 @@ def _select_region_datasets(
 ) -> _RegionDatasetSelection:
     """Choose each region's logical input dataset.
 
-    A US subnational region with a dataset path uses that dataset; every other
-    region uses the reviewed default, and the national region may only name it.
+    A subnational region with a dataset path uses that dataset: US states and
+    districts, and the UK constituencies and local authorities that .py routes
+    to its local-area release. Every other region, such as a UK country that
+    filters the national file, uses the reviewed default, and the national
+    region may only name it. A US subnational region without a path is
+    counted as a fallback.
     """
     selected_dataset_names: set[str] = {reviewed_default}
     region_dataset_names: dict[str, str] = {}
@@ -666,7 +670,7 @@ def _select_region_datasets(
             msg = f"region {code!r} has unsupported type {region_type!r}"
             raise CatalogExtractionError(msg)
         dataset_path = getattr(region, "dataset_path", None)
-        if country_id == "us" and region_type != "national" and dataset_path:
+        if region_type != "national" and dataset_path:
             dataset_name = _dataset_name_from_path(dataset_path)
             selected_dataset_names.add(dataset_name)
         else:

policyengine_api/dataset_display.py

@@ -3,7 +3,9 @@
 from __future__ import annotations
 
 DATASET_FAMILY_DISPLAY_LABELS = (
+    # US Microcosm datasets are named populace_*; UK ones microcosm_*.
     ("populace_", "Microcosm"),
+    ("microcosm_", "Microcosm"),
     ("enhanced_frs_", "Enhanced FRS"),
 )
 DEFAULT_DATASET_DISPLAY_LABEL = "Certified dataset"

policyengine_api/openapi_spec.yaml

@@ -879,7 +879,7 @@ paths:
         # Query parameters are generated from policyengine_api.query_parameters.
       responses:
         200:
-          description: Annual economic impact, progress, or persisted error state. Canonical completed results include actual baseline/reform receipts containing the selected SPM configuration. While computing, the optional stage fields come from the simulation entrypoint and appear only when its worker reports stages.
+          description: Annual economic impact, progress, or persisted error state. Canonical completed results include actual baseline/reform receipts containing the selected SPM configuration. While computing, the optional stage fields come from the simulation entrypoint and appear only when its worker reports stages. With breakdown=local_areas, a completed result is the simulation API's local-area breakdown instead, with kind local_areas, the constituency_impact and local_authority_impact lists, and the dataset and version fields.
           content:
             application/json:
               schema:
@@ -945,6 +945,21 @@ paths:
                     type: string
                   message:
                     type: string
+        502:
+          description: The calculation failed, or the simulation API refused a breakdown=local_areas request it cannot run. The message gives the reason. A failed breakdown=local_areas job, or one whose result is not kind local_areas, keeps its error for two minutes only; after that the next request submits a new job.
+          content:
+            application/json:
+              schema:
+                type: object
+                properties:
+                  status:
+                    type: string
+                    enum: [error]
+                  message:
+                    type: string
+                  result:
+                    type: object
+                    nullable: true
   /{country_id}/economy/{policy_id}/over/{baseline_policy_id}/budget-window:
     get:
       summary: Calculate budget-window economic impacts

policyengine_api/query_parameters.py

@@ -25,9 +25,11 @@
     model_validator,
 )
 
+from policyengine_api.constants import LOCAL_AREA_BREAKDOWN_REGIONS
 from policyengine_api.data.v2.catalog.catalog_selection import (
     validate_policyengine_version,
 )
+from policyengine_api.services.economic_impact_request import LocalAreaBreakdown
 from policyengine_api.spm import SPMSelection
 from policyengine_api.utils.budget_window import (
     BUDGET_WINDOW_MAX_END_YEAR,
@@ -168,6 +170,22 @@ class AnnualEconomyQuery(EconomyQuery):
 
     time_period: EconomyYear
     target: Literal["general", "cliff"] = "general"
+    breakdown: LocalAreaBreakdown | None = Field(
+        default=None,
+        description=(
+            "local_areas computes only the UK constituency and local-authority "
+            "impacts, in a run separate from the national result. UK only, with "
+            f"region one of: {', '.join(LOCAL_AREA_BREAKDOWN_REGIONS)}; "
+            "not with target=cliff"
+        ),
+    )
+
+    def calculation_options(self) -> dict[str, Any]:
+        """Add ``breakdown`` only when requested, so other options hashes stay put."""
+        options = super().calculation_options()
+        if self.breakdown is not None:
+            options["breakdown"] = self.breakdown
+        return options
 
 
 class BudgetWindowEconomyQuery(EconomyQuery):

policyengine_api/runtime_cache/reform_impacts.py

@@ -364,11 +364,15 @@ def set(
         impact: CachedReformImpact,
         *,
         record_execution_id: str | None = None,
+        record_ttl_seconds: int | None = None,
     ) -> bool:
         """Store an impact and index it by scope and recency in one transaction.
 
-        Each index keeps only the newest 1,000 entries. Return False if the
-        write failed; raise ValueError when there is no record identifier.
+        Each index keeps only the newest 1,000 entries. ``record_ttl_seconds``
+        gives this record a shorter lifetime than the usual 30 days; the
+        indexes keep theirs, and an index entry whose record has expired is
+        skipped when read. Return False if the write failed; raise ValueError
+        when there is no record identifier.
         """
         # Terminal errors clear the worker handle in the payload while keeping
         # the original record key and lookup indexes stable.
@@ -388,7 +392,7 @@ def set(
                         REFORM_IMPACT_SCHEMA_VERSION,
                         _impact_to_wire(impact),
                     ),
-                    ex=ttl_seconds,
+                    ex=record_ttl_seconds or ttl_seconds,
                 )
                 for index in indexes:
                     pipeline.zadd(index, {record_key: self._score(impact)})
@@ -506,20 +510,27 @@ def update(
         execution_id: str,
         *,
         clear_execution_id: bool = False,
+        record_ttl_seconds: int | None = None,
         **changes: Any,  # noqa: ANN401  # forwarded unchanged to dataclasses.replace
     ) -> CachedReformImpact | None:
         """Apply field changes to a cached impact and rewrite it under the same key.
 
-        ``clear_execution_id`` drops the worker handle without moving the record.
-        Return the updated impact, or None if it is missing or the write failed.
+        ``clear_execution_id`` drops the worker handle without moving the record,
+        and ``record_ttl_seconds`` shortens its lifetime (see ``set``). Return the
+        updated impact, or None if it is missing or the write failed.
         """
         impact = self.get_by_execution_id(execution_id)
         if impact is None:
             return None
         if clear_execution_id:
             changes["execution_id"] = None
         updated = replace(impact, **changes)
-        return updated if self.set(updated, record_execution_id=execution_id) else None
+        stored = self.set(
+            updated,
+            record_execution_id=execution_id,
+            record_ttl_seconds=record_ttl_seconds,
+        )
+        return updated if stored else None
 
     def delete_matching_computing(
         self,

policyengine_api/services/economic_impact_request.py

@@ -3,6 +3,12 @@
 from dataclasses import dataclass
 from typing import Literal
 
+# The one breakdown the annual economy route can request: the UK
+# constituency and local-authority impacts, computed apart from the national
+# result. The query model, the simulation payload and the service share it.
+LocalAreaBreakdown = Literal["local_areas"]
+LOCAL_AREA_BREAKDOWN: LocalAreaBreakdown = "local_areas"
+
 
 @dataclass(frozen=True)
 class EconomicImpactRequest: