Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/v2-integration-check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -77,8 +77,8 @@ jobs:
run: uv run coverage run --branch -m pytest -q tests/integration/test_v2_catalog_installed.py
env:
RUN_V2_CATALOG_COMPATIBILITY: "1"
- name: Test v2 metadata publication and resource routes
run: uv run coverage run -a --branch -m pytest -q tests/integration/test_v2_catalog_publication.py tests/integration/test_v2_metadata_routes.py
- name: Test v2 metadata publication, resource routes and the countries list
run: uv run coverage run -a --branch -m pytest -q tests/integration/test_v2_catalog_publication.py tests/integration/test_v2_metadata_routes.py tests/integration/test_v2_country_routes.py
- name: Test v2 policy persistence and immediate v1 mirroring
run: uv run coverage run -a --branch -m pytest -q tests/integration/test_v2_policy_persistence.py tests/integration/test_v1_policy_dual_write.py tests/integration/test_mysql_policy_dual_write.py tests/integration/test_v2_user_policy_mirroring.py tests/integration/test_v1_user_policy_dual_write.py
- name: Test signed-in identity provisioning and user associations
Expand Down
1 change: 1 addition & 0 deletions changelog.d/v2-countries-route.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Added `GET /v2/countries`, which lists every row of the countries table: each served country's ID and whether it supports household and society-wide calculations.
1 change: 1 addition & 0 deletions changelog.d/v2-countries-startup-check.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
The Public API refuses to start when the v2 `countries` table and its country list in code name different countries, or when the table or the privilege to read it is missing. The check runs on Cloud Run and wherever a v2 runtime database is configured; when it cannot query the table, it logs why and startup continues.
1 change: 1 addition & 0 deletions changelog.d/v2-countries-table.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Added a v2 `countries` table that records, for each country the Public API serves, whether it supports household and society-wide calculations. Its revision lists the US and the UK, both with both. Users' primary country and the country of every v2 household, policy and user association now reference this table instead of a fixed US-and-UK check.
15 changes: 12 additions & 3 deletions docs/engineering/migration-contracts.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ Generated from `policyengine_api/migration_registry.py` and `tests/contract/regi
| Metric | Count |
| --- | ---: |
| route group count | 9 |
| workflow count | 16 |
| request count | 89 |
| workflow count | 17 |
| request count | 90 |
| db entity count | 6 |
| sim flow count | 3 |

Expand All @@ -18,7 +18,7 @@ Generated from `policyengine_api/migration_registry.py` and `tests/contract/regi
| --- | --- | --- | --- |
| `health` | `health`, `simulation-gateway-check`, `liveness-check`, `readiness-check` | `none` | `none` |
| `specification` | `specification` | `none` | `none` |
| `metadata` | `metadata`, `datasets`, `economy-options`, `parameter-values`, `parameters`, `regions`, `tax-benefit-model-versions`, `tax-benefit-models`, `variables` | `metadata` | `none` |
| `metadata` | `metadata`, `countries`, `datasets`, `economy-options`, `parameter-values`, `parameters`, `regions`, `tax-benefit-model-versions`, `tax-benefit-models`, `variables` | `metadata` | `none` |
| `policy` | `policy`, `policies`, `user-policy`, `user-policies` | `policy` | `none` |
| `household` | `household`, `households`, `user-household`, `user-households`, `calculate`, `calculate-full` | `household` | `household` |
| `economy` | `economy` | `simulation` | `economy` |
Expand Down Expand Up @@ -190,6 +190,15 @@ Generated from `policyengine_api/migration_registry.py` and `tests/contract/regi
| `GET` | `/us/metadata` | 200 | `metadata` | none | `status`, `result.current_law_id`, `result.economy_options.region`, `result.economy_options.time_period` |
| `GET` | `/uk/metadata` | 200 | `metadata` | none | `status`, `result.current_law_id`, `result.economy_options.region`, `result.economy_options.time_period` |

### `country_selection_v2`

- Current contract: `typed_v2_resources`
- Future owner: Countries table and one country list in code

| Method | Path | Status | Route group | Required headers | Stable response fields |
| --- | --- | ---: | --- | --- | --- |
| `GET` | `/v2/countries` | 200 | `metadata` | none | `status`, `message`, `result.items` |

### `metadata_resources_v2_preview`

- Current contract: `typed_v2_resources`
Expand Down
30 changes: 30 additions & 0 deletions docs/engineering/skills/alembic-migrations.md
Original file line number Diff line number Diff line change
Expand Up @@ -294,3 +294,33 @@ last dependent table is dropped. Its only post-generation correction drops the
nine generated `v2_*` enum types at the end of the baseline downgrade. The v2
Postgres lifecycle test covers empty upgrade, downgrade to base, and re-upgrade
so stale enum types cannot make the generated baseline non-reversible.

### API v2 countries rows

Revision `fb1a3e40d3d6` was autogenerated after adding the `countries` table
and replacing the US-and-UK checks on `users.primary_country` and on the
`country_id` of households, policies and the four user association tables
with foreign keys to it. It contains one hand-written operation: it inserts
the `uk` and `us` rows, both household and society-wide capable, between the
generated table creation and the generated foreign keys, which otherwise fail
on any database whose rows already name either country. The rows are exactly
the values the replaced checks allowed. The generated downgrade removes them
with the table, and its restored checks fail if any record names another
country.

Inserting rows is a data operation, which this skill otherwise forbids. The
maintainers approved this insert as a one-off exception, because the new
foreign keys need the rows in the same revision. Unit tests pin the
operation, and the v2 Postgres lifecycle test covers an upgrade that keeps
existing rows, the downgrade, a downgrade refused by another country's
records, and the re-upgrade.

The table must list exactly the countries in
`policyengine_api/supported_countries.py`: the Public API refuses to start
when they differ (`policyengine_api/startup_country_check.py`). Listing
another country later needs a revision that inserts its row, and the same
human migration decision. The deploy runs that revision before it starts
the new release, so instances of the previous release that start afterwards,
and a rollback to it, refuse to start until the revision is downgraded. The
module docstring of `supported_countries.py` lists the other checks a new
country must widen.
24 changes: 22 additions & 2 deletions docs/generated/migration_contracts.json
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
{
"metadata": {
"db_entity_count": 6,
"request_count": 89,
"request_count": 90,
"route_group_count": 9,
"sim_flow_count": 3,
"workflow_count": 16
"workflow_count": 17
},
"route_groups": [
{
Expand All @@ -31,6 +31,7 @@
"name": "metadata",
"path_segments": [
"metadata",
"countries",
"datasets",
"economy-options",
"parameter-values",
Expand Down Expand Up @@ -1348,6 +1349,25 @@
}
]
},
{
"current_contract": "typed_v2_resources",
"future_owner_pr": "Countries table and one country list in code",
"name": "country_selection_v2",
"requests": [
{
"expected_status": 200,
"headers": [],
"method": "GET",
"path": "/v2/countries",
"route_group": "metadata",
"stable_response_fields": [
"status",
"message",
"result.items"
]
}
]
},
{
"current_contract": "typed_v2_resources",
"future_owner_pr": "Later metadata read cutover and v2 route-prefix removal",
Expand Down
4 changes: 2 additions & 2 deletions docs/generated/rules.md
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,7 @@ flowchart LR
| --- | --- | --- | --- | --- | --- | --- | --- |
| `api.catalog.initializer-not-at-startup` | Application startup modules never import the metadata catalog initializer. | mechanical | enforced | yes | pytest `tests/unit/v2/test_catalog_initialization.py` | `pr.yml:test` | **unproven** |
| `api.startup.fail-closed-configuration` | Default Cloud SQL startup needs no Supabase configuration, and a selected v2 runtime without its database URL fails closed. | mechanical | enforced | yes | pytest `tests/unit/v2/test_import_side_effects.py` | `pr.yml:test` | **unproven** |
| `api.startup.no-import-side-effects` | Importing v2 modules, starting up and serving a request open no network connections and create no files or runtime SQLite databases. | mechanical | enforced | yes | pytest `tests/unit/v2/test_import_side_effects.py` | `pr.yml:test` | **unproven** |
| `api.startup.no-import-side-effects` | Importing v2 modules opens no network connections, starting up and serving a request open none unless the process runs on Cloud Run or has a v2 runtime database configured, and none of them creates files or runtime SQLite databases. | mechanical | enforced | yes | pytest `tests/unit/v2/test_import_side_effects.py` | `pr.yml:test` | **unproven** |
| `api.startup.no-runtime-sqlite-modules` | Runtime SQLite modules are absent from the production package. | mechanical | enforced | yes | pytest `tests/unit/v2/test_import_side_effects.py` | `pr.yml:test` | **unproven** |

```mermaid
Expand Down Expand Up @@ -294,7 +294,7 @@ flowchart LR
| `api.quality.command-line-modules-ruff-exemptions` | Application modules that also run as command-line programs (if __name__ == "__main__") may print their output (T201); every other ruff rule applies to them. | mechanical | enforced | yes | ruff per-file ignore of T201 for `policyengine_api/{data/v2/catalog/initialization,data/v2/household_migration_qualification,data/v2/policy_migration_qualification,data/v2/privilege_reconciliation,data/v2/user_association_migration_qualification,worker_spm_release}.py` | `pr.yml:lint` | — |
| `api.quality.command-line-scripts-ruff-exemptions` | Command-line scripts under scripts/ may print their output, run other programs, fetch fixed URLs, stamp today's date and pick random samples (T201, S603, S607, S310, DTZ011, S311); every other ruff rule applies to them. | mechanical | enforced | yes | ruff per-file ignore of DTZ011, S310, S311, S603, S607, T201 for `scripts/**` | `pr.yml:lint` | — |
| `api.quality.environment-variables-load` | Required environment variables and credentials load in a deployed-like environment. | mechanical | enforced | yes | CI job `test_env_vars` | `pr.yml:test_env_vars` | — |
| `api.quality.mypy-v2` | Shared query models, FastAPI dependencies, v2 routes and services, and the user sign-in and account-deletion libraries pass mypy with untyped definitions disallowed. | mechanical | enforced | yes | mypy | `pr.yml:typecheck-v2` | — |
| `api.quality.mypy-v2` | Shared query models, FastAPI dependencies, v2 routes and services, the user sign-in and account-deletion libraries, the country list and its startup check pass mypy with untyped definitions disallowed. | mechanical | enforced | yes | mypy | `pr.yml:typecheck-v2` | — |
| `api.quality.ruff-format` | All Python files are formatted with ruff format. | mechanical | enforced | yes | ruff format | `pr.yml:lint` | — |
| `api.quality.ruff-lint` | All Python code passes ruff check with every ruff rule selected (ALL), with no ratchet, except the ignores in pyproject.toml: COM812 and ISC001, which conflict with ruff format, and D203 and D213, which conflict with the kept D211 and D212. | mechanical | enforced | yes | ruff check (ALL) | `pr.yml:lint` | — |
| `api.quality.runtime-evaluated-models-resolve` | Every package model is one ruff treats as runtime-evaluated (a base resolves, through the import in its own module, to a name in runtime-evaluated-base-classes, or is a same-module class that does), and every Pydantic model and SQLAlchemy mapper resolves its annotations once the package is imported. | mechanical | enforced | yes | pytest `tests/unit/test_runtime_evaluated_annotations.py` | `pr.yml:test` | **unproven** |
Expand Down
195 changes: 195 additions & 0 deletions migrations/v2/versions/fb1a3e40d3d6_add_countries_table.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,195 @@
"""add countries table

Revision ID: fb1a3e40d3d6
Revises: fbb7488ebe8a
Create Date: 2026-10-22 02:16:50.126715
Generation: uv run alembic -c alembic-v2.ini revision --autogenerate
"""

from typing import Sequence, Union

from alembic import op
import sqlalchemy as sa
import sqlmodel


revision: str = "fb1a3e40d3d6"
down_revision: Union[str, None] = "fbb7488ebe8a"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None


def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
countries = op.create_table(
"countries",
sa.Column("id", sqlmodel.sql.sqltypes.AutoString(length=2), nullable=False),
sa.Column("household_enabled", sa.Boolean(), nullable=False),
sa.Column("society_wide_enabled", sa.Boolean(), nullable=False),
sa.CheckConstraint(
"household_enabled OR NOT society_wide_enabled",
name=op.f("ck_countries_society_wide_requires_household"),
),
sa.PrimaryKeyConstraint("id", name=op.f("pk_countries")),
)
# Approved hand-written data operation: the one exception to generated-only
# revisions, decided by the maintainers for this table alone. The foreign
# keys below fail on any database whose rows already name the US or the UK
# unless both countries are listed first, so their rows are inserted here,
# in the same revision. They are the countries the replaced checks
# allowed, and both support household and society-wide calculations. The
# generated downgrade removes them with the table. Listing another
# country later needs its own revision and the same kind of decision.
op.bulk_insert(
countries,
[
{"id": "uk", "household_enabled": True, "society_wide_enabled": True},
{"id": "us", "household_enabled": True, "society_wide_enabled": True},
],
)
op.create_foreign_key(
op.f("fk_households_country_id_countries"),
"households",
"countries",
["country_id"],
["id"],
ondelete="RESTRICT",
)
op.drop_constraint(op.f("ck_households_country"), "households", type_="check")
op.create_foreign_key(
op.f("fk_policies_country_id_countries"),
"policies",
"countries",
["country_id"],
["id"],
ondelete="RESTRICT",
)
op.drop_constraint(op.f("ck_policies_country"), "policies", type_="check")
op.create_foreign_key(
op.f("fk_user_household_associations_country_id_countries"),
"user_household_associations",
"countries",
["country_id"],
["id"],
ondelete="RESTRICT",
)
op.drop_constraint(
op.f("ck_user_household_associations_country"),
"user_household_associations",
type_="check",
)
op.create_foreign_key(
op.f("fk_user_policies_country_id_countries"),
"user_policies",
"countries",
["country_id"],
["id"],
ondelete="RESTRICT",
)
op.drop_constraint(op.f("ck_user_policies_country"), "user_policies", type_="check")
op.create_foreign_key(
op.f("fk_user_report_associations_country_id_countries"),
"user_report_associations",
"countries",
["country_id"],
["id"],
ondelete="RESTRICT",
)
op.drop_constraint(
op.f("ck_user_report_associations_country"),
"user_report_associations",
type_="check",
)
op.create_foreign_key(
op.f("fk_user_simulation_associations_country_id_countries"),
"user_simulation_associations",
"countries",
["country_id"],
["id"],
ondelete="RESTRICT",
)
op.drop_constraint(
op.f("ck_user_simulation_associations_country"),
"user_simulation_associations",
type_="check",
)
op.create_foreign_key(
op.f("fk_users_primary_country_countries"),
"users",
"countries",
["primary_country"],
["id"],
ondelete="RESTRICT",
)
op.drop_constraint(op.f("ck_users_primary_country"), "users", type_="check")
# ### end Alembic commands ###


def downgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.create_check_constraint(
op.f("ck_users_primary_country"),
"users",
"primary_country::text = ANY (ARRAY['us'::character varying, 'uk'::character varying]::text[])",
)
op.drop_constraint(
op.f("fk_users_primary_country_countries"), "users", type_="foreignkey"
)
op.create_check_constraint(
op.f("ck_user_simulation_associations_country"),
"user_simulation_associations",
"country_id::text = ANY (ARRAY['us'::character varying, 'uk'::character varying]::text[])",
)
op.drop_constraint(
op.f("fk_user_simulation_associations_country_id_countries"),
"user_simulation_associations",
type_="foreignkey",
)
op.create_check_constraint(
op.f("ck_user_report_associations_country"),
"user_report_associations",
"country_id::text = ANY (ARRAY['us'::character varying, 'uk'::character varying]::text[])",
)
op.drop_constraint(
op.f("fk_user_report_associations_country_id_countries"),
"user_report_associations",
type_="foreignkey",
)
op.create_check_constraint(
op.f("ck_user_policies_country"),
"user_policies",
"country_id::text = ANY (ARRAY['us'::character varying, 'uk'::character varying]::text[])",
)
op.drop_constraint(
op.f("fk_user_policies_country_id_countries"),
"user_policies",
type_="foreignkey",
)
op.create_check_constraint(
op.f("ck_user_household_associations_country"),
"user_household_associations",
"country_id::text = ANY (ARRAY['us'::character varying, 'uk'::character varying]::text[])",
)
op.drop_constraint(
op.f("fk_user_household_associations_country_id_countries"),
"user_household_associations",
type_="foreignkey",
)
op.create_check_constraint(
op.f("ck_policies_country"),
"policies",
"country_id::text = ANY (ARRAY['us'::character varying, 'uk'::character varying]::text[])",
)
op.drop_constraint(
op.f("fk_policies_country_id_countries"), "policies", type_="foreignkey"
)
op.create_check_constraint(
op.f("ck_households_country"),
"households",
"country_id::text = ANY (ARRAY['us'::character varying, 'uk'::character varying]::text[])",
)
op.drop_constraint(
op.f("fk_households_country_id_countries"), "households", type_="foreignkey"
)
op.drop_table("countries")
# ### end Alembic commands ###
5 changes: 5 additions & 0 deletions policyengine_api/asgi.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
from policyengine_api.observability import get_runtime
from policyengine_api.readiness import mark_not_ready, mark_ready
from policyengine_api.runtime_cache.client import close_runtime_cache_clients
from policyengine_api.startup_country_check import check_countries_table_at_startup
from policyengine_api.warmup import run_startup_warmup


Expand All @@ -24,6 +25,10 @@ def _close_runtime_resources() -> None:
shutdown_callback=_close_runtime_resources,
)

# Refuse to start when the v2 countries table and the country list in code
# disagree (see policyengine_api.startup_country_check).
check_countries_table_at_startup()

# Warm the simulation machinery before serving (see policyengine_api.warmup).
# POLICYENGINE_API_STARTUP_WARMUP=0 skips it.
if os.environ.get("POLICYENGINE_API_STARTUP_WARMUP", "1") != "0":
Expand Down
3 changes: 2 additions & 1 deletion policyengine_api/data/v2/catalog/catalog_selection.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,9 @@
TaxBenefitModel,
TaxBenefitModelVersion,
)
from policyengine_api.supported_countries import SUPPORTED_COUNTRY_IDS

SUPPORTED_V2_COUNTRY_IDS = frozenset({"us", "uk"})
SUPPORTED_V2_COUNTRY_IDS = frozenset(SUPPORTED_COUNTRY_IDS)
MAXIMUM_POLICYENGINE_VERSION_LENGTH = 128


Expand Down
Loading
Loading