Skip to content

Add a v2 countries table that matches one country list in code, GET /v2/countries and a startup check - #4012

Draft
anth-volk wants to merge 6 commits into
masterfrom
v2plan/1g-126-countries-table
Draft

anth-volk wants to merge 6 commits into
masterfrom
v2plan/1g-126-countries-table

Conversation

@anth-volk

@anth-volk anth-volk commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #4009

Summary

  • One country list in code. policyengine_api/supported_countries.py holds SupportedCountryId ("us", "uk") and SUPPORTED_COUNTRY_IDS. The v2 routes' CountryId, catalog extraction's country list, required packages and installed models, the catalog's supported-country check, and the household copy command's --country-id choices all derive from it. It lives in its own module rather than in query_parameters.py, because the query models import the catalog package, which must import the list too.
  • The countries table. One row per served country, with household_enabled and society_wide_enabled (a check requires a society-wide country to be household capable). users.primary_country and the country_id of households, policies and the four user association tables reference countries.id with ON DELETE RESTRICT, replacing their IN ('us', 'uk') checks. The tables that pair v1 records with their v2 copies keep their checks.
  • Revision fb1a3e40d3d6 was autogenerated. Its one hand-written operation inserts the uk and us rows, both capable of both, between the table and the foreign keys, which otherwise fail on any database whose rows already name either country. This is an approved one-off exception to the rule that revisions contain only generated operations; the revision's comment and the Alembic skill say so, and upgrade and downgrade tests cover it. The generated downgrade restores the checks and drops the table.
  • GET /v2/countries lists every row, ordered by ID. It takes no query parameters (422 otherwise) and no credential, answers 503 when the v2 database is not configured or reachable, and a fixed 500 for anything else. The runtime role gets SELECT on countries in the privilege manifest.
  • Startup check. Importing policyengine_api.asgi now checks that the table names exactly the countries in the code list. On Cloud Run (K_SERVICE set), or wherever a v2 runtime database is configured, a worker whose check finds different countries, a missing table or no SELECT privilege raises, so it never serves and the tagged candidate fails its health checks. When the check cannot query the table (unusable v2 configuration, a secret that cannot be resolved, a database that cannot be reached or refuses the credentials), it logs why and startup continues: the v1 routes do not read the v2 database, and readiness already reports unusable v2 configuration as not ready. Local and test startups without v2 configuration open no connection.

Consequence for adding a country later

A deploy runs migrations before it starts the new release. After a revision that adds a country's row, instances of the previous release that start (scale-out during the deploy, or a rollback) refuse to start until that revision is downgraded. The country list's docstring and the Alembic skill say this, and list the US-and-UK checks a new country must also widen (the v1-to-v2 pairing tables and v1 household_mirror_events).

Migration notes

  • Route groups covered: metadata gains /v2/countries (new workflow country_selection_v2 in the contract registry; generated contract documents refreshed).
  • What stays on the Flask/v1 path: every v1 route, unchanged.
  • Newly prepared: the v2 countries table and its foreign keys in Supabase, and the startup check on Cloud Run.
  • User-visible API changes: the new route only. Existing v2 responses are unchanged.

Rules

This PR changes rules/registry.toml, so it needs a review from a core developer other than the author:

  • api.startup.no-import-side-effects now says that starting up opens no network connections unless the process runs on Cloud Run or has a v2 runtime database configured, which is what the startup check makes true. Its tests are unchanged and still cover startup without v2 configuration.
  • api.quality.mypy-v2 adds the country list and the startup check module (also added to [tool.mypy].files).

docs/generated/rules.md is regenerated.

Inline suppressions

One added: # noqa: E402 on the Country import in policyengine_api/data/v2/models/__init__.py. Every model import in that module carries it, because the module must set the metadata naming convention before any table is declared, so the imports cannot move to the top.

Environment variables

The startup check reads:

  • K_SERVICE: set by Cloud Run on every revision; the gunicorn workers inherit it. Not set on GitHub runners.
  • V2_RUNTIME_DATABASE_URL_SECRET_RESOURCE: GitHub environment secret (staging, production). Job env: of the staging and production deploy jobs in push.yml → exported by cloud_run_env.sh → required by validate_cloud_run_deploy_env.sh → set on the revision by deploy_cloud_run_candidate.sh (--set-env-vars). Existing wiring; run_release_tests.sh unsets it, so make test skips the check.
  • V2_SUPABASE_PROJECT_REF, V2_SUPABASE_ENVIRONMENT: GitHub environment variables, same path. Read through the existing load_v2_runtime_database_settings.
  • V2_RUNTIME_DATABASE_URL: read only to decide whether to run the check. Not set on Cloud Run; the one workflow that sets it (process-v1-household-mirror-event.yml, from the V2_DATA_WRITE_DATABASE_URL secret) runs a script that does not import the ASGI module.

The workflow change adds one test file to an existing step of v2-integration-check.yml; that job's V2_MIGRATION_DATABASE_URL, V2_ALEMBIC_DISPOSABLE_TEST and DEPLOYMENT_ENVIRONMENT are literals for its disposable service containers.

Not verified:

  • The runtime role's SELECT on countries in staging and production. It is declared in privileges.py and granted by the deploy's reconcile_v2_privileges.py apply after the schema upgrade and before the candidate starts; nobody has checked it in a deployed database yet. If it is missing, the candidate refuses to start and the release stops before taking traffic.
  • The runtime service accounts' read access to the Secret Manager secret. It is outside this repository; the existing v2 routes already use it.

What this PR waits for

Review only, including a core developer for the rule change. It is the bottom of a stack; #4017 above it removes the unused routes and three countries. Its migration is in the v2 Alembic chain, so if another open PR also adds a v2 revision, one merges first and the other rebases onto the new head.

After it deploys, an operator check is worth doing: GET /v2/countries in staging, then production, should list uk and us.

Testing

Each commit was written test-first:

  • 14a662414 (one country list): failed first because CountryId rejected a country added to the list; then a fresh-interpreter test shows every derived list picks it up, and the catalog's per-country settings cover exactly the listed countries.
  • 2729c37f4 (table and revision): failed first because the countries table and Country model did not exist. Then: model and foreign-key tests, SQLite persistence (an unlisted country is refused, a country in use cannot be deleted, a household-only row lets records name that country), the revision's operations, and the v2 Postgres lifecycle against disposable PostgreSQL 17 (upgrade keeps existing US and UK rows, unlisted countries refused, downgrade and re-upgrade, a downgrade refused while records name another country, the previous release's table usage against the new schema): 10 passed.
  • 0d30696eb (GET /v2/countries): failed first (the countries service package did not exist, the OpenAPI document lacked the path, the generated contract files were stale, the integration workflow did not name the test). Then 306 passed across the route, service, metadata routes, contract, workflow, privilege, CRUD-boundary, migration-flag, import and query-parameter tests; the integration test as the runtime role on disposable PostgreSQL passed.
  • f9a19c11e (startup check): failed first (no validator module). Then 21 unit tests and the runtime-role integration test (the check passes on the revision's rows and fails once an extra row exists) passed, with 203 passing in the related startup, readiness, ASGI and privilege suites.
  • 6b8cbd2bd (review follow-up): failed first (configuration and interface errors raised; the ASGI import exited with an error); then 13 startup tests and 77 in the startup, import, ASGI and readiness suites passed.
  • 975c80fc1 (test isolation): in a full run, the Alembic unit tests load Alembic's logging configuration, which disables the startup check's logger, so its four log tests failed after them (reproduced by running the two files together); an autouse fixture re-enables the logger, and the unit, contract and to-refactor suites pass (3,425).
  • On every commit: ruff format --check ., ruff check ., uv run --frozen --extra dev mypy, python scripts/run_quality_guards.py; python -m scripts.rules guard --base-ref origin/master on the last.

🤖 Generated with Claude Code

anth-volk and others added 5 commits October 22, 2026 16:59
The Public API listed its countries in several places: the v2 routes'
`CountryId` type (`Literal["us", "uk"]`), an unused `SUPPORTED_COUNTRY_IDS`
set in the query models, catalog extraction's `SUPPORTED_COUNTRIES` and
its required packages and installed models, the set of countries the v1
routes copy into v2, and the household copy command's options.

`policyengine_api/supported_countries.py` now holds the one list,
`SupportedCountryId`, and `SUPPORTED_COUNTRY_IDS` derived from it; each of
those places derives from it. The list lives in its own module, not in the
query models, because the query models import the catalog package, which
must import the list too.

Tests: in a fresh interpreter, adding "zz" to the one list adds it to every
derived list (this failed before the change: `CountryId` rejected "zz");
the catalog's per-country settings must cover exactly the listed countries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…reference

Adds the v2 `countries` table: one row per country the Public API serves,
with `household_enabled` and `society_wide_enabled` saying which
calculations it supports (a check requires society-wide countries to be
household capable). `users.primary_country` and the `country_id` of
households, policies and the four user association tables reference it,
with `ON DELETE RESTRICT`, instead of each checking `IN ('us', 'uk')`. The
tables that pair v1 records with their v2 copies keep their checks.

Revision `fb1a3e40d3d6` was autogenerated. Its one hand-written operation
inserts the US and UK rows, both capable of both, between the table and the
foreign keys, which otherwise fail on every database whose rows already
name either country. This insert is an exception to the rule that revisions
contain only generated operations, approved for this revision alone; the
revision's comment and the Alembic skill say so. The generated downgrade
restores the checks and drops the table with its rows.

The privilege manifest gives the countries table no role access yet.

Tests (failed first because the `countries` table and `Country` model did
not exist): model structure and foreign keys; SQLite persistence (an
unlisted country is refused, a listed country in use cannot be deleted, a
household-only row lets records name that country); the revision's
operations; and the Postgres lifecycle (upgrade keeps existing US and UK
rows, unlisted countries are refused, downgrade and re-upgrade, a downgrade
is refused while records name another country, and the previous release's
table usage still works after the upgrade).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`GET /v2/countries` lists every row of the countries table, ordered by ID:
each country's `id`, `household_enabled` and `society_wide_enabled`. It
takes no query parameters (any is refused with a 422), needs no user
credential, and answers 503 when the v2 database is not configured or
cannot be reached, and a secret-safe 500 for anything else, through the
shared v2 error mapping. Requests are logged under the `metadata` route
group.

The route follows the v2 layout: a `countries` route package, and a
service package with a read connector, a session container, a
transformation and a read type. The Public API's runtime role gets SELECT
on `countries` in the privilege manifest, which the deploy grants after the
schema upgrade. The migration contract registry gains the
`country_selection_v2` workflow, and the cross-database integration
workflow runs the new integration test.

Tests (failed first: the route and service tests could not import
`policyengine_api.services.v2.countries`, the OpenAPI test did not find
`/v2/countries`, the generated contract files were stale, and the
integration workflow did not name the new test): route responses, the
query refusal, 405 for other methods, 503 and 500 mapping and the OpenAPI
document; the service against SQLite; and, against migrated disposable
Postgres as the runtime role, the route returning the two rows the
revision inserted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`policyengine_api.asgi` now checks, while the application loads, that the
v2 `countries` table holds one row for each country in
`policyengine_api/supported_countries.py` and no other. A worker that finds
a difference raises `CountryListMismatchError`, naming the listed countries
without a row and the rows not listed, so it never serves a request; on
Cloud Run the revision then fails its health checks.

The check runs on Cloud Run (`K_SERVICE` is set) and wherever a v2 runtime
database URL or its secret resource is configured. A local or test startup
without v2 configuration opens no connection. A missing table, a missing
privilege and unusable v2 configuration also stop startup. A database that
cannot be reached is logged, by error type only, and startup continues: the
v1 routes do not read the v2 database, and their copies into it already
continue without it.

The countries service gains `verify_countries_match_code_list`, which reads
the row IDs through a new read connector and compares them in a validator.

Tests (failed first: `policyengine_api.services.v2.countries.validators`
did not exist): the comparison in every direction; the service against
SQLite; the startup function's selection, each failure that stops startup,
and the logged skip without the error's detail; importing the ASGI module
with an unusable v2 runtime URL failing without printing the password; and,
against migrated disposable Postgres as the runtime role, the check passing
on the revision's rows and failing once an extra row exists.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review found that the startup check sorted its failures by exception type
in a way that stopped startup for a brief Secret Manager outage (reported
as unusable v2 configuration) while only logging rejected credentials (a
connection error). The check now stops startup only when the database
answers and the answer disagrees with the country list: different
countries, a missing table, or no privilege to read it. When it cannot get
an answer, because the v2 configuration cannot be used, the secret cannot
be resolved, or the database cannot be reached or queried, it logs why and
startup continues; the readiness check already reports unusable v2
configuration as not ready and recovers once the secret resolves.

The rule `api.startup.no-import-side-effects` now says what deployed
startup does: it opens no network connections unless the process runs on
Cloud Run or has a v2 runtime database configured. The country list and
the startup check join the mypy file list and rule `api.quality.mypy-v2`.
The country list's docstring and the Alembic skill now say that a deploy
runs a country-adding revision before the new release starts, so the
previous release refuses to start afterwards, and list the US-and-UK
checks a new country must also widen.

Tests (failed first: configuration errors and `InterfaceError` were
raised, and importing the ASGI module with an unusable v2 URL exited with
an error): configuration and secret failures are logged with their
message and startup continues, connection and interface errors are logged
by type only, and importing the ASGI module with an unusable v2 URL logs
the skipped check without the password.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anth-volk
anth-volk requested a review from a team October 22, 2026 13:56
@codecov

codecov Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 83.58974% with 32 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.80%. Comparing base (ee46c7a) to head (975c80f).

Files with missing lines Patch % Lines
policyengine_api/startup_country_check.py 78.78% 6 Missing and 1 partial ⚠️
policyengine_api/fastapi_routes/dependencies.py 40.00% 6 Missing ⚠️
policyengine_api/services/v2/countries/services.py 73.68% 3 Missing and 2 partials ⚠️
...cyengine_api/fastapi_routes/v2/countries/routes.py 86.66% 2 Missing and 2 partials ⚠️
...gine_api/services/v2/countries/database_session.py 71.42% 3 Missing and 1 partial ⚠️
...api/fastapi_routes/v2/countries/response_models.py 90.90% 1 Missing and 1 partial ⚠️
...ngine_api/services/v2/countries/transformations.py 71.42% 1 Missing and 1 partial ⚠️
...licyengine_api/services/v2/countries/validators.py 86.66% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           master    #4012       +/-   ##
===========================================
+ Coverage   50.12%   87.80%   +37.67%     
===========================================
  Files         243      270       +27     
  Lines       14726    16333     +1607     
  Branches     2343     2653      +310     
===========================================
+ Hits         7382    14341     +6959     
+ Misses       6721     1306     -5415     
- Partials      623      686       +63     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

In a full test run, the tests that run Alembic commands load Alembic's
logging configuration, which disables every logger that already exists,
including the startup check's. Its four log tests then captured nothing
and failed, though they passed on their own. An autouse fixture now
re-enables the check's logger for each of its tests.

Tests (failed first: running the Alembic unit tests before the startup
check tests failed all four log tests); with the fixture, both files pass
together (46 passed).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was successfully deployed

1 active deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Record served countries in a v2 countries table that matches one country list in code

1 participant