Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions .github/scripts/cloud_run_env.sh
Original file line number Diff line number Diff line change
Expand Up @@ -29,13 +29,14 @@ cloud_run_set_defaults() {
CLOUD_RUN_CONCURRENCY="${CLOUD_RUN_CONCURRENCY:-6}"
CLOUD_RUN_WEB_CONCURRENCY="${CLOUD_RUN_WEB_CONCURRENCY:-2}"
CLOUD_RUN_PORT="${CLOUD_RUN_PORT:-8080}"
# Readiness gate: /readiness-check stays 503 until the worker has built the
# tax-benefit systems AND run the startup warmup calculate, so boot-to-ready is
# Startup readiness: /health stays 503 until the worker has built the
# tax-benefit systems AND run the startup warmup calculate, and while the
# runtime configuration is invalid, so boot-to-ready is
# now the ~371s p90 import plus the warmup. Window sized to the platform maximum
# (240s + 24x10s = 480s); both halves are capped at 240s. initialDelay is high
# but adds no real scale-out delay because boot exceeds it either way. Sizing
# rationale in docs/migration/cloud-run-operations.md.
CLOUD_RUN_STARTUP_PROBE="${CLOUD_RUN_STARTUP_PROBE:-httpGet.path=/readiness-check,httpGet.port=${CLOUD_RUN_PORT},initialDelaySeconds=240,periodSeconds=10,failureThreshold=24,timeoutSeconds=5}"
CLOUD_RUN_STARTUP_PROBE="${CLOUD_RUN_STARTUP_PROBE:-httpGet.path=/health,httpGet.port=${CLOUD_RUN_PORT},initialDelaySeconds=240,periodSeconds=10,failureThreshold=24,timeoutSeconds=5}"
CLOUD_RUN_POLICYENGINE_DB_PASSWORD_SECRET="${CLOUD_RUN_POLICYENGINE_DB_PASSWORD_SECRET:-}"
CLOUD_RUN_GITHUB_MICRODATA_TOKEN_SECRET="${CLOUD_RUN_GITHUB_MICRODATA_TOKEN_SECRET:-policyengine-api-prod-github-microdata-token:latest}"
CLOUD_RUN_OPENAI_API_KEY_SECRET="${CLOUD_RUN_OPENAI_API_KEY_SECRET:-policyengine-api-prod-openai-api-key:latest}"
Expand Down
14 changes: 7 additions & 7 deletions .github/workflows/push.yml
Original file line number Diff line number Diff line change
Expand Up @@ -501,7 +501,7 @@ jobs:
env:
CLOUD_RUN_TAG: ${{ steps.cloud_run.outputs.revision_tag }}
- name: Wait for Cloud Run staging health
run: bash .github/scripts/health_check.sh "${{ steps.candidate.outputs.url }}/readiness-check"
run: bash .github/scripts/health_check.sh "${{ steps.candidate.outputs.url }}/health"
- name: Run staging integration tests
run: uv run --frozen --extra dev python -m pytest tests/integration/test_cloud_run_candidate.py tests/integration/test_live_v2_metadata.py tests/integration/test_live_v2_policies.py -v
env:
Expand Down Expand Up @@ -543,7 +543,7 @@ jobs:
- name: Install locked live test dependencies
run: make install
- name: Wait for Cloud Run staging candidate health before live tests
run: bash .github/scripts/health_check.sh "${{ needs.deploy-cloud-run-staging.outputs.url }}/readiness-check"
run: bash .github/scripts/health_check.sh "${{ needs.deploy-cloud-run-staging.outputs.url }}/health"
- name: Run live staging tests, one pytest worker per test
run: bash .github/scripts/run_live_staging_tests.sh
env:
Expand Down Expand Up @@ -610,10 +610,10 @@ jobs:
id: stable_health
if: ${{ steps.promote.outcome == 'success' }}
continue-on-error: true
run: bash .github/scripts/health_check.sh "${{ needs.deploy-cloud-run-staging.outputs.stable_url }}/readiness-check"
run: bash .github/scripts/health_check.sh "${{ needs.deploy-cloud-run-staging.outputs.stable_url }}/health"
- name: Restore previous Cloud Run staging revision
if: ${{ steps.promote.outcome == 'failure' || steps.stable_health.outcome == 'failure' }}
run: bash .github/scripts/restore_cloud_run_revision.sh "${{ needs.deploy-cloud-run-staging.outputs.stable_url }}/readiness-check"
run: bash .github/scripts/restore_cloud_run_revision.sh "${{ needs.deploy-cloud-run-staging.outputs.stable_url }}/health"
env:
CLOUD_RUN_TARGET_REVISION: ${{ needs.deploy-cloud-run-staging.outputs.previous_revision }}
CLOUD_RUN_EXPECTED_CURRENT_REVISION: ${{ needs.deploy-cloud-run-staging.outputs.revision }}
Expand Down Expand Up @@ -788,7 +788,7 @@ jobs:
env:
CLOUD_RUN_TAG: ${{ steps.cloud_run.outputs.revision_tag }}
- name: Wait for Cloud Run candidate health
run: bash .github/scripts/health_check.sh "${{ steps.candidate.outputs.url }}/readiness-check"
run: bash .github/scripts/health_check.sh "${{ steps.candidate.outputs.url }}/health"
- name: Install locked Cloud Run smoke test dependencies
run: make install
- name: Run Cloud Run candidate smoke tests
Expand All @@ -813,10 +813,10 @@ jobs:
id: stable_health
if: ${{ steps.promote.outcome == 'success' }}
continue-on-error: true
run: bash .github/scripts/health_check.sh "${{ steps.previous.outputs.stable_url }}/readiness-check"
run: bash .github/scripts/health_check.sh "${{ steps.previous.outputs.stable_url }}/health"
- name: Restore previous Cloud Run production revision
if: ${{ steps.promote.outcome == 'failure' || steps.stable_health.outcome == 'failure' }}
run: bash .github/scripts/restore_cloud_run_revision.sh "${{ steps.previous.outputs.stable_url }}/readiness-check"
run: bash .github/scripts/restore_cloud_run_revision.sh "${{ steps.previous.outputs.stable_url }}/health"
env:
CLOUD_RUN_TARGET_REVISION: ${{ steps.previous.outputs.revision }}
CLOUD_RUN_EXPECTED_CURRENT_REVISION: ${{ steps.candidate.outputs.revision }}
Expand Down
1 change: 1 addition & 0 deletions changelog.d/health-reports-readiness.changed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
`/health` now answers 503 `{"status": "not_ready"}` until the startup warmup has finished and while the runtime configuration is invalid, and 200 `{"status": "healthy"}` after. The Cloud Run startup probe and the deployment health checks poll it instead of `/readiness-check`, and `/liveness-check` and `/readiness-check` are removed.
2 changes: 2 additions & 0 deletions changelog.d/remove-legacy-site-routes.removed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Removed the routes that only legacy.policyengine.org or the API documentation used: the v1 saved-policy routes (`/{country_id}/user-policy`), the user-profile routes (`/{country_id}/user-profile`), the policy search (`GET /{country_id}/policies`), the recent-simulations list (`GET /simulations`), the `GET /` landing page and `GET /specification`; `/v2/openapi.json` documents the v2 resources.
Stopped serving Canada (`ca`), Nigeria (`ng`) and Israel (`il`): requests for them now get the unknown-country 400 response, and `policyengine_canada`, `policyengine-ng` and `policyengine-il` are no longer installed.
14 changes: 7 additions & 7 deletions docs/canonical-spm.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ its existing behavior when settings are omitted and rejects explicit SPM setting
with `SPM_SETTINGS_UNSUPPORTED`, whatever its version. A bundle whose US model
does implement the constructor but ships no certified `measurements.spm`
configuration fails closed with `SPM_CONFIGURATION_UNAVAILABLE`: such a
deployment also fails `/readiness-check`, so the release checks report the
deployment also fails `/health`, so the release checks report the
condition rather than leaving it to individual requests. Other countries retain their
behavior and reject US-only SPM settings.

Expand Down Expand Up @@ -131,7 +131,7 @@ whose measurement configuration this build cannot resolve — no certified
configuration for a model that implements the constructor, a manifest this API
cannot read, or an artifact that will not load — fails every US calculation with
a configuration code, whatever the caller sent, and also fails
`/readiness-check`: it is meant to stop a deployment rather than to be met on a
`/health`: it is meant to stop a deployment rather than to be met on a
request. `GET /us/metadata` still answers, reporting `spm.available` false.

A country receipt this API cannot read in full is the same class of failure and
Expand Down Expand Up @@ -263,8 +263,8 @@ calculation result and SPM receipts. `active_run_id` identifies a pending/runnin
run and becomes null when
inactive; `latest_successful_run_id` identifies the latest successful run. These
run fields and specification metadata may be null on historical records. The
served `/specification` documents these request and response shapes through both
Flask and the native specification route.
legacy `GET /specification` document, which also described these request and
response shapes, has been removed; this section describes them.

HTTP response cache identity includes the normalized selection and model/bundle
versions, and validates certification before reading the cache. The stored
Expand Down Expand Up @@ -321,9 +321,9 @@ annual requests require `time_period`, while budget-window requests require
`start_year` and `window_size` (1–75, ending no later than 2099). Years use four
digits. Optional fields are `version` (installed country model version),
`target` (`general`, or annual-only `cliff`) and the annual-only `breakdown`.
Omitted `spm` inherits certified bundle defaults. The served `/specification`
generates these query declarations from the same typed models as the Flask
parser.
Omitted `spm` inherits certified bundle defaults. The Flask parser's typed
models define these query rules; the legacy `GET /specification` document
that also declared them has been removed.

`breakdown=local_areas` asks the annual route for the UK constituency and
local-authority impacts only. It is a calculation option, so the request has
Expand Down
16 changes: 2 additions & 14 deletions docs/engineering/migration-contracts.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ Generated from `policyengine_api/migration_registry.py` and `tests/contract/regi
| Metric | Count |
| --- | ---: |
| route group count | 9 |
| workflow count | 17 |
| request count | 90 |
| workflow count | 16 |
| request count | 86 |
| db entity count | 6 |
| sim flow count | 3 |

Expand Down Expand Up @@ -37,7 +37,6 @@ Generated from `policyengine_api/migration_registry.py` and `tests/contract/regi
| --- | --- | ---: | --- | --- | --- |
| `POST` | `/us/policy` | 201 | `policy` | none | `status`, `message`, `result.policy_id` |
| `GET` | `/us/policy/{policy_id}` | 200 | `policy` | none | `status`, `message`, `result` |
| `GET` | `/us/policies` | 200 | `policy` | none | `result` |

### `policy_resources_v2`

Expand All @@ -50,17 +49,6 @@ Generated from `policyengine_api/migration_registry.py` and `tests/contract/regi
| `GET` | `/v2/policies/{policy_id}?country_id=us` | 200 | `policy` | none | `status`, `message`, `result.item.id`, `result.item.country_id`, `result.item.tax_benefit_model_id`, `result.item.parameter_values`, `result.item.created_at`, `result.item.updated_at`, `result.item.missing_parameters` |
| `GET` | `/v2/policies?country_id=us` | 200 | `policy` | none | `status`, `message`, `result.items`, `result.offset`, `result.limit`, `result.has_more` |

### `saved_policy_v1_compatibility`

- Current contract: `api_v1_compatible`
- Future owner: PR 10: Policy Migration

| Method | Path | Status | Route group | Required headers | Stable response fields |
| --- | --- | ---: | --- | --- | --- |
| `POST` | `/us/user-policy` | 201 | `policy` | none | `status`, `message`, `result.id`, `result.reform_id`, `result.reform_label`, `result.baseline_id`, `result.user_id` |
| `GET` | `/us/user-policy/{user_id}` | 200 | `policy` | none | `status`, `message`, `result` |
| `PUT` | `/us/user-policy` | 200 | `policy` | none | `status`, `message`, `result.id` |

### `user_policy_associations_v2`

- Current contract: `typed_v2_resources`
Expand Down
4 changes: 1 addition & 3 deletions docs/engineering/skills/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -261,7 +261,6 @@ uv run pytest \
tests/unit/services/test_policy_mirroring.py \
tests/unit/services/test_user_policy_mirroring.py \
tests/unit/routes/test_policy_dual_write_routes.py \
tests/unit/routes/test_user_policy_dual_write_routes.py \
tests/unit/test_migration_flags.py \
tests/unit/test_readiness.py \
tests/contract -q
Expand Down Expand Up @@ -295,8 +294,7 @@ uv run pytest \
tests/contract/test_v1_route_contracts.py \
tests/unit/services/test_policy_service.py \
tests/unit/services/test_user_policy_service.py \
tests/unit/routes/test_policy_dual_write_routes.py \
tests/unit/routes/test_user_policy_dual_write_routes.py -q
tests/unit/routes/test_policy_dual_write_routes.py -q
```

Finally regenerate and validate migration contracts:
Expand Down
61 changes: 2 additions & 59 deletions docs/generated/migration_contracts.json
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
{
"metadata": {
"db_entity_count": 6,
"request_count": 90,
"request_count": 86,
"route_group_count": 9,
"sim_flow_count": 3,
"workflow_count": 17
"workflow_count": 16
},
"route_groups": [
{
Expand Down Expand Up @@ -136,16 +136,6 @@
"message",
"result"
]
},
{
"expected_status": 200,
"headers": [],
"method": "GET",
"path": "/us/policies",
"route_group": "policy",
"stable_response_fields": [
"result"
]
}
]
},
Expand Down Expand Up @@ -207,53 +197,6 @@
}
]
},
{
"current_contract": "api_v1_compatible",
"future_owner_pr": "PR 10: Policy Migration",
"name": "saved_policy_v1_compatibility",
"requests": [
{
"expected_status": 201,
"headers": [],
"method": "POST",
"path": "/us/user-policy",
"route_group": "policy",
"stable_response_fields": [
"status",
"message",
"result.id",
"result.reform_id",
"result.reform_label",
"result.baseline_id",
"result.user_id"
]
},
{
"expected_status": 200,
"headers": [],
"method": "GET",
"path": "/us/user-policy/{user_id}",
"route_group": "policy",
"stable_response_fields": [
"status",
"message",
"result"
]
},
{
"expected_status": 200,
"headers": [],
"method": "PUT",
"path": "/us/user-policy",
"route_group": "policy",
"stable_response_fields": [
"status",
"message",
"result.id"
]
}
]
},
{
"current_contract": "typed_v2_resources",
"future_owner_pr": "PR 12A: User Login and v1-Anchored User Associations",
Expand Down
24 changes: 13 additions & 11 deletions docs/migration/cloud-run-operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,14 +70,16 @@ Values measured and justified in
gcloud inherits unspecified template fields (a mid-campaign CI deploy once shipped an
inherited test concurrency), and `--concurrency default` resolves to the *platform*
default (640), not the historical 80.
- **`--startup-probe httpGet.path=/readiness-check` — pinned on every deploy.**
- **`--startup-probe httpGet.path=/health` — pinned on every deploy.**
Cloud Run's default is a *TCP* probe, which passes the instant gunicorn's master
binds the port, long before a worker finishes importing (the import runs
post-fork; `--preload` is deliberately unset). Cloud Run therefore treated a
booting instance as "started" and routed live traffic onto it, where requests
queued until the 300s timeout — the "no available instance" 500s and 504s seen on
scale-out bursts. Probing `/readiness-check` over HTTP makes Cloud Run withhold
traffic until the app can actually serve.
scale-out bursts. Probing `/health` over HTTP makes Cloud Run withhold
traffic until the app can actually serve. (The probe used `/readiness-check`
until that route and `/liveness-check` were removed; `/health` now has the
readiness behaviour.)

**Sizing it is constrained by measured boot times.** Boot-to-ready across 48
boots over 7 days (2026-07-14→21), measured from each instance's
Expand Down Expand Up @@ -115,14 +117,14 @@ Values measured and justified in
**Startup warmup — why readiness depends on more than the import.** Building the
tax-benefit systems at import does not compile the per-simulation machinery
(parameter-tree materialisation, the formula graph). The **first** calculate on a
fresh worker paid that cost — measured at ~121s — and `/readiness-check` returned
fresh worker paid that cost — measured at ~121s — and the readiness check returned
200 before it, so the probe (and health checks, and smoke tests) saw a "ready"
instance whose first real request still took two minutes.
`policyengine_api.warmup.run_startup_warmup` now runs a throwaway calculate per
country (default US; override with `POLICYENGINE_API_WARMUP_COUNTRIES`, disable
with `POLICYENGINE_API_STARTUP_WARMUP=0`) from `asgi.py` before the worker serves,
and `/readiness-check` returns 503 until it completes
(`policyengine_api.readiness`). Verified locally: after the warmup a cold
and `/health` returns 503 until it completes, or while the runtime
configuration is invalid (`policyengine_api.readiness`). Verified locally: after the warmup a cold
test-payload calculate drops from ~121s to ~10s. This is the fix for the note that
used to live here — deferring the build lazily relocated the cost onto the first
request, where readiness lied; warming it at startup pays the cost off the request
Expand Down Expand Up @@ -208,11 +210,11 @@ Consequences to keep in mind:

- An instance is "started" when its port is bound, before the application can
answer. The HTTP startup probe prevents Cloud Run from routing requests to
that instance until `/readiness-check` succeeds.
- **Cold-start and readiness measurements must be taken from `/readiness-check` turning
that instance until `/health` succeeds.
- **Cold-start and readiness measurements must be taken from `/health` turning
healthy**, not from instance start or port bind — the bind time is no longer meaningful.
- Readiness in CI is governed by `.github/scripts/health_check.sh` polling
`/readiness-check` (defaults: 900s budget, 5s interval, 15s per-request `--max-time`;
`/health` (defaults: 900s budget, 5s interval, 15s per-request `--max-time`;
all env-overridable).

## Automated revision promotion and rollback
Expand Down Expand Up @@ -299,7 +301,7 @@ gcloud run services get-iam-policy policyengine-api-staging \
--project policyengine-api --region us-central1

# Readiness
curl -s -o /dev/null -w '%{http_code}\n' "$SERVICE_URL/readiness-check"
curl -s -o /dev/null -w '%{http_code}\n' "$SERVICE_URL/health"
```

## Public load-balancer routing
Expand All @@ -313,7 +315,7 @@ complete URL map before and after any intended routing change because
gcloud compute url-maps describe lb-api --project policyengine-api \
--format='yaml(defaultService,defaultRouteAction,pathMatchers)'
curl -s -o /dev/null -w '%{http_code}\n' \
https://api.policyengine.org/readiness-check
https://api.policyengine.org/health
```

Application rollback restores the previously serving Cloud Run revision. It
Expand Down
2 changes: 1 addition & 1 deletion docs/migration/pr4-app-engine-rollback-runbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ gcloud compute url-maps describe lb-api --project policyengine-api \

```bash
for i in $(seq 1 200); do
curl -sI --max-time 10 https://api.policyengine.org/liveness-check | tr -d '\r' \
curl -sI --max-time 10 https://api.policyengine.org/health | tr -d '\r' \
| awk -F': ' 'tolower($1)=="x-policyengine-backend" {print $2}'
done | sort | uniq -c # expect 200 app_engine after settle
```
Expand Down
2 changes: 1 addition & 1 deletion docs/migration/pr4-ramp-runbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ it; sampling is curl/server-side only.

```bash
for i in $(seq 1 "$N"); do
curl -sI --max-time 10 "https://$HOST/liveness-check" | tr -d '\r' \
curl -sI --max-time 10 "https://$HOST/health" | tr -d '\r' \
| awk -F': ' 'tolower($1)=="x-policyengine-backend" {print $2}'
done | sort | uniq -c
```
Expand Down
14 changes: 14 additions & 0 deletions docs/migration/stage-10-v2-policies.md
Original file line number Diff line number Diff line change
Expand Up @@ -252,3 +252,17 @@ The normal staging promotion still restores the previously serving Cloud Run
revision when promotion or the stable-URL health check fails. It does not
create persistent policy fixtures or change the configured write selection as
part of each release.

## Removal of the v1 saved-policy routes

`POST /{country_id}/user-policy`, `GET /{country_id}/user-policy/{user_id}`
and `PUT /{country_id}/user-policy` are removed, along with
`GET /{country_id}/policies`. Only legacy.policyengine.org called them; Cloud
Run request logs for the 30 days to 2026-10-19 showed no saved-policy
requests. No new v1 saved policies are created, so none are copied to v2. The
saved-policy copying and v1 saved-policy service modules stay until dual-write
is deleted, and existing Cloud SQL `user_policies` rows are kept.

The Public API also no longer serves Canada, Nigeria and Israel: their v1
policy requests are rejected as unknown countries before any write, so every
v1 policy write is now for a country that v2 supports.
Loading
Loading