Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ To add or change a rule, read `docs/engineering/skills/rules-registry.md`.

### Data source isolation

- Checked: `api.catalog.extractor-import-isolation`, `api.metadata.read-modules-isolated`, `api.reports.v1-persistence-only`, `api.stage12.comparison-tables-isolated`, `api.stage12.retirement-owned`, `api.v2.single-orm-layer`
- Checked: `api.catalog.extractor-import-isolation`, `api.metadata.read-modules-isolated`, `api.reports.v1-persistence-only`, `api.stage12.comparison-tables-isolated`, `api.stage12.retirement-owned`, `api.v2.simulation-tables-stay-removed`, `api.v2.single-orm-layer`

### Migration and schema

Expand Down
1 change: 1 addition & 0 deletions changelog.d/drop-simulation-tables.removed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Dropped the Public API's copies of the 15 simulation-domain v2 tables, their 8 PostgreSQL enum types and the retired `country` and `label` columns of the simulation and report associations. The simulation service has its own replacements for all of these tables except the unused custom-total tables `aggregates` and `change_aggregates`.
37 changes: 37 additions & 0 deletions docs/engineering/skills/alembic-migrations.md
Original file line number Diff line number Diff line change
Expand Up @@ -294,3 +294,40 @@ last dependent table is dropped. Its only post-generation correction drops the
nine generated `v2_*` enum types at the end of the baseline downgrade. The v2
Postgres lifecycle test covers empty upgrade, downgrade to base, and re-upgrade
so stale enum types cannot make the generated baseline non-reversible.

### Simulation-domain table removal

Revision `363f34954a14` was autogenerated after the last schema-only
declarations of the simulation-domain tables were deleted (an earlier release
had already removed their models and stopped reading or writing them). It
drops the 15 simulation-domain tables (`household_jobs`, `simulations`,
`reports`, `report_runs` and the 11 report-run output tables) and the retired
`country` and `label` columns of `user_simulation_associations` and
`user_report_associations`. The simulation service has its own replacements
for all of these tables in its `simulation` schema except `aggregates` and
`change_aggregates`, v2 alpha's custom totals, which were unused and have no
replacement. Its post-generation corrections:

- Alembic emits removed tables in arbitrary order. The upgrade drops each
table before the tables its foreign keys reference, and the downgrade
creates them in the reverse order.
- Alembic does not generate the removal of the dropped tables' eight native
enum types, and the generated downgrade could not create them again while
they remained. The upgrade drops them.
- Reflection strips the outer parentheses of `ck_simulations_filter_pair`,
which leaves invalid SQL in the generated downgrade. They are restored as
the model declared them.

The v2 lifecycle tests cover the upgrade, the downgrade to the previous
revision and the re-upgrade. They check that the downgrade recreates exactly
the tables, columns, constraints, indexes and enum types the earlier
revisions created (empty), and that head serves the previous application
revision, which no longer maps any dropped table or column.

The revision must not deploy while an application revision that maps the
dropped tables or columns can serve traffic, and it may run only after an
operator has confirmed in each environment that the tables are empty. Once it
has run, an application rollback to a revision that maps them, including any
pre-Stage 12A revision, needs the schema downgraded to `fbb7488ebe8a` first.
`api.v2.simulation-tables-stay-removed` stops code from importing the removed
models again, and the v2 metadata validation rejects their table names.
5 changes: 3 additions & 2 deletions docs/engineering/skills/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,10 +55,11 @@ python -m pytest tests/unit/test_cloud_run_deploy_scripts.py tests/unit/test_cap
For v2 platform foundation work, run the smallest applicable group while
iterating, then all groups before qualification.

SQLModel schema, lazy configuration, and import effects:
SQLModel schema, the removed simulation-domain models, lazy configuration, and
import effects:

```bash
uv run pytest tests/unit/v2/test_models.py tests/unit/v2/test_model_persistence.py tests/unit/v2/test_settings.py tests/unit/v2/test_import_side_effects.py -q
uv run pytest tests/unit/v2/test_models.py tests/unit/v2/test_model_persistence.py tests/unit/v2/test_simulation_domain_boundaries.py tests/unit/v2/test_settings.py tests/unit/v2/test_import_side_effects.py -q
```

The generated-only v2 Alembic chain and disposable Postgres lifecycle:
Expand Down
9 changes: 7 additions & 2 deletions docs/generated/rules.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,11 @@ Every rule that governs `policyengine-api` is listed here with how it is enforce

| Tier | enforced | ratcheted | advisory | doc-only | Total |
| --- | ---: | ---: | ---: | ---: | ---: |
| mechanical | 51 | 0 | 1 | 12 | 64 |
| mechanical | 52 | 0 | 1 | 12 | 65 |
| metric | 0 | 0 | 0 | 0 | 0 |
| judgment | 0 | 0 | 0 | 1 | 1 |

**Blocking:** 51 of 65 rules.
**Blocking:** 52 of 66 rules.

## Gaps

Expand Down Expand Up @@ -193,6 +193,7 @@ flowchart LR
| `api.reports.v1-persistence-only` | The existing report route and service import only v1 persistence. | mechanical | enforced | yes | pytest `tests/unit/v2/test_report_v1_runtime_selection.py` | `pr.yml:test` | **unproven** |
| `api.stage12.comparison-tables-isolated` | Serving routes and services never read the Stage 12 comparison tables, and production tables never depend on them. | mechanical | enforced | yes | pytest `tests/unit/v2/test_stage12_comparison_isolation.py` | `pr.yml:test` | **unproven** |
| `api.stage12.retirement-owned` | Retiring the Stage 12 comparison schema stays owned by a named follow-up change in the Stage 12 worker contract. | mechanical | enforced | yes | pytest `tests/unit/v2/test_stage12_comparison_isolation.py` | `pr.yml:test` | **unproven** |
| `api.v2.simulation-tables-stay-removed` | Public API code never imports the removed simulation-domain models (household jobs, simulations, reports, report runs and their outputs), and the v2 metadata never declares the tables a v2 revision dropped with them. | mechanical | enforced | yes | pytest `tests/unit/v2/test_metadata_validation.py`, `tests/unit/v2/test_simulation_domain_boundaries.py` | `pr.yml:test` | yes |
| `api.v2.single-orm-layer` | v2 models never create a parallel SQLAlchemy ORM layer alongside SQLModel. | mechanical | enforced | yes | pytest `tests/unit/v2/test_model_persistence.py` | `pr.yml:test` | **unproven** |

```mermaid
Expand All @@ -207,6 +208,8 @@ flowchart LR
r_api_stage12_comparison_tables_isolated("api.stage12.comparison-tables-isolated (enforced)")
e_pytest_tests_unit_v2_test_stage12_comparison_isolation_py["pytest tests/unit/v2/test_stage12_comparison_isolation.py"]
r_api_stage12_retirement_owned("api.stage12.retirement-owned (enforced)")
r_api_v2_simulation_tables_stay_removed("api.v2.simulation-tables-stay-removed (enforced)")
e_pytest_tests_unit_v2_test_metadata_validation_py__tests_unit_v2_test_simulation_domain_boundaries_py["pytest tests/unit/v2/test_metadata_validation.py, tests/unit/v2/test_simulation_domain_boundaries.py"]
r_api_v2_single_orm_layer("api.v2.single-orm-layer (enforced)")
e_pytest_tests_unit_v2_test_model_persistence_py["pytest tests/unit/v2/test_model_persistence.py"]
r_api_catalog_extractor_import_isolation --> e_pytest_tests_unit_v2_test_catalog_extraction_py
Expand All @@ -218,6 +221,8 @@ flowchart LR
r_api_stage12_comparison_tables_isolated --> e_pytest_tests_unit_v2_test_stage12_comparison_isolation_py
e_pytest_tests_unit_v2_test_stage12_comparison_isolation_py --> j_CI_pr_yml_test
r_api_stage12_retirement_owned --> e_pytest_tests_unit_v2_test_stage12_comparison_isolation_py
r_api_v2_simulation_tables_stay_removed --> e_pytest_tests_unit_v2_test_metadata_validation_py__tests_unit_v2_test_simulation_domain_boundaries_py
e_pytest_tests_unit_v2_test_metadata_validation_py__tests_unit_v2_test_simulation_domain_boundaries_py --> j_CI_pr_yml_test
r_api_v2_single_orm_layer --> e_pytest_tests_unit_v2_test_model_persistence_py
e_pytest_tests_unit_v2_test_model_persistence_py --> j_CI_pr_yml_test
```
Expand Down
4 changes: 2 additions & 2 deletions docs/migration/stage-12-worker-contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -222,8 +222,8 @@ the temporary schema.
The 30-day artifact lifecycle is distinct from final removal of the two
temporary table schemas. After Stage 12 comparison running has ended, a
separately reviewed, autogenerated v2 Alembic revision in Stage 14 removes
those tables. Production `simulations`, `reports`, `report_runs`, and
user-association records must not depend on them.
those tables. No production table, including the user associations, may
depend on them.

The tracked follow-up change identifier is
`retire-stage-12-comparison-schema`. That Stage 14 change may begin only after
Expand Down
36 changes: 22 additions & 14 deletions docs/migration/stage-12a-user-login.md
Original file line number Diff line number Diff line change
Expand Up @@ -311,11 +311,12 @@ from the final SQLModel metadata. It has no post-generation corrections.
kept: `country` becomes nullable and nothing reads or writes either column.
The previous Public API revision still maps them, so dropping them would
break it during the deploy window or an application rollback (see
"Rollback"). A later release stopped mapping them: they stay declared on the
tables, unmapped, only until a revision drops them.
"Rollback"). A later release stopped mapping them, and revision
`363f34954a14` then dropped them (see the Alembic migrations skill).

The models drop the `user_associations` relationships from v2 simulations and
reports. `V2_ALEMBIC_HEAD_REVISION` is `fbb7488ebe8a`.
reports. With this revision, `V2_ALEMBIC_HEAD_REVISION` became `fbb7488ebe8a`;
revision `363f34954a14` later replaced it as head.

### Reshape safety

Expand Down Expand Up @@ -345,6 +346,13 @@ caller can list a user's saved items by `user_id`, read, rename or delete one
by record ID, and create rows for any user. Rolling the application back
would therefore hand users' saved work to anyone.

- **Once revision `363f34954a14` has run in an environment, no pre-Stage 12A
revision can serve traffic there.** Those revisions map the simulation-domain
tables and the retired `country` and `label` association columns that it
drops; the oldest revision that can run is the release that stopped mapping
them. A rollback to a pre-Stage 12A revision then first needs the v2 schema
downgraded to `fbb7488ebe8a`, which recreates those tables and columns
empty, and only after that do the steps below apply.
- **Before any user-owned record exists, an application rollback is safe
without a database downgrade.** Signed-out saving needs no configuration, so
treat user-owned records as existing from the moment this revision serves
Expand Down Expand Up @@ -403,7 +411,9 @@ row with a NULL `primary_country` does not fail there.

The lifecycle test `test_stage_12a_schema_serves_the_previous_revision`
exercises this rather than only comparing columns. It reflects the schema at
the parent revision with SQLAlchemy and upgrades to head. Then, through the
the parent revision with SQLAlchemy and upgrades to `fbb7488ebe8a`, not to
head: revision `363f34954a14` later dropped tables the parent revision maps
(see "Rollback"). Then, through the
reflected parent tables, it does the following for every table the parent
revision maps:

Expand All @@ -413,13 +423,13 @@ revision maps:
- rewrites every non-key column;
- deletes the row.

Rows of the two dormant tables are inserted through head's tables, since only
new code writes them, and are then read, rewritten, and deleted through the
parent's. The test also deletes, through the parent's `users` table, a user who
Rows of the two dormant tables are inserted through the Stage 12A tables, since
only new code writes them, and are then read, rewritten, and deleted through
the parent's. The test also deletes, through the parent's `users` table, a user who
owns rows in all four association tables and a `user_identities` row, and it
checks that every one of them cascades away. It fails if head drops a parent
column, or adds a required column without a server default to a table the
parent revision writes.
checks that every one of them cascades away. It fails if the Stage 12A schema
drops a parent column, or adds a required column without a server default to a
table the parent revision writes.

## Settings

Expand Down Expand Up @@ -531,7 +541,8 @@ Configure staging first, then production:
(`docs/migration/armor/proposed/20261013-pol-api-lb-stage-12a-rollback-guard.yaml`)
ready. From the first user-owned record on, signed in or not, it is a
precondition for any application rollback to a pre-Stage 12A revision (see
"Rollback").
"Rollback"). Once revision `363f34954a14` has run, such a rollback also
needs the v2 schema downgraded to `fbb7488ebe8a` first.

User-owned routes must never become less restrictive during a cutover. The
migration-contract guard rejects any `/v2/me` or `/v2/user-` contract that
Expand Down Expand Up @@ -578,9 +589,6 @@ The disposable-Postgres suite proves the following:

## Deferred

- A later autogenerated revision that drops the retired `country` and `label`
columns from `user_simulation_associations` and `user_report_associations`,
once no deployable Public API revision maps them.
- A retention policy for unauthenticated users whose browsers never return.
Nothing expires them today.
- An explicit claim flow for legacy-app users (`POST /v2/me/legacy-claims`).
Expand Down
Loading
Loading