Skip to content

Stop using the simulation-domain v2 tables and retired association columns - #4007

Draft
anth-volk wants to merge 4 commits into
masterfrom
v2plan/1h-130-stop-using-simulation-tables
Draft

anth-volk wants to merge 4 commits into
masterfrom
v2plan/1h-130-stop-using-simulation-tables

Conversation

@anth-volk

Copy link
Copy Markdown
Collaborator

Fixes #4006

Summary

The simulation service owns simulations, reports, report runs and their outputs in its own simulation schema. A later autogenerated v2 revision (the next PR in this stack) drops the Public API's copies of those 15 tables, their 8 PostgreSQL enum types and the retired country and label columns of the simulation and report associations. The release workflow upgrades the schema before the new Cloud Run revision serves traffic, so the release serving traffic while that revision runs must never touch them. This PR produces that release. It adds no Alembic revision, and the tables and columns stay in the database.

  • Catalog publication no longer counts rows in simulations, reports and report_runs to prove it changed none; it counts only dataset_versions. The catalog writer's declared SELECT on those three tables is removed from the privilege manifest. The deploy step never revokes, so the existing grants stay (reported as undeclared in the job summary) until the next PR drops the tables.
  • The association models no longer declare country and label. A new module, models/retired_tables.py, appends the two columns to the two tables after their mappers exist, so no mapper selects, inserts or updates them.
  • The simulation-domain models are removed: models/reports.py, models/simulations.py, HouseholdJob, the unused report_runs module and their tests, and every relationship the kept models (users, policies, dynamics, households, datasets, regions, model versions) had to them. Deleting a user through the ORM no longer selects from reports; the foreign keys and their ON DELETE actions stay in the schema.
  • retired_tables.py declares the 15 tables as plain SQLAlchemy tables that no model maps. Their compiled PostgreSQL DDL (columns, types, defaults, keys, checks, indexes, enum names and values) is identical to the removed models'. The v2 metadata therefore still matches the schema at head (alembic check reports no operations), and deleting that module is what lets autogeneration produce the next PR's revision.
  • The household migration qualification counts simulations, reports and household_jobs through table() constructs instead of the models. It still skips tables a target does not have.
  • Tests of the retired tables' design rules are removed, and the Stage 9, Stage 12A and testing documents are updated.

Migration scope: no route group, workflow or user-visible API contract changes. Nothing moves off the Flask/API v1 path.

Inline suppressions added: one # noqa: E402 on the retired_tables import in policyengine_api/data/v2/models/__init__.py. The package must set the metadata naming convention before any table is declared, so every import in that file follows the assignment and carries the same suppression; the next PR removes this import.

Environment variables

None. The changed code reads and declares no environment variable. The changed privilege declarations reach staging and production through the existing scripts/reconcile_v2_privileges.py apply step in .github/workflows/seed-v2-database.yml, whose variables and wiring this PR does not change.

Merge requirements

  • CI and an approving review. No other PR or operator step blocks this one.
  • It must be deployed to staging and production before the next PR in the stack (which drops the tables) merges.

Testing

  • uv run pytest tests/unit/v2 tests/contract -q: 1036 passed. The rest of tests/unit and tests/to_refactor: 2331 passed.
  • New tests record every SQL statement the ORM sends and fail if any names a retired table, or names country or label on the association tables: association insert, read, update and delete, and user deletion. They fail on master (the association insert lists country and label; user deletion selects from reports).
  • Model tests: no model maps a retired table, the models package exports none of the removed names, and no kept model has a relationship to a retired table.
  • Against disposable PostgreSQL: alembic -c alembic-v2.ini check reports no new operations; tests/integration/test_alembic_v2_lifecycle.py, test_v2_catalog_publication.py (including a new test that publishes as the catalog writer after its access to the three tables is revoked), test_v2_user_identity_persistence.py, test_v2_privilege_reconciliation.py, test_v2_stage12_comparison_persistence.py, test_v2_household_persistence.py, test_v2_policy_persistence.py, plus the cross-database dual-write tests against disposable MySQL: all passed.
  • mypy, ruff format --check, ruff check, scripts/run_quality_guards.py and scripts/export_migration_contracts.py (no changes): all passed.

🤖 Generated with Claude Code

anth-volk and others added 4 commits October 22, 2026 16:59
Catalog publication counted the rows of simulations, reports and
report_runs before and after it wrote the catalog, to prove it changed none
of them. The simulation service owns those tables in its own schema, and a
later v2 revision drops the Public API's copies while this release serves
traffic, so publication must not read them. It now counts only
dataset_versions rows.

The catalog writer role no longer needs SELECT on those three tables, so
the privilege manifest declares no access to them. The deploy step never
revokes, so existing grants stay until the revision drops the tables. The
dynamics entry moves out of the simulation-domain group it was listed in.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The simulation and report association tables still have the country and
label columns that Stage 12A retired. A later v2 revision drops them while
this release serves traffic, so this release must never select, insert or
update them.

The association models no longer declare the two columns. A new module,
retired_tables, appends them to the two tables after the models are
mapped, so the v2 metadata still matches the schema (alembic check and the
head-schema validation compare the two) and this change needs no
revision, but no mapper includes them. Deleting that module is what lets
autogeneration produce the revision that drops them. The direct-SQLAlchemy
categories list this use.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The simulation service owns household jobs, simulations, reports, report
runs and their outputs in its own simulation schema. A later v2 revision
drops the Public API's copies of those 15 tables while this release
serves traffic, so this release must never read or write them.

Remove their models (models/reports.py, models/simulations.py and
HouseholdJob), the unused report_runs module and their tests, and the
relationships the kept models (users, policies, dynamics, households,
datasets, regions and model versions) had to them. Deleting a user
through the ORM no longer selects from reports to clear their owner; the
foreign keys, with their ON DELETE actions, stay in the schema.

retired_tables now also declares the 15 tables as plain SQLAlchemy tables
that no model maps, identical in DDL to the removed models, so the v2
metadata still matches the schema and this change needs no revision.

The household migration qualification counts simulations, reports and
household_jobs through table() constructs instead of the models, for a
target that still has them. Households imports the PostgreSQL dialect
itself: it no longer imports policies, whose import loaded it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Five tests in test_models, and part of a sixth, asserted design rules of
the reports, report_runs, simulations and run output tables: which columns
reports and report runs have, the report-run uniqueness and current-output
index, and that outputs reference report runs. Those tables no longer have
models, and their remaining declarations exist only to match the schema
until a later revision drops them, which the metadata-to-database
comparison in the integration tests already checks. Remove the tests.

The Stage 9 metadata document no longer names the removed Simulation,
Report and ReportRun models.

Tests and documentation only: no behavior changes, so no test can fail
before this change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.27586% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 87.72%. Comparing base (ee46c7a) to head (943601c).

Files with missing lines Patch % Lines
policyengine_api/data/v2/catalog/publication.py 75.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           master    #4007       +/-   ##
===========================================
+ Coverage   50.12%   87.72%   +37.59%     
===========================================
  Files         243      257       +14     
  Lines       14726    15768     +1042     
  Branches     2343     2616      +273     
===========================================
+ Hits         7382    13833     +6451     
+ Misses       6721     1265     -5456     
- Partials      623      670       +47     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch was successfully deployed

1 active deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stop using the Public API's copies of the simulation-domain v2 tables before they are dropped

1 participant