Repository navigation
feat(migrations): keep incremental and snapshot history when models are renamed - #813
Merged
Merged
Conversation
…te_from Add migrate_from and migrate_force MODEL header keys for incremental and snapshot models. Direct-mode planning decides each migration per run from append-only _sqlbuild_migrations events plus live relation state (done, migrate, superseded replace, forced replace, conflict, origin missing), checks the origin against the model with the normal incremental schema rules, and plans the destination from the migrated history so it is not treated as a first run. Builds clone the origin over the destination with one adapter statement before any model runs and then record the event.
sqb plan --as <target> resolves configuration and namespaces for another configured target and inspects them through the active target's connection, following the single-connection rule used by defer and diff. It is inspection-only and shows migration decisions for that target.
Store a local migration fingerprint in each model fingerprint row and, when a model has no fingerprint on the target, match it against removed models whose relations still exist. Matching resolves renamed upstream refs bottom-up, ignores local CTE and alias names, comments, formatting and storage-only settings, and only accepts unique one-to-one matches; an explicit migrate_from always wins. Incremental and snapshot matches run the manual migration path and are recorded with discovery 'automatic'. Also bring the migration tests and helpers in line with the repository's dataclass-parameterized test conventions, and share the relation-location mapping used by planning, discovery and execution.
A pending migration whose destination exists without any recorded event and without a fingerprint of its own is now a 'redo': the clone runs again and the event is recorded, so a crash between clone and record converges on a plain re-run. A destination with its own build history still conflicts (M103) until migrate_force is set. A migrate_from origin that does not exist, with no recorded migration into the destination, now blocks the build with M102 and tells the user to remove migrate_from if the migration already happened elsewhere or is no longer needed. A recorded migration still reports done.
Local-name canonicalization now rewrites CTE definitions, unqualified CTE references, relation alias definitions, and alias qualifiers of columns and stars. Column names and other identifiers are never rewritten, so a.a and b.b no longer fingerprint alike.
Every model that declares migrate_from, qualified or not and selected or not, is excluded from automatic matching and its resolved origin leaves the candidate pool. Recorded moves into models not yet built resume as done and seed the rename map, so an interrupted coordinated rename continues downstream. Matching and ambiguity use all unbuilt project models; only selected destinations migrate.
BigQuery has no CREATE OR REPLACE TABLE ... CLONE, so it now raises the same unsupported-migrations error as Postgres and SQL Server.
Move the M105 check from inline graph resolution into _plan_virtual_build, and cover it with a virtual build integration test.
Plan text now renders error-severity diagnostics, such as M102 missing origins and M103 conflicts, in an 'Errors (n)' section ahead of 'Warnings (n)'. JSON output already carried the severity and is unchanged. Integration tests also pin that build prints each M102/M103 error exactly once.
Migration fingerprints are no longer produced for queries whose local names could resolve differently by position: nested or non-top-level WITH, recursive WITH, CTE bodies that reference their own or a later CTE name, and aliases that reuse a physical relation name. Such models are not eligible for automatic matching. Discovery now seeds the rename map from the latest recorded migration event per destination model on the current target, ignoring origins that are still project models, even after the destination has been built. It fingerprints each candidate with and without these recorded renames, so a retried partial chain keeps downstream history and later renames of models rebuilt under their new names still match.
# Conflicts: # tests/unit/src/sqlbuild/adapters/_test_types.py
…chived displacement Each moving migration now stages the origin into a fresh _sqb_archive__<ts>__migration_stage__ relation with the destination's table type, verifies it (existence, table type, columns, row count for copies), and promotes it with the shared staged-relation promotion. A live destination is kept as _sqb_archive__<ts>__migration_previous__ so janitor expires it by name. Transactional adapters commit the renames, PostgreSQL dependent-view rebinds, and the event together.
A table or view matched by fingerprint to a removed table or view is still rebuilt, but inherits the old fingerprint and plans as `renamed` with forward-only backfill, so downstream replay_on_change does not rebuild over migrated history. Each handover is recorded as a `renamed` event so a retry after the renamed model was built keeps resolving the rename. Plans list renames in their own Renamed section.
…dered SQL Migration state-table DDL now comes from each adapter, so SQL Server uses its guarded create-if-missing instead of an invalid CREATE TABLE IF NOT EXISTS. Event writes follow the janitor-event pattern: an existence check then INSERT ... VALUES with text-timestamp literals, avoiding ANSI TIMESTAMP casts that SQL Server rejects.
…e refusals The BigQuery stage fallback previously copied after any clone failure. Adapters now supply a clone-refusal classifier; BigQuery accepts only HTTP 400 invalid/invalidQuery errors whose details describe a clone limitation, so transient, quota, and permission errors are re-raised. If the stage exists after a refusal it is never adopted. Plan text and JSON show the copy fallback, and build output reports the transfer used.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Renaming an incremental or snapshot model rebuilt it from scratch under the new name, losing history the source data can no longer reproduce. Refactors should keep that history without manual steps.
Changes
migrate_from/migrate_forcemodel headers (direct mode, incremental and snapshot models). Per-run decisions: migrate, done, redo, superseded replace, forced replace, conflict (M103) and origin missing (M102), plus compatibility under normalon_schema_changerules (M104)._sqlbuild_fingerprintsmetadata. Unique matches only (M107 otherwise); explicitmigrate_fromwins._sqb_archive__<ts>__migration_stage__<name>with the destination's table type, verify, promote by swap or rename (transactional where supported), keep a replaced destination as…migration_previous…, then record. Janitor expires abandoned stages and replaced tables by name. The origin is never modified._sqlbuild_migrationsstate,sqb plan --as <target>, and an Errors heading in plan text.Verification
make testandmake check-cipass locally;fensu checkreports 0 faults.