Repository navigation
feat(migrations): preserve column data on renames in incremental models - #860
Merged
Merged
Conversation
Add column-level migrate_from for incremental, microbatch and snapshot models. Exact expression renames are detected automatically, near matches produce a plan hint, and declared renames run as in-place RENAME COLUMN before the delta with append-only, deterministic-ID events in _sqlbuild_column_migrations. Pure renames no longer trigger replay_on_change. Refs CHI-393
…erences Refs CHI-393
A planned rename of the cursor column made the first cursor-bounds read query the new column name, which does not exist yet, and log a misleading 'Failed cursor bounds' line before the old column was read. Skip the target read when inspected columns show the cursor column is missing; the outcome is unchanged and no failure is reported. Refs CHI-393
Maintenance notices such as M101 and M108 were the last visible output of a successful sqb plan. Text plan output now ends with a 'Plan complete' line after the Warnings section; JSON is unchanged. Refs CHI-393
…e whole change Automatic detection matched projection text alone, so an expression rebound by a changed CTE, join, or filter was renamed in place and bypassed on_schema_change. Replay suppression compared projections by position and treated alias references as query changes, so a pure rename with ORDER BY on the new alias, or a reordered projection list, replayed and overwrote history. Prove renames against the canonical AST: everything except the output names must match, projections are matched by name through the rename map, ORDER BY and QUALIFY references to a renamed alias are restored to the old name, and reorders are refused under positional grouping or ordering. Unproven changes get no automatic rename and only a 'similar to' hint; declared renames keep the model's replay decision unchanged. Refs CHI-393
Live columns were read once per model, so an overlapping build that renamed a later column between two renames made the second ALTER fail. Re-read them immediately before each rename and record a column that is already renamed. Refs CHI-393
…ally Refs CHI-393
… as equivalent ORDER BY and QUALIFY references to a renamed output were rewritten to the old name without proving they bind to the alias. When the new name was also an input column, DuckDB bound QUALIFY to the input, so the queries differed but the rename was applied automatically and replay was suppressed. Substitute only when the binding is proven: every relation the SELECT reads must have known columns (inspected physical columns of sources and upstream models, or named CTE and subquery outputs), the new name must not be an input column, and the old name must be absent from the inputs or passed through unchanged. Otherwise the change is not a rename, so nothing is renamed automatically and declared renames follow the normal replay policy. Refs CHI-393
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Renaming a column in an incremental or snapshot model looked like a drop plus an add: history was split across two columns or dropped, and fixing it needed a replay or hand-written SQL.
Changes
migrate_from(revenue (migrate_from amount)) renames the warehouse column in place before the delta, keeping every historical value. Native renames per adapter; SQL Server usessp_rename; Databricks requires column mapping.on_schema_change. Near matches show a one-time plan hint._sqlbuild_column_migrationstable; retries decide from live columns and re-check before each rename.sqb planshowsColumn migrations (N)and ends with a completion line so notices are never the last output. The cursor-column rename no longer logs a false failure.Verification
make test,make check-ciandmake test-real-postgrespass; hygiene scan clean.