feat(migrations): preserve column data on renames in incremental models

fensu.toml

@@ -93,6 +93,12 @@ rule = "FFR601"
 path = "src/sqlbuild/adapters/sqlserver/classes/sqlserver_adapter.py"
 reason = "The SQL Server adapter intentionally keeps overrideable backend behavior on one class."
 
+[[targets.python.rule_exceptions]]
+rule = "XSB051"
+path = "src/sqlbuild/executor/migrations/_helpers/columns.py"
+symbols = ["apply_model_column_renames"]
+reason = "Each in-place column rename must re-read live columns immediately before its ALTER so overlapping builds reconcile instead of failing."
+
 [[targets.python.rule_exceptions]]
 rule = "FFS120"
 path = "src/sqlbuild/integrations/dbt/_helpers/profile/render.py"

src/sqlbuild/.agents/skills/sqlbuild/references/docs/CONTENTS.md

@@ -39,6 +39,7 @@ Bundled copies of every page on the SQLBuild documentation site, matching the in
 - [Type Enforcement](concepts/models/type-enforcement.md) (`concepts/models/type-enforcement`) - Understand declared model types, static checks, and runtime casting.
 - [Contracts](concepts/models/contracts.md) (`concepts/models/contracts`) - Validate required or exact model output schemas.
 - [Model migrations](concepts/models/migrations.md) (`concepts/models/migrations`) - Keep the history of an incremental or snapshot model when you rename it.
+- [Column migrations](concepts/models/column-migrations.md) (`concepts/models/column-migrations`) - Keep a column's history when you rename it in an incremental or snapshot model.
 - [Hooks](concepts/models/hooks.md) (`concepts/models/hooks`) - Run SQL or Python lifecycle hooks around model materialization.
 - [SQL Hooks](concepts/models/hooks/sql.md) (`concepts/models/hooks/sql`) - Define, parameterize, compile, and invoke reusable or inline SQL lifecycle hooks.
 - [Python Hooks](concepts/models/hooks/python.md) (`concepts/models/hooks/python`) - Define Python lifecycle hooks with runtime context, providers, SQL access, and skips.

src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/plan.md

@@ -6,6 +6,13 @@
 
 Online: https://sqlbuild.com/docs/cli/plan/
 
+## Contents
+
+- Usage
+- Flags
+- Output
+- Missing upstream dependencies
+
 Shows the execution plan without making any changes. Useful for inspecting change detection, backfill policies, and selector scope before building.
 
 ## Usage
@@ -55,6 +62,8 @@ First run (12)
 
 Seeds (1)
   waffle_types
+
+✓ Plan complete  13 selected
 ```
 
 Steady state:
@@ -72,10 +81,24 @@ Normal (12)
 
 Seeds (1)
   waffle_types
+
+✓ Plan complete  13 selected
 ```
 
 When query or schema changes are detected, the plan shows the affected models with backfill actions and cascade information.
 
+Warnings and maintenance notices, such as `migrate_from can be removed`, are listed in a `Warnings` section before the final `Plan complete` line. `sqb plan --json` output is unchanged.
+
+Work that runs before models build is listed first. [Model migrations](../concepts/models/migrations.md) appear under `Migrations`, and in-place [column renames](../concepts/models/column-migrations.md) under `Column migrations`:
+
+```
+Column migrations (1)
+└── fact_orders  migrate columns
+    └── amount -> revenue  rename in place
+```
+
+When a removed column and an added column look related but are not an exact rename, the schema diff shows a hint such as `similar to amount; if this is a rename, add revenue (migrate_from amount)` until the change is built.
+
 ## Missing upstream dependencies
 
 Planning a scoped selection whose upstream inputs were never built in the target fails with `S301` rather than producing a plan that would break at build time:

src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/incremental.md

@@ -393,3 +393,5 @@ Controls how schema differences are handled at execution time when the increment
 | `sync_all_columns` | Add, drop, and alter columns to match the delta |
 | `ignore` | Log and continue without schema changes |
 | `fail` | Reject the build with an error |
+
+Columns are compared by name, so a renamed column looks like one column removed and one added. [Column migrations](models/column-migrations.md) rename the warehouse column in place before `on_schema_change` runs, so its history keeps its values; the policy then applies only to the differences that remain. Pure renames such as `amount` becoming `amount AS revenue` are detected automatically, and `sqb plan` suggests `migrate_from` for near matches.

src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/column-migrations.md

@@ -0,0 +1,137 @@
+<!-- generated-by: sqlbuild skills -->
+
+# Column migrations
+
+> Keep a column's history when you rename it in an incremental or snapshot model.
+
+Online: https://sqlbuild.com/docs/concepts/models/column-migrations/
+
+## Contents
+
+- Declaring a rename
+- Automatic detection
+- Decisions
+- How a rename runs
+- State
+- Adapter support
+
+`on_schema_change` compares columns by name, so renaming a column looks like one column removed and another added. With the default `append_new_columns`, the new column starts empty and older rows keep their values under the old name; with `sync_all_columns`, the old column and its data are dropped. A column migration renames the column in the warehouse instead, so every row keeps its value under the new name, with no leftover column and no replay.
+
+Column migrations apply to incremental models, including microbatch models, and snapshot models. Tables and views are rebuilt on every change and don't need them.
+
+## Declaring a rename
+
+Add `migrate_from` to the renamed column in the model header, naming the old column:
+
+```sql
+MODEL (
+  materialized incremental,
+  incremental_strategy delete_insert,
+  unique_key order_id,
+  cursor order_date,
+  columns (
+    revenue (migrate_from amount),
+  ),
+);
+
+SELECT order_id, order_date, amount AS revenue
+FROM __source("raw_orders")
+```
+
+On the next build, SQLBuild renames `amount` to `revenue` in the existing table, records the rename, and then builds the model incrementally, so the new delta lands in `revenue`.
+
+A column entry may contain only `migrate_from`, or combine it with `type`, `nullable`, `description`, and `audits`. When the model binds a reusable schema with `model_schema`, an inherited column may add `migrate_from` without redeclaring its metadata.
+
+After the rename is recorded, `migrate_from` has no further effect, and SQLBuild reports (`M108`) that it can be removed. Keep it until every target that needs the rename has built. On a target where the table doesn't exist yet, the declaration does nothing: the table is created with the new column name.
+
+A declaration is rejected when compiling if:
+
+- the model is not incremental or snapshot;
+- `migrate_from` names the column itself, or a qualified name instead of one column;
+- two columns declare the same `migrate_from`;
+- a declared column migrates from another column that itself declares `migrate_from`. Chains and swaps can't be renamed in place safely; rename through a name no other column uses, or rebuild with `--full-refresh`.
+
+## Automatic detection
+
+A pure rename is detected without a declaration: exactly one column that disappeared from the model has exactly the same expression as exactly one new column, for example `amount` becoming `amount AS revenue`, or `ROUND(amount, 2) AS revenue` becoming `ROUND(amount, 2) AS net_revenue`. Expressions are compared ignoring formatting, comments, and keyword and unquoted identifier case. Detection compares the query the table was last built from with the current query, so it applies to the build that makes the change.
+
+A rename is only automatic when it is the whole change. Apart from the renamed output names, the query must be the same: the same CTEs, `FROM`, joins, filters, grouping, and other columns. References to the new name in `ORDER BY` or `QUALIFY` count as references to the old name only when SQLBuild can prove they bind to the output column: the columns of every relation the query reads must be known, from the inspected warehouse tables of its sources and upstream models or from CTEs and subqueries with named columns, the new name must not be an input column, and the old name must either not be an input column or be passed through unchanged. Otherwise the change is not treated as a rename, because engines differ on whether such a name refers to the output alias or the input column. The output columns may be reordered unless the query groups, orders, or uses `DISTINCT ON` by position. If anything else changed, an unchanged expression such as `amount` may now read different data, for example from a CTE that redefines `amount`, so nothing is renamed automatically and the change follows `on_schema_change`. Declare `migrate_from` to rename the column anyway.
+
+Detection never guesses:
+
+- Columns are matched by expression, not by position. Two columns renamed at once each pair with the column whose expression they kept; two columns that swap names keep their names and are not renamed.
+- If more than one removed column has the same expression, or more than one new column does, nothing is migrated automatically.
+- A near match, where the expressions reference the same columns but differ or the rest of the query changed, for example `amount` becoming `ROUND(amount, 2) AS revenue`, is not a rename. It follows the model's `on_schema_change` policy, and `sqb plan` shows a hint next to the added column until the change has been built:
+
+```text
+    ├── schema diff:
+      + revenue   (added)
+        similar to amount; if this is a rename, add revenue (migrate_from amount)
+      - amount   (removed)
+```
+
+Add the suggested `migrate_from` to keep the history. A declared rename with a changed expression or other query changes is still renamed in place, but it is a query change, so `replay_on_change` decides how much history is rebuilt, exactly as it would without the rename.
+
+## Decisions
+
+`sqb plan` and `sqb build` list column migrations before the models they change:
+
+```text
+Column migrations (1)
+└── fact_orders  migrate columns
+    └── amount -> revenue  rename in place
+```
+
+Automatically detected renames are marked `(automatic)`. The decision is based on the table's current columns first, then on recorded renames:
+
+| Decision | Meaning |
+|----------|---------|
+| `rename in place` | The old column exists and the new one doesn't. Rename it. |
+| `already renamed, record` | The new column exists and the old one doesn't, but no rename is recorded, which is what an interrupted rename leaves behind. Record it without renaming again. |
+| `done` | The rename is recorded and the new column exists. Nothing to do. |
+| `conflict` | Both columns exist. The build stops (`M110`); drop one of them, or remove `migrate_from` and let `on_schema_change` handle the columns. |
+| `source missing` | Neither column exists and no rename is recorded. The build stops (`M109`); remove `migrate_from` if the rename happened elsewhere or is no longer needed. |
+| `source still produced` | The query still selects the old column. The build stops (`M111`). |
+| `unsupported` | The warehouse can't rename this table's columns in place (see [Adapter support](#adapter-support)). The build stops (`M112`). |
+
+Automatically detected renames never stop a build: when they can't be applied, the columns follow `on_schema_change` as before.
+
+## How a rename runs
+
+Renames run after [model migrations](migrations.md) and before any model builds, one model at a time:
+
+1. The table's current columns are read again. A column that is already renamed is not renamed twice.
+2. The column is renamed with `ALTER TABLE ... RENAME COLUMN` or the adapter's equivalent.
+3. The rename is recorded.
+
+On adapters with transactional DDL (PostgreSQL, SQL Server, DuckDB, MotherDuck), each model's renames and their records commit together. On other adapters, a build interrupted between the rename and the record is recovered by the `already renamed, record` decision on the next build. A build that fails after the rename, for example while applying the delta, resumes on the next build without renaming again.
+
+After the rename, the remaining differences follow `on_schema_change`: `append_new_columns` adds any other new columns, `sync_all_columns` also drops removed ones and changes types, and `fail` only fails when a difference other than the rename remains.
+
+A pure rename is not a query change. This applies to declared renames too, as long as the rename is the whole change by the rule above. It doesn't trigger `replay_on_change` for the model, and it doesn't cascade a replay to downstream models. Downstream models that you edit to use the new name have query changes of their own and follow their own `replay_on_change`. If the renamed column is the model's cursor, the next delta starts from the watermark stored under the old name.
+
+## State
+
+Each rename is recorded in an append-only `_sqlbuild_column_migrations` table in the model's schema, with the model, relation, old and new column, discovery (`manual` or `automatic`), and decision (`rename` or `record`). Janitor never archives it.
+
+## Adapter support
+
+| Adapter | Statement |
+|---------|-----------|
+| Snowflake, PostgreSQL, DuckDB, MotherDuck, BigQuery | `ALTER TABLE ... RENAME COLUMN ... TO ...` |
+| Databricks | `ALTER TABLE ... RENAME COLUMN ... TO ...`, which requires Delta column mapping |
+| SQL Server | `EXEC sp_rename '<schema>.<table>.<column>', '<new name>', 'COLUMN'` |
+
+Databricks renames a Delta table column only when the table uses name-based column mapping (`delta.columnMapping.mode = 'name'`). SQLBuild checks this during planning and doesn't change the table protocol for you. A declared rename on a table without column mapping stops the build (`M112`) with the statement that enables it:
+
+```sql
+ALTER TABLE <table> SET TBLPROPERTIES (
+  'delta.columnMapping.mode' = 'name',
+  'delta.minReaderVersion' = '2',
+  'delta.minWriterVersion' = '5'
+);
+```
+
+Enabling column mapping upgrades the table's Delta protocol, which older readers may not support. An automatically detected rename on such a table is skipped with a warning (`M112`) and follows `on_schema_change`.
+
+On PostgreSQL, views that select the renamed column keep working and keep their own output column names. On SQL Server, `sp_rename` warns that renaming can break scripts and stored procedures that reference the old name. On PostgreSQL, the new name is created in lower case, as an unquoted identifier would be.

src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/configuration.md

@@ -23,7 +23,7 @@ Online: https://sqlbuild.com/docs/concepts/models/configuration/
 | `materialized` | `view`, `table`, `incremental`, `snapshot`, or a custom materialization name |
 | `tags` | Tags used by selectors |
 | `description` | Human-readable model description |
-| `columns` | Model-local column declarations or inherited-column audit augmentation |
+| `columns` | Model-local column declarations, inherited-column audit augmentation, or column renames with `migrate_from` (incremental and snapshot models; see [Column migrations](column-migrations.md)) |
 | `model_schema` | Reusable column schema name |
 | `audits` | Model-level audit instances |
 | `enums` | Model-local enum declarations; names must begin with `_` |

src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/migrations.md

@@ -8,7 +8,7 @@ Online: https://sqlbuild.com/docs/concepts/models/migrations/
 
 Renaming an incremental or snapshot model normally means rebuilding it from scratch under the new name, which loses any history the source data can no longer reproduce. A model migration moves the existing relation's data to the new name instead, then continues building incrementally.
 
-Migrations apply to incremental and snapshot models.
+Migrations apply to incremental and snapshot models. To keep a column's history when you rename the column, see [Column migrations](column-migrations.md).
 
 ## Declaring a migration
 

src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/schemas.md

@@ -39,6 +39,8 @@ MODEL (
 
 See [Audits](../audits.md) for built-in audits, custom audits, arguments, severity, and incremental run scope.
 
+In incremental and snapshot models, an inline column can also declare `migrate_from <old column>` to rename the existing warehouse column in place and keep its history. See [Column migrations](column-migrations.md). Reusable `SCHEMA()` declarations don't accept `migrate_from`; a bound model adds it to an inherited column in its own header.
+
 ## Reusable schemas
 
 When multiple models implement the same relation shape, declare it once in a schema role. SQLBuild discovers schema files recursively in each role.
@@ -147,6 +149,6 @@ Reusable schemas intentionally support a narrow ownership model:
 
 - One optional parent, with transitive inheritance.
 - Additive child and model-local output columns.
-- Audit-only model augmentation of inherited columns.
+- Audit-only model augmentation of inherited columns, plus `migrate_from` column renames.
 - No general column overrides, multiple inheritance, composition, mixins, parameters, or generated projections.
 - No physical output ordinal enforcement.

src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/snapshots.md

@@ -200,6 +200,9 @@ Reappearing keys create a new active version.
 | `valid_to_column` | No | Override generated column name. Default `valid_to`. |
 | `initial_valid_from` | No | First-version start time: `updated_at`, `observed_at`, or `execution_time`. See defaults below. |
 | `snapshot_full_refresh` | No | Model-level full-refresh safety: `deny`, `require_confirmation`, or `allow`. |
+| `columns (<new> (migrate_from <old>))` | No | Rename a snapshot column in place so every version keeps its value. See [Column migrations](models/column-migrations.md). |
+
+Renaming a column in the snapshot query, such as `amount` becoming `amount AS revenue`, renames the existing snapshot column in place, so earlier versions keep their values under the new name. Renames are detected automatically when the expression is unchanged; declare `migrate_from` for anything else. See [Column migrations](models/column-migrations.md).
 
 ### Initial valid_from defaults
 

src/sqlbuild/.agents/skills/sqlbuild/references/docs/quickstart.md

@@ -68,6 +68,8 @@ First run (12)
 
 Seeds (1)
   waffle_types
+
+✓ Plan complete  13 selected
 ```
 
 Source data is defined as inline expressions in `sources/raw.yml`, so SQLBuild resolves everything at compile time - no seeding step needed.
@@ -159,6 +161,8 @@ Normal (12)
 
 Seeds (1)
   waffle_types
+
+✓ Plan complete  13 selected
 ```
 
 All models now show `Normal` instead of `First run`.

src/sqlbuild/adapter/contract/classes/base_adapter.py

@@ -775,6 +775,16 @@ def render_alter_column_types(
             for col in columns
         )
 
+    def render_rename_column(
+        self, *, destination: str, old_name: str, new_name: str
+    ) -> tuple[str, ...]:
+        """Render SQL that renames one table column in place, keeping its data."""
+
+        return (
+            f"ALTER TABLE {destination} RENAME COLUMN {self.render_identifier(old_name)} "
+            f"TO {self.render_identifier(new_name)}",
+        )
+
     def render_merge(
         self,
         *,
@@ -1350,6 +1360,23 @@ def drop_columns(
     ) -> None:
         raise AdapterUserError(message="drop_columns requires an engine-specific implementation")
 
+    def rename_column(
+        self,
+        *,
+        connection: Any,
+        destination: str,
+        old_name: str,
+        new_name: str,
+        statement_recorder: StatementRecorder,
+    ) -> None:
+        raise AdapterUserError(message="rename_column requires an engine-specific implementation")
+
+    def column_rename_unavailable_reason(self, *, connection: Any, destination: str) -> str | None:
+        """Return why this table cannot rename a column in place, or None when it can."""
+
+        del connection, destination
+        return None
+
     def alter_column_types(
         self,
         *,
@@ -2133,6 +2160,23 @@ def render_create_migration_state_table_sql(self, *, database: str | None, schem
             transient=self.state_tables_transient,
         )
 
+    def render_create_column_migration_state_table_sql(
+        self, *, database: str | None, schema: str
+    ) -> str:
+        """Render DDL that creates the column migration event table when it is missing."""
+
+        from sqlbuild.compiler.migrations.main.column_create_table_sql import (
+            build_column_migration_state_create_table_sql,
+        )
+
+        return build_column_migration_state_create_table_sql(
+            database=database,
+            schema=schema,
+            render_qualified_name=self.render_qualified_name,
+            render_framework_type=self.render_framework_type,
+            transient=self.state_tables_transient,
+        )
+
     def render_prune_fingerprint_history_sql(
         self,
         *,