You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 6701700
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: src/sqlbuild/.agents/skills/sqlbuild/references/docs/CONTENTS.md
+2-1Lines changed: 2 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -36,6 +36,7 @@ Bundled copies of every page on the SQLBuild documentation site, matching the in
36
36
-[Schemas](concepts/models/schemas.md) (`concepts/models/schemas`) - Declare model columns inline or reuse canonical inherited schemas.
37
37
-[Type Enforcement](concepts/models/type-enforcement.md) (`concepts/models/type-enforcement`) - Understand declared model types, static checks, and runtime casting.
38
38
-[Contracts](concepts/models/contracts.md) (`concepts/models/contracts`) - Validate required or exact model output schemas.
39
+
-[Model migrations](concepts/models/migrations.md) (`concepts/models/migrations`) - Keep the history of an incremental or snapshot model when you rename it.
39
40
-[Hooks](concepts/models/hooks.md) (`concepts/models/hooks`) - Run SQL or Python lifecycle hooks around model materialization.
40
41
-[SQL Hooks](concepts/models/hooks/sql.md) (`concepts/models/hooks/sql`) - Define, parameterize, compile, and invoke reusable or inline SQL lifecycle hooks.
41
42
-[Python Hooks](concepts/models/hooks/python.md) (`concepts/models/hooks/python`) - Define Python lifecycle hooks with runtime context, providers, SQL access, and skips.
@@ -137,7 +138,7 @@ Bundled copies of every page on the SQLBuild documentation site, matching the in
137
138
-[dag](cli/dag.md) (`cli/dag`) - Generate the static DAG artifact for Dagster and other integrations.
138
139
-[query](cli/query.md) (`cli/query`) - Run ad hoc SQL queries against the project database.
139
140
-[debug](cli/debug.md) (`cli/debug`) - Validate project configuration and test the warehouse connection.
140
-
-[janitor](cli/janitor.md) (`cli/janitor`) - Clean up stale warehouse relations.
141
+
-[janitor](cli/janitor.md) (`cli/janitor`) - Archive and then delete stale warehouse relations.
141
142
-[clean](cli/clean.md) (`cli/clean`) - Remove compiled artifacts from the target directory.
142
143
-[dbt](cli/dbt.md) (`cli/dbt`) - Coordinate dbt and SQLBuild projects.
143
144
-[state](cli/state.md) (`cli/state`) - Manage the virtual mode state store.
Copy file name to clipboardExpand all lines: src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/janitor.md
+82-10Lines changed: 82 additions & 10 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,13 +2,27 @@
2
2
3
3
# janitor
4
4
5
-
> Clean up stale warehouse relations.
5
+
> Archive and then delete stale warehouse relations.
6
6
7
7
Online: https://docs.sqlbuild.com/cli/janitor
8
8
9
+
## Contents
10
+
11
+
- Usage
12
+
- Flags
13
+
- Direct-mode lifecycle
14
+
- Relation age
15
+
- Audit record
16
+
- Restoring an archive
17
+
- Configuration
18
+
- Examples
19
+
- State history pruning
20
+
- Virtual environments
21
+
- Safety
22
+
9
23
# sqb janitor
10
24
11
-
Identifies and removes stale warehouse relations that are no longer part of the project. Requires `janitor.enabled = true` in `sqlbuild_project.toml`.
25
+
Finds stale warehouse relations that are no longer part of the project, archives them, and deletes archives once they expire. Requires `janitor.enabled = true` in `sqlbuild_project.toml`.
|`--auto-approve`| Skip the confirmation prompt and delete immediately |
24
-
|`--retention-days`| Override the configured retention period (days) |
37
+
|`--auto-approve`| Skip the confirmation prompt and run the planned actions immediately |
38
+
|`--retention-days`| Override the configured retention period (days) before a stale relation is archived |
39
+
|`--direct-state-history-versions`| Override how many state-history versions are kept per identity |
40
+
41
+
## Direct-mode lifecycle
42
+
43
+
In direct mode, the janitor never drops a live relation directly. Each stale relation goes through two stages:
44
+
45
+
1.**Archive.** Once a relation has been stale for `retention_days`, the janitor renames it in place, in the same schema, to:
46
+
47
+
```
48
+
_SQB_ARCHIVE__<UTC timestamp>__<original name>
49
+
```
50
+
51
+
For example, `stg_orders` becomes `_SQB_ARCHIVE__20260924T101500Z__stg_orders`. A rename copies no data, and keeps the relation's table type, grants and time travel.
52
+
2.**Delete.** Once an archive is older than `archive_retention_days`, the janitor drops it.
53
+
54
+
Stale relations are identified exactly as before. A relation is a candidate only if it is in a schema the project writes to, it is not produced by the current complete project, it was tracked by SQLBuild (when `delete_tracked_only = true`), it matches no exclude pattern, and it is older than the retention period.
55
+
56
+
The age of an archive comes from the timestamp in its name, so deletion needs no other state. The janitor only deletes relations whose names **strictly** match the archive format and that are in a schema the project writes to. Names that only look similar are reported and never touched. The janitor also only acts on relations whose names are plain lowercase identifiers (letters, digits and underscores), and that don't collide case-insensitively with another relation in the same schema. Anything else is reported and left alone.
57
+
58
+
Archive names are written as unquoted lowercase identifiers, so warehouses that fold unquoted names show them in their own case. On Snowflake, for example, the archive appears as `_SQB_ARCHIVE__20260924T101500Z__STG_ORDERS`.
59
+
60
+
Long names are shortened with a short hash so they fit the adapter's identifier limit. The prefix and timestamp are never shortened.
61
+
62
+
Setting both `retention_days` and `archive_retention_days` to `0` archives and deletes stale relations in the same run.
63
+
64
+
## Relation age
65
+
66
+
`retention_days` is measured from the last time a relation was written, as reported by the warehouse. For a relation the project no longer builds, that is roughly when it stopped being built.
67
+
68
+
| Adapter | Age source |
69
+
|---------|------------|
70
+
| Snowflake |`INFORMATION_SCHEMA.TABLES``CREATED` and `LAST_ALTERED`, which change on every write |
71
+
| Databricks |`DESCRIBE DETAIL``lastModified` for tables, and `INFORMATION_SCHEMA.TABLES` for views |
72
+
| BigQuery |`__TABLES__``creation_time` and `last_modified_time`|
73
+
| PostgreSQL, SQL Server, DuckDB, MotherDuck | Not available |
74
+
75
+
When the warehouse can't report a relation's age, the janitor skips that relation whenever `retention_days` is greater than `0`. SQL Server is in this group because its modification date ignores inserts and updates, so a recently written table could look old.
76
+
77
+
Ages are only looked up for relations that are otherwise candidates, and not at all when `retention_days` is `0`.
78
+
79
+
## Audit record
80
+
81
+
Every archive and every delete is recorded in an append-only `_sqlbuild_janitor_events` table in the affected schema. It records the original name, the archive name, the time and the invocation. The janitor never reads this table to decide what to do. It exists for auditing, and for restoring archives whose names were shortened.
82
+
83
+
## Restoring an archive
84
+
85
+
Rename the archive back to its original name before it expires, using your warehouse's rename statement. For example, on Snowflake:
86
+
87
+
```sql
88
+
ALTERTABLEanalytics._SQB_ARCHIVE__20260924T101500Z__stg_orders RENAME TO analytics.stg_orders;
89
+
```
90
+
91
+
If the archive's name was shortened, look up the original name in `_sqlbuild_janitor_events`. After an archive has been deleted, your warehouse's own time travel or undrop feature may still be able to recover it within its retention window.
25
92
26
93
## Configuration
27
94
@@ -30,20 +97,21 @@ Configure janitor behavior in `sqlbuild_project.toml`:
30
97
```toml
31
98
[janitor]
32
99
enabled = true
33
-
retention_days = 30
100
+
retention_days = 14
101
+
archive_retention_days = 14
34
102
delete_tracked_only = true
35
103
exclude_patterns = ["audit_*", "tmp_*"]
36
104
```
37
105
38
-
See [Project Configuration](../concepts/project-configuration.md) for details on janitor settings.
106
+
When `retention_days` is not set, direct mode uses `14` and virtual-environment mode uses `30`. See [Project Configuration](../concepts/project-configuration.md#janitor) for every janitor setting.
39
107
40
108
## Examples
41
109
42
110
```bash
43
-
# Interactive mode (prompts for confirmation)
111
+
# Interactive mode (shows the plan and prompts for confirmation)
44
112
sqb janitor
45
113
46
-
#Auto-approve deletion
114
+
#Run the planned archives and deletes without prompting
In addition to cleaning up stale relations, the janitor prunes old rows from `_sqlbuild_fingerprints` and `_sqlbuild_source_freshness` tables, retaining only the latest record per identity. This keeps state tables compact without affecting change detection.
123
+
In addition to archiving stale relations, the janitor prunes old rows from `_sqlbuild_fingerprints` and `_sqlbuild_source_freshness`, keeping the latest `direct_state_history_versions` rows per identity (default `20`). This is configured separately from relation retention.
124
+
125
+
## Virtual environments
126
+
127
+
In virtual-environment mode, the janitor also cleans up expired environments, old checkpoints and unreferenced physical versions. See [Virtual environments: janitor](../concepts/virtual-environments/janitor.md). Direct-mode archiving doesn't apply there. In both modes, relations whose names differ only by case from another relation in the same schema are reported and never dropped, and exclude patterns match regardless of case.
56
128
57
129
## Safety
58
130
59
-
Janitor prompts for confirmation before deleting. The confirmation requires typing an exact string to prevent accidental deletion. Use `--auto-approve` only in CI or when you're certain.
131
+
The janitor shows every planned archive and delete, with exact names, ages and expiry, then asks you to type an exact confirmation phrase. Use `--auto-approve` only in automation, or when you're certain.
Copy file name to clipboardExpand all lines: src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/plan.md
+1Lines changed: 1 addition & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -25,6 +25,7 @@ sqb --project-dir <path> plan [flags]
25
25
|`--changes-only`| Virtual mode only: show work not already bound to its expected VDE version |
26
26
|`--no-python`| Exclude read-side Python tasks and assets from the plan |
27
27
|`--defer-to`| Resolve unselected model references against another target |
28
+
|`--as`| Compile for another target and show its plan, including [model migration](../concepts/models/migrations.md) decisions, using the current connection. Inspection only. |
28
29
|`--json`| Output the plan as JSON |
29
30
|`--full-refresh`| Plan a full rebuild for selected models unless a model sets `full_refresh false`; `full_refresh true` forces a model without this flag |
30
31
|`--start-cursor-ts`| Override start cursor for timestamp incremental models (ISO format) |
Copy file name to clipboardExpand all lines: src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/configuration.md
+2Lines changed: 2 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -81,6 +81,8 @@ Table promotion mode is a project setting rather than a `MODEL()` field. Staged
81
81
|`full_refresh`| Optional model execution override: `false` always runs incrementally, `true` always full-refreshes, and omission follows the command |
82
82
|`on_schema_change`|`append_new_columns`, `sync_all_columns`, `ignore`, or `fail`|
83
83
|`replay_on_change`|`forward`, `full`, or `bounded-<duration>`|
84
+
|`migrate_from`| Old model name (or `schema.name`) whose data this model takes over; see [Model migrations](migrations.md). Also valid on snapshot models |
85
+
|`migrate_force`|`true` replaces a destination that already has its own build history during a migration |
84
86
85
87
See [Incremental](../incremental.md) for full semantics.
Renaming an incremental or snapshot model normally means rebuilding it from scratch under the new name, which loses any history the source data can no longer reproduce. A model migration moves the existing relation's data to the new name instead, then continues building incrementally.
12
+
13
+
Migrations apply to incremental and snapshot models in direct mode. Virtual-environment builds reject `migrate_from`.
14
+
15
+
## Declaring a migration
16
+
17
+
Add `migrate_from` to the renamed model's header, naming the old model:
18
+
19
+
```sql
20
+
MODEL (
21
+
materialized incremental,
22
+
migrate_from stg_orders,
23
+
cursor order_date,
24
+
...
25
+
);
26
+
```
27
+
28
+
Use `schema.name` when the old relation is in a different schema of the same database.
29
+
30
+
On the next build, SQLBuild copies the old relation's data to the new name through a staging table (see [How a move runs](#how-a-move-runs)), records the move, and builds the model incrementally from the migrated state. There is no first-run rebuild, and no `replay_on_change` replay caused by the rename. The old relation is never modified or dropped. Once it is no longer part of the project, [janitor](../../cli/janitor.md) archives it like any other stale relation.
31
+
32
+
After the move is recorded, `migrate_from` has no further effect, and SQLBuild tells you it can be removed. Keep it until every target that needs the move has built.
33
+
34
+
## Decisions
35
+
36
+
`sqb plan` shows the decision for every migration:
37
+
38
+
| Decision | Meaning |
39
+
|----------|---------|
40
+
|`migrate`| The new relation doesn't exist. Move the old one's data to it. |
41
+
|`done`| The move is already recorded. Nothing to do. |
42
+
|`redo`| The new relation exists but has no build history and no recorded move, which is what an interrupted migration leaves behind. Move it again. |
43
+
|`superseded replace`| The new relation was itself the origin of the latest recorded move, for example after renaming a model back. Replace it with the newer data. |
44
+
|`forced replace`| The new relation has its own build history and `migrate_force true` is set. Replace it. |
45
+
|`conflict`| The new relation has its own build history. The build stops (`M103`) until you set `migrate_force true` or remove `migrate_from`. |
46
+
|`origin missing`| The old relation doesn't exist and no move was recorded. The build stops (`M102`). |
47
+
48
+
The origin and destination must be compatible under the model's normal `on_schema_change` rules; otherwise the build stops (`M104`).
49
+
50
+
`migrate_force true` only affects the `conflict` case and is safe to leave in the header.
51
+
52
+
## Automatic discovery
53
+
54
+
If a selected model has never been built, SQLBuild compares it with models that were removed from the project but whose relations still exist. When exactly one removed model has equivalent logic, it is migrated automatically. Equivalent means the same query and configuration, ignoring the model's own name, CTE and table alias names, comments, formatting, and storage-only settings. Renamed upstream models are matched first, so a renamed chain of models is migrated together. Removed models whose data was already moved on by a recorded migration are not candidates, so a model renamed several times matches its latest table.
55
+
56
+
Renamed tables and views are matched the same way. They are still rebuilt under the new name, since they hold no history, but they keep their identity for change detection, so incremental models downstream of a renamed view are not replayed or rebuilt. These renames are recorded as `renamed` events.
57
+
58
+
Automatic discovery never guesses. If a match is ambiguous, SQLBuild warns (`M107`) and builds from scratch; declare `migrate_from` to choose. An explicit `migrate_from` always wins. Automatic discovery covers renames within the project's schemas; use `migrate_from` to move a model to another schema. Relations last built before this feature carry no stored fingerprint and aren't matched automatically.
59
+
60
+
## How a move runs
61
+
62
+
A move never overwrites the destination directly:
63
+
64
+
1. The old relation is cloned or copied into a new staging table named `_sqb_archive__<UTC timestamp>__migration_stage__<name>`, with the destination's configured table type.
65
+
2. The staging table is checked: it must exist, have the expected columns and table type, and, for a physical copy, the same row count as the old relation.
66
+
3. The staging table is swapped in as the destination. If a destination already existed, for example after `superseded replace` or `forced replace`, it is kept as `_sqb_archive__<UTC timestamp>__migration_previous__<name>` rather than dropped.
67
+
4. The move is recorded, then the model builds as usual.
68
+
69
+
The old relation is never modified. Staging and replaced tables use the [janitor](../../cli/janitor.md) archive format, so an abandoned staging table or a replaced destination is deleted by janitor after `archive_retention_days`. A staging table left by an interrupted run is never reused; the next run stages again. Don't run janitor at the same time as a build in the same schema.
70
+
71
+
`sqb plan` shows how each move will run, for example `physical copy, transient -> permanent, promote by swap`.
72
+
73
+
## State
74
+
75
+
Each move is recorded in an append-only `_sqlbuild_migrations` table in the destination schema. A missing record means completion is unrecorded; staging or promotion may already have happened. Re-running after an interruption recovers through the migration decisions above.
76
+
77
+
## Adapter support
78
+
79
+
| Adapter | Staging | Promotion |
80
+
|---------|---------|-----------|
81
+
| Snowflake | Zero-copy clone with `COPY GRANTS`. Transient → permanent is a physical copy, because Snowflake can't clone a transient table into a permanent one. |`ALTER TABLE … SWAP WITH` for an existing destination; rename for a missing destination |
82
+
| BigQuery |`CREATE TABLE … CLONE`, or `CREATE TABLE … COPY` when BigQuery refuses the clone | Two renames; the destination is briefly missing between them, and an interrupted run recovers on retry |
83
+
| Databricks |`DEEP CLONE`| Two renames, as for BigQuery |
84
+
| PostgreSQL, SQL Server, DuckDB, MotherDuck | Physical copy | Renames and the migration record in one transaction |
85
+
86
+
On PostgreSQL, views that depend on the destination are re-pointed to the new table in the same transaction, keeping their options, owner and grants. Materialized views are not re-pointed.
87
+
88
+
## Previewing another target
89
+
90
+
`sqb plan --as <target>` compiles the project for another target, for example production, and shows its migration decisions using your current connection. It only inspects; nothing is written.
Copy file name to clipboardExpand all lines: src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/project-configuration.md
+9-3Lines changed: 9 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -372,22 +372,28 @@ See [Macros](macros.md) for details on variable substitution and how variables i
372
372
373
373
## Janitor
374
374
375
-
Configuration for the `sqb janitor` command, which cleans up stale warehouse relations:
375
+
Configuration for the `sqb janitor` command, which archives and then deletes stale warehouse relations:
376
376
377
377
```toml
378
378
[janitor]
379
379
enabled = false
380
-
retention_days = 30
380
+
retention_days = 14
381
+
archive_retention_days = 14
381
382
delete_tracked_only = true
382
383
exclude_patterns = ["audit_*", "tmp_*"]
383
384
```
384
385
385
386
| Field | Default | Description |
386
387
|-------|---------|-------------|
387
388
|`enabled`|`false`| Whether janitor is active |
388
-
|`retention_days`|`30`| Only clean relations older than this many days |
389
+
|`retention_days`|`14` in direct mode, `30` in virtual-environment mode | How many days a relation must be stale before it is archived (direct mode) or cleaned up (virtual-environment mode) |
390
+
|`archive_retention_days`|`14`| Direct mode only: how many days an archive is kept before it is deleted. `0` deletes it in the same run. |
389
391
|`delete_tracked_only`|`true`| Only clean relations that appear in fingerprint tracking |
390
392
|`exclude_patterns`|`[]`| Glob patterns for relations to skip |
393
+
|`direct_state_history_versions`|`20`| How many state-history rows to keep per identity in `_sqlbuild_fingerprints` and `_sqlbuild_source_freshness`|
394
+
|`max_checkpoints`|`20`| Virtual-environment mode: how many checkpoints to keep |
395
+
396
+
See [janitor](../cli/janitor.md) for the archive and delete lifecycle.
0 commit comments