Skip to content

Commit d6b254c

Browse files
kvlongekvlonge
committed
docs: describe compile reuse of unchanged inputs
1 parent b23b2ce commit d6b254c

2 files changed

Lines changed: 147 additions & 0 deletions

File tree

  • src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli
  • website/src/content/docs/docs/cli

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/compile.md‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ Online: https://sqlbuild.com/docs/cli/compile/
1212
- Flags
1313
- What compile does
1414
- Focused compilation
15+
- Reusing an unchanged compile
1516
- Static analysis
1617
- Output
1718
- Column lineage modes
@@ -69,6 +70,78 @@ with selected model, seed, and function counts.
6970
Manifest and DAG outputs remain full-project artifacts. Focused compilation changes analysis and
7071
reporting scope; it does not produce a partial project graph.
7172

73+
## Reusing an unchanged compile
74+
75+
When nothing that can change the result has changed since the last compile, `sqb compile` replays
76+
that result instead of compiling again. It prints one line to stderr, for example
77+
`Inputs unchanged; reused the previous compile (0.1 s)`, then the same report and exit code as the
78+
previous run. Failed compiles are replayed too, with the same diagnostics.
79+
80+
A compile is reused only when all of these match the stored run:
81+
82+
- Every file and directory under the project, compared by type, size, modification and change
83+
times, and inode. When those differ, SQLBuild compares content only for files whose content
84+
digest it already knows; otherwise the compile runs in full. Digests are recorded for files that
85+
changed within two seconds of a compile and for files whose timestamps moved since the previous
86+
compile, so large unchanged files are never read. After a checkout that only moves timestamps,
87+
the next compile runs in full once, and later timestamp-only changes are reused.
88+
- The command line: project directory, working directory, flags, selectors, `--vars`, target, and
89+
whether color output is active.
90+
- Environment variables that SQLBuild reads: every `SQLBUILD_*` and `SQB_*` variable, every
91+
variable that a template or `@@ENV` read during the stored compile, and every variable a
92+
provider's settings can read, including `env_prefix`, aliases, and nested-delimiter variables.
93+
Values are stored only as hashes.
94+
- The contents of each provider settings `env_file` and `secrets_dir`.
95+
- The Python interpreter, the SQLBuild version and native extension, `sys.path` entries, and every
96+
loaded Python module file outside the project.
97+
- Every artifact the stored compile wrote under `target/compiled/`, and the `--dag` file. If one is
98+
missing or changed, compile runs in full and rewrites it. A compile is stored only when these
99+
artifacts still hold the bytes it wrote, so a concurrent compile cannot leave another compile's
100+
artifacts behind a reused result.
101+
102+
These project paths are not inputs: `target/`, `logs/`, `venv/`, `.git/` and `__pycache__/`
103+
anywhere, and these folders at the project root: `.cache`, `.fensu`, `.hg`, `.idea`,
104+
`.mypy_cache`, `.nox`, `.pytest_cache`, `.ruff_cache`, `.sqlbuild`, `.svn`, `.tox`, `.venv`, and
105+
`.vscode`. Other hidden folders are inputs, because custom rules can read them. DuckDB database
106+
files count only by presence.
107+
108+
Reuse assumes macros, hooks, providers, adapters, and custom rules are deterministic: the same inputs
109+
must produce the same output. A compile whose templates read `run.id` is never reused.
110+
111+
Compile always runs in full with `--no-cache`, `--manifest`, `--debug`, or a profiling flag, when
112+
`SQLBUILD_DISABLE_COMPILE_CACHE=1` or `SQLBUILD_DISABLE_COMPILE_REUSE=1` is set, when the project has
113+
no `sqlbuild_project.toml` or `sqlbuild_project.yml`, when the target sets `compile_cache = false`,
114+
and when a provider's settings use sources SQLBuild cannot list exactly, such as a custom
115+
`settings_customise_sources` or command-line parsing. The reason is logged at debug level.
116+
117+
SQLBuild keeps one stored result per project and target under
118+
`target/cache/compiler/project-reuse-v1/`. It is replaced atomically; a damaged or unreadable entry is
119+
ignored and compile runs in full. Storing a result normally takes a fraction of a second after the
120+
report is ready. When it has to hash a lot of data or the project has very many files, compile
121+
prints `Recording compile for reuse...` and a completion line to stderr before the report.
122+
123+
Known limits:
124+
125+
- Python packages outside the project are checked by the stamps of the module files the compile
126+
loaded. A newly added submodule or package data file that the compile reads without importing it
127+
first is not tracked.
128+
- On network filesystems that cache file attributes, or when clocks differ between machines,
129+
timestamps may not reflect a recent edit.
130+
- On Windows, a file's change time is its creation time. An in-place rewrite that keeps the same
131+
size and restores the previous modification time, such as `touch -r`, `rsync -t`, or
132+
`robocopy /COPY:T`, is not detected unless the file changed within two seconds of the stored
133+
compile. The same applies to filesystems without a real change time, such as some FUSE, SMB, and
134+
exFAT mounts.
135+
- In these cases, or whenever a result looks stale, run `sqb compile --no-cache` or set
136+
`SQLBUILD_DISABLE_COMPILE_REUSE=1`.
137+
138+
A replayed JSON report is byte-identical to the previous one except for `compile_timings`, which
139+
holds `project_reuse_hits`, `project_reuse_misses`, `project_reuse_bypasses`,
140+
`project_reuse_check_ms`, and `total_ms`. Full compiles report the same reuse counters alongside
141+
the phase timings. In text mode, a replayed compile does not print the per-phase progress lines.
142+
Lifecycle sinks receive only the invocation events for a replayed compile; per-phase `operation`
143+
events are not emitted because no compile phase runs.
144+
72145
## Static analysis
73146

74147
When SQL analysis is enabled (default), compile performs static analysis on your models without connecting to the warehouse:

‎website/src/content/docs/docs/cli/compile.mdx‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,80 @@ Manifest and DAG outputs remain full-project artifacts. Focused compilation chan
6161
reporting scope; it does not produce a partial project graph.
6262
</Note>
6363

64+
## Reusing an unchanged compile
65+
66+
When nothing that can change the result has changed since the last compile, `sqb compile` replays
67+
that result instead of compiling again. It prints one line to stderr, for example
68+
`Inputs unchanged; reused the previous compile (0.1 s)`, then the same report and exit code as the
69+
previous run. Failed compiles are replayed too, with the same diagnostics.
70+
71+
A compile is reused only when all of these match the stored run:
72+
73+
- Every file and directory under the project, compared by type, size, modification and change
74+
times, and inode. When those differ, SQLBuild compares content only for files whose content
75+
digest it already knows; otherwise the compile runs in full. Digests are recorded for files that
76+
changed within two seconds of a compile and for files whose timestamps moved since the previous
77+
compile, so large unchanged files are never read. After a checkout that only moves timestamps,
78+
the next compile runs in full once, and later timestamp-only changes are reused.
79+
- The command line: project directory, working directory, flags, selectors, `--vars`, target, and
80+
whether color output is active.
81+
- Environment variables that SQLBuild reads: every `SQLBUILD_*` and `SQB_*` variable, every
82+
variable that a template or `@@ENV` read during the stored compile, and every variable a
83+
provider's settings can read, including `env_prefix`, aliases, and nested-delimiter variables.
84+
Values are stored only as hashes.
85+
- The contents of each provider settings `env_file` and `secrets_dir`.
86+
- The Python interpreter, the SQLBuild version and native extension, `sys.path` entries, and every
87+
loaded Python module file outside the project.
88+
- Every artifact the stored compile wrote under `target/compiled/`, and the `--dag` file. If one is
89+
missing or changed, compile runs in full and rewrites it. A compile is stored only when these
90+
artifacts still hold the bytes it wrote, so a concurrent compile cannot leave another compile's
91+
artifacts behind a reused result.
92+
93+
These project paths are not inputs: `target/`, `logs/`, `venv/`, `.git/` and `__pycache__/`
94+
anywhere, and these folders at the project root: `.cache`, `.fensu`, `.hg`, `.idea`,
95+
`.mypy_cache`, `.nox`, `.pytest_cache`, `.ruff_cache`, `.sqlbuild`, `.svn`, `.tox`, `.venv`, and
96+
`.vscode`. Other hidden folders are inputs, because custom rules can read them. DuckDB database
97+
files count only by presence.
98+
99+
Reuse assumes macros, hooks, providers, adapters, and custom rules are deterministic: the same inputs
100+
must produce the same output. A compile whose templates read `run.id` is never reused.
101+
102+
Compile always runs in full with `--no-cache`, `--manifest`, `--debug`, or a profiling flag, when
103+
`SQLBUILD_DISABLE_COMPILE_CACHE=1` or `SQLBUILD_DISABLE_COMPILE_REUSE=1` is set, when the project has
104+
no `sqlbuild_project.toml` or `sqlbuild_project.yml`, when the target sets `compile_cache = false`,
105+
and when a provider's settings use sources SQLBuild cannot list exactly, such as a custom
106+
`settings_customise_sources` or command-line parsing. The reason is logged at debug level.
107+
108+
SQLBuild keeps one stored result per project and target under
109+
`target/cache/compiler/project-reuse-v1/`. It is replaced atomically; a damaged or unreadable entry is
110+
ignored and compile runs in full. Storing a result normally takes a fraction of a second after the
111+
report is ready. When it has to hash a lot of data or the project has very many files, compile
112+
prints `Recording compile for reuse...` and a completion line to stderr before the report.
113+
114+
Known limits:
115+
116+
- Python packages outside the project are checked by the stamps of the module files the compile
117+
loaded. A newly added submodule or package data file that the compile reads without importing it
118+
first is not tracked.
119+
- On network filesystems that cache file attributes, or when clocks differ between machines,
120+
timestamps may not reflect a recent edit.
121+
- On Windows, a file's change time is its creation time. An in-place rewrite that keeps the same
122+
size and restores the previous modification time, such as `touch -r`, `rsync -t`, or
123+
`robocopy /COPY:T`, is not detected unless the file changed within two seconds of the stored
124+
compile. The same applies to filesystems without a real change time, such as some FUSE, SMB, and
125+
exFAT mounts.
126+
- In these cases, or whenever a result looks stale, run `sqb compile --no-cache` or set
127+
`SQLBUILD_DISABLE_COMPILE_REUSE=1`.
128+
129+
<Note>
130+
A replayed JSON report is byte-identical to the previous one except for `compile_timings`, which
131+
holds `project_reuse_hits`, `project_reuse_misses`, `project_reuse_bypasses`,
132+
`project_reuse_check_ms`, and `total_ms`. Full compiles report the same reuse counters alongside
133+
the phase timings. In text mode, a replayed compile does not print the per-phase progress lines.
134+
Lifecycle sinks receive only the invocation events for a replayed compile; per-phase `operation`
135+
events are not emitted because no compile phase runs.
136+
</Note>
137+
64138
## Static analysis
65139

66140
When SQL analysis is enabled (default), compile performs static analysis on your models without connecting to the warehouse:

0 commit comments

Comments
 (0)