Repository navigation
perf: compile only the changed models after an edit - #1157
Merged
Merged
Conversation
Store an image file next to each reuse slot, written between the output and the slot so the slot rename stays the commit point, and keep images only for the two most recently used slots. A missed reuse check now also reports which project files changed by content, the input a delta compile starts from.
…whole project Model analysis sessions can keep their final relation facts and run in a given dependency order, the built-in rules request can be kept without its SQL text and evaluated again from cached findings only, and model and SQL test discovery can be limited to named files. A delta compile builds on these to analyse only the models that changed.
A delta base now carries the stored inputs and image with the identity of the output they name, so the large stored report is read after the delta has run rather than before every attempt.
…stored compile On a reuse miss where only model or SQL test files changed, rerun the compile for those files with the rest of the project injected from the stored compile's image, splice their report items into the stored report and record the result as a normal reusable compile. Full compiles capture the image as they run. Any change the delta cannot reproduce exactly falls back to the full compile. SQLBUILD_DISABLE_DELTA_COMPILE=1 turns delta compiles off, and SQLBUILD_VERIFY_DELTA_COMPILE=1 runs each delta beside the full compile and exits 70 when their reports, exit codes or artifacts differ.
…decoding the image A delta compile now reads a gate section (format, output mode, stored errors, models, tests, header keys and custom-rule fact digests) to rule itself out before decoding the image's facts and rules, which it decodes in parallel. It moves the image's whole-project facts into the injection instead of cloning them each round, stops as soon as analysis or contracts report an error, and reruns the target model of a changed SQL test. Header digests and the stored rules request are cheaper to produce on every full compile.
…file in delta compiles The rules engine now retains a compile's request as JSON, encoded in place without cloning its models, so full compiles pay less for their image and the image stores those bytes as they are. A delta compile hands its decoded request to the rules round instead of cloning it, lints its changed files without walking and canonicalising every SQL file, encodes the patched image while it splices the report, reads the stored report only in JSON mode, and skips the macro call store's project fingerprint.
Edits that introduce or fix error diagnostics, and changed files that stop the compile with an error, now take the delta path; the verifier compares them like any other delta.
The image's gate and facts are now length-prefixed native records with a string table instead of JSON, and the rules engine assembles its retained request from the row and request JSON it already serializes for its digests, so a full compile pays about 1.7% for its image instead of 3.4%.
When no stored diagnostic concerns a file outside models and SQL tests, a delta compile builds records only for the sources its rerun models read and splices their report items into the stored source list.
…t rules - Index scope resources once instead of scanning them for every SQL test. - Summarise each SQL-test fixture once per evaluation and keep the summaries beside the bulk rules cache, so unchanged fixtures are not parsed again. - Drop unread source entries from loaded source YAML before building them. - Skip the rules response memo in delta compiles, whose request never repeats. - Record a delta compile's kept artifacts with the stamps the reuse check verified instead of walking the compiled tree again. - Scan string bytes a word at a time when splicing the stored report.
A delta compile stores each source YAML document it loads, keyed by the digest of its text, in a native store beside the reuse slots, and decodes it instead of parsing it when the text is unchanged.
Decode the stored rules request on its own thread while the delta compile discovers and analyses, and write a stored compile's output and image at once.
…nd tests read A delta compile that discovers only some sources now also keeps those that functions and audits read and those its rerun SQL tests read, falls back early when a model edit changes which sources it names, and falls back when anything still reports an unknown source.
…round deltas - [settings] delta_compile (default true, overridable locally) stops compiles capturing a project image, so no delta compile follows. - Record a delta compile's kept artifacts with the reuse check's stamps only where they still match; a multi-round delta rewrites some of them, and the stale stamp made recording drop the stored compile. - Fall back before compiling when an edited model's authored references change, which a delta compile would otherwise only find after its first round. - Unit tests for the report splice, the project image round trip and damaged images, and the stored YAML documents.
- A seeded series of edits to a 300-model DuckDB project, in JSON and text, checks every delta against a full compile and every expected fallback. - Cover replaying a delta, multi-round deltas followed by another edit, and the delta_compile setting and environment opt-outs. - Fall back before compiling when the stored compile reports non-rule problems in files the delta would not compile again, which it cannot reproduce, and log which diagnostic differs when one still does. - Export the delta compile environment variable names.
…line with workspace policy - Serialize retained rules rows without their SQL text directly instead of scanning each row's JSON for the SQL keys, which a full compile paid for. - Single-line doc comments, constants ahead of behavior, explicit handling of unreadable stores, the image section alias in types.rs, one entry function per module, and table-driven tests with helpers in helpers.rs.
… instead of digesting every model's - Full compiles no longer scan every model's authored references for the image; a delta compile scans only the changed models and falls back when one names a model, source or seed its stored compile did not read. - Find the retained request's model list at its first match and key the image string table with the fast hasher.
- Split the delta driver into preparing, running rounds and finishing; pass owned values instead of mutable references; read recorded output and graph edges through their native types instead of Python attributes; log through one compile reuse debug helper. - Keep only text, not Python strings or the request, in finished deltas, and keep a compile's phases as a tuple as before. - Handle unreadable stores, images and JSON explicitly. - Move the delta compile unit tests to bindings/tests with shared helpers and table-driven cases, and the seeded E2E edit series into scripts/delta_compile_series.
Full compiles serialized every model's rules row a second time as JSON, without its SQL, for the project image. The rows are now written once in the native length-prefixed record format next to the model-free request JSON and read back in sqlbuild-rules, cutting the image overhead of a full compile after an edit to under 2%.
Delta compiles diverged from full compiles in three ways: - Partial semantic-check reasons came only from the rerun models, so the text report dropped notices for unchanged models (models with SQL analysis off, dependents of an invalid model), and JSON reports with partial models always fell back. Each model's coverage facts are now kept in the project image (format v8). A delta merges the stored facts with the rerun models' fresh facts and recomputes the invalid-dependency reasons over the whole project. Text and JSON both render from that merged set, and the JSON splice takes the delta's section. - A SQL test edited to expect or mock another model was compiled without that model, which gave a false missing-model error. An edited test now reruns its stored chain, the chain's direct upstream models and every model its text names. The delta falls back when the test's chain resolves differently, or when a round reports a project model it did not compile as missing. - A fallback after a delta round printed the lifecycle lines twice. The progress projector now holds a delta's lines back. It writes only the longest round's lines, and only once the delta finishes; delta stderr notes print after them. The edit series gains SQL-analysis, invalid-dependency and SQL-test retarget edits, a dialect without type checks, and a mode that compares each compile's stdout and stderr with a full compile of a copy of the project.
The dense all-rules jobs at 3k and 10k models now compile the project once, then edit one model body, one model header and one SQL test, compiling after each edit. Each of those compiles must: - take the delta path, so it analyses only a handful of models according to its compile_timings counters; - stay under a wall cap; - keep its CPU within a ratio of a delta-disabled full compile on the same runner. Then project-config edits alternate between delta-enabled compiles, which fall back, and delta-disabled ones. The median CPU of the fallbacks must stay within 5% of the delta-disabled compiles. The dense job's timeout grows to 30 minutes to fit the 10k loop.
…l budgets The wall budgets sat two to three times above what CI measures. Each budget is now the larger of 1.3 times the median and 1.15 times the maximum of the last 20 successful CI runs of the job, rounded up to half a second.
…ment again The delta driver refactor formatted the count into the message, so tools and tests reading the compile reuse debug log's "digested %d project files" records no longer saw the check's reads. The message is logged with the count as its argument again, through the shared compile reuse debug helper.
…alls A delta compile's reuse record listed only the module files its own process imported. A stored macro call that the delta did not run kept its lazily imported module out of the record, so editing that module replayed stale output. The delta's record now also keeps every module file of the compile it patched. Three related problems made the delta fall back, or cost the following full compile its stored calls: - A SQL test rerun because its chain touched a rerun model was compiled without the chain's other models, which gave false missing-model errors. A rerun test now brings its whole stored chain into the rerun. - Sources filled by a loader were dropped with the other unread sources, so their loaders were reported as undescribed resources. Such sources are now always discovered. - A delta round claimed the process's first compile, the only one allowed to use the macro call store, so the full compile after a fallback re-ran every stored macro. The claim now passes on to that full compile and is kept only when the delta finishes.
kvlonge
enabled auto-merge (squash)
October 19, 2026 11:17
…lta compile A delta compile discovers only the models it reruns, so cross-file validation saw a path_defaults key whose folder held no rerun model as unmatched and raised a discovery conflict. Every edit on such a project fell back to a full compile, slightly slower than compiling in full outright. Path defaults are now matched against the whole project's model paths from the stored compile, which a delta shares because added or removed files fall back.
…nner edit ratio A delta compile analyses only the models it reruns, without the analysis cache, so the ratio script's cache precondition rejected head's one-model edit as an uncached compile. Each run now records how many models its analysis counters account for. An edit run that analysed fewer models than the project has is a delta, and passes when it analysed at least one model afresh. Full compiles keep the existing checks, so a bypassing full compile is still rejected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A one-model edit on a dense 10,000-model project took ~16 s to recompile, because any change re-ran the whole project. The edit loop should take seconds (CHI-628).
Changes
[settings] delta_compile = falseorSQLBUILD_DISABLE_DELTA_COMPILE=1. Docs describe what is reused and the fallbacks.Verification
delta_compileadded to the allowed settings list).