Repository navigation
perf: cut edit-loop compile overhead in the reuse store, hashing and walks - #1144
Merged
Merged
Conversation
A reuse check read and SHA-256-verified the whole stored compile, including the full report, on every check, and a hit that refreshed racy stamps read and rewrote all of it again. On a dense 10,000-model project that is a 96 MB file: about 0.5 s per check and 1.3 s per refresh. Each slot is now a small `<slot>.store` file holding the recorded inputs and the identity of a `<slot>.output` file holding the report. Both use a new section format with BLAKE3 checksums per section; readers skip sections they do not know and a refresh carries them forward, so later releases can add sections without another format break. A check decodes only the slot, the output is read only once the inputs match, and a refresh rewrites only the slot. A damaged, missing or mismatched output file is a miss. Stored compiles move to `project-reuse-v3`; the v1 and v2 folders are retired. Recording now stats the compiled tree once and re-stats only the artifacts whose content it had to read, instead of walking the tree a second time. Native store frames are checksummed with BLAKE3 (format version 2), so older store files open empty once.
…piles Several per-compile costs on a one-model edit scaled with the whole project although every per-model cache hit: - Background SQL test planning held the interpreter while it SHA-256 hashed an identity for every model and test, which blocked the main thread for about 0.6 s. Identities are now BLAKE3 digests of the same payload text, hashed in one batch without the interpreter, and the planning loops let other threads run every 64 items. The test artifact cache is version 5. - The macro call store listed model paths through `PurePath.relative_to` (0.5 s for 10,000 models); a string prefix check now covers the common case, with the old path as the fallback. Saving the store skips the second project fingerprint when the compile recorded no new calls, and the fingerprint uses BLAKE3. - Lint selection resolved every project SQL file through the filesystem one by one; folders shared by many files are now resolved once. - The stale compiled-file walk visited every artifact on every compile. It now leaves a stamp of each compiled folder in the compile cache and is skipped while the managed file set is the same and no folder changed; folders changed within the racy window are never trusted.
The stale-file walk stamp judged the racy window against the time it was recorded, after the walk. A walk longer than the window could stamp a folder another process changed after the walk listed it, and later compiles would then never remove that stray file. The window is now measured from before the walk starts. A folder that cannot be stamped, such as a linked `target/compiled` root, or one the walk could not read, was silently left out of the stamp, so stray files directly in it were never removed. Such a tree is now never stamped and is walked on every compile. Folders the walk removed are no longer listed. Pruning the reuse store also removes `.output` files with no slot once they are ten minutes old, so a compile that stops between writing the output and the slot no longer leaves the output behind for good. The compile reference now describes `project-reuse-v3` with its `.store` and `.output` files.
kvlonge
enabled auto-merge (squash)
October 18, 2026 15:33
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
On a dense 10k-model project a one-model edit compile took ~17 s and the compile right after an edit 2.7 s, even with every per-model cache hitting. Much of that was fixed overhead: a 96 MB reuse store fully checksummed with software SHA-256 on every read and rewritten on every racy refresh, repeated tree walks, a duplicate project fingerprint, pathlib conversions and SQL-test planning holding the GIL (CHI-628, first step).
Changes
<slot>.store(inputs plus output identity) and<slot>.output(report), with per-section BLAKE3 checksums; checks read only the slot, refreshes rewrite only the slot. Directoryproject-reuse-v3; older stores miss once and are removed. Orphaned outputs are pruned._model_pathswithout pathlib.Verification