Repository navigation
perf(compile): faster cold compiles - #922
Merged
Merged
Conversation
…ying Background artifact preparation wrote every compiled file to a system temp directory and the write phase then walked, read and rewrote each one into target/. Stage beside target/ on the same filesystem instead, move the whole tree into place when target/compiled is absent, and otherwise rename new files and rewrite only changed ones, keeping the unchanged-file and stale-file semantics. Each staging directory is claimed by locking a sibling owner lock before the directory exists, held for the whole compile. At the start of a compile, staging directories are removed only while this compile holds their owner lock for the entire removal; cleanup stops and logs instead of continuing after a failed removal. Publication verifies the owner marker and the staged file set before moving anything or deleting stale files, and fails the compile rather than deleting valid artifacts if staging changed. New files that cannot be renamed across filesystems (EXDEV, for example target/compiled symlinked to another mount) are copied through a sibling temporary file and replaced atomically.
…eds them SQL-test planning rescanned each model's comments and quoted text once per marker kind even when that kind never occurs, and checked every match against every range. Scan lazily on the first candidate match, share one scan across the unresolved-reference checks, and look ranges up by binary search. Rendered test SQL is byte-identical.
…mand A cold compile keeps millions of objects alive until it finishes, so every full cyclic collection rescans that heap without freeing it (about 2-4 s of GIL-held time on the 3,000- and 5,000-model dense benchmarks). Disable the cyclic collector for the duration of the compile command and restore the previous state afterwards; reference counting still frees acyclic garbage.
…covery Native SQL lint, header lint and model discovery each kept a private copy of the line-start table and offset-to-position mapping; the native SQL lint copy scanned character by character in Python. Give discovery the single owner using str.find and reuse it from both lint modules. Positions are unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Cold compiles spent seconds re-reading and rewriting artifacts from a system temp directory, scanning every SQL test for comments and quotes, and running cyclic garbage collection, which alone took 18–25 s per 10k-model compile (CHI-441 phase 2, part of CHI-447/448).
Changes
--no-cacheartifacts are staged undertarget/and published by rename. The owner lock is taken before the staging dir exists; cleanup of leftovers from killed runs holds the lock throughout; publish verifies its staging tree before moving or deleting anything. Cross-device destinations fall back to an atomic copy.sqb compilepauses cyclic GC for the command and restores it afterwards.Verification
make test,make check-ci.