Repository navigation
perf: read native records directly and shorten the compile tail - #1139
Merged
Merged
Conversation
Compile-phase consumers (analysis session requests, declared relations, source keys, lineage counts, SQL test assembly, scope index, model loop) now read native model, source and compiled-model records directly instead of materialising Python record objects. Generic record helpers resolve specs by type pointer. Output is byte-identical; main-thread instructions on the dense 10k cold compile drop by about 10G (13.5%).
Serialising the 85 MB dense compile report copied every nested container's text into its parent and allocated a string per escaped character. Thread one buffer through the emitter instead; the output is byte-identical.
The --json report computed the selected models' semantic coverage twice: once for the stderr notice and again for semantic_checks_partial. Compute it once and hand the notice to the caller before the report is assembled.
write_static_target alternated between queueing a group on the main thread and waiting for its files to be written. Model, function and audit groups now go to one writer thread in order while the next group is queued; the SQL test group's synchronous flush waits for them, so records, stale cleanup and the manifest still see every file on disk. The writer stops at the first failure and no later group is written.
The launched sqlbuild-python process exits right after its command, yet each compile spent about half a second freeing the project graph and collecting it during interpreter finalization. The launcher now asks compiles to keep that state alive and freeze it out of collections; in-process entries still free it normally.
When a step after a background group failed, the artifact batch discarded the writer's outcome on drop, so a later error could mask an earlier OSError and defeat the staged path's direct-write fallback. Errors from the queueing section now wait for the writer and report its write error first, as the synchronous writes did. Waiting for the writer releases the GIL and checks for signals. An interrupt, or any non-Exception error, stops the writer after the group it is writing, skips groups not yet started, and then propagates.
kvlonge
enabled auto-merge (squash)
October 18, 2026 08:06
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
After the graph-phase changes, a cold dense 10k compile still spent long stretches on the main thread building and reading Python records the native side already held, and its last seconds ran on one core: JSON text was re-copied at every nesting level, semantic coverage was computed twice, artifact groups were written strictly after they were queued, and the process freed the whole project graph just before exiting.
Changes
Verification