Repository navigation
perf: cut serial compile work and duplicated parsing on dense projects - #1130
Merged
Merged
Conversation
…thon enum reads natively and read record fields from the native spec
…llel rule cache keys
…num and factory lookups Header columns, audit instances and measurement thresholds are built with positional arguments in their declared field order, verified once against the registered record specs, instead of a per-object keyword dict. The model loop keeps header columns and audits as vectors rather than round-tripping through temporary tuples, derives direct audit identities from the parsed nodes, skips enum column resolution when no enums are visible, and lists audit factories only for headers that reference them.
A run selecting no rules now validates its rules configuration through a model-free native request on the calling thread instead of spawning the native rules thread, rebuilding the scope index and every model's rules facts, and replaying the project's models to count unevaluated ones.
…und trip The built-in rules request converted the native scope metadata projection into Python dictionaries only to read them straight back into JSON values. Convert the projection to the same sorted JSON value natively instead.
Rule-cache, custom-rule-cache, rows-memo and model-finding identities are internal, build-scoped keys. Replace software SHA-256 with BLAKE3 and stream model JSON into the hasher instead of buffering it. Bump the three cache versions so old entries are ignored.
Large batches of distinct files are written on four threads. The first failure in batch order is still the one reported, unchanged files are still counted, and batches that repeat a path stay sequential so the last queued contents win.
Shared-result keys and compact-batch template deduplication re-serialised the batch's function return types for every member. Hash the batch-wide settings once and keep them out of the per-member template identity; the request sent to the analysis pool is unchanged.
Model files keep their contents, query, header values and path text as Rust data and build the Python dict, strings and pathlib objects only when Python reads them. Model input records share the native query and expansion texts instead of Python strings, and internal readers use the native fields directly.
Before the ordered model loop, a fixed four-worker pool reads every model's native header off the GIL: template and macro presence and config-reuse checks per header entry, and the header's columns. The ordered loop uses a fact only for a config value that is the header's own object, so values that defaults, templates or hooks replaced are still read from Python. Header columns parsed natively build their Python records only when Python reads them.
Model facts share the model file's query and authored text instead of copying them. Compiled artifacts, metadata checks and resource SQL requests read a native model's name, path and SQL directly, and Python reads of a model's query and file text share one lazily built string. Header facts and their native node reader move to the discovery crate with Rust tests.
Compiles with several failing models report the first failure in model order, native and templated header columns produce the same contract diagnostics, and native column audits run against DuckDB.
Model file paths build their pathlib objects natively on first read, header column parsing helpers move under the discovery helpers, the unused whole-config macro walk is removed in favour of the per-entry walk, natively parsed columns use the default contract classes, and pending Python-crossing budgets drop to the new counts.
A failed pool creation was cached and failed every later compile in the process. Compute the header facts on the calling thread instead, so compiling never depends on spawning the pre-pass workers.
Add a speculative rows evaluation to the rules engine: it runs on its own thread, holds its bulk cache and response memo writes, and answers the real request only when that request has the same key, making the held writes in the original order. A stop flag ends waiting and model evaluation early.
Add early_rules_build_ms, early_rules_wait_ms, early_rules_hits and early_rules_misses to compile_timings.
After type recovery the compile builds the built-in rules request from the recovered project and starts its evaluation on a background thread. The rules phase builds its request as before and uses the early result only when the request keys match. Projects with custom rules, model selections and failed graphs keep the existing path.
A thread that cannot start, or a speculative evaluation that fails, no longer fails the compile: the rules phase evaluates the request as before.
kvlonge
changed the title
kvlonge
marked this pull request as ready for review
October 17, 2026 22:33
kvlonge
enabled auto-merge (squash)
October 17, 2026 22:33
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Cold compiles of large dense projects spent most of their wall time in serial main-thread work and duplicated parsing. The dense 10k all-rules CI measurement was 44.1 s on 0.157.0.
Changes
Verification