Repository navigation
feat(native): batch and cache macro calls behind the preview engine - #1013
Merged
Merged
Conversation
kvlonge
enabled auto-merge (squash)
October 7, 2026 21:02
kvlonge
force-pushed
the
feat/native-macro-bridge
branch
from
October 7, 2026 21:40
82be72d to
899e93b
Compare
…ro calls The sqlbuild-render crate ports the Python macro call scanner byte for byte (quotes, comments, dollar quotes, the identifier/whitespace quirks, Python Unicode classes), splices rendered calls with code-point spans, and holds the in-compile memo of recorded calls and their events. Anything the scanner cannot reproduce exactly defers to Python.
…thon A macro_bridge module in the PyO3 crate registers scan_macro_call_sites and splice_macro_calls (run with the GIL released) and the MacroCallMemo class, whose recorded events cross as plain (tag, text, text) rows.
With the native engine, the compile-input render runs every expand_sql_macros_result call through the macro bridge: Rust scans the top-level call sites, Python resolves their call class and executes memo misses in order with the existing _evaluate_macro_call, recorded calls replay their ordered events (macro uses, declaration reads, generated-reference candidates, typed argument references) for each consumer, and Rust splices the results. Relation placeholders render exactly as in the serial path. Any native render failure re-runs the Python stage, which raises the exact error; a re-run that succeeds is reported as a divergence. The Python engine is unchanged.
The sqlbuild-cache crate is the one store and fingerprinting scheme for native caches: a content digest, a project file fingerprint that skips the folders project reuse skips, and one checksummed, atomically replaced file per cache kind whose entries are immutable and dropped after 64 saves unused. A damaged, foreign or older file opens empty. The macro call memo can attach a store: calls of a persistent class are looked up by a digest of their class text and call text, and recorded results are stored. The Python extension exposes the digest, the project fingerprint and the memo's store methods.
The native macro bridge now loads stored call results when the compile cache is enabled and saves the results it used or recorded after the render stage succeeds. A stored result is valid only in the same environment: native build, interpreter, import search path, SQLBUILD_* and SQB_* variables, and the content of every project file except model SQL files; loaded module files are stamped and rechecked before use. Each call is keyed by its text, the name, file, declaration identity and source of every macro in the call, the relations passed so far, and for context macros the whole context by value (vars, target, adapter, visible constants and enums). Values without stable text make the call uncached. An edit-sequence e2e test compiles each edit with the store and with --no-cache, with and without project reuse, and a broken store key must make it fail. The compiler differential compiles every authored and generated project a second time with a warm store.
… tier The macro bridge and the macro call store are registered as preview-tier native stages, so the default native engine and the Python engine never batch macro calls or open the store. The bindings move under the grouped bindings layout (render and cache domains). An e2e test compiles twice under each engine and checks macro executions and store files; the edit sequence also changes an @@env value passed to a macro.
Collecting the names of nested macro calls recursed once per nesting level and rescanned the arguments at each level, so deep nesting was quadratic and very deep nesting overflowed the native stack. One pass with an explicit stack now covers any depth in linear time.
Compile reuse, the per-compile macro call memo and the macro call cache all reuse a macro's result for the same call and context. Document the contract, which inputs are compared, and that environment variables read directly are compared only for SQLBUILD_ and SQB_ names.
…tent Module stamps were read when the store was saved, so a module imported before its file changed could be stored under the new file's stamps and replayed by later processes; unreadable modules (zip members, moved files) were skipped, and timestamps can be preserved or coarse. Only the first compile of a process now uses and saves the store, so code loaded by earlier compiles can no longer differ from disk. Loaded modules outside the interpreter and SQLBuild are identified by a BLAKE3 digest of their backing file, at attach, after recorded calls and at save; a module whose code cannot be read from a plain file disables the store for that compile. Installed compiled extensions are covered by their distributions' RECORD files, and the project fingerprint must be unchanged between attach and save. The process start time is no longer read, so the store also works without /proc.
kvlonge
force-pushed
the
feat/native-macro-bridge
branch
from
October 7, 2026 23:22
899e93b to
331adbc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Macro calls are the next render step to move native. Heavy, stable calls (large generated CTE
blocks) repeat within a compile and across edits, so the native path should run each distinct
call once and keep results between compiles, without risking a stale compile. Both ship in the
preview tier only.
Changes
sqlbuild-rendercrate: a byte-for-byte port of the macro call scanner (linear, no recursionat any nesting depth), code-point splicing, and an in-compile memo replaying recorded events.
macro_callsandmacro_call_store: one top-level call runs per distinctclass and text; any native failure re-runs the Python stage for the exact error. Default
nativeandpythonnever run the bridge or open the store.sqlbuild-cachecrate: the shared native store and fingerprinting scheme.ctxmacros, the whole context by value, plus build, interpreter, import path, installedpackages,
SQLBUILD_*/SQB_*variables and non-model project files.by BLAKE3 content digest at attach, during render and at save; unreadable module code (zip
members, moved files, custom loaders) disables the store for that compile.
Verification
--no-cache; broken keys fail it; a gate test checks no bridge or store outside preview;zip, moved, backdated, mid-compile and same-process module edits never replay stale results.
per project.
cargo test, clippy,make check-ci,make test, compile/lineage/scope e2e.