feat(native): batch and cache macro calls behind the preview engine (#1013)

* feat(native): add a render crate that scans, splices and memoizes macro calls

The sqlbuild-render crate ports the Python macro call scanner byte for
byte (quotes, comments, dollar quotes, the identifier/whitespace quirks,
Python Unicode classes), splices rendered calls with code-point spans,
and holds the in-compile memo of recorded calls and their events.
Anything the scanner cannot reproduce exactly defers to Python.

* feat(native): expose macro call scanning, splicing and the memo to Python

A macro_bridge module in the PyO3 crate registers scan_macro_call_sites
and splice_macro_calls (run with the GIL released) and the MacroCallMemo
class, whose recorded events cross as plain (tag, text, text) rows.

* feat(native): batch macro calls through a native render bridge

With the native engine, the compile-input render runs every
expand_sql_macros_result call through the macro bridge: Rust scans the
top-level call sites, Python resolves their call class and executes
memo misses in order with the existing _evaluate_macro_call, recorded
calls replay their ordered events (macro uses, declaration reads,
generated-reference candidates, typed argument references) for each
consumer, and Rust splices the results. Relation placeholders render
exactly as in the serial path. Any native render failure re-runs the
Python stage, which raises the exact error; a re-run that succeeds is
reported as a divergence. The Python engine is unchanged.

* feat(native): add a shared native store and carry macro calls in it

The sqlbuild-cache crate is the one store and fingerprinting scheme for
native caches: a content digest, a project file fingerprint that skips
the folders project reuse skips, and one checksummed, atomically
replaced file per cache kind whose entries are immutable and dropped
after 64 saves unused. A damaged, foreign or older file opens empty.

The macro call memo can attach a store: calls of a persistent class are
looked up by a digest of their class text and call text, and recorded
results are stored. The Python extension exposes the digest, the
project fingerprint and the memo's store methods.

* feat(native): cache macro call results across compiles

The native macro bridge now loads stored call results when the compile
cache is enabled and saves the results it used or recorded after the
render stage succeeds. A stored result is valid only in the same
environment: native build, interpreter, import search path, SQLBUILD_*
and SQB_* variables, and the content of every project file except model
SQL files; loaded module files are stamped and rechecked before use.
Each call is keyed by its text, the name, file, declaration identity
and source of every macro in the call, the relations passed so far, and
for context macros the whole context by value (vars, target, adapter,
visible constants and enums). Values without stable text make the call
uncached.

An edit-sequence e2e test compiles each edit with the store and with
--no-cache, with and without project reuse, and a broken store key
must make it fail. The compiler differential compiles every authored
and generated project a second time with a warm store.

* feat(native): run the macro bridge and call store only in the preview tier

The macro bridge and the macro call store are registered as preview-tier
native stages, so the default native engine and the Python engine never
batch macro calls or open the store. The bindings move under the grouped
bindings layout (render and cache domains). An e2e test compiles twice
under each engine and checks macro executions and store files; the
edit sequence also changes an @@env value passed to a macro.

* fix(native): scan nested macro calls in one pass without recursion

Collecting the names of nested macro calls recursed once per nesting
level and rescanned the arguments at each level, so deep nesting was
quadratic and very deep nesting overflowed the native stack. One pass
with an explicit stack now covers any depth in linear time.

* docs: state that macros must be deterministic

Compile reuse, the per-compile macro call memo and the macro call cache
all reuse a macro's result for the same call and context. Document the
contract, which inputs are compared, and that environment variables read
directly are compared only for SQLBUILD_ and SQB_ names.

* fix(native): trust stored macro calls only for code identified by content

Module stamps were read when the store was saved, so a module imported
before its file changed could be stored under the new file's stamps and
replayed by later processes; unreadable modules (zip members, moved
files) were skipped, and timestamps can be preserved or coarse.

Only the first compile of a process now uses and saves the store, so
code loaded by earlier compiles can no longer differ from disk. Loaded
modules outside the interpreter and SQLBuild are identified by a BLAKE3
digest of their backing file, at attach, after recorded calls and at
save; a module whose code cannot be read from a plain file disables the
store for that compile. Installed compiled extensions are covered by
their distributions' RECORD files, and the project fingerprint must be
unchanged between attach and save. The process start time is no longer
read, so the store also works without /proc.

* test(native): replay macro-returned reference call errors for every consumer

* refactor(native): raise the shared native stage mismatch error from the macro bridge

Author: kvlonge

Cargo.lock

@@ -1,11 +1,13 @@
 [workspace]
 members = [
     "crates/sqlbuild-core",
+    "crates/sqlbuild-cache",
     "crates/sqlbuild-sqltext",
     "crates/sqlbuild-config",
     "crates/sqlbuild-discovery",
     "crates/sqlbuild-scopes",
     "crates/sqlbuild-model-config",
+    "crates/sqlbuild-render",
     "crates/sqlbuild-analysis",
     "crates/sqlbuild-rules",
     "crates/sqlbuild-python",
@@ -18,11 +20,13 @@ resolver = "2"
 [workspace.metadata.native-layers]
 order = [
     "sqlbuild-core",
+    "sqlbuild-cache",
     "sqlbuild-sqltext",
     "sqlbuild-config",
     "sqlbuild-discovery",
     "sqlbuild-scopes",
     "sqlbuild-model-config",
+    "sqlbuild-render",
     "sqlbuild-analysis",
     "sqlbuild-rules",
     "sqlbuild-python",
@@ -61,6 +65,7 @@ polyglot-sql = { git = "https://github.com/chio-labs/polyglot.git", rev = "9ff0a
 polyglot-sql-function-catalogs = { git = "https://github.com/chio-labs/polyglot.git", rev = "9ff0af282829d92efeeb36b6a820be2271e1da49", default-features = false, features = [
     "dialect-duckdb",
 ] }
+blake3 = { version = "1", features = ["rayon"] }
 quote = "1"
 rayon = "1"
 saphyr-parser = "0.1"
@@ -70,9 +75,11 @@ serde = { version = "1", features = ["derive"] }
 serde_json = { version = "1", features = ["preserve_order"] }
 sha2 = "0.10"
 sqlbuild-analysis = { path = "crates/sqlbuild-analysis" }
+sqlbuild-cache = { path = "crates/sqlbuild-cache" }
 sqlbuild-config = { path = "crates/sqlbuild-config" }
 sqlbuild-core = { path = "crates/sqlbuild-core" }
 sqlbuild-discovery = { path = "crates/sqlbuild-discovery" }
+sqlbuild-render = { path = "crates/sqlbuild-render" }
 sqlbuild-rules = { path = "crates/sqlbuild-rules" }
 sqlbuild-model-config = { path = "crates/sqlbuild-model-config" }
 sqlbuild-scopes = { path = "crates/sqlbuild-scopes" }

Cargo.toml

@@ -7,11 +7,13 @@ that can be tested and benchmarked with `cargo test` alone.
 | Crate | Responsibility |
 |---|---|
 | `sqlbuild-core` | Constants shared by several crates, the panic boundary that turns an unwinding native failure into a named compiler error, passive identity, diagnostic and SQL value types, text positions, and a JSON emitter whose output matches Python's `json.dumps` and `orjson` byte for byte. |
+| `sqlbuild-cache` | The shared native store every native cache uses (one atomically replaced, checksummed file per cache kind, keyed by content digests) and the one fingerprinting scheme for cache keys: the content digest and the project file fingerprint. |
 | `sqlbuild-sqltext` | Lexical SQL text without polyglot: comment, quote and parenthesis scanning, the rules quote policy, model header tokenization and matching, static variable substitution, static reference extraction, and the general reference scan with table-function call arguments under each adapter's lexical rules. |
 | `sqlbuild-config` | Configuration file reading without Python: `tomllib`-compatible TOML, PyYAML `safe_load`-compatible YAML 1.1, and typed project and local config readers for the fields discovery needs. Errors carry their kind and position; YAML outside the supported subset is an `Unsupported` error. |
 | `sqlbuild-discovery` | Native project discovery and the only implementation of the model, SQL test, scenario, source and schema file collections and the declaration layout: the shared directory walk with Python's glob and sort semantics, file reading with Python's newline handling and read errors, and parsing of authored files with Python's exact discovery messages. Results are plain data the Python discovery facade materialises. |
 | `sqlbuild-scopes` | Declaration scopes: the scope index with Python's record orders and diagnostics, declaration visibility, relationship grants, the scope lookup groups, and the dialect-aware scan of SQL tests and scenarios for their expected models. Anything it cannot reproduce exactly defers to Python. |
 | `sqlbuild-model-config` | Model configuration: MODEL header columns and audits, `${...}` template expansion with its environment and context reads, and the template and macro presence scans over authored config values, all read through a trait over the caller's values. Anything it cannot reproduce exactly defers to Python, which also raises every model config error. |
+| `sqlbuild-render` | Native rendering: the macro call scanner (a byte-for-byte port of Python's), splicing of rendered calls with code-point spans, and the in-compile memo of recorded macro calls and their replayable events, which can carry results across compiles through the shared native store. Anything it cannot reproduce exactly defers to Python. |
 | `sqlbuild-analysis` | SQL analysis over polyglot: SQL tokens, query analysis, the binding catalog, semantic validation and usage, column references, and SQL-test extraction, planning and rendering. |
 | `sqlbuild-rules` | Built-in rules and the rules engine, the custom-rule host, SQL lint, quality checks and formatting, rules configuration and the request models. It also owns the build identity script. |
 | `sqlbuild-python` | The only PyO3 crate: the `_native` module, its Python classes and functions, conversions from Python objects, and the process allocator. |
@@ -22,8 +24,9 @@ Dependencies point one way, from the top of this graph to the bottom:
 
 ```text
 sqlbuild-python
-  -> sqlbuild-rules -> sqlbuild-analysis -> sqlbuild-model-config -> sqlbuild-scopes
-  -> sqlbuild-discovery -> sqlbuild-config -> sqlbuild-sqltext -> sqlbuild-core
+  -> sqlbuild-rules -> sqlbuild-analysis -> sqlbuild-model-config -> sqlbuild-render
+  -> sqlbuild-scopes -> sqlbuild-discovery -> sqlbuild-config -> sqlbuild-sqltext -> sqlbuild-cache
+  -> sqlbuild-core
 ```
 
 `sqlbuild-config` and `sqlbuild-model-config` do not depend on the crates below them today, and

crates/README.md

@@ -0,0 +1,21 @@
+[package]
+name = "sqlbuild-cache"
+version.workspace = true
+description = "The shared native store and fingerprinting scheme for SQLBuild's compiler caches."
+edition.workspace = true
+license.workspace = true
+publish.workspace = true
+rust-version.workspace = true
+
+[dependencies]
+blake3.workspace = true
+rayon.workspace = true
+sha2.workspace = true
+tempfile.workspace = true
+walkdir.workspace = true
+
+[dev-dependencies]
+tempfile.workspace = true
+
+[lints]
+workspace = true

crates/sqlbuild-cache/Cargo.toml

@@ -0,0 +1 @@
+pub(crate) mod walk;

crates/sqlbuild-cache/src/digest/_helpers/mod.rs

@@ -0,0 +1,72 @@
+//! List the project files a fingerprint covers, in a stable order.
+
+use std::collections::HashSet;
+use std::path::{Path, PathBuf};
+use walkdir::{DirEntry, WalkDir};
+
+use crate::digest::constants::{
+    EXCLUDED_DIRECTORIES, EXCLUDED_ROOT_DIRECTORIES, PRESENCE_ONLY_FILE_SUFFIXES,
+};
+use crate::digest::errors::FingerprintError;
+
+/// One covered file: its project-relative path, location and whether only presence counts.
+pub(crate) struct CoveredFile {
+    pub(crate) relative_path: String,
+    pub(crate) path: PathBuf,
+    pub(crate) presence_only: bool,
+}
+
+/// Every file below `root` outside the excluded folders and `excluded_files`, links followed.
+pub(crate) fn covered_files(
+    root: &Path,
+    excluded_files: &HashSet<String>,
+) -> Result<Vec<CoveredFile>, FingerprintError> {
+    let mut files: Vec<CoveredFile> = Vec::new();
+    let walk = WalkDir::new(root)
+        .follow_links(true)
+        .sort_by_file_name()
+        .into_iter()
+        .filter_entry(|entry| !excluded_directory(entry));
+    for entry in walk {
+        let entry: DirEntry = entry.map_err(FingerprintError::Walk)?;
+        if !entry.file_type().is_file() {
+            continue;
+        }
+        let relative_path: String = relative_text(root, entry.path())?;
+        if excluded_files.contains(&relative_path) {
+            continue;
+        }
+        let presence_only: bool = PRESENCE_ONLY_FILE_SUFFIXES
+            .iter()
+            .any(|suffix| relative_path.ends_with(suffix));
+        files.push(CoveredFile {
+            relative_path,
+            path: entry.into_path(),
+            presence_only,
+        });
+    }
+    Ok(files)
+}
+
+fn excluded_directory(entry: &DirEntry) -> bool {
+    if !entry.file_type().is_dir() || entry.depth() == 0 {
+        return false;
+    }
+    let Some(name) = entry.file_name().to_str() else {
+        return false;
+    };
+    EXCLUDED_DIRECTORIES.contains(&name)
+        || (entry.depth() == 1 && EXCLUDED_ROOT_DIRECTORIES.contains(&name))
+}
+
+fn relative_text(root: &Path, path: &Path) -> Result<String, FingerprintError> {
+    let relative: &Path = path
+        .strip_prefix(root)
+        .map_err(|_| FingerprintError::OutsideRoot(path.to_path_buf()))?;
+    let non_utf8 = || FingerprintError::NonUtf8Path(path.to_path_buf());
+    let parts: Vec<&str> = relative
+        .components()
+        .map(|component| component.as_os_str().to_str().ok_or_else(non_utf8))
+        .collect::<Result<Vec<&str>, FingerprintError>>()?;
+    Ok(parts.join("/"))
+}

crates/sqlbuild-cache/src/digest/_helpers/walk.rs

@@ -0,0 +1,32 @@
+//! What the project fingerprint walk skips; the same folders project reuse leaves out.
+
+/// Folders skipped only directly below the project root.
+pub const EXCLUDED_ROOT_DIRECTORIES: &[&str] = &[
+    "logs",
+    "target",
+    "venv",
+    ".cache",
+    ".fensu",
+    ".hg",
+    ".idea",
+    ".mypy_cache",
+    ".nox",
+    ".pytest_cache",
+    ".ruff_cache",
+    ".sqlbuild",
+    ".svn",
+    ".tox",
+    ".venv",
+    ".vscode",
+];
+
+/// Folders skipped at any depth.
+pub const EXCLUDED_DIRECTORIES: &[&str] = &[".git", "__pycache__"];
+
+/// Files that count only by presence because every build rewrites them.
+pub const PRESENCE_ONLY_FILE_SUFFIXES: &[&str] = &[".duckdb", ".duckdb.wal"];
+
+/// Digest tag of a file whose bytes count.
+pub const CONTENT_TAG: &[u8] = b"content";
+/// Digest tag of a file that counts only by presence.
+pub const PRESENCE_TAG: &[u8] = b"presence";

crates/sqlbuild-cache/src/digest/constants.rs

@@ -0,0 +1,25 @@
+//! Why a project fingerprint could not cover every file.
+
+use std::fmt;
+
+/// A file the fingerprint must cover could not be listed or read.
+#[derive(Debug)]
+pub enum FingerprintError {
+    Walk(walkdir::Error),
+    NonUtf8Path(std::path::PathBuf),
+    OutsideRoot(std::path::PathBuf),
+    Read(std::path::PathBuf, std::io::Error),
+}
+
+impl fmt::Display for FingerprintError {
+    fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
+        match self {
+            Self::Walk(error) => write!(formatter, "cannot list project files: {error}"),
+            Self::NonUtf8Path(path) => write!(formatter, "path is not UTF-8: {}", path.display()),
+            Self::OutsideRoot(path) => {
+                write!(formatter, "path is outside the project: {}", path.display())
+            }
+            Self::Read(path, error) => write!(formatter, "cannot read {}: {error}", path.display()),
+        }
+    }
+}