refactor(native): move discovery, scopes and the model config build into plain-Rust crates (#1182)

* refactor(native): move discovery's header keys and header changes into sqlbuild-discovery

The header key sets each statement accepts and the options built from
them now live in `sqlbuild-discovery` (`constants`, `model_file_options`,
`declaration_file_options`, `sql_test_file_options`), and the bindings
call them.

The delta compile's header comparison moves too: `header_changes` digests
each header key, decides how an edit changed a model header and whether a
custom rule reads a changed key. The bindings keep only the recording of
digests into the compile image. The comparison gains unit tests.

* refactor(native): move plain scope logic from the bindings into sqlbuild-scopes

The conversions from a cached full index to the report and projection
views (`cached_report_records`, `cached_projection_index` and their
parts), the scope path and qualified identity messages, the `as_posix`
path text helpers, and the merge of new runtime usages
(`merged_usages`, with `UsageKey`) now live in `sqlbuild-scopes`. The
bindings read Python objects into those types and convert the results
back, nothing more.

`distinct_positions` and `merged_dependencies` become helpers of the
usage merge, their only remaining caller.

* feat(native): add a typed model config value and layer described models with it

`sqlbuild-model-config` gains `config_values`: `ConfigNode<H>` holds the
builtin Python values config is made of (None, bool, int, float, str,
list, tuple, dict) as Rust data, and every other value as an opaque host
value carrying its handle and the facts config logic reads of it, taken
once with the same rules as before. It implements `AuthoredNode`, so the
existing config scans read it unchanged, and `layered_config` merges one
layer over another as `_merged_with_tag_union` does.

The bindings convert Python values into nodes and project nodes back;
opaque values project to the very same object. Externally described
models now layer defaults, path defaults and headers through the typed
merge, the first consumer.

* refactor(native): build model configs as typed values in sqlbuild-model-config

The model config build (layering with tag, row-diff and tolerance unions,
hook checks, the three model template passes, the target namespace and
its templates, storage key removal, the no-macros check and header keys)
now runs in `sqlbuild-model-config::model_config_build` over
`ConfigNode`s. It asks its caller only for environment reads, the
project-var text of non-string values and the exact error of a value
that is not UTF-8 text, through `ConfigBuildHost`.

`NativeModelConfigBuilder` converts the project inputs once per compile
and each header once per model, and projects the built values back.
Opaque values are carried as their own Python objects. Which header
entries are still the header's own is tracked by provenance instead of
object identity, so native header facts apply exactly where they did.

Output is unchanged: the compiler differential passes, and a new
integration test pins values, types, order, header keys and errors for
builtin, subclassed, str-enum, date and opaque header values against the
previous build. The build runs about 2% fewer instructions on a
600-model templated project.

* refactor(native): read planner config values as typed nodes

`sqlbuild-model-config` gains the typed config reads (`checked_text`,
`checked_flag`, `checked_integer`, `checked_texts`,
`checked_cursor_bound`) with the type rules and error contents of the
`get_config_*` readers; the Python-facing readers now check through
them and still return the caller's own objects.

A config record keeps its values as typed nodes, converted from
`values` on first read and cached on the record. The planner's phase
objects, change detection, display plan, column migrations, lineage and
SQL test cursor window read through it instead of the Python dict.

* fix(native): keep header-owned config values as the header's objects

Since configs are built as typed nodes, every built container was
projected as a fresh Python object. The model loop reuses natively
parsed header columns only when `values["columns"]` is the header's own
object, so any model with `columns` beside another config key fell back
to parsing its columns again in Python.

The build now reports which header keys still hold the header's own
values (the provenance it already tracks, with set-aside hooks
included), and the projection uses the header's objects for them, so
identity holds exactly where it did before.

Stage counters record whether a model's header columns were reused or
parsed again, and an integration test pins the native path for models
with `columns` beside `materialized`, `tags` or a description.

* fix(native): read config value text through Python when only Python prints it exactly

Opaque config values were read eagerly at conversion: `str()` twice,
`repr()`, a date check and three hook-entry checks, with a raising
`__repr__` read as empty. Only cheap facts (kind, text, numbers) are now
read when a value is converted. Date checks and display text are asked
of the value's handle when a scan needs them, and the build asks its
host for `str()` and hook-entry checks, so their errors raise as before.

Text is produced in Rust only where it is exact: `None`, booleans,
integers, floats and a string's own text. A string's repr, containers
and opaque values use Python's `str()`/`repr()` of the projected object
for template operands and tag/row-diff unions. That replaces an
approximate Rust repr that printed some format, private-use and
unassigned characters unescaped.

Hook-template context values read only `database`, `schema` and `alias`
instead of converting the whole values dict.

Integration tests pin container text, item errors in template operands
and unions, and that the build never reads an opaque value's text.

* perf(native): carry header containers the config build never looks into whole

Every header value was converted into config nodes, then cloned through
layering and expansion, even large `columns` trees the build only
passes through. A header container whose native parse found no template
or macro, under a key the build never reads item by item (anything but
tags, row-diff, hook and storage keys), is now carried as the header's
own object. Dense compiles return to the instruction count they had
before the typed build.

Because such containers stay the header's own objects, models whose
other config values hold templates also reuse their natively parsed
columns now; an integration case pins it.

* perf(native): convert typed config values key by key on first read

The first typed read of a config record converted every value in
`config.values` into typed nodes, just to read one scalar key such as
`incremental_mode`. A config record now keeps a per-key cache: a value
is converted the first time its key is read, through the same
`values.get(key)`, and kept for later reads. Checked reads keep their
type rules and errors.

Lineage's microbatch check compares the mode text as Python's `==`
does again, so a string that cannot be encoded reads as not-microbatch
instead of raising.

A stage counter records each typed conversion, and an integration test
pins that keys no planner reader reads are never converted.

Author: kvlonge

Cargo.lock

@@ -9,6 +9,7 @@ rust-version.workspace = true
 
 [dependencies]
 rayon.workspace = true
+sqlbuild-cache.workspace = true
 sqlbuild-config.workspace = true
 sqlbuild-core.workspace = true
 sqlbuild-model-config.workspace = true

crates/sqlbuild-discovery/Cargo.toml

@@ -4,5 +4,6 @@ pub(crate) mod locations;
 pub(crate) mod pool;
 pub(crate) mod reading;
 pub(crate) mod scoped_paths;
+pub(crate) mod sorted_keys;
 pub(crate) mod statement_headers;
 pub(crate) mod suggestions;

crates/sqlbuild-discovery/src/_helpers/mod.rs

@@ -1,4 +1,4 @@
-//! Header key sets as the sorted lists native discovery options take.
+//! Header key sets as the sorted lists the discovery options take.
 
 /// A header key set as owned keys, sorted by code point as Python sorts them.
 pub(crate) fn sorted_keys(keys: &[&str]) -> Vec<String> {

…ndings/_helpers/discovery/sorted_keys.rs …ld-discovery/src/_helpers/sorted_keys.rscrates/sqlbuild-python/src/bindings/_helpers/discovery/sorted_keys.rs renamed to crates/sqlbuild-discovery/src/_helpers/sorted_keys.rs crates/sqlbuild-python/src/bindings/_helpers/discovery/sorted_keys.rs renamed to crates/sqlbuild-discovery/src/_helpers/sorted_keys.rs

@@ -47,3 +47,124 @@ pub const AUDIT_ROLE_DIRECTORY: &str = "audits";
 pub const LOCAL_AUDIT_ROLE_DIRECTORY: &str = "_audits";
 /// The singular audit directory below an audit role.
 pub const SINGULAR_AUDIT_DIRECTORY: &str = "singular";
+/// Header keys a `MODEL` statement accepts, sorted by code point.
+pub const SQL_MODEL_HEADER_KEYS: [&str; 73] = [
+    "alias",
+    "append_cursor_inclusive",
+    "audit_factories",
+    "audits",
+    "batch_concurrency",
+    "batch_size",
+    "check_columns",
+    "columns",
+    "config",
+    "constants",
+    "contract",
+    "cursor",
+    "cursor_end",
+    "cursor_future_action",
+    "cursor_future_max_distance",
+    "cursor_grain",
+    "cursor_inputs",
+    "cursor_start",
+    "cursor_start_max_action",
+    "cursor_start_max_ahead",
+    "cursor_type",
+    "cursor_watermark_mode",
+    "database",
+    "description",
+    "dynamic_columns",
+    "enabled",
+    "engine",
+    "enums",
+    "full_refresh",
+    "historical_input",
+    "incremental_mode",
+    "incremental_strategy",
+    "initial_valid_from",
+    "invalidate_hard_deletes",
+    "lookback",
+    "materialized",
+    "max_duration",
+    "max_microbatches",
+    "merge_exclude_columns",
+    "microbatch_limit",
+    "microbatch_strategy",
+    "migrate_force",
+    "migrate_from",
+    "model_schema",
+    "observed_at",
+    "old_name_view",
+    "on_schema_change",
+    "order_by",
+    "partition_by",
+    "placeholders",
+    "post_hooks",
+    "pre_hooks",
+    "primary_key",
+    "replay_on_change",
+    "row_diff_exclude_columns",
+    "row_diff_sample_rows",
+    "row_diff_sample_seed",
+    "row_diff_tolerances",
+    "schema",
+    "snapshot_full_refresh",
+    "snapshot_schema_change",
+    "snapshot_strategy",
+    "sql_analysis",
+    "sql_validation",
+    "table_settings",
+    "table_type",
+    "tags",
+    "time_travel_retention",
+    "unaccounted_partition_policy",
+    "unique_key",
+    "updated_at",
+    "valid_from_column",
+    "valid_to_column",
+];
+/// `MODEL` header keys that were removed and fail with a dedicated message.
+pub const REMOVED_SQL_MODEL_HEADER_KEYS: [&str; 1] = ["run_despite_unchanged"];
+/// Header keys a `FUNCTION` statement accepts.
+pub const SQL_FUNCTION_HEADER_KEYS: [&str; 6] = [
+    "arguments",
+    "database",
+    "description",
+    "returns",
+    "schema",
+    "tags",
+];
+/// Header keys an `AUDIT` statement accepts.
+pub const SQL_AUDIT_HEADER_KEYS: [&str; 11] = [
+    "always_run",
+    "evaluation",
+    "minimum_samples",
+    "name",
+    "run_scope",
+    "sample_count",
+    "sample_unit",
+    "severity",
+    "sql_analysis",
+    "thresholds",
+    "value",
+];
+/// Header keys a `HOOK` statement accepts.
+pub const SQL_HOOK_HEADER_KEYS: [&str; 1] = ["description"];
+/// Header keys a `TEST` statement accepts.
+pub const SQL_TEST_HEADER_KEYS: [&str; 8] = [
+    "cases",
+    "cursor_end",
+    "cursor_start",
+    "mode",
+    "name",
+    "parameters",
+    "sql_analysis",
+    "timeout",
+];
+/// Header keys a `SCENARIO` statement accepts.
+pub const SQL_SCENARIO_HEADER_KEYS: [&str; 4] =
+    ["description", "expected_columns", "tags", "timeout"];
+/// `SCENARIO (expected_columns ...)` values: compare listed columns, or require every column.
+pub const SCENARIO_EXPECTED_COLUMNS_KEY: &str = "expected_columns";
+pub const SCENARIO_EXPECTED_COLUMNS_VALUES: [&str; 2] = ["listed", "all"];
+pub const SCENARIO_EXPECTED_COLUMNS_ALL: &str = "all";

crates/sqlbuild-discovery/src/constants.rs

@@ -0,0 +1,15 @@
+//! The options declaration discovery parses headers with.
+
+use crate::_helpers::sorted_keys::sorted_keys;
+use crate::constants::{SQL_AUDIT_HEADER_KEYS, SQL_FUNCTION_HEADER_KEYS, SQL_HOOK_HEADER_KEYS};
+use crate::declaration_files::models::DeclarationFileOptions;
+
+/// The header keys `FUNCTION`, `AUDIT` and `HOOK` statements accept.
+#[must_use]
+pub fn declaration_file_options() -> DeclarationFileOptions {
+    DeclarationFileOptions {
+        function_keys: sorted_keys(&SQL_FUNCTION_HEADER_KEYS),
+        audit_keys: sorted_keys(&SQL_AUDIT_HEADER_KEYS),
+        hook_keys: sorted_keys(&SQL_HOOK_HEADER_KEYS),
+    }
+}

crates/sqlbuild-discovery/src/declaration_files/main/declaration_file_options.rs

@@ -1,4 +1,5 @@
 pub(crate) mod bare_number_failure;
+pub mod declaration_file_options;
 pub mod discover_declaration_collection;
 pub mod parse_declaration_text;
 pub mod python_function_files;

crates/sqlbuild-discovery/src/declaration_files/main/mod.rs

@@ -0,0 +1,2 @@
+pub(crate) mod references;
+pub(crate) mod value_encoding;

crates/sqlbuild-discovery/src/header_changes/_helpers/mod.rs

@@ -0,0 +1,39 @@
+//! Whether an edited query still references only what the stored model read.
+
+use sqlbuild_sqltext::sql_references::main::extract_sql_references::extract_sql_references;
+use sqlbuild_sqltext::sql_references::models::ReferenceExtraction;
+use sqlbuild_sqltext::sql_scan::models::LexicalSyntax;
+
+use crate::header_changes::constants::{
+    LINE_COMMENT, MODEL_KIND, REF_CALL, SEED_CALL, SEED_KIND, SOURCE_CALL, SOURCE_KIND,
+};
+
+/// Whether every model, source and seed `query_sql` names was among the stored `upstream`.
+pub(crate) fn references_read(query_sql: &str, upstream: &[(String, String)]) -> bool {
+    let syntax: LexicalSyntax = LexicalSyntax {
+        line_comment_prefixes: vec![LINE_COMMENT.to_owned()],
+        ..LexicalSyntax::default()
+    };
+    let ReferenceExtraction::Extracted(scan) = extract_sql_references(query_sql, &syntax) else {
+        return false;
+    };
+    scan.references.iter().all(|reference| {
+        let kind: Option<&str> = match reference.kind {
+            REF_CALL => Some(MODEL_KIND),
+            SOURCE_CALL => Some(SOURCE_KIND),
+            SEED_CALL => Some(SEED_KIND),
+            _ => None,
+        };
+        match (kind, &reference.package) {
+            (Some(kind), None) => stored_upstream(upstream, kind, &reference.name),
+            _ => true,
+        }
+    })
+}
+
+/// Whether `upstream` holds the `kind` resource `name`.
+fn stored_upstream(upstream: &[(String, String)], kind: &str, name: &str) -> bool {
+    upstream
+        .iter()
+        .any(|(stored_kind, stored_name)| stored_kind == kind && stored_name == name)
+}

crates/sqlbuild-discovery/src/header_changes/_helpers/references.rs

@@ -0,0 +1,79 @@
+//! Header values as short digests: an unambiguous byte encoding, hashed and truncated.
+
+use sqlbuild_cache::digest::main::bytes_digest::bytes_digest;
+use sqlbuild_sqltext::compiler::models::AuthoredValue;
+
+use crate::header_changes::constants::{HEX, KEPT_DIGEST_BYTES};
+
+/// An unambiguous encoding of header values: a tag byte, then length-prefixed parts.
+#[derive(Default)]
+pub(crate) struct ValueEncoding {
+    bytes: Vec<u8>,
+}
+
+impl ValueEncoding {
+    fn text(&mut self, tag: u8, text: &str) {
+        self.bytes.push(tag);
+        self.bytes
+            .extend_from_slice(&(text.len() as u64).to_le_bytes());
+        self.bytes.extend_from_slice(text.as_bytes());
+    }
+
+    fn list(&mut self, tag: u8, items: &[AuthoredValue]) {
+        self.bytes.push(tag);
+        self.bytes
+            .extend_from_slice(&(items.len() as u64).to_le_bytes());
+        for item in items {
+            self.value(item);
+        }
+    }
+
+    fn pairs(&mut self, tag: u8, items: &[(String, AuthoredValue)]) {
+        self.bytes.push(tag);
+        self.bytes
+            .extend_from_slice(&(items.len() as u64).to_le_bytes());
+        for (key, item) in items {
+            self.text(b'k', key);
+            self.value(item);
+        }
+    }
+
+    pub(crate) fn value(&mut self, value: &AuthoredValue) {
+        match value {
+            AuthoredValue::Null => self.bytes.push(b'n'),
+            AuthoredValue::Boolean(flag) => self.bytes.extend_from_slice(&[b'b', u8::from(*flag)]),
+            AuthoredValue::BareWord(word) => self.text(b'w', word),
+            AuthoredValue::String(string) => self.text(b's', string),
+            AuthoredValue::List(items) => self.list(b'l', items),
+            AuthoredValue::Map(items) => self.pairs(b'm', items),
+            AuthoredValue::Set(items) => self.list(b'e', items),
+            AuthoredValue::Tuple(items) => self.list(b't', items),
+            AuthoredValue::TypedConstant(items) => self.pairs(b'c', items),
+            AuthoredValue::InlineSqlHook(sql) => self.text(b'i', sql),
+            AuthoredValue::NamedSqlHook(name, items) => {
+                self.text(b'h', name);
+                self.pairs(b'a', items);
+            }
+            AuthoredValue::PythonHook(name, items) => {
+                self.text(b'p', name);
+                self.pairs(b'a', items);
+            }
+        }
+    }
+}
+
+pub(crate) fn short_hex(digest: &[u8]) -> String {
+    let mut hex: String = String::with_capacity(KEPT_DIGEST_BYTES * 2);
+    for byte in &digest[..KEPT_DIGEST_BYTES] {
+        hex.push(char::from(HEX[usize::from(byte >> 4)]));
+        hex.push(char::from(HEX[usize::from(byte & 0x0f)]));
+    }
+    hex
+}
+
+/// The short digest of one header value.
+pub(crate) fn value_digest(value: &AuthoredValue) -> String {
+    let mut encoding: ValueEncoding = ValueEncoding::default();
+    encoding.value(value);
+    short_hex(&bytes_digest(&encoding.bytes))
+}