feat(native): extract model references natively behind the preview engine (#1008)

* feat(native): add the general SQL reference scan with table-function call arguments

Port Python's general reference extraction to sqlbuild-sqltext: every
reference kind including __table_fn with its call-argument count, under
each adapter's lexical rules, with Python's exact error messages for
argument counts, empty arguments, malformed names and unclosed text.
A non-ASCII character that decides the outcome, or an unsupported line
comment prefix, defers the text to Python.

* feat(native): extract model references natively behind the preview engine

Register reference extraction as a preview-tier native stage. Under
native-preview every SQL text goes through the native scan, for every
adapter's lexical rules; a deferral runs the Python scanner. A native
error stands only when Python raises the same message, which is
re-raised from Python; otherwise the compile fails with a
NativeStageMismatchError instead of silently falling back. The python
and native engines are unchanged.

The lexical-syntax mapping for native scanners is shared through
SqlLexicalSyntax.native_mapping and one binding input type.

* feat(native): compare reference extraction in the compiler differential

The failure corpus gains reference cases: wrong argument counts, an empty
argument, an expression name, unclosed calls, and table functions without
an argument list, with a single-quoted name or with an empty call
argument. A harness test compares a project with table functions and
references hidden in comments, strings and dollar quotes under both
engines.

Author: kvlonge

crates/README.md

@@ -7,7 +7,7 @@ that can be tested and benchmarked with `cargo test` alone.
 | Crate | Responsibility |
 |---|---|
 | `sqlbuild-core` | Constants shared by several crates, the panic boundary that turns an unwinding native failure into a named compiler error, passive identity, diagnostic and SQL value types, text positions, and a JSON emitter whose output matches Python's `json.dumps` and `orjson` byte for byte. |
-| `sqlbuild-sqltext` | Lexical SQL text without polyglot: comment, quote and parenthesis scanning, the rules quote policy, model header tokenization and matching, static variable substitution and static reference extraction. |
+| `sqlbuild-sqltext` | Lexical SQL text without polyglot: comment, quote and parenthesis scanning, the rules quote policy, model header tokenization and matching, static variable substitution, static reference extraction, and the general reference scan with table-function call arguments under each adapter's lexical rules. |
 | `sqlbuild-config` | Configuration file reading without Python: `tomllib`-compatible TOML, PyYAML `safe_load`-compatible YAML 1.1, and typed project and local config readers for the fields discovery needs. Errors carry their kind and position; YAML outside the supported subset is an `Unsupported` error. |
 | `sqlbuild-discovery` | Native project discovery and the only implementation of the model, SQL test, scenario, source and schema file collections and the declaration layout: the shared directory walk with Python's glob and sort semantics, file reading with Python's newline handling and read errors, and parsing of authored files with Python's exact discovery messages. Results are plain data the Python discovery facade materialises. |
 | `sqlbuild-scopes` | Declaration scopes: the scope index with Python's record orders and diagnostics, declaration visibility, relationship grants, the scope lookup groups, and the dialect-aware scan of SQL tests and scenarios for their expected models. Anything it cannot reproduce exactly defers to Python. |

crates/sqlbuild-python/src/bindings/_helpers/scopes/relationship_names.rs

@@ -1,39 +1,15 @@
 //! Expected-model relationship names of SQL tests and scenarios, scanned natively.
 
 use pyo3::prelude::{Bound, PyModule, PyModuleMethods, PyResult, Python};
-use pyo3::{FromPyObject, pyfunction, wrap_pyfunction};
+use pyo3::{pyfunction, wrap_pyfunction};
 use sqlbuild_scopes::relationship_names::main::expected_model_names::expected_model_names;
 use sqlbuild_scopes::relationship_names::models::ExpectedNames;
 use sqlbuild_sqltext::sql_scan::models::LexicalSyntax;
 
 use crate::bindings::_helpers::boundary::panics::compiler_error;
+use crate::bindings::_helpers::sqltext::lexical_syntax::LexicalSyntaxInput;
 use crate::bindings::types::CompilerDetach;
 
-/// One adapter's `SqlLexicalSyntax`, read from a Python mapping.
-#[derive(FromPyObject)]
-#[pyo3(from_item_all)]
-struct LexicalSyntaxInput {
-    backslash_escape_quotes: Vec<String>,
-    escape_string_prefix: bool,
-    raw_string_prefix: bool,
-    triple_quoted_strings: bool,
-    nested_block_comments: bool,
-    line_comment_prefixes: Vec<String>,
-}
-
-impl From<LexicalSyntaxInput> for LexicalSyntax {
-    fn from(input: LexicalSyntaxInput) -> Self {
-        Self {
-            backslash_escape_quotes: input.backslash_escape_quotes,
-            escape_string_prefix: input.escape_string_prefix,
-            raw_string_prefix: input.raw_string_prefix,
-            triple_quoted_strings: input.triple_quoted_strings,
-            nested_block_comments: input.nested_block_comments,
-            line_comment_prefixes: input.line_comment_prefixes,
-        }
-    }
-}
-
 /// Return each body's expected-model names, or `None` where Python must extract them.
 #[pyfunction]
 fn scope_expected_model_names(

crates/sqlbuild-python/src/bindings/_helpers/sqltext/lexical_syntax.rs

@@ -0,0 +1,29 @@
+//! One adapter's `SqlLexicalSyntax`, read from a Python mapping.
+
+use pyo3::FromPyObject;
+use sqlbuild_sqltext::sql_scan::models::LexicalSyntax;
+
+/// One adapter's `SqlLexicalSyntax`, read from a Python mapping.
+#[derive(FromPyObject)]
+#[pyo3(from_item_all)]
+pub(crate) struct LexicalSyntaxInput {
+    backslash_escape_quotes: Vec<String>,
+    escape_string_prefix: bool,
+    raw_string_prefix: bool,
+    triple_quoted_strings: bool,
+    nested_block_comments: bool,
+    line_comment_prefixes: Vec<String>,
+}
+
+impl From<LexicalSyntaxInput> for LexicalSyntax {
+    fn from(input: LexicalSyntaxInput) -> Self {
+        Self {
+            backslash_escape_quotes: input.backslash_escape_quotes,
+            escape_string_prefix: input.escape_string_prefix,
+            raw_string_prefix: input.raw_string_prefix,
+            triple_quoted_strings: input.triple_quoted_strings,
+            nested_block_comments: input.nested_block_comments,
+            line_comment_prefixes: input.line_comment_prefixes,
+        }
+    }
+}

crates/sqlbuild-python/src/bindings/_helpers/sqltext/mod.rs

@@ -1,3 +1,6 @@
 pub(crate) mod authored_values;
+pub(crate) mod lexical_syntax;
 pub(crate) mod model_headers;
+pub(crate) mod registration;
+pub(crate) mod sql_references;
 pub(crate) mod static_sql;

crates/sqlbuild-python/src/bindings/_helpers/sqltext/registration.rs

@@ -0,0 +1,12 @@
+//! Register the lexical SQL text functions and classes.
+
+use pyo3::prelude::{Bound, PyModule, PyResult};
+
+use crate::bindings::_helpers::sqltext::{model_headers, sql_references, static_sql};
+
+pub(crate) fn register(module: &Bound<'_, PyModule>) -> PyResult<()> {
+    model_headers::register(module)?;
+    static_sql::register(module)?;
+    sql_references::register(module)?;
+    Ok(())
+}

crates/sqlbuild-python/src/bindings/_helpers/sqltext/sql_references.rs

@@ -0,0 +1,59 @@
+//! Logical SQL references extracted natively under one adapter's lexical rules.
+
+use pyo3::prelude::{Bound, PyModule, PyModuleMethods, PyResult};
+use pyo3::{pyclass, pymethods};
+use sqlbuild_sqltext::sql_references::main::extract_sql_references::extract_sql_references;
+use sqlbuild_sqltext::sql_references::models::{ReferenceExtraction, SqlReference};
+use sqlbuild_sqltext::sql_scan::models::LexicalSyntax;
+
+use crate::bindings::_helpers::boundary::panics::compiler_guard;
+use crate::bindings::_helpers::sqltext::lexical_syntax::LexicalSyntaxInput;
+
+/// `(kind, name, package, call_argument_count)` for one reference.
+type ReferenceRow = (&'static str, String, Option<String>, Option<usize>);
+/// The references, or Python's error message; `None` defers the text to Python.
+type ExtractionRow = Option<(Option<Vec<ReferenceRow>>, Option<String>)>;
+
+/// Extracts references under one adapter's lexical rules, read once per syntax.
+#[pyclass(module = "sqlbuild._native", frozen)]
+pub(crate) struct SqlReferenceScanner {
+    syntax: LexicalSyntax,
+}
+
+#[pymethods]
+impl SqlReferenceScanner {
+    #[new]
+    fn new(syntax: LexicalSyntaxInput) -> Self {
+        Self {
+            syntax: syntax.into(),
+        }
+    }
+
+    /// Return `(references, None)`, `(None, message)` when Python raises, or `None` to defer.
+    fn extract(&self, sql: &str) -> PyResult<ExtractionRow> {
+        compiler_guard(|| {
+            Ok(match extract_sql_references(sql, &self.syntax) {
+                ReferenceExtraction::Extracted(references) => Some((
+                    Some(references.into_iter().map(reference_row).collect()),
+                    None,
+                )),
+                ReferenceExtraction::Failed(message) => Some((None, Some(message))),
+                ReferenceExtraction::Deferred => None,
+            })
+        })
+    }
+}
+
+fn reference_row(reference: SqlReference) -> ReferenceRow {
+    (
+        reference.kind,
+        reference.name,
+        reference.package,
+        reference.call_argument_count,
+    )
+}
+
+pub(crate) fn register(module: &Bound<'_, PyModule>) -> PyResult<()> {
+    module.add_class::<SqlReferenceScanner>()?;
+    Ok(())
+}

crates/sqlbuild-python/src/bindings/main/register.rs

@@ -24,8 +24,7 @@ pub fn _native(module: &Bound<'_, PyModule>) -> PyResult<()> {
     analysis::normalization::register(module)?;
     analysis::queries::register(module)?;
     analysis::sql_tests::register(module)?;
-    sqltext::model_headers::register(module)?;
-    sqltext::static_sql::register(module)?;
+    sqltext::registration::register(module)?;
     boundary::oracles::register(module)?;
     discovery::project_files::register(module)?;
     scopes::scope_index::register(module)?;

crates/sqlbuild-sqltext/src/lib.rs

@@ -4,4 +4,5 @@
 
 pub mod compiler;
 mod constants;
+pub mod sql_references;
 pub mod sql_scan;

crates/sqlbuild-sqltext/src/sql_references/_helpers/mod.rs

@@ -0,0 +1 @@
+pub(crate) mod scan;