refactor(native): delete the Python discovery implementation (#1007)

* refactor(native): read bounded discovery entry points through native discovery

Scope inspection, selected contract discovery, model description inputs
and SQL test fixture formatting now read model, SQL test, scenario and
source files and the declaration layout through the native discovery
instead of the Python discovery helpers. Native discovery gains model
name and test path selection, per-file fault reporting, in-memory SQL
test and scenario parsing, and declaration scans isolated to one kind,
so these callers keep their exact outcomes. The Python engine's
discovery stage is unchanged and remains the differential oracle.

* feat(native): replace native discovery deferrals with native errors

Model, SQL test, scenario, source and schema files and the declaration
layout are now discovered natively under both compiler engines, with no
Python fallback:

- Unreadable files raise Python's own read errors natively: the exact
  UnicodeDecodeError for invalid UTF-8 and OSError for refused reads,
  with Windows error codes mapped the way Python maps them.
- Invalid YAML in sources files (D006) and seed declarations or
  schema.yml (D005) names the file, line and column. YAML outside the
  supported plain subset (tabs, directives, Python-only tags, recursive
  aliases and similar) fails with the same codes and a hint.
- A model, SQL test, scenario, source or schema file whose path is not
  UTF-8 fails with D016, showing invalid bytes as \xNN escapes; callers
  that tolerate bad files report it per file. Other files and directories
  with such names are ignored and declaration files are read by their
  real names, as before; every name keeps its own identity even when
  names share a lossy spelling. A non-UTF-8 project directory fails.
- Native discovery runs on Windows: paths print with backslashes, sort
  case-insensitively as Python's Windows paths do, and junctions are
  walked like directories. A Windows CI job covers it.
- An unknown Python or Unicode version fails with D017 naming the
  supported Python versions.

The discovery fact cache, which only served the Python fallbacks, is
removed.

* refactor(native): delete the Python discovery implementation

Model, SQL test, scenario, source and schema file discovery and the
declaration layout scan now exist only in native discovery, for both
compiler engines. Delete the Python walkers, parsers, YAML loaders and
the discovery fact cache they used, keep the Python semantic layers that
build discovered objects from native results, and point the remaining
behaviour tests at native discovery. Parity oracles that compared the
two implementations are removed with the Python side.

* build: support Python 3.12 to 3.14 only

Native discovery ships Unicode and text tables for Python 3.12, 3.13 and
3.14 and refuses other versions, so declare that range in
requires-python and the lockfile, and say "Python 3.12–3.14" in the
quickstart and contributing guide.

* fix(native): show Windows lone surrogates as units and test Windows raw names

A raw-segment test built its names from POSIX byte hex, which Windows
decodes as UTF-16 units, so it failed there. The product round-trips
Windows names correctly; only the display was off: a lone surrogate in
U+DC80-U+DCFF was shown as a \xNN byte. Windows now always shows lone
surrogates as \uXXXX, in Rust and in Python's own messages.

Decoding and display take the platform as a parameter so Linux tests
cover both the POSIX and the Windows branches, and a Windows-only test
walks real files whose names hold lone surrogates.

* fix(cli): write CLI output as UTF-8 under legacy code pages

When stdout or stderr was opened with a non-UTF-8 encoding, such as the
Windows cp1252 code page used for pipes and redirects, sqb crashed with
E001 on the first non-ASCII character of its summary output.

The CLI entry point now reconfigures any non-UTF-8 standard stream to
UTF-8 with backslashreplace, so output never fails to encode and JSON
output stays valid. Library use and the rule-engine host process are
unaffected.

Author: kvlonge

.github/workflows/verify.yml

@@ -335,14 +335,61 @@ jobs:
           uv run --no-sync pytest
           tests/integration/src/sqlbuild/compiler/test_native_python_compat_oracle.py
           -n auto --dist loadfile
-      - name: Compare python with the shipped native default on fixtures, examples, the failure corpus and a few seeds
+      - name: Compare python with native-preview on fixtures, examples, the failure corpus and a few seeds
         env:
           PYTHONUNBUFFERED: "1"
         run: >-
           uv run --no-sync python -m scripts.run_compiler_differential
-          --engines python native
+          --engines python native-preview
           --corpus fixtures examples failures seeds --seeds 4 --jobs 4
 
+  native_discovery_windows:
+    name: Native discovery on Windows
+    needs: [changes, lockfile]
+    if: needs.changes.outputs.compiler_differential == 'true'
+    runs-on: windows-2025
+    timeout-minutes: 40
+    defaults:
+      run:
+        shell: bash
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+        with:
+          version: "0.12.3"
+          enable-cache: true
+          cache-dependency-glob: |
+            uv.lock
+            Cargo.lock
+      - uses: dtolnay/rust-toolchain@master
+        with:
+          toolchain: stable
+      - name: Test the native discovery crate (walk, junctions, ordering, read errors)
+        run: cargo test --locked -p sqlbuild-discovery
+      - name: Install Python
+        run: uv python install 3.12
+      - name: Install SQLBuild and pytest
+        run: |
+          uv sync --locked --no-dev --python 3.12
+          uv pip install "pytest>=8.4.2" "pytest-xdist>=3.8.0"
+      - name: Compare native text, YAML and TOML readers with this Python
+        run: >-
+          uv run --no-sync pytest
+          tests/integration/src/sqlbuild/compiler/test_native_python_compat_oracle.py
+          tests/integration/src/sqlbuild/compiler/discovery/test_native_yaml_oracle.py
+          tests/integration/src/sqlbuild/compiler/discovery/test_native_toml_oracle.py
+          -n auto --dist loadfile
+      - name: Compile and build playground projects
+        run: |
+          project="${RUNNER_TEMP}/waffle-shop"
+          uv run --no-sync sqb playground "${project}"
+          uv run --no-sync sqb --project-dir "${project}" --no-color compile
+          SQLBUILD_COMPILER_ENGINE=python uv run --no-sync sqb --project-dir "${project}" --no-color compile
+          uv run --no-sync sqb --project-dir "${project}" --no-color build
+          nodes="${RUNNER_TEMP}/python-nodes"
+          uv run --no-sync sqb playground --template python_nodes "${nodes}"
+          uv run --no-sync sqb --project-dir "${nodes}" --no-color compile
+
   format_performance:
     name: Formatter performance guards
     needs: [changes, lockfile]
@@ -762,6 +809,7 @@ jobs:
         e2e_shards,
         compiler_differential,
         compiler_differential_python_versions,
+        native_discovery_windows,
         format_performance,
         performance,
         cold_compile_performance,
@@ -783,6 +831,7 @@ jobs:
           E2E_SHARDS_RESULT: ${{ needs.e2e_shards.result }}
           COMPILER_DIFFERENTIAL_RESULT: ${{ needs.compiler_differential.result }}
           COMPILER_DIFFERENTIAL_PYTHON_VERSIONS_RESULT: ${{ needs.compiler_differential_python_versions.result }}
+          NATIVE_DISCOVERY_WINDOWS_RESULT: ${{ needs.native_discovery_windows.result }}
           FORMAT_PERFORMANCE_RESULT: ${{ needs.format_performance.result }}
           PERFORMANCE_RESULT: ${{ needs.performance.result }}
           COLD_COMPILE_PERFORMANCE_RESULT: ${{ needs.cold_compile_performance.result }}
@@ -806,6 +855,7 @@ jobs:
             "${E2E_SHARDS_RESULT}" \
             "${COMPILER_DIFFERENTIAL_RESULT}" \
             "${COMPILER_DIFFERENTIAL_PYTHON_VERSIONS_RESULT}" \
+            "${NATIVE_DISCOVERY_WINDOWS_RESULT}" \
             "${FORMAT_PERFORMANCE_RESULT}" \
             "${PERFORMANCE_RESULT}" \
             "${COLD_COMPILE_PERFORMANCE_RESULT}" \

CONTRIBUTING.md

@@ -4,7 +4,7 @@ Thanks for helping improve SQLBuild. Keep changes focused, tested, and easy to r
 
 ## Setup
 
-SQLBuild requires Python 3.12 or newer.
+SQLBuild supports Python 3.12, 3.13 and 3.14.
 
 Install dependencies with:
 

Cargo.lock

@@ -40,6 +40,8 @@ rust-version = "1.88"
 [workspace.dependencies]
 fensu-policy = "=0.22.0"
 globset = "0.4"
+# Windows only: tells NTFS junctions, which Python walks as directories, from symbolic links.
+junction = { version = "2.1.0", default-features = false }
 # Python loads the extension with dlopen, so mimalloc must not use initial-exec TLS.
 mimalloc = { version = "0.1", default-features = false, features = ["local_dynamic_tls"] }
 proc-macro2 = { version = "1", features = ["span-locations"] }

Cargo.toml

@@ -8,8 +8,8 @@ that can be tested and benchmarked with `cargo test` alone.
 |---|---|
 | `sqlbuild-core` | Constants shared by several crates, the panic boundary that turns an unwinding native failure into a named compiler error, passive identity, diagnostic and SQL value types, text positions, and a JSON emitter whose output matches Python's `json.dumps` and `orjson` byte for byte. |
 | `sqlbuild-sqltext` | Lexical SQL text without polyglot: comment, quote and parenthesis scanning, the rules quote policy, model header tokenization and matching, static variable substitution and static reference extraction. |
-| `sqlbuild-config` | Configuration file reading without Python: `tomllib`-compatible TOML, PyYAML `safe_load`-compatible YAML 1.1, and typed project and local config readers for the fields discovery needs. Any error means "defer to Python", which re-parses for exact messages. |
-| `sqlbuild-discovery` | Native project discovery: the shared directory walk with Python's glob and sort semantics, file reading with Python's newline handling, and parsing of authored files with Python's exact discovery messages. Results are plain data the Python discovery facade materialises; anything it cannot reproduce exactly defers to Python. |
+| `sqlbuild-config` | Configuration file reading without Python: `tomllib`-compatible TOML, PyYAML `safe_load`-compatible YAML 1.1, and typed project and local config readers for the fields discovery needs. Errors carry their kind and position; YAML outside the supported subset is an `Unsupported` error. |
+| `sqlbuild-discovery` | Native project discovery and the only implementation of the model, SQL test, scenario, source and schema file collections and the declaration layout: the shared directory walk with Python's glob and sort semantics, file reading with Python's newline handling and read errors, and parsing of authored files with Python's exact discovery messages. Results are plain data the Python discovery facade materialises. |
 | `sqlbuild-scopes` | Declaration scopes: the scope index with Python's record orders and diagnostics, declaration visibility, relationship grants, the scope lookup groups, and the dialect-aware scan of SQL tests and scenarios for their expected models. Anything it cannot reproduce exactly defers to Python. |
 | `sqlbuild-model-config` | Model configuration: MODEL header columns and audits, `${...}` template expansion with its environment and context reads, and the template and macro presence scans over authored config values, all read through a trait over the caller's values. Anything it cannot reproduce exactly defers to Python, which also raises every model config error. |
 | `sqlbuild-analysis` | SQL analysis over polyglot: SQL tokens, query analysis, the binding catalog, semantic validation and usage, column references, and SQL-test extraction, planning and rendering. |

crates/README.md

@@ -15,7 +15,7 @@ pub enum ConfigErrorKind {
     InvalidField,
 }
 
-/// A configuration failure; the Python reader re-parses the file to report its exact message.
+/// A configuration failure, at its one-based line and column where known.
 #[derive(Clone, Debug, PartialEq, Eq)]
 pub struct ConfigError {
     pub kind: ConfigErrorKind,
@@ -34,4 +34,16 @@ impl ConfigError {
             column: None,
         }
     }
+
+    /// This error at a one-based line and column, unless it already has a position.
+    pub fn at(self, line: usize, column: usize) -> Self {
+        if self.line.is_some() {
+            return self;
+        }
+        Self {
+            line: Some(line),
+            column: Some(column),
+            ..self
+        }
+    }
 }

crates/sqlbuild-config/src/errors.rs

@@ -64,10 +64,15 @@ fn syntax_error(error: &ScanError) -> ConfigError {
         kind: ConfigErrorKind::Syntax,
         message: error.info().to_owned(),
         line: Some(error.marker().line()),
-        column: Some(error.marker().col()),
+        column: Some(error.marker().col() + 1),
     }
 }
 
+/// The one-based line and column where `span` starts.
+fn start_position(span: Span) -> (usize, usize) {
+    (span.start.line(), span.start.col() + 1)
+}
+
 fn full_tag(tag: Option<&Cow<'_, Tag>>) -> Option<String> {
     tag.map(|tag| format!("{}{}", tag.handle, tag.suffix))
 }
@@ -92,8 +97,12 @@ impl<'input> Composer<'input> {
         }
     }
 
-    fn push(&mut self, tag: String, content: NodeContent) -> usize {
-        self.document.nodes.push(Node { tag, content });
+    fn push(&mut self, tag: String, content: NodeContent, span: Span) -> usize {
+        self.document.nodes.push(Node {
+            tag,
+            content,
+            position: start_position(span),
+        });
         self.document.nodes.len() - 1
     }
 
@@ -103,7 +112,7 @@ impl<'input> Composer<'input> {
             return Ok(());
         }
         let name = anchor_name(&self.chars, &self.code_ends, span.start.index())
-            .ok_or_else(|| unsupported("anchor name could not be recovered"))?;
+            .ok_or_else(|| unsupported("this anchor"))?;
         if !is_python_anchor_name(&name) {
             return Err(ConfigError::new(
                 ConfigErrorKind::Syntax,
@@ -130,7 +139,7 @@ impl<'input> Composer<'input> {
 
     fn alias(&self, anchor_id: usize) -> Result<usize, ConfigError> {
         if self.open_anchors.contains(&anchor_id) {
-            return Err(unsupported("recursive aliases are left to Python"));
+            return Err(unsupported("recursive aliases"));
         }
         self.anchors
             .get(&anchor_id)
@@ -212,7 +221,7 @@ impl<'input> Composer<'input> {
             && matches!(self.char_at(colon + 1), Some(',' | ']' | '}'));
         if placement.flow && adjacent_value {
             return Err(unsupported(
-                "LibYAML rejects ':' directly before a flow indicator",
+                "':' directly before ',', ']' or '}' in a flow collection",
             ));
         }
         let explicit_key = explicit_key_before(&self.chars, &self.code_ends, start);
@@ -281,7 +290,7 @@ impl<'input> Composer<'input> {
         span: Span,
         placement: Placement,
     ) -> Result<String, ConfigError> {
-        let deferred = || unsupported("this block scalar is left to Python");
+        let deferred = || unsupported("this literal or folded block scalar layout");
         if placement.flow || placement.key {
             return Err(deferred());
         }
@@ -304,14 +313,26 @@ impl<'input> Composer<'input> {
             .ok_or_else(deferred)
     }
 
+    /// Compose one node; an error without a position is placed at the node's start.
     fn compose_node(
         &mut self,
         event: Event<'input>,
         span: Span,
         placement: Placement,
+    ) -> Result<usize, ConfigError> {
+        let (line, column) = start_position(span);
+        self.compose_node_at(event, span, placement)
+            .map_err(|error| error.at(line, column))
+    }
+
+    fn compose_node_at(
+        &mut self,
+        event: Event<'input>,
+        span: Span,
+        placement: Placement,
     ) -> Result<usize, ConfigError> {
         if placement.depth > MAX_NESTING_DEPTH {
-            return Err(unsupported("nesting is deeper than the native limit"));
+            return Err(unsupported("nesting deeper than 256 levels"));
         }
         match event {
             Event::Alias(anchor_id) => self.alias(anchor_id),
@@ -321,7 +342,7 @@ impl<'input> Composer<'input> {
                     && value.contains(FLOW_KEY_INDICATOR)
                 {
                     return Err(unsupported(
-                        "PyYAML ends a flow scalar at '?' and LibYAML does not",
+                        "'?' inside a plain scalar in a flow collection",
                     ));
                 }
                 if style == ScalarStyle::Plain
@@ -340,9 +361,7 @@ impl<'input> Composer<'input> {
                     ));
                 }
                 if placement.key && self.is_long_implicit_key(span) {
-                    return Err(unsupported(
-                        "implicit keys near PyYAML's 1024-character limit are left to Python",
-                    ));
+                    return Err(unsupported("implicit keys longer than 1000 characters"));
                 }
                 if placement.key && placement.flow && self.spans_lines(span) {
                     return Err(ConfigError::new(
@@ -361,13 +380,11 @@ impl<'input> Composer<'input> {
                 };
                 let non_specific = full_tag(tag.as_ref()).as_deref() == Some(NON_SPECIFIC_TAG);
                 if non_specific && value.is_empty() && style == ScalarStyle::Plain {
-                    return Err(unsupported(
-                        "LibYAML and PyYAML differ on an empty '!' scalar",
-                    ));
+                    return Err(unsupported("an empty scalar tagged '!'"));
                 }
                 self.open_anchor(anchor_id, span)?;
                 let tag = scalar_tag(&value, style, full_tag(tag.as_ref()));
-                let node = self.push(tag, NodeContent::Scalar(value.into_owned()));
+                let node = self.push(tag, NodeContent::Scalar(value.into_owned()), span);
                 Ok(self.close_anchor(anchor_id, node))
             }
             Event::SequenceStart(anchor_id, tag) => {
@@ -398,7 +415,7 @@ impl<'input> Composer<'input> {
             items.push(self.compose_node(event, item_span, item_placement)?);
         }
         let tag = collection_tag(full_tag(tag.as_ref()), SEQ_TAG);
-        let node = self.push(tag, NodeContent::Sequence(items));
+        let node = self.push(tag, NodeContent::Sequence(items), span);
         Ok(self.close_anchor(anchor_id, node))
     }
 
@@ -426,7 +443,7 @@ impl<'input> Composer<'input> {
             entries.push((key, value));
         }
         let tag = collection_tag(full_tag(tag.as_ref()), MAP_TAG);
-        let node = self.push(tag, NodeContent::Mapping(entries));
+        let node = self.push(tag, NodeContent::Mapping(entries), span);
         Ok(self.close_anchor(anchor_id, node))
     }