refactor(native): split the native crate into a layered workspace (#983)

* refactor(native): rename the native crate to sqlbuild-python under a versioned workspace

The extension crate becomes the Python boundary crate of a Cargo workspace. The release
version moves to the workspace package so every native crate shares it, and maturin, uv
cache keys, release automation and fensu paths follow the new location.

* refactor(native): keep PyO3 classes and conversions in the bindings module

ProjectCatalog, CompactAnalysisJob, BindingPositions and ParsedRulesRequest become thin
Python classes in bindings that wrap plain native models, and catalog mapping requests are
converted from Python at the boundary. Method names, signatures, GIL release, panic
handling and exception types are unchanged, so later crates can stay free of PyO3.

* refactor(native): extract sqlbuild-core with shared constants and the panic boundary

The constants used by more than one future crate and the PyO3-free panic catcher move to a
dependency-free sqlbuild-core crate, with the panic tests. The Python crate keeps the named
exception and guards, and the build identity now hashes every workspace crate.

* refactor(native): extract sqlbuild-sqltext for lexical scanning and model headers

SQL scanning, the rules quote policy, model header tokenization and matching, static
variable substitution and static reference extraction move with their tests to a
polyglot-free sqlbuild-sqltext crate. Items other crates use become public; behaviour is
unchanged.

* refactor(native): extract sqlbuild-analysis for polyglot query analysis

Query analysis, the binding catalog, semantic validation and usage, column references,
SQL tokens and SQL-test extraction, planning and rendering move with their tests and the
function-name data to sqlbuild-analysis, the lowest crate that depends on polyglot. A
test-support feature exposes one test-only helper to dependent crates' tests, and the
workspace states the insertion-ordered JSON feature every crate already relied on.

* refactor(native): extract sqlbuild-rules for rules, lint, quality and formatting

Built-in rules, the rules engine and custom-rule host, SQL lint, quality checks, the
formatter, rules configuration and the request models move with their tests to
sqlbuild-rules. It owns the build identity script, which still hashes every workspace
crate, so sqlbuild-python is left with only the PyO3 bindings.

* refactor(native): route cross-crate use through main entries and declare the crates to fensu

Items other crates use are reached through main entry points, models and types modules
instead of helpers: the static reference type moves to types.rs, the rules dialect and the
test-only built-in function list gain main entries, and the panic boundary splits into one
entry per function with its message in constants. fensu now declares every workspace
crate and its exceptions follow the moved files.

* refactor(native): enforce one-way crate layering and matching versions in check-ci

The workspace declares its crates from the bottom up, and a check run by make check-ci
fails when a crate depends on a later layer (resolving aliased workspace dependencies to
their packages), when any crate other than sqlbuild-python reaches PyO3, or when a crate
below sqlbuild-analysis reaches polyglot. It also fails when the workspace, pyproject and
release-please versions differ, since the version is part of the build identity.
crates/README.md documents the crates, the graph and the check.

* fix(native): include the workspace manifest in the build identity

Dependency features and release profiles live in the workspace Cargo.toml, so changing them
left BUILD_IDENTITY and every cache keyed on it unchanged. The build script now hashes and
watches the workspace manifest next to the lockfile and exports the hashed workspace files,
which a test asserts.

Author: kvlonge

.github/workflows/release-please.yml

@@ -186,7 +186,7 @@ jobs:
           done
 
           unexpected_files="$(gh pr view "${RELEASE_PR}" --repo "${GITHUB_REPOSITORY}" \
-            --json files --jq '[.files[].path | select(. != ".release-please-manifest.json" and . != "CHANGELOG.md" and . != "pyproject.toml" and . != "uv.lock" and . != "Cargo.lock" and . != "crates/sqlbuild-rules-native/Cargo.toml")] | join("\n")')"
+            --json files --jq '[.files[].path | select(. != ".release-please-manifest.json" and . != "CHANGELOG.md" and . != "pyproject.toml" and . != "uv.lock" and . != "Cargo.lock" and . != "Cargo.toml")] | join("\n")')"
           if [[ -n "${unexpected_files}" ]]; then
             echo "Release pull request contains non-metadata changes:" >&2
             echo "${unexpected_files}" >&2

Cargo.lock

@@ -1,10 +1,29 @@
 [workspace]
 members = [
-    "crates/sqlbuild-rules-native",
+    "crates/sqlbuild-core",
+    "crates/sqlbuild-sqltext",
+    "crates/sqlbuild-analysis",
+    "crates/sqlbuild-rules",
+    "crates/sqlbuild-python",
 ]
 resolver = "2"
 
+# Native crates from the bottom up: each crate may depend only on crates listed before it.
+# `make check-ci` enforces this, keeps PyO3 in the Python boundary crate and keeps polyglot
+# out of every crate below the polyglot floor. See crates/README.md.
+[workspace.metadata.native-layers]
+order = [
+    "sqlbuild-core",
+    "sqlbuild-sqltext",
+    "sqlbuild-analysis",
+    "sqlbuild-rules",
+    "sqlbuild-python",
+]
+python-boundary = "sqlbuild-python"
+polyglot-floor = "sqlbuild-analysis"
+
 [workspace.package]
+version = "0.136.0"
 edition = "2024"
 license = "Apache-2.0"
 publish = false
@@ -36,8 +55,13 @@ quote = "1"
 rayon = "1"
 regex = "1"
 serde = { version = "1", features = ["derive"] }
-serde_json = "1"
+# Every crate relies on insertion-ordered JSON objects, which the extension has always built with.
+serde_json = { version = "1", features = ["preserve_order"] }
 sha2 = "0.10"
+sqlbuild-analysis = { path = "crates/sqlbuild-analysis" }
+sqlbuild-core = { path = "crates/sqlbuild-core" }
+sqlbuild-rules = { path = "crates/sqlbuild-rules" }
+sqlbuild-sqltext = { path = "crates/sqlbuild-sqltext" }
 stacker = "0.1"
 sqlparser = { version = "0.58", features = ["visitor"] }
 syn = { version = "2", features = ["full", "visit", "extra-traits"] }

Cargo.toml

@@ -1,6 +1,6 @@
 SHELL := /bin/bash
 
-.PHONY: verify verify-quick verify-pg coverage cli-preview rust-check \
+.PHONY: verify verify-quick verify-pg coverage cli-preview rust-check check-native-layering \
 	check-e2e-shards test-e2e-duckdb test-e2e-duckdb-build-core \
 	test-e2e-duckdb-build-incremental \
 	test-e2e-duckdb-cli-data test-e2e-duckdb-cli test-e2e-duckdb-inspection \
@@ -566,14 +566,18 @@ verify-quick:
 	exit $$status
 
 
-check-ci: check-e2e-shards
+check-ci: check-e2e-shards check-native-layering
 	uv run ruff format --check .
 	uv run ruff check .
 	uv run ty check src tests
 	uv run pytest tests/unit/src/sqlbuild/adapter/contract/classes/strict_adapter/test_strict_adapter.py -q
 	uv run fensu check
 
 
+check-native-layering:
+	uv run python -m scripts.check_native_layering
+
+
 check-pr-metadata:
 	@test -n "$(PR_TITLE)" || { echo "PR_TITLE is required" >&2; exit 2; }
 	@test -n "$(PR_BODY_FILE)" || { echo "PR_BODY_FILE is required" >&2; exit 2; }

Makefile

@@ -0,0 +1,48 @@
+# Native crates
+
+SQLBuild's native code is a Cargo workspace that builds one Python extension module,
+`sqlbuild._native`. Only `sqlbuild-python` knows about Python; every other crate is plain Rust
+that can be tested and benchmarked with `cargo test` alone.
+
+| Crate | Responsibility |
+|---|---|
+| `sqlbuild-core` | Constants shared by several crates and the panic boundary that turns an unwinding native failure into a named compiler error. |
+| `sqlbuild-sqltext` | Lexical SQL text without polyglot: comment, quote and parenthesis scanning, the rules quote policy, model header tokenization and matching, static variable substitution and static reference extraction. |
+| `sqlbuild-analysis` | SQL analysis over polyglot: SQL tokens, query analysis, the binding catalog, semantic validation and usage, column references, and SQL-test extraction, planning and rendering. |
+| `sqlbuild-rules` | Built-in rules and the rules engine, the custom-rule host, SQL lint, quality checks and formatting, rules configuration and the request models. It also owns the build identity script. |
+| `sqlbuild-python` | The only PyO3 crate: the `_native` module, its Python classes and functions, conversions from Python objects, and the process allocator. |
+
+## Layering
+
+Dependencies point one way, from the top of this graph to the bottom:
+
+```text
+sqlbuild-python
+  -> sqlbuild-rules -> sqlbuild-analysis -> sqlbuild-sqltext -> sqlbuild-core
+```
+
+Each crate may depend only on crates below it, and may also skip layers. The order is declared
+in `[workspace.metadata.native-layers]` in the workspace `Cargo.toml`, and
+`make check-native-layering` (part of `make check-ci`) fails when:
+
+- a crate depends on a crate that is not in an earlier layer, or a workspace crate is missing
+  from the order;
+- any crate other than `sqlbuild-python` reaches `pyo3`, directly or through another package;
+- a crate below `sqlbuild-analysis` reaches polyglot.
+
+The same check fails when the workspace version, the `pyproject.toml` version and
+`.release-please-manifest.json` disagree, because the version is part of the build identity.
+
+`fensu check` applies the Rust structure rules inside each crate: other crates use items through
+`main/` entry points, `models.rs`, `types.rs` and `constants.rs`, not through `_helpers/`.
+
+## Working on the crates
+
+- Tests live next to the module they cover; run them for one crate with
+  `cargo test -p <crate>` or for all crates with `cargo test --workspace`.
+- `sqlbuild-analysis` has a `test-support` feature that exposes test-only helpers to the tests of
+  dependent crates. Enable it only from `[dev-dependencies]`.
+- The build identity (`sqlbuild._native.BUILD_IDENTITY`) hashes the sources of every crate in
+  this directory, so editing any crate invalidates the native caches keyed on it.
+- maturin builds `crates/sqlbuild-python`; the release version lives in
+  `[workspace.package]` and is shared by every crate.

crates/README.md

@@ -0,0 +1,27 @@
+[package]
+name = "sqlbuild-analysis"
+version.workspace = true
+description = "Polyglot-based SQL analysis, binding and semantic validation for SQLBuild."
+edition.workspace = true
+license.workspace = true
+publish.workspace = true
+rust-version.workspace = true
+
+[features]
+default = []
+# Expose test-only helpers to the tests of dependent crates.
+test-support = []
+
+[dependencies]
+polyglot-sql.workspace = true
+polyglot-sql-function-catalogs.workspace = true
+rayon.workspace = true
+regex.workspace = true
+serde.workspace = true
+serde_json.workspace = true
+sha2.workspace = true
+sqlbuild-core.workspace = true
+sqlbuild-sqltext.workspace = true
+
+[lints]
+workspace = true

crates/sqlbuild-analysis/Cargo.toml

@@ -0,0 +1,3 @@
+pub fn analyze_json(request_json: &str) -> Result<String, String> {
+    crate::column_references::references::analyze_json(request_json)
+}

…ules-native/function_names/snowflake.txt …ld-analysis/function_names/snowflake.txtcrates/sqlbuild-rules-native/function_names/snowflake.txt renamed to crates/sqlbuild-analysis/function_names/snowflake.txt crates/sqlbuild-rules-native/function_names/snowflake.txt renamed to crates/sqlbuild-analysis/function_names/snowflake.txt

@@ -0,0 +1 @@
+pub mod analyze;