Skip to content

Commit fac17f3

Browse files
authored
refactor: port clone, load, check, test, seed and diff commands, rule caches and state helpers to the native extension (#1102)
* refactor(cli): run sqb clone natively * chore: update the Python boundary ratchet * refactor(state): build migration event IDs, event SQL and freshness versions natively The new plain-Rust sqlbuild-state crate computes the content-addressed migration, column-rename and old-name view event IDs, renders the idempotent event lookup and typed INSERT statements, and normalizes, hashes, compares and age-checks source freshness data versions. Stored IDs and hashes are pinned to values the Python implementation wrote. Reads and writes still go through the adapter, and warehouse orchestration stays in Python. * fix: read native freshness helpers from the extension in the freshness command * chore: update the Python boundary ratchet * refactor: normalize dbt profiles and write dbt init config natively * refactor: build and render the sqb dbt plan natively * refactor: index dbt manifests and resolve dbt refs natively * chore: update the Python boundary ratchet * refactor(cli): run sqb load natively * chore: update the Python boundary ratchet * refactor: register command handler bindings with the CLI bindings * fix(cli): route progress to stderr for sqb dbt --json * chore: update the Python boundary ratchet * refactor(diagnostics): redact diagnostic secrets natively Secret redaction of diagnostic text and structured logging fields runs in sqlbuild-core with Rust regex. The Python logging handlers, formatters, filters and the LogRecord field collection stay in Python and are listed as stays in python-boundary.toml. * refactor(rules): key and look up the SQL rule result cache natively SQL rule cache keys (lint settings, rule codes, model SQL and dynamic column contract, non-model file text), stored-entry lookup and the cache file now live in sqlbuild-rules as pure functions with a new key format (sql-rules-v6), so the first run after upgrading misses once. A new integration test pins cold misses, warm hits and invalidation on model, threshold, exception-policy and rule-selection edits. * chore: update the Python boundary ratchet * fix: drop command binding imports from the main registration * refactor(cli): run sqb check natively * chore: update the Python boundary ratchet * refactor(cli): render sqb diff output and resolve its exit codes natively * chore: update the Python boundary ratchet * refactor: import janitor, load, clone and freshness Python by complete module paths * test: check native Python module paths resolve and are never built at runtime * feat: run sqb test natively * chore: update the Python boundary ratchet * refactor(rules): key, look up and store the custom-rule result cache natively Custom-rule identities (code identity, host runtime, rule metadata, source fingerprint, rules settings, dialect, project root), subject identities, read-set lookup, result merging, finding validation and ordering, and the cache file now live in sqlbuild-rules as pure functions behind a CustomRuleCache binding, with a new key format (custom-rules-native-v1), so the first run after upgrading misses once. Fact digests stay with the Python fact views the rule hosts read. A new integration test pins cold misses, warm hits, a hit through an unread model edit, and invalidation on a rule source edit. * refactor(rules): load rules configuration natively The native loader now returns the RulesConfig models directly and raises RulesError or RulesConfigError itself; the allow_exceptions policy check moves to sqlbuild-rules. The JSON configuration hand-off and its Python normalisation are removed. Selector resolution against the compiled project stays with the planner scope in Python. * refactor(rules): convert rule findings and SQL test rows natively The rules request binding now reads compiled SQL tests, scenarios and initial findings straight from the Python objects, and evaluation, finalisation, the SQL rule cache lookup and the custom-rule cache return Finding models built natively. Finalisation keeps each original finding's fixable mark natively, and undecodable cached SQL findings are dropped as misses inside the lookup, so its handled-error entry is removed. The Python row builders and finding decoders are deleted; a new NativeRulesRequest.sql_facts_json accessor lets tests inspect the facts sent to the engine. * refactor(rules): separate rule-run thread and custom-rule dispatch modules Thread pools, futures and stop events around a rules run move to run/rule_threads.py, and custom-rule submission beside native evaluation moves to engine/custom_dispatch.py. Run planning, SQL rule evaluation and result finalisation stay in run/rules.py as plain functions. The two new modules and engine/custom_rules.py, which dispatches to the rule-host worker pool, are listed as user-code stays. * refactor(rules): build the built-in rule catalogue natively Built-in Rule models, their declared options and guidance are built by a native builtin_rules binding from the sqlbuild-rules catalogue, and the early-lint SQL code selection runs natively. The JSON catalogue hand-off and the Python option table are removed. Loading custom rule modules and validating their declared options stay in Python with the user code. * chore: update the Python boundary ratchet * refactor(cli): run sqb diff validation, orchestration and query lifecycle natively * chore: update the Python boundary ratchet * feat: run sqb seed natively * chore: update the Python boundary ratchet * fix(cli): render sqb check rows before encoding check metadata
1 parent 74fcac8 commit fac17f3

530 files changed

Lines changed: 22337 additions & 11498 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Cargo.lock‎

Lines changed: 13 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
members = [
33
"crates/sqlbuild-core",
44
"crates/sqlbuild-cursor",
5+
"crates/sqlbuild-state",
56
"crates/sqlbuild-pysource",
67
"crates/sqlbuild-cache",
78
"crates/sqlbuild-sqltext",
@@ -26,6 +27,7 @@ resolver = "2"
2627
order = [
2728
"sqlbuild-core",
2829
"sqlbuild-cursor",
30+
"sqlbuild-state",
2931
"sqlbuild-pysource",
3032
"sqlbuild-cache",
3133
"sqlbuild-sqltext",
@@ -104,6 +106,7 @@ sqlbuild-model-config = { path = "crates/sqlbuild-model-config" }
104106
sqlbuild-pysource = { path = "crates/sqlbuild-pysource" }
105107
sqlbuild-scopes = { path = "crates/sqlbuild-scopes" }
106108
sqlbuild-sqltext = { path = "crates/sqlbuild-sqltext" }
109+
sqlbuild-state = { path = "crates/sqlbuild-state" }
107110
stacker = "0.1"
108111
sqlparser = { version = "0.58", features = ["visitor"] }
109112
syn = { version = "2", features = ["full", "visit", "extra-traits"] }

‎crates/README.md‎

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ that can be tested and benchmarked with `cargo test` alone.
88
|---|---|
99
| `sqlbuild-core` | Constants shared by several crates, the panic boundary that turns an unwinding native failure into a named compiler error, passive identity, diagnostic and SQL value types, text positions, and a JSON emitter whose output matches Python's `json.dumps` and `orjson` byte for byte. |
1010
| `sqlbuild-cursor` | Microbatch cursor algebra: typed cursor values, grain flooring and stepping, half-open interval clamping, merging and batch splitting, timestamp text normalization and calendar durations, with Python `datetime` results and errors. Zones other than fixed offsets are resolved through a caller-supplied lookup. |
11+
| `sqlbuild-state` | Append-only warehouse state rows: content-addressed migration event IDs, typed event SQL literals and statements, and source freshness data-version normalization, hashing, equivalence and age policy, byte-compatible with the rows Python wrote. |
1112
| `sqlbuild-pysource` | Static reading of project Python source with the ruff parser: macro module structure, literal SQL arguments, definition locations and string-literal edits. It never imports or runs user code; CPython stays the authority on whether a file is valid Python. |
1213
| `sqlbuild-cache` | The shared native store every native cache uses (one atomically replaced, checksummed file per cache kind, keyed by content digests) and the one fingerprinting scheme for cache keys: the content digest and the project file fingerprint. |
1314
| `sqlbuild-sqltext` | Lexical SQL text without polyglot: comment, quote and parenthesis scanning, the rules quote policy, model header tokenization and matching, static variable substitution, static reference extraction, and the general reference scan with table-function call arguments under each adapter's lexical rules. |
@@ -31,7 +32,7 @@ Dependencies point one way, from the top of this graph to the bottom:
3132
sqlbuild-python
3233
-> sqlbuild-refactor -> sqlbuild-rules -> sqlbuild-analysis -> sqlbuild-attachments
3334
-> sqlbuild-model-config -> sqlbuild-scopes -> sqlbuild-render -> sqlbuild-discovery -> sqlbuild-config
34-
-> sqlbuild-sqltext -> sqlbuild-cache -> sqlbuild-pysource -> sqlbuild-cursor -> sqlbuild-core
35+
-> sqlbuild-sqltext -> sqlbuild-cache -> sqlbuild-pysource -> sqlbuild-state -> sqlbuild-cursor -> sqlbuild-core
3536
sqlbuild-cli -> sqlbuild-cache, sqlbuild-core
3637
```
3738

@@ -56,6 +57,21 @@ The same check fails when the workspace version, the `pyproject.toml` version an
5657
`fensu check` applies the Rust structure rules inside each crate: other crates use items through
5758
`main/` entry points, `models.rs`, `types.rs` and `constants.rs`, not through `_helpers/`.
5859

60+
## Python module paths
61+
62+
`sqlbuild-python` and `sqlbuild-cli` import Python by module path strings. The repository test
63+
`tests/unit/scripts/native_module_paths` (`scripts/native_module_paths`) keeps them honest:
64+
65+
- every `"sqlbuild...."` literal must be a complete dotted path that imports as a module or
66+
resolves as a module attribute; constants whose name contains `LOGGER` hold `logging`
67+
logger names and are not imported;
68+
- attributes are checked where a literal module, or a module constant defined in the same file,
69+
is passed with a literal name to an import helper (`imported(py, MODULE, "name")`), to
70+
`import(MODULE)?.getattr("name")`, or written as a `("sqlbuild.module", "name")` pair;
71+
- module paths are never built at runtime: a `format!` or `concat!` template may not start with
72+
`sqlbuild.` or with a `{..PACKAGE..}.`/`{..MODULE..}.` constant. Write the complete module
73+
path as a constant instead.
74+
5975
## Compiler
6076

6177
The compiler runs discovery, rendering and analysis natively. Generic compiler stages do not
Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
//! Bounded, difference-aware rendering of example values and keys.
2+
3+
use sqlbuild_cache::digest::main::sha256_hex::sha256_hex;
4+
5+
use crate::diff_report::_helpers::numbers::grouped;
6+
use crate::diff_report::constants::MAX_RENDERED_KEY_LENGTH;
7+
use crate::diff_report::models::{DiffExampleOptions, RenderedValue};
8+
9+
fn first_difference(left: &[char], right: &[char]) -> Option<usize> {
10+
if let Some(index) = left.iter().zip(right).position(|(a, b)| a != b) {
11+
return Some(index);
12+
}
13+
(left.len() != right.len()).then(|| left.len().min(right.len()))
14+
}
15+
16+
fn render_value(
17+
text: &[char],
18+
first_difference: Option<usize>,
19+
options: DiffExampleOptions,
20+
) -> RenderedValue {
21+
let original_length: usize = text.len();
22+
let full: String = text.iter().collect();
23+
if options.suppress_values {
24+
return RenderedValue {
25+
text: None,
26+
original_length,
27+
truncated: false,
28+
suppressed: true,
29+
first_difference,
30+
};
31+
}
32+
let Some(limit) = options
33+
.max_value_length
34+
.filter(|limit| original_length > *limit)
35+
else {
36+
return RenderedValue {
37+
text: Some(full),
38+
original_length,
39+
truncated: false,
40+
suppressed: false,
41+
first_difference,
42+
};
43+
};
44+
let difference: usize = first_difference.unwrap_or(0);
45+
let context_before: usize = (limit / 3).min(limit.saturating_sub(1));
46+
let start: usize = difference
47+
.saturating_sub(context_before)
48+
.min(original_length - limit);
49+
let end: usize = start + limit;
50+
let fragment: String = text[start..end].iter().collect();
51+
RenderedValue {
52+
text: Some(format!(
53+
"{}{fragment}{}",
54+
if start > 0 { "…" } else { "" },
55+
if end < original_length { "…" } else { "" }
56+
)),
57+
original_length,
58+
truncated: true,
59+
suppressed: false,
60+
first_difference,
61+
}
62+
}
63+
64+
/// Two unequal values rendered around their first differing character.
65+
pub(crate) fn render_example_pair(
66+
left: &str,
67+
right: &str,
68+
options: DiffExampleOptions,
69+
) -> (RenderedValue, RenderedValue) {
70+
let left: Vec<char> = left.chars().collect();
71+
let right: Vec<char> = right.chars().collect();
72+
let difference: Option<usize> = first_difference(&left, &right);
73+
(
74+
render_value(&left, difference, options),
75+
render_value(&right, difference, options),
76+
)
77+
}
78+
79+
/// One bounded value for terminal output.
80+
pub(crate) fn display_example_value(value: &RenderedValue) -> String {
81+
if value.suppressed {
82+
return format!(
83+
"<value suppressed; {} chars>",
84+
grouped(value.original_length as i64)
85+
);
86+
}
87+
let rendered: &str = value.text.as_deref().unwrap_or_default();
88+
if value.truncated {
89+
return format!(
90+
"{rendered} [truncated; {} chars]",
91+
grouped(value.original_length as i64)
92+
);
93+
}
94+
rendered.to_owned()
95+
}
96+
97+
/// Ordinary keys complete; exceptionally long keys shortened and identified by digest.
98+
pub(crate) fn render_key_value(text: &str) -> String {
99+
let length: usize = text.chars().count();
100+
if length <= MAX_RENDERED_KEY_LENGTH {
101+
return text.to_owned();
102+
}
103+
let digest: String = sha256_hex(text.as_bytes());
104+
let prefix: String = text.chars().take(MAX_RENDERED_KEY_LENGTH).collect();
105+
format!(
106+
"{prefix}… [truncated; {} chars; sha256:{}]",
107+
grouped(length as i64),
108+
&digest[..12]
109+
)
110+
}
Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
//! Whether a diff found differences, and the resulting outcome.
2+
3+
use crate::diff_report::constants::{FINDINGS_OUTCOME, PASS_OUTCOME};
4+
use crate::diff_report::models::DiffModelFacts;
5+
6+
pub(crate) fn has_schema_differences(model: &DiffModelFacts) -> bool {
7+
!(model.added_columns.is_empty()
8+
&& model.removed_columns.is_empty()
9+
&& model.type_changes.is_empty())
10+
}
11+
12+
/// Whether any model has schema or row differences.
13+
pub(crate) fn has_findings(models: &[DiffModelFacts]) -> bool {
14+
models.iter().any(|model| {
15+
has_schema_differences(model)
16+
|| model.rows.as_ref().is_some_and(|rows| {
17+
rows.unequal_count != 0 || rows.left_only_count != 0 || rows.right_only_count != 0
18+
})
19+
})
20+
}
21+
22+
/// The explicit outcome, or `findings`/`pass` from the results.
23+
pub(crate) fn resolved_outcome(models: &[DiffModelFacts], outcome: Option<&str>) -> String {
24+
outcome
25+
.filter(|outcome| !outcome.is_empty())
26+
.map(str::to_owned)
27+
.unwrap_or_else(|| {
28+
if has_findings(models) {
29+
FINDINGS_OUTCOME
30+
} else {
31+
PASS_OUTCOME
32+
}
33+
.to_owned()
34+
})
35+
}

0 commit comments

Comments
 (0)