Skip to content

Commit 74fcac8

Browse files
authored
refactor: port CLI commands, progress and output, rules pieces and cursor arithmetic to the native extension (#1101)
* refactor: write and decode integration-result records natively * chore: update the Python boundary ratchet * refactor(cli): project lifecycle progress and render progress text natively Move the lifecycle progress projector, the connection and planning progress reporters, the verbose inspection writer, execution headers and SQL expectation rows into the native extension and delete the Python they replace. * chore: update the Python boundary ratchet * refactor(cli): run sqb scope natively Load the offline scope index, qualify bare targets, run the scope query and render its text in Rust, route sqb scope to the native handler, and delete the Python command, target and output modules. * refactor(lint): scan, lint and format DSL headers natively Header scanning, body ranges, the header lint rules (parse, description presence and length, trailing whitespace) and the header formatting fixes (whitespace, leading-comment relocation, description and audit wrapping) run natively in sqlbuild-rules, reproducing Python's textwrap and string semantics. Header values are still parsed by parse_header_values. Delete the Python header scanner and native header rules. * refactor(rules): render rules results and duplicate-literal hints natively Rules result text and JSON rendering and the SQBRSQL044 duplicate-literal hints run natively in sqlbuild-rules, with PurePosixPath ordering and json.dumps output reproduced exactly. Delete the Python implementations. * chore: update the Python boundary ratchet * refactor: store and report run costs and shared CLI text natively * chore: drop the duplicate section style and update the Python boundary ratchet * refactor(cli): run sqb lineage natively Select relation and column lineage, read and write the structural cache, and render tree, list and JSON output in Rust, route sqb lineage to the native handler, resolve rename and move targets with the native target reader, and delete the Python lineage command, selection, cache and output modules. * chore: update the Python boundary ratchet * refactor: create playgrounds and install skills natively * chore: update the Python boundary ratchet * refactor(cli): run sqb rename and sqb mv natively Read the target, build planner facts and the declaration placement host, plan, stage, verify, migrate and commit in Rust, render the edit tree, the layer-move suggestion and the JSON report natively, route rename and mv to the native handler, and delete the Python command, facts, output, target and plan-decoding modules. The offline compile stays in Python. * chore: update the Python boundary ratchet * refactor(cli): run sqb janitor natively Resolve the invocation and settings, render the plan preview, --as preview framing, confirmation prompt and completion summary in Rust, and route sqb janitor to the native handler. Compilation, warehouse connection, planning and cleanup stay in the Python janitor runtime. * refactor(rules): read custom-rule evidence natively Custom-rule import closures, implementation fingerprints and statically associated harness test cases are read natively with the CPython-mirrored ruff tree in sqlbuild-pysource. Loading and running custom rules stays in Python. Delete the Python evidence module. * refactor(spec): read typed config values natively The typed authored-config readers (string, boolean, integer, string tuple and cursor bound) run natively and raise the same ConfigValueTypeError. Delete the Python helper and its five public wrappers; callers import the readers from sqlbuild._native. * style: satisfy clippy and fensu in custom-rule evidence source reading * chore: update the Python boundary ratchet * refactor: move sqb debug into the native extension * refactor: move command connection and warehouse resolution into the native extension * refactor(cli): run sqb freshness natively * chore: drop a module the freshness port deleted and update the Python boundary ratchet * refactor(cursor): run the cursor algebra natively Cursor parsing, rendering, comparison, grain flooring and stepping, observed partitions, interval clamping, merging, batch splitting, caps, interval validation and calendar durations run in the new plain-Rust sqlbuild-cursor crate with thin bindings. Python datetime objects stay at the boundary: results keep the input's datetime type and tzinfo, fixed offsets are computed natively and other zones are resolved through the input tzinfo. The cursor value models, sentinels and grain constants stay in Python; callers import the functions from sqlbuild._native. * style: satisfy clippy in integer cursor text parsing * chore: update the Python boundary ratchet * test: raise the native lineage memory budget
1 parent a3c431c commit 74fcac8

817 files changed

Lines changed: 29455 additions & 17276 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Cargo.lock‎

Lines changed: 8 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
[workspace]
22
members = [
33
"crates/sqlbuild-core",
4+
"crates/sqlbuild-cursor",
45
"crates/sqlbuild-pysource",
56
"crates/sqlbuild-cache",
67
"crates/sqlbuild-sqltext",
@@ -24,6 +25,7 @@ resolver = "2"
2425
[workspace.metadata.native-layers]
2526
order = [
2627
"sqlbuild-core",
28+
"sqlbuild-cursor",
2729
"sqlbuild-pysource",
2830
"sqlbuild-cache",
2931
"sqlbuild-sqltext",
@@ -93,6 +95,7 @@ sqlbuild-cache = { path = "crates/sqlbuild-cache" }
9395
sqlbuild-cli = { path = "crates/sqlbuild-cli" }
9496
sqlbuild-config = { path = "crates/sqlbuild-config" }
9597
sqlbuild-core = { path = "crates/sqlbuild-core" }
98+
sqlbuild-cursor = { path = "crates/sqlbuild-cursor" }
9699
sqlbuild-discovery = { path = "crates/sqlbuild-discovery" }
97100
sqlbuild-render = { path = "crates/sqlbuild-render" }
98101
sqlbuild-refactor = { path = "crates/sqlbuild-refactor" }

‎crates/README.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,10 +7,11 @@ that can be tested and benchmarked with `cargo test` alone.
77
| Crate | Responsibility |
88
|---|---|
99
| `sqlbuild-core` | Constants shared by several crates, the panic boundary that turns an unwinding native failure into a named compiler error, passive identity, diagnostic and SQL value types, text positions, and a JSON emitter whose output matches Python's `json.dumps` and `orjson` byte for byte. |
10+
| `sqlbuild-cursor` | Microbatch cursor algebra: typed cursor values, grain flooring and stepping, half-open interval clamping, merging and batch splitting, timestamp text normalization and calendar durations, with Python `datetime` results and errors. Zones other than fixed offsets are resolved through a caller-supplied lookup. |
1011
| `sqlbuild-pysource` | Static reading of project Python source with the ruff parser: macro module structure, literal SQL arguments, definition locations and string-literal edits. It never imports or runs user code; CPython stays the authority on whether a file is valid Python. |
1112
| `sqlbuild-cache` | The shared native store every native cache uses (one atomically replaced, checksummed file per cache kind, keyed by content digests) and the one fingerprinting scheme for cache keys: the content digest and the project file fingerprint. |
1213
| `sqlbuild-sqltext` | Lexical SQL text without polyglot: comment, quote and parenthesis scanning, the rules quote policy, model header tokenization and matching, static variable substitution, static reference extraction, and the general reference scan with table-function call arguments under each adapter's lexical rules. |
13-
| `sqlbuild-config` | Configuration file reading without Python: TOML, YAML 1.2 with the core schema and merge keys, and typed project and local config readers for the fields discovery needs. Errors carry their kind and position; YAML outside the supported subset is an `Unsupported` error. |
14+
| `sqlbuild-config` | Configuration file reading without Python: TOML, YAML 1.2 with the core schema and merge keys, and typed project and local config readers for the fields discovery needs. It also writes the workspace files `sqb playground` and `sqb skills` create from packaged templates and skills. Errors carry their kind and position; YAML outside the supported subset is an `Unsupported` error. |
1415
| `sqlbuild-discovery` | Native project discovery and the only implementation of the model, SQL test, scenario, source and schema file collections and the declaration layout: the shared directory walk with Python's glob and sort semantics, file reading with Python's newline handling and read errors, and parsing of authored files with Python's exact discovery messages. Results are plain data the Python discovery facade materialises. |
1516
| `sqlbuild-scopes` | Declaration scopes, visibility, relationship grants, placement, report queries and the offline scope cache, including dialect-aware SQL-test and scenario scans. |
1617
| `sqlbuild-model-config` | MODEL configuration, columns and audits, template expansion over caller values, and dynamic-column name patterns using Rust regex syntax. |
@@ -30,7 +31,7 @@ Dependencies point one way, from the top of this graph to the bottom:
3031
sqlbuild-python
3132
-> sqlbuild-refactor -> sqlbuild-rules -> sqlbuild-analysis -> sqlbuild-attachments
3233
-> sqlbuild-model-config -> sqlbuild-scopes -> sqlbuild-render -> sqlbuild-discovery -> sqlbuild-config
33-
-> sqlbuild-sqltext -> sqlbuild-cache -> sqlbuild-pysource -> sqlbuild-core
34+
-> sqlbuild-sqltext -> sqlbuild-cache -> sqlbuild-pysource -> sqlbuild-cursor -> sqlbuild-core
3435
sqlbuild-cli -> sqlbuild-cache, sqlbuild-core
3536
```
3637

‎crates/sqlbuild-analysis/src/lib.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ pub mod explicit_references;
1616
pub mod fingerprint_storage;
1717
pub mod graph;
1818
pub mod lineage;
19+
pub mod lineage_command;
1920
pub mod lineage_graph;
2021
pub mod manifest;
2122
pub mod python_plan;
Lines changed: 175 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,175 @@
1+
//! Human column lineage: dependency lists and column-trace trees.
2+
3+
use std::collections::{HashMap, HashSet};
4+
use std::convert::Infallible;
5+
6+
use sqlbuild_core::cli_text::models::CliStyle;
7+
8+
use crate::lineage_command::constants::{
9+
ALREADY_SHOWN, DOWNSTREAM_DIRECTION, HUMAN_COLUMN_TRACE_LIMIT, STAR_TRANSFORM,
10+
STAR_TRANSFORM_LABEL,
11+
};
12+
use crate::lineage_command::models::{ColumnTrace, LineageColumn, LineageColumnEdge};
13+
use crate::lineage_graph::main::trace_limit_note::trace_limit_note;
14+
use crate::lineage_graph::main::tree_lines::tree_lines;
15+
use crate::lineage_graph::models::TreeLine;
16+
use crate::lineage_graph::types::TreeVisitor;
17+
18+
/// `resource.column`.
19+
fn column_id(column: &LineageColumn) -> String {
20+
format!("{}.{}", column.resource_name, column.column_name)
21+
}
22+
23+
fn transform_label(edge: &LineageColumnEdge) -> &str {
24+
if edge.transform == STAR_TRANSFORM {
25+
STAR_TRANSFORM_LABEL
26+
} else {
27+
&edge.transform
28+
}
29+
}
30+
31+
fn displayed(trace: &ColumnTrace) -> &[LineageColumnEdge] {
32+
&trace.trace[..trace.trace.len().min(HUMAN_COLUMN_TRACE_LIMIT)]
33+
}
34+
35+
fn limit_note(trace: &ColumnTrace, style: CliStyle) -> Vec<String> {
36+
let notes: Vec<String> = trace_limit_note(
37+
i64::try_from(trace.trace.len()).unwrap_or(i64::MAX),
38+
i64::try_from(HUMAN_COLUMN_TRACE_LIMIT).unwrap_or(i64::MAX),
39+
);
40+
if notes.is_empty() {
41+
return notes;
42+
}
43+
let mut lines: Vec<String> = vec![String::new()];
44+
lines.extend(notes.iter().map(|note| style.muted(note)));
45+
lines
46+
}
47+
48+
/// The trace as `source -> target transform` lines.
49+
pub(crate) fn column_list(trace: &ColumnTrace, style: CliStyle) -> String {
50+
if trace.trace.is_empty() {
51+
return format!(
52+
"Column dependencies\n\n{}",
53+
style.section(&column_id(&trace.target))
54+
);
55+
}
56+
let edges: &[LineageColumnEdge] = displayed(trace);
57+
let width = |side: fn(&LineageColumnEdge) -> &LineageColumn| -> usize {
58+
edges
59+
.iter()
60+
.map(|edge| column_id(side(edge)).chars().count())
61+
.max()
62+
.unwrap_or(0)
63+
};
64+
let left: usize = width(|edge| &edge.source);
65+
let right: usize = width(|edge| &edge.target);
66+
let mut lines: Vec<String> = vec![style.title("Column dependencies"), String::new()];
67+
for edge in edges {
68+
let source: String = column_id(&edge.source);
69+
let target: String = column_id(&edge.target);
70+
lines.push(format!(
71+
"{}{} {} {}{} {}",
72+
style.section(&source),
73+
" ".repeat(left - source.chars().count()),
74+
style.muted("->"),
75+
style.section(&target),
76+
" ".repeat(right - target.chars().count()),
77+
style.muted(transform_label(edge))
78+
));
79+
}
80+
lines.extend(limit_note(trace, style));
81+
lines.join("\n")
82+
}
83+
84+
/// The trace as a tree from the target column.
85+
pub(crate) fn column_tree(trace: &ColumnTrace, style: CliStyle) -> String {
86+
let target: String = column_id(&trace.target);
87+
let mut lines: Vec<String> = vec![
88+
format!(
89+
"{} {} {}",
90+
style.title("Column trace"),
91+
style.section(&target),
92+
style.muted(&trace.direction)
93+
),
94+
String::new(),
95+
];
96+
if trace.trace.is_empty() {
97+
lines.push(style.muted(" No column dependencies found"));
98+
return lines.join("\n");
99+
}
100+
let downstream: bool = trace.direction == DOWNSTREAM_DIRECTION;
101+
let mut deps: HashMap<String, Vec<&LineageColumnEdge>> = HashMap::new();
102+
for edge in displayed(trace) {
103+
let key: &LineageColumn = if downstream {
104+
&edge.source
105+
} else {
106+
&edge.target
107+
};
108+
deps.entry(column_id(key)).or_default().push(edge);
109+
}
110+
let mut tree = ColumnTree {
111+
deps: &deps,
112+
downstream,
113+
style,
114+
lines: Vec::new(),
115+
};
116+
let roots: Vec<&LineageColumnEdge> = tree.sorted_edges(&target);
117+
let Ok(()) = tree_lines(roots, HashSet::from([target]), &mut tree);
118+
lines.extend(tree.lines);
119+
lines.extend(limit_note(trace, style));
120+
lines.join("\n")
121+
}
122+
123+
/// Column-trace edges shown once per related column, sorted by that column.
124+
struct ColumnTree<'a> {
125+
deps: &'a HashMap<String, Vec<&'a LineageColumnEdge>>,
126+
downstream: bool,
127+
style: CliStyle,
128+
lines: Vec<String>,
129+
}
130+
131+
impl<'a> ColumnTree<'a> {
132+
fn related_id(&self, edge: &LineageColumnEdge) -> String {
133+
column_id(if self.downstream {
134+
&edge.target
135+
} else {
136+
&edge.source
137+
})
138+
}
139+
140+
fn sorted_edges(&self, id: &str) -> Vec<&'a LineageColumnEdge> {
141+
let mut edges: Vec<&'a LineageColumnEdge> = self.deps.get(id).cloned().unwrap_or_default();
142+
edges.sort_by_cached_key(|edge| self.related_id(edge));
143+
edges
144+
}
145+
}
146+
147+
impl<'a> TreeVisitor<&'a LineageColumnEdge, String, Infallible> for ColumnTree<'a> {
148+
fn key(&mut self, edge: &&'a LineageColumnEdge) -> Result<String, Infallible> {
149+
Ok(self.related_id(edge))
150+
}
151+
152+
fn line(&mut self, line: TreeLine<'_, &'a LineageColumnEdge>) -> Result<(), Infallible> {
153+
let suffix: String = if line.repeated {
154+
self.style.muted(ALREADY_SHOWN)
155+
} else {
156+
String::new()
157+
};
158+
let edge: &LineageColumnEdge = line.node;
159+
self.lines.push(format!(
160+
"{}{}{} {}{suffix}",
161+
line.prefix,
162+
self.style.muted(line.branch),
163+
self.style.section(&self.related_id(edge)),
164+
self.style.muted(&format!("({})", transform_label(edge)))
165+
));
166+
Ok(())
167+
}
168+
169+
fn children(
170+
&mut self,
171+
edge: &&'a LineageColumnEdge,
172+
) -> Result<Vec<&'a LineageColumnEdge>, Infallible> {
173+
Ok(self.sorted_edges(&self.related_id(edge)))
174+
}
175+
}
Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
//! Breadth-first closures and selected-graph assembly over a lineage index.
2+
3+
use std::collections::{HashMap, HashSet, VecDeque};
4+
5+
use crate::lineage_command::constants::{
6+
BOTH_DIRECTIONS, DOWNSTREAM_DIRECTION, UPSTREAM_DIRECTION,
7+
};
8+
use crate::lineage_command::models::{LineageGraph, LineageIndex, LineageKey, LineageNode};
9+
10+
/// Edge lists by key.
11+
pub(crate) type EdgeMap<'a> = HashMap<&'a LineageKey, &'a [LineageKey]>;
12+
13+
/// `edges` as a lookup by key.
14+
pub(crate) fn edge_map(edges: &[(LineageKey, Vec<LineageKey>)]) -> EdgeMap<'_> {
15+
edges
16+
.iter()
17+
.map(|(key, targets)| (key, targets.as_slice()))
18+
.collect()
19+
}
20+
21+
/// Keys within `max_depth` edges of any start; a start is included only when reached again.
22+
pub(crate) fn closure(
23+
edges: &EdgeMap<'_>,
24+
starts: &[LineageKey],
25+
max_depth: Option<i64>,
26+
) -> HashSet<LineageKey> {
27+
let mut visited: HashSet<LineageKey> = HashSet::new();
28+
let mut frontier: VecDeque<(&LineageKey, i64)> =
29+
starts.iter().map(|start| (start, 0)).collect();
30+
while let Some((current, depth)) = frontier.pop_front() {
31+
if max_depth.is_some_and(|maximum| depth >= maximum) {
32+
continue;
33+
}
34+
for neighbor in edges.get(current).copied().unwrap_or_default() {
35+
if visited.insert(neighbor.clone()) {
36+
frontier.push_back((neighbor, depth + 1));
37+
}
38+
}
39+
}
40+
visited
41+
}
42+
43+
/// The keys `direction` reaches from `keys` within `depth`.
44+
pub(crate) fn expand(
45+
index: &LineageIndex,
46+
keys: &[LineageKey],
47+
(direction, depth): (&str, Option<i64>),
48+
) -> HashSet<LineageKey> {
49+
let mut expanded: HashSet<LineageKey> = HashSet::new();
50+
if direction == UPSTREAM_DIRECTION || direction == BOTH_DIRECTIONS {
51+
expanded.extend(closure(&edge_map(&index.upstream), keys, depth));
52+
}
53+
if direction == DOWNSTREAM_DIRECTION || direction == BOTH_DIRECTIONS {
54+
expanded.extend(closure(&edge_map(&index.downstream), keys, depth));
55+
}
56+
expanded
57+
}
58+
59+
/// `keys` sorted by resource type, then name.
60+
pub(crate) fn sorted_keys<'a>(keys: impl IntoIterator<Item = &'a LineageKey>) -> Vec<LineageKey> {
61+
let mut sorted: Vec<LineageKey> = keys.into_iter().cloned().collect();
62+
sorted.sort();
63+
sorted.dedup();
64+
sorted
65+
}
66+
67+
/// Display nodes and the upstream edges between selected keys.
68+
pub(crate) fn build_graph(
69+
index: &LineageIndex,
70+
selected: &HashSet<LineageKey>,
71+
(focus_keys, direction): (Vec<LineageKey>, Option<&str>),
72+
) -> LineageGraph {
73+
let ordered: Vec<LineageKey> = sorted_keys(selected);
74+
let upstream: EdgeMap<'_> = edge_map(&index.upstream);
75+
let mut edges: Vec<(LineageKey, LineageKey)> = Vec::new();
76+
for downstream in &ordered {
77+
for upstream_key in upstream.get(downstream).copied().unwrap_or_default() {
78+
if selected.contains(upstream_key) {
79+
edges.push((upstream_key.clone(), downstream.clone()));
80+
}
81+
}
82+
}
83+
LineageGraph {
84+
nodes: ordered
85+
.iter()
86+
.map(|key| {
87+
index
88+
.nodes
89+
.get(key)
90+
.cloned()
91+
.unwrap_or_else(|| LineageNode {
92+
key: key.clone(),
93+
relative_path: None,
94+
qualified_name: None,
95+
})
96+
})
97+
.collect(),
98+
edges,
99+
focus_keys,
100+
direction: direction.map(str::to_owned),
101+
}
102+
}

0 commit comments

Comments
 (0)