Skip to content

Commit 845980a

Browse files
authored
refactor: move runtime capture, history and orchestration logic to Rust (#1114)
* refactor: move compile timings, built-in audits and code identity to native * refactor: run dbt subprocesses, event streams and project init natively * refactor: orchestrate custom rules and their host processes natively * test: prove custom rules split across hosts match one host * fix: stop split custom rule hosts on interrupt and restore multi-host coverage * refactor: validate dbt project and profile YAML natively * chore: drop stale imports after orchestration integration * refactor: host compute log storage natively Move LocalFilesystemComputeLogStorage, metadata encoding and validation, inventory and pruning into sqlbuild-cli's compute log module and expose the storage as a native class. The replay path and the storage now share one metadata encoder, inventory and prune implementation. Python keeps the stream tees, the scoped capture and the public shapes. * fix: build compute log storage errors after releasing the state lock Constructing a Python exception imports the exceptions module, which can release the GIL. Doing that while holding the storage state mutex let another thread take the GIL and block on the mutex, deadlocking a late diagnostic append against storage close. Every Python call now happens after the guard is dropped. * refactor: host command-output capture natively Move command-output text normalization, chunking, line buffering, the bounded priority queue, record validation, wire JSON, the invocation context and the delivery dispatcher into Rust. The dispatcher keeps Python threading primitives so delivery still runs on a named Python daemon thread. Python keeps the stream tees, the scope that installs them, and the public record shapes. * refactor: host execution history backends natively * fix: apply SQLite history file modes only on Unix
1 parent 624b408 commit 845980a

352 files changed

Lines changed: 15594 additions & 7605 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Cargo.lock‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎crates/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ that can be tested and benchmarked with `cargo test` alone.
88
|---|---|
99
| `sqlbuild-core` | Constants shared by several crates, the panic boundary that turns an unwinding native failure into a named compiler error, passive identity, diagnostic and SQL value types, text positions, a JSON emitter whose output matches Python's `json.dumps` and `orjson` byte for byte, and run-cost allocation with exact Python `decimal` default-context arithmetic and Snowflake query-history SQL. |
1010
| `sqlbuild-cursor` | Microbatch cursor algebra: typed cursor values, grain flooring and stepping, half-open interval clamping, merging and batch splitting, timestamp text normalization and calendar durations, with Python `datetime` results and errors. Zones other than fixed offsets are resolved through a caller-supplied lookup. |
11-
| `sqlbuild-state` | Append-only warehouse state rows: content-addressed migration event IDs, typed event SQL literals and statements, and source freshness data-version normalization, hashing, equivalence, age policy and latest-state read SQL, byte-compatible with the rows Python wrote. |
11+
| `sqlbuild-state` | Append-only warehouse state rows: content-addressed migration event IDs, typed event SQL literals and statements, and source freshness data-version normalization, hashing, equivalence, age policy and latest-state read SQL, byte-compatible with the rows Python wrote. It also owns execution history: lifecycle event batch digests and conflicts, run projection, event and run cursors, filter and page validation, and the SQLite and PostgreSQL statements the native history backends run. |
1212
| `sqlbuild-pysource` | Static reading of project Python source with the ruff parser: macro module structure, literal SQL arguments, definition locations and string-literal edits. It never imports or runs user code; CPython stays the authority on whether a file is valid Python. |
1313
| `sqlbuild-cache` | The shared native store every native cache uses (one atomically replaced, checksummed file per cache kind, keyed by content digests) and the one fingerprinting scheme for cache keys: the content digest and the project file fingerprint. |
1414
| `sqlbuild-sqltext` | Lexical SQL text without polyglot: comment, quote and parenthesis scanning, the rules quote policy, model header tokenization and matching, static variable substitution, static reference extraction, and the general reference scan with table-function call arguments under each adapter's lexical rules. |

‎crates/sqlbuild-attachments/src/audits/constants.rs‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,3 +13,22 @@ pub(crate) const FUNCTION_RESOURCE: &str = "function";
1313
pub(crate) const CROSS_MODEL_MINIMUM: usize = 2;
1414
/// The run scope Python falls back to without an instance or project value.
1515
pub(crate) const FALLBACK_RUN_SCOPE: &str = "delta_and_final";
16+
/// Built-in generic audit names and SQL bodies, which project definitions override by name.
17+
pub const BUILTIN_AUDITS: [(&str, &str); 4] = [
18+
(
19+
"accepted_values",
20+
"SELECT @column\nFROM @relation\nWHERE @column IS NOT NULL\n AND @column NOT IN (@'values')",
21+
),
22+
(
23+
"not_null",
24+
"SELECT @column\nFROM @relation\nWHERE @column IS NULL",
25+
),
26+
(
27+
"relationships",
28+
"SELECT @column\nFROM @relation\nWHERE @column IS NOT NULL\n AND @column NOT IN (\n SELECT @field\n FROM @to\n WHERE @field IS NOT NULL\n )",
29+
),
30+
(
31+
"unique",
32+
"SELECT @column, COUNT(*) AS duplicate_count\nFROM @relation\nWHERE @column IS NOT NULL\nGROUP BY @column\nHAVING COUNT(*) > 1",
33+
),
34+
];

‎crates/sqlbuild-attachments/src/audits/mod.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
pub(crate) mod _helpers;
2-
pub(crate) mod constants;
2+
pub mod constants;
33
pub mod main;
44
pub mod models;
55

‎crates/sqlbuild-cache/Cargo.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ rust-version.workspace = true
1010
[dependencies]
1111
blake3.workspace = true
1212
rayon.workspace = true
13+
serde_json.workspace = true
1314
sha2.workspace = true
1415
sqlbuild-core.workspace = true
1516
tempfile.workspace = true
Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
//! `importlib.metadata` distribution lookup over search paths, for code identity facts.
2+
3+
use std::io::ErrorKind;
4+
use std::path::{Path, PathBuf};
5+
6+
use crate::digest::constants::{METADATA_DIRECTORY_SUFFIXES, METADATA_FILES};
7+
8+
/// The first metadata folder of `name` along `search_paths`, as `importlib.metadata` finds it.
9+
pub(crate) fn distribution_directory(search_paths: &[PathBuf], name: &str) -> Option<PathBuf> {
10+
let wanted: String = normalized(name);
11+
for root in search_paths {
12+
let directory: &Path = if root.as_os_str().is_empty() {
13+
Path::new(".")
14+
} else {
15+
root
16+
};
17+
if let Some(child) = children(directory)
18+
.into_iter()
19+
.find(|child| metadata_name(child).is_some_and(|found| normalized(&found) == wanted))
20+
{
21+
return Some(root.join(child));
22+
}
23+
}
24+
None
25+
}
26+
27+
/// The `Version` header of a distribution's metadata, when it has one.
28+
pub(crate) fn distribution_version(directory: &Path) -> Option<String> {
29+
for file in METADATA_FILES {
30+
if let Some(text) = optional_text(&directory.join(file)) {
31+
return header_value(&text, "version");
32+
}
33+
}
34+
None
35+
}
36+
37+
/// A file's bytes, or `None` where `importlib.metadata` treats the file as absent.
38+
pub(crate) fn optional_bytes(path: &Path) -> Option<Vec<u8>> {
39+
match std::fs::read(path) {
40+
Ok(bytes) => Some(bytes),
41+
Err(error) if absent(&error) => None,
42+
Err(_) => None,
43+
}
44+
}
45+
46+
/// Entry names of a search path folder; folders Python cannot list have none.
47+
fn children(directory: &Path) -> Vec<String> {
48+
let entries: std::fs::ReadDir = match std::fs::read_dir(directory) {
49+
Ok(entries) => entries,
50+
Err(error) if absent(&error) => return Vec::new(),
51+
Err(_) => return Vec::new(),
52+
};
53+
let mut names: Vec<String> = Vec::new();
54+
for entry in entries.flatten() {
55+
names.push(entry.file_name().to_string_lossy().into_owned());
56+
}
57+
names
58+
}
59+
60+
fn optional_text(path: &Path) -> Option<String> {
61+
optional_bytes(path).map(|bytes| String::from_utf8_lossy(&bytes).into_owned())
62+
}
63+
64+
fn absent(error: &std::io::Error) -> bool {
65+
matches!(
66+
error.kind(),
67+
ErrorKind::NotFound | ErrorKind::NotADirectory | ErrorKind::IsADirectory
68+
)
69+
}
70+
71+
/// The distribution name a metadata folder entry names, before its version.
72+
fn metadata_name(child: &str) -> Option<String> {
73+
let lowered: String = child.to_lowercase();
74+
METADATA_DIRECTORY_SUFFIXES
75+
.iter()
76+
.any(|suffix| lowered.ends_with(suffix))
77+
.then(|| {
78+
let stem: &str = lowered.rsplit_once('.').map_or("", |(stem, _)| stem);
79+
stem.split('-').next().unwrap_or_default().to_owned()
80+
})
81+
}
82+
83+
/// `re.sub(r"[-_.]+", "-", name).lower().replace("-", "_")`.
84+
fn normalized(name: &str) -> String {
85+
let mut text: String = String::with_capacity(name.len());
86+
let mut separator: bool = false;
87+
for character in name.chars() {
88+
if matches!(character, '-' | '_' | '.') {
89+
if !separator {
90+
text.push('_');
91+
}
92+
separator = true;
93+
} else {
94+
text.extend(character.to_lowercase());
95+
separator = false;
96+
}
97+
}
98+
text
99+
}
100+
101+
/// The first value of a case-insensitive header in an RFC 822 style header block.
102+
fn header_value(text: &str, name: &str) -> Option<String> {
103+
let mut value: Option<String> = None;
104+
for line in text.lines() {
105+
if line.is_empty() {
106+
break;
107+
}
108+
if line.starts_with([' ', '\t']) {
109+
if let Some(current) = value.as_mut() {
110+
current.push_str(line);
111+
}
112+
continue;
113+
}
114+
if value.is_some() {
115+
break;
116+
}
117+
if let Some((header, rest)) = line.split_once(':')
118+
&& header.eq_ignore_ascii_case(name)
119+
{
120+
value = Some(rest.trim_start().to_owned());
121+
}
122+
}
123+
value.map(|text| text.trim_end().to_owned())
124+
}
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,3 @@
11
pub(crate) mod code_files;
2+
pub(crate) mod distributions;
23
pub(crate) mod walk;

‎crates/sqlbuild-cache/src/digest/constants.rs‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,3 +35,16 @@ pub const PRESENCE_TAG: &[u8] = b"presence";
3535
pub(crate) const CODE_SOURCE_SUFFIXES: &[&str] = &["py", "so", "pyd"];
3636
/// The bytecode folder an editable install's code walk skips.
3737
pub(crate) const BYTECODE_CACHE_DIRECTORY: &str = "__pycache__";
38+
39+
/// Distributions whose installed versions identify the code behind cached results.
40+
pub const CODE_IDENTITY_DISTRIBUTIONS: [&str; 3] = ["sqlbuild", "polyglot-sql-chio", "PyYAML"];
41+
/// The distribution an editable install's `direct_url.json` describes.
42+
pub(crate) const SQLBUILD_DISTRIBUTION: &str = "sqlbuild";
43+
/// The version reported for a distribution no search path holds.
44+
pub(crate) const UNKNOWN_VERSION: &str = "unknown";
45+
/// Metadata folder suffixes `importlib.metadata` searches, lowercased.
46+
pub(crate) const METADATA_DIRECTORY_SUFFIXES: [&str; 2] = [".dist-info", ".egg-info"];
47+
/// Metadata files read for a distribution's version, in order.
48+
pub(crate) const METADATA_FILES: [&str; 2] = ["METADATA", "PKG-INFO"];
49+
/// The installer's record of where a distribution was installed from.
50+
pub(crate) const DIRECT_URL_FILE: &str = "direct_url.json";
Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
//! The installed distribution versions and editable flag behind the installed code identity.
2+
3+
use std::path::PathBuf;
4+
5+
use crate::digest::_helpers::distributions::{
6+
distribution_directory, distribution_version, optional_bytes,
7+
};
8+
use crate::digest::constants::{
9+
CODE_IDENTITY_DISTRIBUTIONS, DIRECT_URL_FILE, SQLBUILD_DISTRIBUTION, UNKNOWN_VERSION,
10+
};
11+
12+
/// Each identity distribution's version along `search_paths`, and whether SQLBuild is editable.
13+
#[must_use]
14+
pub fn installed_code_facts(search_paths: &[PathBuf]) -> (Vec<String>, bool) {
15+
let versions: Vec<String> = CODE_IDENTITY_DISTRIBUTIONS
16+
.iter()
17+
.map(|name| {
18+
distribution_directory(search_paths, name)
19+
.and_then(|directory| distribution_version(&directory))
20+
.unwrap_or_else(|| UNKNOWN_VERSION.to_owned())
21+
})
22+
.collect();
23+
let editable: bool =
24+
distribution_directory(search_paths, SQLBUILD_DISTRIBUTION).is_none_or(|directory| {
25+
optional_bytes(&directory.join(DIRECT_URL_FILE))
26+
.is_some_and(|bytes| editable_direct_url(&bytes))
27+
});
28+
(versions, editable)
29+
}
30+
31+
/// Whether `direct_url.json` marks an editable install; unreadable payloads count as editable.
32+
fn editable_direct_url(bytes: &[u8]) -> bool {
33+
let Ok(payload) = serde_json::from_slice::<serde_json::Value>(bytes) else {
34+
return true;
35+
};
36+
let Some(payload) = payload.as_object() else {
37+
return true;
38+
};
39+
payload
40+
.get("dir_info")
41+
.and_then(serde_json::Value::as_object)
42+
.and_then(|dir_info| dir_info.get("editable"))
43+
.is_some_and(|editable| editable == &serde_json::Value::Bool(true))
44+
}

‎crates/sqlbuild-cache/src/digest/main/mod.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,5 +4,6 @@ pub mod content_digest;
44
pub mod digest_files;
55
pub mod fingerprint_project_files;
66
pub mod hex_digest;
7+
pub mod installed_code_facts;
78
pub mod installed_code_identity;
89
pub mod sha256_hex;

0 commit comments

Comments
 (0)