Skip to content

Commit bbd4e53

Browse files
authored
refactor: host remaining Rust-only shapes, event export and janitor planning natively (#1115)
* test: drop stale imports of removed rule and diagnostics modules * refactor: host lint, lineage, scope, discovery and analysis-session records and errors natively * refactor: host compile, discovery, rule, planner and dbt value records natively * chore: record staying executor, adapter and API shapes and move migration relation naming to the executor * refactor: declare compile input protocols in the extension stub and record the staying compile view shapes * chore: reconcile boundary after data-model leftovers * chore: drop stale stub imports after data-model integration * refactor: add native lifecycle-event export policy and priority queue Add the destination-neutral export policy derived from the lifecycle catalogue, severity comparison and narrowing, exception-name sanitisation, the bounded finite-priority queue with displacement, per-sink routing and accounting snapshots to sqlbuild-core, with unit tests. * refactor: host lifecycle-event export natively Move the event exporter dispatcher (priority queue, per-sink accounting, drain deadlines, invocation timeouts, failure notifications and periodic and final health summaries), the export policy, severity narrowing, error-name sanitisation and the command scope with its context variable into Rust. The dispatcher hosts its work on named Python daemon threads under a Python condition, joins its health thread at shutdown and stops its threads at interpreter exit. Each lifecycle sink call runs on a thread whose target is the Python invocation shim, so user code never runs above native frames that finalization would abort. Python keeps the public sink decorators and readers, the shapes re-exported by sqlbuild.sinks and the two shims that call user sink functions. * fix: stop command-output delivery before interpreter finalization A command-output sink still running when the interpreter finalized ran above native frames on the capture worker thread, and CPython ending that daemon thread aborted the process. Each command-output sink call now runs on its own daemon thread whose target is the Python invocation shim, which reports completion back to the native export. Writes from those threads stay uncaptured. At exit, hooks set a shared exit signal that ends native waits, then stop and join the capture worker and the event exporter threads with a bounded wait. Delivered records and accounting are unchanged. * chore: drop duplicated stub imports after event export integration * fix: join background SQL test planning before interpreter exit When planning failed before the SQL test plan was awaited, the background planning thread could still be running native code as the interpreter finalized, and CPython ending that daemon thread aborted the process instead of reporting the planning error. Execution planning now joins the planning thread on every exit path, and an exit hook joins any still-running planning thread with a bounded wait. Planned output is unchanged. * test: import native SqlTestSelection in planning exit test * refactor: plan janitor cleanup natively around adapter callbacks * refactor: build imported project Python discovery records natively around a module shim * refactor: parse Dagster invocation arguments, logs and payloads natively * refactor: build integration loader records natively around runtime loader factories * fix: keep provider settings unloaded during plain compile discovery * fix: run command-output sinks on one thread per event exporter dispatcher Each command-output sink call started its own daemon thread, so a provider used only by output sinks that creates a thread-bound resource on first use (such as a SQLite connection) failed on the next record. All command-output sink calls for a dispatcher now run in order on a single long-lived thread whose target is a Python shim waiting on a Python queue, so no native frame is on that thread when interpreter finalization can end it. Records, ordering, failure counting, filtering and provider teardown are unchanged.
1 parent 845980a commit bbd4e53

707 files changed

Lines changed: 20371 additions & 10332 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
pub(crate) mod name_parts;
Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
//! Fixed-width ASCII segments of reserved relation names.
2+
3+
/// The rest of `name` after an ASCII case-insensitive `prefix`.
4+
pub(crate) fn strip_prefix_folded<'a>(name: &'a str, prefix: &str) -> Option<&'a str> {
5+
let head: &str = name.get(..prefix.len())?;
6+
head.eq_ignore_ascii_case(prefix)
7+
.then(|| name.get(prefix.len()..))
8+
.flatten()
9+
}
10+
11+
/// `width` leading ASCII digits as a number, and the rest of `text`.
12+
pub(crate) fn leading_number(text: &str, width: usize) -> Option<(u32, &str)> {
13+
let digits: &str = text.get(..width)?;
14+
if !digits.bytes().all(|byte| byte.is_ascii_digit()) {
15+
return None;
16+
}
17+
let value: u32 = digits
18+
.bytes()
19+
.fold(0, |total, byte| total * 10 + u32::from(byte - b'0'));
20+
Some((value, text.get(width..)?))
21+
}
22+
23+
/// The rest of `text` after one ASCII case-insensitive `marker` character.
24+
pub(crate) fn after_marker(text: &str, marker: char) -> Option<&str> {
25+
let mut characters = text.chars();
26+
characters
27+
.next()
28+
.filter(|character| character.eq_ignore_ascii_case(&marker))
29+
.map(|_| characters.as_str())
30+
}
Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
//! Janitor planning names, prefixes and skip reasons.
2+
3+
/// SQLBuild state tables the janitor never acts on.
4+
pub const BUILT_IN_EXCLUDE_PATTERNS: [&str; 9] = [
5+
"_sqlbuild_fingerprints",
6+
"_sqlbuild_microbatches",
7+
"_sqlbuild_source_freshness",
8+
"_sqlbuild_node_results",
9+
"_sqlbuild_audit_results",
10+
"_sqlbuild_janitor_events",
11+
"_sqlbuild_migrations",
12+
"_sqlbuild_column_migrations",
13+
"_sqlbuild_old_name_views",
14+
];
15+
/// Prefix of every strict janitor archive name.
16+
pub const ARCHIVE_NAME_PREFIX: &str = "_SQB_ARCHIVE__";
17+
/// Separator between archive name segments.
18+
pub const ARCHIVE_NAME_SEPARATOR: &str = "__";
19+
/// Folded prefix reserving a name for janitor archive handling.
20+
pub const ARCHIVE_LOOKALIKE_PREFIX: &str = "_sqb_archive";
21+
/// Label used in archive name fitting errors.
22+
pub const ARCHIVE_ARTIFACT_LABEL: &str = "Janitor archive";
23+
/// Reason for names that may require quoting.
24+
pub const UNQUOTED_ADDRESSING_REASON: &str = "relation name is not a plain lowercase identifier and may require quoting; janitor does not act on it";
25+
/// Reason for names that collide case-insensitively.
26+
pub const CASE_COLLISION_REASON: &str =
27+
"relation name collides case-insensitively with another relation; janitor does not act on it";
28+
/// Reason for archive lookalikes outside the strict grammar.
29+
pub const MALFORMED_ARCHIVE_REASON: &str =
30+
"name resembles a janitor archive but does not match the strict archive grammar";
31+
/// Reason for query-diff artifacts left to fingerprint expiry cleanup.
32+
pub const QUERY_DIFF_ARTIFACT_REASON: &str = "query-diff artifacts use fingerprint expiry cleanup";
33+
/// Reason for reserved query-diff names without ownership evidence.
34+
pub const UNTRACKED_QUERY_DIFF_REASON: &str =
35+
"reserved query-diff name lacks matching ownership evidence";
36+
/// Default reason for protected relations.
37+
pub const PROTECTED_REASON: &str = "relation is protected from cleanup";
38+
/// Reason for relations without a SQLBuild fingerprint.
39+
pub const UNTRACKED_REASON: &str = "relation is not tracked by SQLBuild";
40+
/// Reason when the adapter has no relation age metadata.
41+
pub const AGE_UNSUPPORTED_REASON: &str = "adapter does not expose relation age metadata";
42+
/// Reason when a relation's age is unknown.
43+
pub const AGE_UNAVAILABLE_REASON: &str = "relation age is unavailable";
44+
/// Reason protecting compatibility views at old model names.
45+
pub const OLD_NAME_VIEW_REASON: &str = "compatibility view at an old model name";
46+
/// Parts of a fully qualified migration origin.
47+
pub const QUALIFIED_ORIGIN_PARTS: usize = 3;
48+
/// Default janitor relation retention in days.
49+
pub const DEFAULT_RETENTION_DAYS: i64 = 14;
50+
/// Prefix of reserved query-diff artifact names.
51+
pub const QUERY_DIFF_ARTIFACT_PREFIX: &str = "__sqlbuild_query_diff_";
52+
/// Sides a query-diff artifact may hold.
53+
pub const QUERY_DIFF_SIDES: [&str; 2] = ["left", "right"];
54+
/// Hex digits ending a query-diff run id.
55+
pub const ARTIFACT_RUN_HEX_DIGITS: usize = 12;
56+
/// Digits of a year in a compact UTC timestamp.
57+
pub const YEAR_DIGITS: usize = 4;
58+
/// Digits of a month or day in a compact UTC timestamp.
59+
pub const DATE_DIGITS: usize = 2;
60+
/// Digits of an hour, minute or second in a compact UTC timestamp.
61+
pub const TIME_DIGITS: usize = 2;
62+
/// Largest month number.
63+
pub const LAST_MONTH: u32 = 12;
64+
/// Largest hour.
65+
pub const LAST_HOUR: u32 = 23;
66+
/// Largest minute.
67+
pub const LAST_MINUTE: u32 = 59;
68+
/// Largest second; leap seconds are rejected as Python's `datetime` rejects them.
69+
pub const LAST_SECOND: u32 = 59;
Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
//! Fitted, case-folded janitor archive names.
2+
3+
use crate::identifier_fitting::main::fit_artifact_logical_name::fit_artifact_logical_name;
4+
use crate::janitor_planning::constants::{
5+
ARCHIVE_ARTIFACT_LABEL, ARCHIVE_NAME_PREFIX, ARCHIVE_NAME_SEPARATOR,
6+
};
7+
use crate::janitor_planning::models::ArchiveTimestamp;
8+
9+
/// The archive name keeping prefix, timestamp and kind intact; `Err` carries the fitting error.
10+
pub fn build_archive_name(
11+
(original_name, archived_at): (&str, ArchiveTimestamp),
12+
identifier_limit: i64,
13+
kind: Option<&str>,
14+
) -> Result<String, String> {
15+
let kind_segment: String = kind.map_or_else(String::new, |kind| {
16+
format!("{kind}{ARCHIVE_NAME_SEPARATOR}")
17+
});
18+
let fixed_prefix: String = format!(
19+
"{ARCHIVE_NAME_PREFIX}{:04}{:02}{:02}T{:02}{:02}{:02}Z{ARCHIVE_NAME_SEPARATOR}{kind_segment}",
20+
archived_at.year,
21+
archived_at.month,
22+
archived_at.day,
23+
archived_at.hour,
24+
archived_at.minute,
25+
archived_at.second,
26+
);
27+
let logical_part: String = fit_artifact_logical_name(
28+
(original_name, &fixed_prefix),
29+
identifier_limit,
30+
ARCHIVE_ARTIFACT_LABEL,
31+
)?;
32+
Ok(format!("{fixed_prefix}{logical_part}").to_lowercase())
33+
}
Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
//! Folded names shared by several relations in one listing.
2+
3+
use std::collections::{HashMap, HashSet};
4+
5+
/// Case-folded names that occur more than once in `names`.
6+
pub fn case_colliding_names<'a>(names: impl IntoIterator<Item = &'a str>) -> HashSet<String> {
7+
let mut counts: HashMap<String, usize> = HashMap::new();
8+
for name in names {
9+
*counts.entry(name.to_lowercase()).or_insert(0) += 1;
10+
}
11+
counts
12+
.into_iter()
13+
.filter(|(_, count)| *count > 1)
14+
.map(|(name, _)| name)
15+
.collect()
16+
}
Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
//! Relations whose folded name is shared in their listing.
2+
3+
use std::collections::HashSet;
4+
5+
use crate::janitor_planning::constants::CASE_COLLISION_REASON;
6+
7+
/// Why a relation must not be dropped because its folded name collides, if it does.
8+
#[must_use]
9+
pub fn case_collision_reason(
10+
name: &str,
11+
colliding_names: &HashSet<String>,
12+
) -> Option<&'static str> {
13+
colliding_names
14+
.contains(&name.to_lowercase())
15+
.then_some(CASE_COLLISION_REASON)
16+
}
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
//! Names reserved for janitor archive handling.
2+
3+
use crate::janitor_planning::constants::ARCHIVE_LOOKALIKE_PREFIX;
4+
5+
/// Whether a relation name starts, ignoring case, with the archive prefix.
6+
#[must_use]
7+
pub fn is_archive_lookalike_name(name: &str) -> bool {
8+
name.to_lowercase().starts_with(ARCHIVE_LOOKALIKE_PREFIX)
9+
}
Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
//! Janitor exclude patterns matched against relation and qualified names.
2+
3+
use sqlbuild_core::text::main::glob_match::glob_match;
4+
5+
/// The first pattern matching the name or the qualified display name, ignoring case.
6+
#[must_use]
7+
pub fn matching_exclude_pattern<'a>(
8+
(name, display_name): (&str, &str),
9+
patterns: &'a [String],
10+
) -> Option<&'a str> {
11+
let name: String = name.to_lowercase();
12+
let display_name: String = display_name.to_lowercase();
13+
patterns
14+
.iter()
15+
.find(|pattern| {
16+
let folded: String = pattern.to_lowercase();
17+
glob_match(&name, &folded) || glob_match(&display_name, &folded)
18+
})
19+
.map(String::as_str)
20+
}
Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
//! Relations a `migrate_from` value may name.
2+
3+
use crate::janitor_planning::constants::QUALIFIED_ORIGIN_PARTS;
4+
use crate::janitor_planning::models::OriginKey;
5+
6+
/// Origin keys for `raw`: a bare name in each sorted schema, else the one qualified relation.
7+
#[must_use]
8+
pub fn migration_origin_keys(
9+
raw: &str,
10+
destination_database: Option<&str>,
11+
sorted_schemas: &[(Option<String>, Option<String>)],
12+
) -> Vec<OriginKey> {
13+
let parts: Vec<String> = raw
14+
.split('.')
15+
.map(|part| {
16+
part.trim()
17+
.trim_matches(|c| c == '"' || c == '`')
18+
.to_owned()
19+
})
20+
.collect();
21+
if let [name] = parts.as_slice() {
22+
return sorted_schemas
23+
.iter()
24+
.map(|(database, schema)| OriginKey {
25+
database: database.clone(),
26+
schema: schema.clone(),
27+
name: name.clone(),
28+
})
29+
.collect();
30+
}
31+
let count: usize = parts.len();
32+
vec![OriginKey {
33+
database: if count == QUALIFIED_ORIGIN_PARTS {
34+
Some(parts[0].clone())
35+
} else {
36+
destination_database.map(str::to_owned)
37+
},
38+
schema: Some(parts[count - 2].clone()),
39+
name: parts[count - 1].clone(),
40+
}]
41+
}
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
pub mod build_archive_name;
2+
pub mod case_colliding_names;
3+
pub mod case_collision_reason;
4+
pub mod is_archive_lookalike_name;
5+
pub mod matching_exclude_pattern;
6+
pub mod migration_origin_keys;
7+
pub mod old_name_request_matches;
8+
pub mod parse_archive_name;
9+
pub mod query_diff_artifact_parts;
10+
pub mod resolve_retention_days;
11+
pub mod unaddressable_relation_reason;

0 commit comments

Comments
 (0)