Skip to content

Commit dfdadd8

Browse files
authored
feat: bound SQL unit test and model execution cost and attach audit metadata (#1141)
* feat: bound unit test, scenario and model execution cost Limit the real models one SQL unit test may execute ([settings] unit_test_max_real_models, default 10) with a compile error that names every real model, its dependency path, the inputs to mock, when a scenario may fit better and the exact key to raise. Evaluate each compared actual output once into a temporary relation (DuckDB, MotherDuck, PostgreSQL; Snowflake reads RESULT_SCAN) so counts, both difference directions, column probes and samples read it. Add per-item time limits: test_timeout (default 30s) and scenario_timeout in [settings] and target execution_limits, TEST()/SCENARIO() timeout, --test-timeout and --scenario-timeout, and per-model max_duration in MODEL(), [path_defaults] and [defaults]. At expiry SQLBuild cancels the running statement, reports TIMEOUT with the limit, its source, the cancellation outcome and query IDs, and continues with independent work. Every built-in adapter now enforces deadlines, including the target's whole-build max_duration: Snowflake, PostgreSQL, BigQuery and Databricks bound statements server-side, and all adapters cancel in-flight work. * docs: document unit test cost guards and per-item time limits * feat: attach structured meta to audit cases and attachments Audit cases and SQL audit attachments accept an optional JSON mapping that is reported with each result but never reaches audit SQL, evaluation, thresholds or audit fingerprints. Results store it in a new nullable audit_meta_json column, added to existing result tables only when a result carries meta; audit_completed events and JSON check output include it as audit_meta. Non-JSON meta is a compile error that names the offending value. * fix: keep unit-test output storage optional for project adapters StrictAdapter now provides create_sql_test_output and drop_sql_test_output defaults that store nothing, so the executor keeps the inline comparison query and project adapters need no new methods. Built-in adapters that support single-read comparison override them. * feat: count timed-out models and tests separately in build and test summaries The sqb build footer reports TIMEOUT= apart from FAIL=, and build JSON adds summary.timeout_count, with failure_count excluding timeouts. sqb test JSON adds timeout_count likewise. A timeout still fails the run: the exit code is non-zero and the JSON status is failed. * fix: keep JSON failure counts inclusive of timeouts Build JSON summary.failure_count and test JSON summary.fail_count count timed-out items again, as before; timeout_count is an additive subset. The human-readable footers still show FAIL= and TIMEOUT= separately, and the CLI docs explain the difference. * fix: write audit results without meta when the result columns cannot be read A failing column read while checking for the additive meta column now logs a warning and writes the records without meta instead of failing the write. Remove the unused Python build_read_latest_sql builder. * fix: roll back, recover and bound only running time when an item times out Rollback and unit-test cleanup now run with every deadline suspended, so a model or scenario whose limit expires inside a transaction rolls it back before the timeout propagates. A connection whose statement was interrupted is probed before reuse by the next item, on build, test and scenario workers alike, and replaced when it is no longer usable. Item limits now count running time from the first warehouse statement and stop while cleanup or a queued BigQuery job waits. Snowflake relies on its statement limit, which excludes queueing, with a client backstop after a grace margin; BigQuery items are bounded by cancellation because a job timeout counts from job creation. Nested limits apply the earliest one, cancellation is issued only while the same statement is still registered, and work that finished before expiry keeps its result. PostgreSQL unit tests store their outputs in an ON COMMIT DROP table of one transaction bounded by SET LOCAL statement_timeout, so no setting outlives a pooled transaction, and a stored output that can no longer be read falls back to the inline comparison. * docs: describe per-item running-time limits and upgrade notes Add the per-adapter enforcement matrix for item limits, the PostgreSQL transaction-scoped statement limit and single-read transaction, and upgrade notes for the new real-model compile check and for whole-build max_duration being accepted on every built-in adapter. * fix: keep pool slots, test transactions and shared clocks correct after timeouts A connection that can neither be recovered nor replaced after a timeout is now returned to its pool marked broken, so later work fails with the driver's error instead of waiting forever for a lost slot; each later return retries the replacement, and microbatch subworker slots are always released. A PostgreSQL unit test claims its transaction only after BEGIN succeeds, so a refused BEGIN can no longer make a later release roll back a caller's transaction. A queued BigQuery job pauses an item's clock only while every statement under that deadline is queued, so one queued microbatch worker no longer pauses a model whose other workers are running. * test: list the new execution limit keys in config error goldens
1 parent e6a4a44 commit dfdadd8

218 files changed

Lines changed: 6994 additions & 334 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎crates/sqlbuild-analysis/src/compiler/_helpers/sql_tests/mod.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ pub mod markers;
1616
pub mod mock_reads;
1717
pub mod planning;
1818
pub mod reader_references;
19+
pub(crate) mod real_model_limit;
1920
pub mod reference_calls;
2021
pub mod relation_markers;
2122
pub mod rendering;

‎crates/sqlbuild-analysis/src/compiler/_helpers/sql_tests/planning.rs‎

Lines changed: 72 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,9 @@ use crate::compiler::_helpers::sql_tests::reader_references::{
3131
CompilerReads, assertion_ref_targets, calls_reference, unresolvable_call,
3232
unresolved_reference_error,
3333
};
34+
use crate::compiler::_helpers::sql_tests::real_model_limit::{
35+
RealModelChain, real_model_limit_message,
36+
};
3437
use crate::compiler::_helpers::sql_tests::relation_markers::relation_marker_calls;
3538
use crate::compiler::_helpers::sql_tests::rendering::{
3639
RenderRequest, render_comparison_sql, render_dialect, rendered_chain_steps,
@@ -94,6 +97,7 @@ struct ProjectContext {
9497
patterns: SqlTestPatterns,
9598
render_dialect: Arc<Dialect>,
9699
rejects_nested_with: bool,
100+
max_real_models: Option<usize>,
97101
/// Each model query's reserved CTE name, since every test reading the model checks it.
98102
reserved_ctes: Arc<Mutex<HashMap<String, Option<String>>>>,
99103
}
@@ -469,7 +473,7 @@ pub(crate) fn resolve_chains(request: SqlTestChainBatch) -> Result<Vec<Vec<Strin
469473
let fixtures = classify_fixtures(authored_ctes, expected_ctes, assertion_ctes);
470474
let expected_names =
471475
chain_root_names(expected_model_names, &reads, &fixtures, &patterns);
472-
Ok(topo_sort_model_chain(TopoSortRequest {
476+
Ok(topo_sort_model_chain(&TopoSortRequest {
473477
expected_names: &expected_names,
474478
models: &models,
475479
overrides: &model_query_overrides,
@@ -537,6 +541,7 @@ pub(crate) fn plan_batch(request: SqlTestPlanBatch) -> Result<SqlTestPlanBatchOu
537541
patterns: SqlTestPatterns::new(request.lexical_syntax)?,
538542
render_dialect,
539543
rejects_nested_with,
544+
max_real_models: request.max_unit_test_real_models,
540545
reserved_ctes: Arc::new(Mutex::new(HashMap::new())),
541546
};
542547
let workers = request.workers.clamp(1, MAX_WORKERS);
@@ -683,6 +688,8 @@ fn plan_direct_test(
683688
set_difference_operator: context.set_difference_operator.clone(),
684689
sql_analysis_dialect: Some(context.dialect.clone()),
685690
probe_step_index: None,
691+
actual_relations: HashMap::new(),
692+
output_step_index: None,
686693
};
687694
Ok(PlannedResponse {
688695
request,
@@ -755,13 +762,50 @@ fn plan_model_test(
755762
&fixtures,
756763
&context.patterns,
757764
);
758-
let ordered_names = topo_sort_model_chain(TopoSortRequest {
765+
let topo_request = TopoSortRequest {
759766
expected_names: &expected_names,
760767
models: &context.models,
761768
overrides: &plan.model_query_overrides,
762769
mock_refs: &fixtures.mock_refs,
763770
patterns: &context.patterns,
764-
});
771+
};
772+
let ordered_names = topo_sort_model_chain(&topo_request);
773+
if let Some(limit) = context.max_real_models {
774+
let real_models: Vec<String> = ordered_names
775+
.iter()
776+
.filter(|name| context.models.contains_key(*name))
777+
.cloned()
778+
.collect();
779+
let chain = RealModelChain {
780+
test_name: &plan.test_name,
781+
file_label: &plan.file_label,
782+
roots: &expected_names,
783+
real_models: &real_models,
784+
limit,
785+
};
786+
if let Some(message) =
787+
real_model_limit_message(&chain, |name| real_model_inputs(&topo_request, name))
788+
{
789+
return Ok(PlannedResponse {
790+
request: RenderRequest {
791+
chain: Vec::new(),
792+
assertions: Vec::new(),
793+
sql_analysis_enabled: context.sql_analysis_enabled,
794+
set_difference_operator: context.set_difference_operator.clone(),
795+
sql_analysis_dialect: Some(context.dialect.clone()),
796+
probe_step_index: None,
797+
actual_relations: HashMap::new(),
798+
output_step_index: None,
799+
},
800+
model_names: real_models,
801+
warnings: vec![SqlTestPlanWarning {
802+
model_name: None,
803+
severity: "error",
804+
message,
805+
}],
806+
});
807+
}
808+
}
765809
let mut warnings: Vec<SqlTestPlanWarning> = Vec::new();
766810
let mut reported_missing_mocks: HashSet<(&'static str, String)> = HashSet::new();
767811
let mut reachable_mocks: HashSet<String> = HashSet::new();
@@ -928,6 +972,8 @@ fn plan_model_test(
928972
set_difference_operator: context.set_difference_operator.clone(),
929973
sql_analysis_dialect: Some(context.dialect.clone()),
930974
probe_step_index: None,
975+
actual_relations: HashMap::new(),
976+
output_step_index: None,
931977
};
932978
Ok(PlannedResponse {
933979
request,
@@ -1302,28 +1348,31 @@ fn classify_fixtures(
13021348
fixtures
13031349
}
13041350

1305-
fn topo_sort_model_chain(request: TopoSortRequest<'_>) -> Vec<String> {
1351+
fn topo_sort_model_chain(request: &TopoSortRequest<'_>) -> Vec<String> {
13061352
let mut roots: Vec<String> = request.expected_names.to_vec();
13071353
roots.sort();
1308-
depth_first_postorder(roots, |name| {
1309-
let Some(model) = request.models.get(name) else {
1310-
return Vec::new();
1311-
};
1312-
let mut dependencies: Vec<String> = match request.overrides.get(name) {
1313-
Some(override_sql) => marker_names(
1314-
&request.patterns.reference,
1315-
&request.patterns.lexical,
1316-
override_sql,
1317-
),
1318-
None => model.model_dependencies.clone(),
1319-
};
1320-
dependencies.retain(|dependency| {
1321-
!request.mock_refs.contains_key(dependency) && request.models.contains_key(dependency)
1322-
});
1323-
dependencies.sort();
1324-
dependencies.dedup();
1325-
dependencies
1326-
})
1354+
depth_first_postorder(roots, |name| real_model_inputs(request, name))
1355+
}
1356+
1357+
/// The unmocked project models one chain model reads, sorted and deduplicated.
1358+
fn real_model_inputs(request: &TopoSortRequest<'_>, name: &str) -> Vec<String> {
1359+
let Some(model) = request.models.get(name) else {
1360+
return Vec::new();
1361+
};
1362+
let mut dependencies: Vec<String> = match request.overrides.get(name) {
1363+
Some(override_sql) => marker_names(
1364+
&request.patterns.reference,
1365+
&request.patterns.lexical,
1366+
override_sql,
1367+
),
1368+
None => model.model_dependencies.clone(),
1369+
};
1370+
dependencies.retain(|dependency| {
1371+
!request.mock_refs.contains_key(dependency) && request.models.contains_key(dependency)
1372+
});
1373+
dependencies.sort();
1374+
dependencies.dedup();
1375+
dependencies
13271376
}
13281377

13291378
fn analyze_and_resolve_sql(
Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
//! The compile error for a unit test that executes more real models than the project allows.
2+
3+
use std::collections::{BTreeSet, HashMap, VecDeque};
4+
5+
/// The project setting that bounds how many real models one unit test may execute.
6+
pub(crate) const REAL_MODEL_LIMIT_KEY: &str = "unit_test_max_real_models";
7+
8+
/// One unit test's chain, measured against the configured real-model limit.
9+
pub(crate) struct RealModelChain<'a> {
10+
pub(crate) test_name: &'a str,
11+
pub(crate) file_label: &'a str,
12+
pub(crate) roots: &'a [String],
13+
pub(crate) real_models: &'a [String],
14+
pub(crate) limit: usize,
15+
}
16+
17+
/// The explanation of an oversized chain, or `None` when the chain is within the limit.
18+
pub(crate) fn real_model_limit_message<F>(chain: &RealModelChain<'_>, inputs: F) -> Option<String>
19+
where
20+
F: Fn(&str) -> Vec<String>,
21+
{
22+
let count = chain.real_models.len();
23+
if count <= chain.limit {
24+
return None;
25+
}
26+
let roots: Vec<&String> = chain
27+
.roots
28+
.iter()
29+
.filter(|root| chain.real_models.contains(root))
30+
.collect();
31+
let paths = dependency_paths(&roots, &inputs);
32+
let width = chain.real_models.iter().map(String::len).max().unwrap_or(0);
33+
let mut lines: Vec<String> = vec![
34+
format!(
35+
"SQL unit test '{}' executes {count} real models (limit {}).",
36+
chain.test_name, chain.limit
37+
),
38+
format!(" Test file: {}", chain.file_label),
39+
format!(
40+
" Model under test: {}",
41+
roots
42+
.iter()
43+
.map(|root| root.as_str())
44+
.collect::<Vec<_>>()
45+
.join(", ")
46+
),
47+
String::new(),
48+
"Real models executed, and the dependency path that pulls each one in:".to_owned(),
49+
];
50+
for model in chain.real_models.iter().rev() {
51+
let reason = match paths.get(model.as_str()) {
52+
Some(path) if path.len() == 1 => "model under test".to_owned(),
53+
Some(path) => path.join(" -> "),
54+
None => "read by the test".to_owned(),
55+
};
56+
lines.push(format!(" {model:<width$} {reason}"));
57+
}
58+
lines.push(String::new());
59+
lines.push("Options:".to_owned());
60+
lines.extend(mock_advice(&roots, chain.real_models, &inputs));
61+
lines.extend(scenario_advice());
62+
lines.extend([
63+
" 3. Raise the limit in sqlbuild_project.toml (or sqlbuild_local.toml):".to_owned(),
64+
" [settings]".to_owned(),
65+
format!(" {REAL_MODEL_LIMIT_KEY} = {count}"),
66+
]);
67+
Some(lines.join("\n"))
68+
}
69+
70+
/// Why a scenario may suit a multi-model check, as the second option.
71+
fn scenario_advice() -> [String; 4] {
72+
[
73+
" 2. If this test checks behaviour across several models, consider a scenario".to_owned(),
74+
" (tests/scenarios/). A unit test inlines every real model into one statement,"
75+
.to_owned(),
76+
" so the warehouse compiles and optimises the whole chain at once; a scenario"
77+
.to_owned(),
78+
" builds each model step by step, compiling each statement separately.".to_owned(),
79+
]
80+
}
81+
82+
/// The direct real inputs of each model under test, which a test mocks to cut the chain.
83+
fn mock_advice<F>(roots: &[&String], real_models: &[String], inputs: &F) -> Vec<String>
84+
where
85+
F: Fn(&str) -> Vec<String>,
86+
{
87+
let mut seen: BTreeSet<String> = BTreeSet::new();
88+
let mut mocks: Vec<String> = Vec::new();
89+
for root in roots {
90+
for input in inputs(root) {
91+
let is_root = roots.iter().any(|candidate| **candidate == input);
92+
if !is_root && real_models.contains(&input) && seen.insert(input.clone()) {
93+
mocks.push(input);
94+
}
95+
}
96+
}
97+
if mocks.is_empty() {
98+
return vec![
99+
" 1. Mock models further up the paths above with __ref__<model> CTEs.".to_owned(),
100+
];
101+
}
102+
let mut lines: Vec<String> =
103+
vec![" 1. Mock the direct inputs of the model under test to cut the chain:".to_owned()];
104+
for mock in mocks {
105+
lines.push(format!(" __ref__{mock} AS (SELECT ...)"));
106+
}
107+
lines
108+
}
109+
110+
/// The shortest read path from a model under test to each real model, breadth first.
111+
fn dependency_paths<F>(roots: &[&String], inputs: &F) -> HashMap<String, Vec<String>>
112+
where
113+
F: Fn(&str) -> Vec<String>,
114+
{
115+
let mut paths: HashMap<String, Vec<String>> = HashMap::new();
116+
let mut queue: VecDeque<String> = VecDeque::new();
117+
for root in roots {
118+
if !paths.contains_key(root.as_str()) {
119+
paths.insert((*root).clone(), vec![(*root).clone()]);
120+
queue.push_back((*root).clone());
121+
}
122+
}
123+
while let Some(model) = queue.pop_front() {
124+
let path = paths.get(&model).cloned().unwrap_or_default();
125+
for input in inputs(&model) {
126+
if paths.contains_key(&input) {
127+
continue;
128+
}
129+
let mut extended = path.clone();
130+
extended.push(input.clone());
131+
paths.insert(input.clone(), extended);
132+
queue.push_back(input);
133+
}
134+
}
135+
paths
136+
}

0 commit comments

Comments
 (0)