Repository navigation
feat: bound SQL unit test and model execution cost and attach audit metadata - #1141
Merged
Merged
Conversation
Limit the real models one SQL unit test may execute ([settings] unit_test_max_real_models, default 10) with a compile error that names every real model, its dependency path, the inputs to mock, when a scenario may fit better and the exact key to raise. Evaluate each compared actual output once into a temporary relation (DuckDB, MotherDuck, PostgreSQL; Snowflake reads RESULT_SCAN) so counts, both difference directions, column probes and samples read it. Add per-item time limits: test_timeout (default 30s) and scenario_timeout in [settings] and target execution_limits, TEST()/SCENARIO() timeout, --test-timeout and --scenario-timeout, and per-model max_duration in MODEL(), [path_defaults] and [defaults]. At expiry SQLBuild cancels the running statement, reports TIMEOUT with the limit, its source, the cancellation outcome and query IDs, and continues with independent work. Every built-in adapter now enforces deadlines, including the target's whole-build max_duration: Snowflake, PostgreSQL, BigQuery and Databricks bound statements server-side, and all adapters cancel in-flight work.
Audit cases and SQL audit attachments accept an optional JSON mapping that is reported with each result but never reaches audit SQL, evaluation, thresholds or audit fingerprints. Results store it in a new nullable audit_meta_json column, added to existing result tables only when a result carries meta; audit_completed events and JSON check output include it as audit_meta. Non-JSON meta is a compile error that names the offending value.
StrictAdapter now provides create_sql_test_output and drop_sql_test_output defaults that store nothing, so the executor keeps the inline comparison query and project adapters need no new methods. Built-in adapters that support single-read comparison override them.
…ummaries The sqb build footer reports TIMEOUT= apart from FAIL=, and build JSON adds summary.timeout_count, with failure_count excluding timeouts. sqb test JSON adds timeout_count likewise. A timeout still fails the run: the exit code is non-zero and the JSON status is failed.
Build JSON summary.failure_count and test JSON summary.fail_count count timed-out items again, as before; timeout_count is an additive subset. The human-readable footers still show FAIL= and TIMEOUT= separately, and the CLI docs explain the difference.
…be read A failing column read while checking for the additive meta column now logs a warning and writes the records without meta instead of failing the write. Remove the unused Python build_read_latest_sql builder.
…s out Rollback and unit-test cleanup now run with every deadline suspended, so a model or scenario whose limit expires inside a transaction rolls it back before the timeout propagates. A connection whose statement was interrupted is probed before reuse by the next item, on build, test and scenario workers alike, and replaced when it is no longer usable. Item limits now count running time from the first warehouse statement and stop while cleanup or a queued BigQuery job waits. Snowflake relies on its statement limit, which excludes queueing, with a client backstop after a grace margin; BigQuery items are bounded by cancellation because a job timeout counts from job creation. Nested limits apply the earliest one, cancellation is issued only while the same statement is still registered, and work that finished before expiry keeps its result. PostgreSQL unit tests store their outputs in an ON COMMIT DROP table of one transaction bounded by SET LOCAL statement_timeout, so no setting outlives a pooled transaction, and a stored output that can no longer be read falls back to the inline comparison.
Add the per-adapter enforcement matrix for item limits, the PostgreSQL transaction-scoped statement limit and single-read transaction, and upgrade notes for the new real-model compile check and for whole-build max_duration being accepted on every built-in adapter.
…er timeouts A connection that can neither be recovered nor replaced after a timeout is now returned to its pool marked broken, so later work fails with the driver's error instead of waiting forever for a lost slot; each later return retries the replacement, and microbatch subworker slots are always released. A PostgreSQL unit test claims its transaction only after BEGIN succeeds, so a refused BEGIN can no longer make a later release roll back a caller's transaction. A queued BigQuery job pauses an item's clock only while every statement under that deadline is queued, so one queued microbatch worker no longer pauses a model whose other workers are running.
kvlonge
enabled auto-merge (squash)
October 18, 2026 10:11
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A SQL unit test could silently inline a long chain of real models into one statement; on Snowflake one such test compiled for up to 33 minutes with nothing bounding it. The comparison also evaluated the tested output three to five times. Audits also lacked structured context (CHI-620, CHI-619).
Changes
[settings] unit_test_max_real_models(default 10): an oversized unit test fails compilation with the dependency chain, inputs to mock, scenario guidance and the setting to raise.max_duration) opt-in. All built-in adapters cancel at the deadline; rollback and cleanup always run first, interrupted connections are checked or replaced. TIMEOUT status and timeout_count; JSON failure counts still include timeouts.execution_limits.max_durationworks on every built-in adapter.metaon audit cases and attachments, stored in a new nullable results column, lifecycle events and --json; not part of evaluation identity.Verification