Repository navigation
Commit dfdadd8
authored
feat: bound SQL unit test and model execution cost and attach audit metadata (#1141)
* feat: bound unit test, scenario and model execution cost
Limit the real models one SQL unit test may execute
([settings] unit_test_max_real_models, default 10) with a compile error
that names every real model, its dependency path, the inputs to mock,
when a scenario may fit better and the exact key to raise.
Evaluate each compared actual output once into a temporary relation
(DuckDB, MotherDuck, PostgreSQL; Snowflake reads RESULT_SCAN) so counts,
both difference directions, column probes and samples read it.
Add per-item time limits: test_timeout (default 30s) and scenario_timeout
in [settings] and target execution_limits, TEST()/SCENARIO() timeout,
--test-timeout and --scenario-timeout, and per-model max_duration in
MODEL(), [path_defaults] and [defaults]. At expiry SQLBuild cancels the
running statement, reports TIMEOUT with the limit, its source, the
cancellation outcome and query IDs, and continues with independent work.
Every built-in adapter now enforces deadlines, including the target's
whole-build max_duration: Snowflake, PostgreSQL, BigQuery and Databricks
bound statements server-side, and all adapters cancel in-flight work.
* docs: document unit test cost guards and per-item time limits
* feat: attach structured meta to audit cases and attachments
Audit cases and SQL audit attachments accept an optional JSON mapping
that is reported with each result but never reaches audit SQL,
evaluation, thresholds or audit fingerprints. Results store it in a new
nullable audit_meta_json column, added to existing result tables only
when a result carries meta; audit_completed events and JSON check
output include it as audit_meta. Non-JSON meta is a compile error that
names the offending value.
* fix: keep unit-test output storage optional for project adapters
StrictAdapter now provides create_sql_test_output and
drop_sql_test_output defaults that store nothing, so the executor keeps
the inline comparison query and project adapters need no new methods.
Built-in adapters that support single-read comparison override them.
* feat: count timed-out models and tests separately in build and test summaries
The sqb build footer reports TIMEOUT= apart from FAIL=, and build JSON
adds summary.timeout_count, with failure_count excluding timeouts.
sqb test JSON adds timeout_count likewise. A timeout still fails the run:
the exit code is non-zero and the JSON status is failed.
* fix: keep JSON failure counts inclusive of timeouts
Build JSON summary.failure_count and test JSON summary.fail_count count
timed-out items again, as before; timeout_count is an additive subset.
The human-readable footers still show FAIL= and TIMEOUT= separately,
and the CLI docs explain the difference.
* fix: write audit results without meta when the result columns cannot be read
A failing column read while checking for the additive meta column now
logs a warning and writes the records without meta instead of failing
the write. Remove the unused Python build_read_latest_sql builder.
* fix: roll back, recover and bound only running time when an item times out
Rollback and unit-test cleanup now run with every deadline suspended, so
a model or scenario whose limit expires inside a transaction rolls it
back before the timeout propagates. A connection whose statement was
interrupted is probed before reuse by the next item, on build, test and
scenario workers alike, and replaced when it is no longer usable.
Item limits now count running time from the first warehouse statement
and stop while cleanup or a queued BigQuery job waits. Snowflake relies
on its statement limit, which excludes queueing, with a client backstop
after a grace margin; BigQuery items are bounded by cancellation because
a job timeout counts from job creation. Nested limits apply the
earliest one, cancellation is issued only while the same statement is
still registered, and work that finished before expiry keeps its result.
PostgreSQL unit tests store their outputs in an ON COMMIT DROP table of
one transaction bounded by SET LOCAL statement_timeout, so no setting
outlives a pooled transaction, and a stored output that can no longer
be read falls back to the inline comparison.
* docs: describe per-item running-time limits and upgrade notes
Add the per-adapter enforcement matrix for item limits, the PostgreSQL
transaction-scoped statement limit and single-read transaction, and
upgrade notes for the new real-model compile check and for whole-build
max_duration being accepted on every built-in adapter.
* fix: keep pool slots, test transactions and shared clocks correct after timeouts
A connection that can neither be recovered nor replaced after a timeout
is now returned to its pool marked broken, so later work fails with the
driver's error instead of waiting forever for a lost slot; each later
return retries the replacement, and microbatch subworker slots are
always released.
A PostgreSQL unit test claims its transaction only after BEGIN
succeeds, so a refused BEGIN can no longer make a later release roll
back a caller's transaction.
A queued BigQuery job pauses an item's clock only while every statement
under that deadline is queued, so one queued microbatch worker no
longer pauses a model whose other workers are running.
* test: list the new execution limit keys in config error goldens1 parent e6a4a44 commit dfdadd8
218 files changed
Lines changed: 6994 additions & 334 deletions
File tree
- crates
- sqlbuild-analysis/src/compiler
- _helpers/sql_tests
- tests
- single_read
- sqlbuild-cache/src/stage_capture
- sqlbuild-cli/src
- argv
- dispatch
- sqlbuild-config/src
- effective
- project
- _helpers
- sqlbuild-core/src
- build_aggregate
- main
- build_progress
- _helpers
- main
- cli_text
- lifecycle_events
- progress
- main
- test_command
- main
- sqlbuild-discovery/src/model_files
- _helpers
- sqlbuild-model-config/src
- execution_limits
- main
- tests
- header_metadata
- _helpers
- main
- tests
- model_validation
- _helpers
- main
- sql_tests
- _helpers
- main
- sqlbuild-python/src/bindings/_helpers
- analysis
- authored_values
- build_command
- compile_records
- compiled_project
- contract_records
- discovery
- model_config
- native_host
- planner_records
- planner_strategy
- records
- result_output
- rules
- scenario_command
- sql_test_command
- sql_test_glue
- src/sqlbuild
- .agents/skills/sqlbuild/references/docs/concepts
- adapters
- bigquery
- classes
- databricks/classes
- postgres
- classes
- snowflake
- classes
- sqlserver/classes
- adapter/contract
- _helpers
- classes
- compiler/compile
- executor
- audit_results
- _helpers
- main
- auditing
- _helpers
- build
- _helpers
- classes
- pipeline/_helpers
- run/_helpers/reuse
- scenario
- _helpers/execution
- scheduling/classes
- testing
- _helpers
- classes
- main
- python_nodes
- runtime
- execution_limits
- _helpers
- classes
- main
- observability/classes
- tests
- e2e/src/sqlbuild/cli/commands/main
- audit
- build
- no_tests_no_audits
- test
- goldens/compiler/failure
- integration/src/sqlbuild
- compiler/compiled_project/expected
- executor/audit_results/main
- unit/src/sqlbuild
- adapters
- postgres
- adapter/contract/classes/strict_adapter
- cli
- commands/main/execution
- output/main/build_execution_json
- compiler/auditing/main
- executor
- audit_results/_helpers
- auditing/main
- pipeline/_helpers
- scheduling/classes
- python_nodes/main
- runtime/execution_limits
- website/src/content/docs/docs
- cli
- concepts
- models
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
| 19 | + | |
19 | 20 | | |
20 | 21 | | |
21 | 22 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
31 | 31 | | |
32 | 32 | | |
33 | 33 | | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
34 | 37 | | |
35 | 38 | | |
36 | 39 | | |
| |||
94 | 97 | | |
95 | 98 | | |
96 | 99 | | |
| 100 | + | |
97 | 101 | | |
98 | 102 | | |
99 | 103 | | |
| |||
469 | 473 | | |
470 | 474 | | |
471 | 475 | | |
472 | | - | |
| 476 | + | |
473 | 477 | | |
474 | 478 | | |
475 | 479 | | |
| |||
537 | 541 | | |
538 | 542 | | |
539 | 543 | | |
| 544 | + | |
540 | 545 | | |
541 | 546 | | |
542 | 547 | | |
| |||
683 | 688 | | |
684 | 689 | | |
685 | 690 | | |
| 691 | + | |
| 692 | + | |
686 | 693 | | |
687 | 694 | | |
688 | 695 | | |
| |||
755 | 762 | | |
756 | 763 | | |
757 | 764 | | |
758 | | - | |
| 765 | + | |
759 | 766 | | |
760 | 767 | | |
761 | 768 | | |
762 | 769 | | |
763 | 770 | | |
764 | | - | |
| 771 | + | |
| 772 | + | |
| 773 | + | |
| 774 | + | |
| 775 | + | |
| 776 | + | |
| 777 | + | |
| 778 | + | |
| 779 | + | |
| 780 | + | |
| 781 | + | |
| 782 | + | |
| 783 | + | |
| 784 | + | |
| 785 | + | |
| 786 | + | |
| 787 | + | |
| 788 | + | |
| 789 | + | |
| 790 | + | |
| 791 | + | |
| 792 | + | |
| 793 | + | |
| 794 | + | |
| 795 | + | |
| 796 | + | |
| 797 | + | |
| 798 | + | |
| 799 | + | |
| 800 | + | |
| 801 | + | |
| 802 | + | |
| 803 | + | |
| 804 | + | |
| 805 | + | |
| 806 | + | |
| 807 | + | |
| 808 | + | |
765 | 809 | | |
766 | 810 | | |
767 | 811 | | |
| |||
928 | 972 | | |
929 | 973 | | |
930 | 974 | | |
| 975 | + | |
| 976 | + | |
931 | 977 | | |
932 | 978 | | |
933 | 979 | | |
| |||
1302 | 1348 | | |
1303 | 1349 | | |
1304 | 1350 | | |
1305 | | - | |
| 1351 | + | |
1306 | 1352 | | |
1307 | 1353 | | |
1308 | | - | |
1309 | | - | |
1310 | | - | |
1311 | | - | |
1312 | | - | |
1313 | | - | |
1314 | | - | |
1315 | | - | |
1316 | | - | |
1317 | | - | |
1318 | | - | |
1319 | | - | |
1320 | | - | |
1321 | | - | |
1322 | | - | |
1323 | | - | |
1324 | | - | |
1325 | | - | |
1326 | | - | |
| 1354 | + | |
| 1355 | + | |
| 1356 | + | |
| 1357 | + | |
| 1358 | + | |
| 1359 | + | |
| 1360 | + | |
| 1361 | + | |
| 1362 | + | |
| 1363 | + | |
| 1364 | + | |
| 1365 | + | |
| 1366 | + | |
| 1367 | + | |
| 1368 | + | |
| 1369 | + | |
| 1370 | + | |
| 1371 | + | |
| 1372 | + | |
| 1373 | + | |
| 1374 | + | |
| 1375 | + | |
1327 | 1376 | | |
1328 | 1377 | | |
1329 | 1378 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
0 commit comments