Repository navigation
fix(scenario): honour promotion mode and run scenarios concurrently - #984
Merged
Merged
Conversation
Scenario models were always materialized with immediate table promotion, so any table model with an enforced contract failed with K011 and declared type enforcement failed with R004 even when the project configured staged promotion. Scenarios now resolve the same promotion mode as build (project table_promotion_mode, else the adapter default) and use staged promotion whenever a model's contract is enforced, so the runtime contract check runs and the scenario proceeds to its expectations. Cleanup also drops the staging relation a failed staged promotion keeps for inspection.
Scenario runs issued a blind DROP IF EXISTS for every planned relation both before and after each scenario, ensured the destination schema before every fixture and model, and ran strictly one scenario at a time. - Cleanup reads the catalog once per database and drops only the scenario-owned relations that exist, using the catalogued relation type, so a retained relation whose materialization changed is still replaced. When the catalog cannot be read, cleanup falls back to the previous blind drops. - Scenario destination schemas are ensured once per invocation. - An interrupt or unexpected error inside a scenario still drops that scenario's relations before propagating (unless --retain). - Independent scenarios run concurrently up to --concurrency or the project concurrency setting, each on its own connection; results are rendered in selection order. Scenario artifacts are already namespaced per scenario. - Text output reports each scenario's wall time and the total; --json adds scenarios[].duration_ms and summary.duration_ms. Nested model progress rows are labelled with their scenario so concurrent rows stay unambiguous.
Scenario planning already expands SELECT * FROM __empty_fixture() into the relation's declared, typed columns, as unit tests do, but the SQL Rules engine only exempted the marker in TEST files, so scenarios using it failed SQBRSQL021. Scenario files now receive the same exemption.
Staging, type-enforcement, delta and snapshot-delta relations appended a suffix to the target name without fitting the result. On Postgres (63-char identifiers) a long model name truncated the staging name to the target name, so staged promotion failed with R007, and slightly shorter names leaked a truncated staging table that cleanup could not find. Every auxiliary name now goes through fit_auxiliary_relation_name, which keeps the suffix and replaces the overflow with a stable hash. Build, load and scenario cleanup all use the same fitted names.
Adapters now declare whether list_relations covers a scope. Cleanup lists only authoritative scopes, matches schema and name case-insensitively, and falls back to DROP IF EXISTS for every other target, including the type-enforcement relation.
…ch build K011 The first Ctrl-C prints one notice, cancels queued scenarios and stops running ones at their next step boundary so they clean up before the connections close; repeat presses are absorbed. Enforced contracts with immediate promotion now fail with K011 exactly as sqb build does.
An enforced contract on a table or non-microbatch incremental model under immediate table promotion is now a compile diagnostic, so compile, plan, build and scenario test (including --local) fail before any connection or statement. The help names the file and current value, shows the exact staged TOML, and offers removing the enforced contract. Build and plan failures now carry the help when every error shares it. The run-time guard stays as a defensive check for paths the compile check does not cover.
… bytes Running every scenario through a worker pool and absorbing repeat interrupts made long statements uninterruptible, even at concurrency 1. Concurrency 1, --local replay and capture now run on the main thread again, so Ctrl-C reaches the driver as before. A SIGINT handler records the stop and cancels in-flight statements through the new adapter interrupt_connection (DuckDB interrupt, Postgres cancel_safe); without it a DuckDB query interrupted only by the signal kept running in the background and blocked cleanup. Concurrent runs cancel queued scenarios, cancel or close in-flight worker connections, wait for teardown, and sweep started scenarios on a valid or fresh connection. A second Ctrl-C prints one notice and exits at once; daemon workers never block exit, and the next run of the same scenarios drops what was left behind. Identifier fitting now measures UTF-8 bytes and cuts on character boundaries, so non-ASCII names fit Postgres's 63-byte limit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
sqb scenario testran every scenario one at a time with many redundant statements, failed enforced-contract models with K011 even when staged promotion was available, and rejected__empty_fixture(). Auxiliary relation names (__staging,__enforced,__delta) could also exceed the identifier limit and be truncated onto the target (R007 on Postgres).Changes
compile,plan,buildandscenario testbefore any warehouse work. Scenarios otherwise resolve promotion likesqb build.--concurrency N), schemas are ensured once per run, and cleanup reads the catalog once and drops only existing relations. Rows and the summary show wall time; JSON addsduration_ms.fit_auxiliary_relation_namekeeps the suffix and a stable hash within the adapter limit, measured in UTF-8 bytes; build, load and scenario cleanup share it.lists_relations_authoritatively; cleanup falls back toDROP ... IF EXISTSoutside authoritative scopes and matches names case-insensitively.--localand capture run on the main thread.__empty_fixture()is allowed in scenario fixtures.Verification
ruff,ty check src tests,fensu check,fensu dupes,make check-ci.