Repository navigation
feat(verify): replay only changed cases and run one required Verify check - #162
Merged
Merged
Conversation
The hazard case generator reads crates/sqlquack-policy/policy.toml since the policy moved there, but its tests still wrote a root sqlquack.toml, so all 14 failed before generating anything.
… in parallel `sqlquack verify --replay-cache <file>` keeps each converted case's judgement under a digest of everything it is computed from: the case, its recording's metadata, the policy, the rules' failure modes, the converted SQL with its setup, provenance, raises, Python functions and layout, and a caller-given --replay-scope. A later run reuses a judgement whose key is unchanged and replays only the rest, so a rule fix replays only the cases whose SQL it changed. Cases expecting a divergence always replay, because their row check converts each row on its own. A file of another format is ignored. scripts/replay_scope declares the paths whose change can alter a judgement without changing any conversion: the lock files and toolchain, the evidence pin, and the replay, comparison, policy and verify code. Its digest of their contents is the scope, so a change under them replays every case. It also tells a change that needs no replay, because it touches no crate the sqlquack binary is built from, and one that needs only the site's checks. check-ci fails when a declared path matches no tracked file. Verify now judges cases in parallel, and skips the replay of the laid-out SQL when DuckDB parses it to the same tree as the single-line SQL. Verify on every recorded case takes 338 s instead of 942 s, and 120 s when every judgement is reused, with the same report, statements, sizes and coverage. verify-recorded and translation-results run the gate build's binaries instead of building again in another profile. New make targets: verify-changed, test-crate CRATE=<crate>, and help entries for check-merge and python-api.
CI runs on pull requests, on a nightly schedule and on dispatch. A first job runs the clutter guard on every change and classifies it with scripts/replay_scope, and Verify passes only when every job the change needs passed, a skipped job counting only when it was not needed: - documentation and website changes build and check the site, through the new reusable Website Build workflow that Website also deploys from; - code changes run check-merge, the Python tests and the replay, which restores the replay cache saved on the base branch and replays only cases whose judgement inputs changed, or every case after a full-replay change; - the nightly run replays every case and saves the cache, converts DuckShop within its time budget and fuzzes. The DuckShop job no longer compares with polyglot, which SQLQuack no longer uses. Release Please dispatches CI, the version guard and the metadata check on each release pull request, which a token-opened pull request does not trigger, so every release replays every case first. Pull Request Metadata validates the branch, Conventional title and Why, Changes and Verification sections; make check-pr-metadata runs it locally. Block automatic 1.0 releases fails when the workspace version reaches 1.0.0.
Distributions built every platform and the sdists, two of them on macOS, for any pull request touching packaging or Cargo.lock. Pull requests now build and smoke test the linux-x86_64 wheels, as every push to main does, with the DuckDB-compiled convert tests; every platform and the sdists build nightly, for each release and on dispatch.
…acks it The Rust build cache keeps libduckdb-sys' recorded build-script output but not target/duckdb-download, where that build script puts libduckdb.so. A warm run therefore never downloads the library, and make python-module cannot copy it. Before building, the checks and DuckShop jobs now remove the fingerprint of each recorded run whose library is missing, so Cargo runs the build script again.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Every change replayed all 12,236 recorded cases, and CI was switched off. This brings back CI that can be required, sized to each change.
Changes
verify --replay-cachereuses the judgement of every case whose conversion and recording are unchanged, keyed on a digest of what the judgement is computed from. A rule fix replays only the cases whose SQL it changed.scripts/replay_scopedeclares the paths that force a full replay (lock files, evidence pin, replay, comparison and policy code); their digest scopes the cache. Changes outside the binary's crates replay nothing; docs and website changes run only the site checks.Verifythat resolves when jobs skip. Nightly replays everything, refreshes the cache, converts DuckShop and fuzzes.Verification