Repository navigation
feat!: remove virtual data environments and validate models at compile time - #852
Merged
Merged
Conversation
Add code-owned help, authored spans, schema and operand notes, SQL-test snippets, and detailed cold/warm partial-check notices. Preserve independent downstream errors with column-specific recovery. Avoid all-project schema construction per model and reuse identical SQL-test body analysis. The 1000-model guard measures 12.14s versus 8.73s on bc9a5fc, within the unchanged 14s budget. Update the dense JSON golden explicitly for the intentional empty semantic_checks_partial field: 99f2f5c6 becomes 1a9ce1be. Fingerprint normalization and time/RSS thresholds remain unchanged.
Accept temporal cursor families and honor Snowflake quoted-identifier session settings in offline and physical binding, including cache identity. Replace repeated character-level diagnostic alignment with lazy cached token maps. Reuse native root diagnostics while propagating unknown types through output lineage; retain independent downstream errors and avoid cascading contract warnings. Add synthetic integration coverage for temporal types, target settings, physical rebinding, repetitive SQL spans, transitive recovery, and warm-cache parity. Preserve the existing performance thresholds and semantic golden.
Fuse full semantic validation with compact native analysis when input schemas are unchanged. Preserve required rebinding and cache identity, skip empty batches, and batch SQL-test interface inference. Add CI-wired guards for large diagnostic-heavy models, position-mapping scaling, and deep poisoned DAGs. Match repeated diagnostic messages by occurrence and attribute cascades through the failing output's lineage. Verify cold fusion, unchanged warm-cache reuse, and schema invalidation with synthetic integration tests. Keep existing dense performance ceilings and semantic fingerprints unchanged.
BREAKING CHANGE: Projects run only in direct mode. Virtual commands, flags, configuration keys, state backends, and prepare_version hooks are removed. Concurrent microbatch event IDs changed; drop existing _sqlbuild_microbatches tables before the next concurrent run. Sequential runs never use this table.
Remove unused public facades and their exclusive planning helpers, models, and tests. Preserve live direct-mode sibling interfaces as compiler-internal entrypoints and route direct identity hashing to its existing implementation.
Compose nested normalization edits and translate native unwinds to named compiler errors. Preserve inferred and physical column identity, apply dialect-aware metadata coercion and field matching, and rebind affected downstream shapes before execution.
Compared normalized CLI JSON and every compiled artifact between release base 7b710a9 (0.119.1) and the combined branch for 3000, 5000 and 10000 models. The sole JSON difference is the intentional empty semantic_checks_partial field; compiled artifact paths and content hashes are identical. Base fingerprints match the former expectations in all three cases. The 1000 entry already accounts for this field. No time or RSS budgets changed.
Fingerprint comparisons exposed stale compact projections replacing completed derived-star output on unrelated edits, and CTE star projections losing their syntactic has_star flag. Reuse completed entry results for partial as well as complete batch hits, retain root-star identity, and invalidate older analysis cache records. Real CLI regressions compare edited cached output with an uncached oracle.
…lysis Compared normalized base 7b710a9 and HEAD JSON plus SHA-256 maps of every compiled artifact for fresh and cached 3000/5000/10000 projects and all varied-1000 edits. Varied output differs only by the intentional empty semantic_checks_partial field. Fresh/cache differences are that field plus CHI-373 inferred-schema completion: 150/250/500 formerly zero-column derived-star shapes and 185/312/628 recovered lineage edge counts. SQL, dependency lists, all other output fields, and compiled artifact maps agree between engines in every phase. Repair invalid benchmark SQL first by qualifying input.id in projections, predicates and macro arguments. DuckDB rejects the previous unqualified id when a seed is joined. Both engines used this same corrected fixture for comparison. Partial-cache projection drift and the conflation of syntactic stars with unresolved expansion were fixed separately. Leaf edits now change only their authored query, matching the uncached result. No time, RSS or cache-size budget changed.
Carry native bound-star completeness through compact analysis and entry caches separately from the syntactic has_star flag. Preserve native EXCLUDE and REPLACE provenance instead of appending excluded source columns during Python fallback. Verified the real CLI/DuckDB bound projection regressions and cached-versus-uncached shape regressions.
The legacy docs folder was retired in favour of the website docs. Port the semantic compilation reference and the direct-mode and implicit-conversion upgrade notes to website pages, add them to the sidebar, drop the branch's remaining legacy docs files, and regenerate the bundled skill references from the website.
Absolute compile budgets must absorb large hosted-runner variance, so real regressions of 30-50% passed or failed depending on the runner. Build the pull request's base and head in one job, generate the dense or fresh 3,000-model project once, warm both builds, then alternate three cold compiles each. Fail when the head's median wall or CPU time exceeds the base's by more than 10%, and report medians, ratios, the runner CPU and per-phase timings in the job summary. Refs CHI-403.
Polyglot 0.13.5 type-checks star-sourced columns in every query through ordered CTE and source star expansion instead of requalifying only statements with a set operation; 0.13.6 skips requalification for stars no scope consumes.
…ning The rebase onto main added the dollar_quotes quote-policy field. SQL analysis normalization already skips $$ literals itself; opt into the compiler's dollar-quote handling so tagged literals are skipped the same way, and cover a tagged literal containing an apostrophe end to end.
The removed configuration keys already fail with explicit errors, and the implicit-conversion guidance lives on the semantic compilation page, which applies to new and existing projects alike.
make check-ci and make rust-check reported two fensu faults introduced by the star-lineage work: a multi-line docstring on _complete_inferred_bindings (FFH001) and a filtered generator in the new lineage integration test (FFT104). Name the native-expansion condition instead of documenting it, and index compiled models by name in the test. No behaviour change.
…r compile Assembly inferred each expression source in its own native call, and the coverage, recovery and detail passes re-inferred every source under a default profile. Batch the sources and key the compile catalog's cache by expression, so later passes reuse the shapes compile validated against.
The CTE namespace lowercased and stored every token of every test's SQL to avoid collisions, although only tokens with the generated-name prefix can collide. Keep only those, and move CTE bodies into the rename instead of copying them. Rendered test artifacts are unchanged.
Binding-mode models whose relation-stubbed SQL and effective analysis and binding input shapes are equal now share one native query and template. A shared result is reused only when the representative analysis succeeded without any diagnostic; otherwise each member is re-analysed with its own relation names. Qualified, quoted, colliding or otherwise lossy names keep the per-model query. Batches with recomputed members are not published to the compact-batch cache.
Exact representative results of shared binding queries are remembered per binding catalog, keyed by stubbed SQL, effective input shapes and template inputs, and projected onto later members through their own resource names without another native call or fact decoding. Only results that analysed successfully without diagnostics are remembered, and batches that publish the compact-batch cache do not reuse them. Candidates are gated by a cheap compile-wide relation-marker pre-key so projects that barely share skip the full key.
… SQL The virtual-state path removal dropped the PostgreSQL case pattern, which left the suite-selection script unparsable on pull requests. Separately, binding-query sharing only checked reference names for the stub prefix, so an authored CTE with a stub name could capture the shared reference.
…rors Implicit conversions are now blocking errors, and literal casts carry no source position, so the workload's literal-cast warnings collapsed into one statement-level error. Use column comparisons so every model still produces 100 independently positioned errors.
…sured values Chained SQL test CTE isolation renames lifted CTEs in compiled test artifacts, which changes every compile fingerprint; compile output is otherwise unchanged. Fresh cold compiles still cost more than before semantic validation, so the fresh-process and cache budgets and the fresh same-runner ratio limit are set to the measured values until the remaining gap is closed. Dense limits are unchanged.
kvlonge
enabled auto-merge (squash)
September 27, 2026 12:15
…ttle fresh budgets Dependency-wave analysis caches each wave as its own batch, so the edit compile republishes the edited wave and the next compile is served entirely from batch hits instead of one entry hit. Assert a complete cache hit, as the macro-edit path already does. Raise the fresh 5000 and cache 5000 cold budgets to just above the slowest CI runs.
…5000 budget Wave-level analysis batches serve every model outside the edited one from batch hits, so a macro edit re-analyses exactly one model with no entry hits. The miss count and complete-hit checks still catch any extra re-analysis.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Virtual data environments added a second execution model that SQLBuild no longer needs, and offline compile accepted models whose columns or types did not match their inputs, so errors surfaced in the warehouse.
Changes
_sqlbuild_microbatchestables once, before the first concurrent run).BREAKING CHANGE: virtual data environments are removed, and implicit type conversions now fail compile.
Verification
make testandmake check-cipass; real-warehouse PostgreSQL suite passes.