Repository navigation
refactor(compiler): remove the Python engine and remaining M4 algorithms - #1081
Merged
Merged
Conversation
…ranches Every native stage is shipped, so the python engine switch, the NativeStage members it gated and their native-answer counters no longer select anything. SQLBUILD_COMPILER_ENGINE=python and --compiler-engine python now stop with an error that explains the removal. Python branches that only the python engine reached are deleted: JSON SQL-test planning and chain resolution, Python promotion conflicts, the whole-project contract fallback and the attachment else-branches. Native deferrals still hand their inputs to the Python helpers. The compiler differential now compares native with native-preview plus the goldens, and the phase-CPU ratio jobs compare against the base commit.
…ge on native Tests that compared the removed python engine with native now either compare native with native-preview, keeping every explicit expectation, or compare native with the Python fallback it still defers to (project assembly, SQL-test assembly, contracts, semantic completion, type normalisation). Pure parity tests whose Python side was deleted are removed; goldens and the corpus pin that output. The Rust JSON planning entry points become test-only helpers now that their bindings are gone.
…swer counts The shipped stages no longer report native answers, so their <stage>.native entries leave the allow-list. No fallback entry or count changes and every golden is unchanged.
A ref cycle no longer hands the whole project to Python. Like Python's dataflow, the session analyses every model in one unordered wave over the shapes known up front and, like Python's completion pass on a CycleError, keeps those analyses without enrichment or binding re-validation. The cycle flag is part of the analysis-cache key.
…'s Unicode semantics Generated casefold/upper tables per Unicode version, the running Python's text semantics recorded at import, all Polyglot dialects compiled in, arbitrary-size type parameters, stack growth for deep types, and Python's errors for unknown dialects. Deletes the Python type normalization and per-model contract fallbacks.
…ly top-level type fields
…l adapter nullability rules back Python's casefold/upper in CTE facts, legacy analysis and dynamic pivot proofs; pivot parse errors reported as Python reports them; shipped safe_cast rule as data; an adapter's own rules run through an explicit callback whose exceptions reach the caller, bypassing the analysis cache as Python's did.
…red projects Unknown dialect names raise Python's ValueError, projects without an analysis catalog run on a fresh analysis pool, and column references are walked iteratively. Deletes the whole-project Python FAST lineage fallback; the per-model handback for native failures stays.
…atively Project assembly answers per model whether its SQL parses instead of handing the whole project back, and Python's syntax messages come from one native validate/parse_one entry point.
Native model analysis no longer hands models, enrichments, dynamic pivot proofs or whole sessions back to Python. An internal native failure (a panic or a broken invariant) raises NativeCompilerError with its context and fails the compile; adapter rule exceptions and the ValueError Python's analysis raised for a dialect alias keep their original types. Non-ASCII function names now resolve declared return types natively. Deletes the Python whole-project analysis driver, binding waves and dataflow, the SQLite analysis cache, the Python dynamic pivot proof and their fallback-only tests; parity tests now compare uncached, cached and re-proven native runs.
…he wheel A parser panic or unreadable tree in native FAST lineage raises NativeCompilerError instead of re-parsing with the Polyglot wheel, which aborts the process on the same input; invalid SQL keeps its unparsed log. Deletes the per-model Python FAST builder. Analysis-session enrichment no longer maps an analyze_query options rejection to Python's ValueError: compile passes the alias-mapped dialect (postgres -> postgresql), so the rejection is internal.
…puts Native-vs-native cache comparisons prove cache consistency only, so the generated analysis, FAST lineage, type and contract tests now also match digests recorded once from the pre-port Python implementations. Restores the deleted shared-binding tests' explicit diagnostics and lineage expectations as native end-to-end tests, keeps postgres, motherduck and sqlserver alias routes covered at the compile boundary, drops tests and helpers that patched deleted Python seams, and brings the native stubs in line with the raising session API.
…ct and text Ports the semantic completion stage's remaining deferrals: Python's \w, \d, \s, re.IGNORECASE and $ semantics, splitlines boundaries, code-point offsets, casefold and upper beyond ASCII. A missing dialect raises the wheel's TypeError and an unknown one its ValueError, as Python's completion did; an internal failure (a panic, an unreadable tree, a location outside the authored lines, a project without its analysis catalog) raises NativeCompilerError. Deletes Python's recover_output_types, explain_diagnostics, recover_diagnostics, opt-out rejection and metadata checks. The native tests now match outputs recorded from those implementations, including non-ASCII projects and the dialects the full parser build added.
Untyped expression sources take Python's legacy analysis natively, running adapter nullability rules as the session does; a project restored without its catalog infers on a fresh one. Deletes Python's expression-source shape inference and the deferred-shapes fixture.
Models native analysis cannot project (CREATE TABLE AS, INSERT ... SELECT, deeply nested calls) keep the columns Python's parsed-tree inference gave them; table-function bodies, SQL-test fixture interfaces, source rebinding and planner fixture inference run the matching native analysis. A model the session leaves unanalysed is a broken invariant and raises NativeCompilerError. Deletes Python's compact, parsed-tree and CTE fact analyses; the few reference and placeholder helpers left move to reference_names, binding_requests and cte_scopes. The CTE fact parity test matches facts recorded from Python's recovery.
…all natively SQL-test assembly no longer hands tests to Python: non-ASCII contents (Python casefold, \w and re.IGNORECASE keys from generated tables), deep CTE trees, decimal rounding/underflow/overflow under Python's default context, reference and macro scan errors are native. Tests mocking macros keep their model queries from the user's macros, reported as the sql_test_glue.macro_mock_queries extension callback. The macro bridge scans every string natively and reports where Python's scan raises; strings with an unresolvable macro tree or a scan error evaluate their calls in order without the memo. Deletes Python's SQL-test assembly, scope deps and case identity.
Models send name, directory and tags, seeds their tags, and SQL functions their kind, name and tags, so the assembly that computes dependencies can build the project graph.
…rings through native call sites find_macro_call_names, find_nested_macro_call_names and the final macro output check use the native scanner. Strings expanded outside compile inputs evaluate their native call sites in order without the memo, as bridged strings with an unresolvable tree do, so Python's top-level call scanner and its macro_calls.unbridged_expansion fallback are deleted. The bridge requires a supported Python, as discovery already does, instead of falling back to the Python expansion.
…ished analysis session The finished session keeps each model's output columns with their types and nullability, not just names, and each model's dynamic pivot proof from the session's own relation facts, so native consumers read them without re-analysis.
…cture checks Doc comments are single summary lines; entry modules expose one function with helpers, errors and constants in their roles; placeholder defaults live in sqlbuild-sqltext; fallible probes match their errors. Test helpers move out of test modules and carry no conditional control flow, the golden comparison moves into the compiler test helpers, and the analysis-code scan reads the contract codes from the native crate. Two rule exceptions record the process-wide Python text semantics and the nullability callback's manual Debug.
…c proof FinishedSession::model_analysis returns the typed columns compilation publishes for a model the session decides (a proven pivot's fixed columns, else a successful analysis's) and the session's dynamic proof, borrowed from the finished session, for the native compiled-project owner to record without re-analysis.
Each call's arguments are scanned by the native call-site scanner, which now reports where a call's arguments start and the name of a call Python's evaluation fails in, so every call, top level or nested, runs from native bounds. A failing call raises its macro lookup error first, then the scan's message, as Python's evaluation did. Python's macro call scanner, name parser and parenthesis matching are deleted.
Literal types, NULL literals and quoted aliases of a fixture query's first SELECT come from the native parse instead of the wheel's to_dict walk in Python. A golden recorded from Python's _simple_projection_facts covers literals, structured values, quoted non-ASCII aliases, nested and set-operation selects and parse errors.
Python's regex substitute_placeholder_defaults now calls the native placeholder substitution the syntax checks already use, so one implementation serves both.
…ied-reference scans Native analysis sessions share binding results themselves, so Python's binding-share prekeys, shared result keys and the qualified-reference scan they used no longer run anywhere, along with the constants only the deleted Python analyses read.
…mn inference Per-query column inference outside the session normalizes its parse and legacy SQL itself, in query order and with the unaliased dialect Python used, raising the same ValueError on rejection, so Python no longer round-trips each query through the normalizer first.
…erence scan The usage walk for macro SQL tests now reads the native @enum/@const scan in a mode that resumes past malformed references, as Python's walk did, and raises the same unclosed quote or comment errors. Python's reference start walk and its patterns are deleted. A golden recorded from Python's walk covers malformed, quoted, commented and unclosed text.
Relation names in Python-node SQL and the runtime relation guard come from the native parse and a port of Python's dict walk (table payloads, CTE names and temporary tables per statement, first-seen order). An unknown dialect still raises the wheel's ValueError and unparseable SQL still yields None. A golden from Python's walk covers qualifiers, CTE shadowing, temporary tables, DML, quoting and non-ASCII names; the site recorder test now records the rule document's wheel parse.
The helper CTE graph's parsed reads (unqualified tables naming a helper key, minus CTEs a body defines at any depth) come from the native parse and Python's dict walk port instead of the wheel. A golden from Python's _parsed_reads covers qualification, shadowing, quoting, non-ASCII keys, set operations, subqueries and parse failures.
…g in Python A parser panic during native project assembly or the native syntax check used to be reported as a 'panic' deferral, so Python re-ran the whole assembly through the wheel, which crashes the process on the same SQL. Both now raise NativeCompilerError with their context, with no Python retry and no deferral record.
ModelColumnsInput::model_columns applies the session result, the parse fallback for a failed analysis, the session or standalone pivot proof and type-recovery poisoning in _assemble_model's and _recovered_model's order, so the owner store can finalize typed model columns without Python. SessionModelAnalysis now reports whether the analysis succeeded.
… deferring Native project assembly now reports the first source, seed or audit Python's assembly raises at, with Python's exact CompileInputError/ValueError messages, and renders variables only Python renders from their precomputed str() and render_project_var_text outcomes. Python's target, dependency and fallback helpers are deleted; the goldens record the pre-port Python outcomes.
…umer Relationships, test CTE extraction and scenario expected names now use the native top-level CTE scanner, which classifies identifier characters and upper-cases keywords with Python's own string tables. The Python scanner and its dead identifier helpers are deleted; goldens record the pre-port Python outcomes, including Unicode identifier edges.
Attached-audit gate cycles (P005), hook-read validation (unknown reads and P007 cycles) and Python-SQL hard-coded relation names (P008: project-relation index with old names, qualifier matching, temporary tables, f-string placeholders, ownership exemptions, dedup and ordering) now run in sqlbuild-analysis. Python only collects facts, including the literal SQL introspected from user functions. The replaced Python routines are deleted; a golden records the pre-port Python outcomes.
kvlonge
enabled auto-merge (squash)
October 11, 2026 00:26
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Complete M4 by replacing reachable Python analysis/assembly algorithms.
Changes
Verification
Independent review/follow-up; Windows hook paths corrected. Focused Rust/Python tests, historical oracle goldens, sabotage and CI-toolchain lints. make compiler-baselines: 360 identical projects, full discovery/render/analysis coverage. uv sync --all-extras and make check-ci passed; regeneration left no diff.
Expectation changes: