Skip to content

feat(native): build fast column lineage natively behind the preview engine - #1029

Merged
kvlonge merged 2 commits into
mainfrom
feat/native-fast-lineage
Oct 8, 2026
Merged

kvlonge merged 2 commits into
mainfrom
feat/native-fast-lineage

Conversation

@kvlonge

@kvlonge kvlonge commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Why

FAST column lineage still ran in Python and called the Polyglot wheel's parse_one for each model without compact facts, such as models with sql_analysis false. This PR builds the same lineage natively under native-preview. The python and native engines are unchanged.

Changes

  • sqlbuild-analysis::lineage: star expansion and the parse fallback, using the crate parser. It ports the reference matching, alias maps, confidence and transform rules, union branches, and the pythonize-shaped to_dict walk.
  • The stage runs on the compile catalog's deep-stack analysis pool. Set-operation chains are serialized once and walked iteratively, so deep UNION ALL chains stay linear in cost and depth.
  • A model is returned to Python, which builds it exactly as before and records a {"kind","site"} deferral, when:
    • its dialect is not compiled into the crate;
    • the parser panics;
    • the project has no analysis catalog.
  • Parse errors are logged with Python's exact debug event.
  • build_fast_column_lineage binding and stub, plus lineage failure-corpus cases.

Verification

  • Rust unit tests, including a 512-branch chain at the parser guard limit, started from a 256 KiB thread.
  • Parity property test, Python vs native, on generated projects with and without compact facts, across 4 dialects and model selections, requiring a minimum number of native answers. Further tests cover parse-error logging and the deferral records.
  • DuckDB CLI tests on all three engines comparing compile --json and lineage --mode fast, including a 480-branch UNION ALL beside a second opted-out model. That project segfaulted before this fix.
  • Random-SQL fuzz with deep set-operation chains: no mismatches.
  • cargo test --workspace --all-features, clippy and fmt 1.99.0, make compiler-differential with CI flags (fast_columns wheel calls 204 to 0, no deferrals), -shipped, make test, make check-ci and the hygiene scan all pass.

@kvlonge
kvlonge enabled auto-merge (squash) October 8, 2026 12:14
@kvlonge
kvlonge merged commit e074bd1 into main Oct 8, 2026
38 checks passed
@kvlonge
kvlonge deleted the feat/native-fast-lineage branch October 8, 2026 12:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant