Skip to content

refactor: move discovery validation and YAML parsing to Rust - #1091

Merged
kvlonge merged 7 commits into
mainfrom
feat/native-discovery-stage
Oct 12, 2026
Merged

kvlonge merged 7 commits into
mainfrom
feat/native-discovery-stage

Conversation

@kvlonge

@kvlonge kvlonge commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

Why

First milestone of the Rust-hosted compile (CHI-589): move discovery logic out of Python. Each port deletes the Python it replaces and keeps behaviour identical, including error text, exception classes and which error is reported first.

Changes

  • Model files are native records (sqlbuild._native.DiscoveredSqlModelFile) built directly by native discovery; the model loop reads them without Python unpacking. Records still copy, pickle and support lint-fix edits.
  • Cross-file discovery validation (identities, name conflicts, Python node dependencies and cycles, seed CSV headers, path defaults) runs in sqlbuild-discovery::project_validation. The seed header reader matches Python's csv reader; the running interpreter still judges CSV dialect settings, so 3.13/3.14 errors are unchanged.
  • Source and seed YAML declarations, including freshness, ingestr and dlt sources, are parsed in sqlbuild-discovery::yaml_entries.
  • Python-exact strip, casefold, isalnum, isdigit and isspace helpers in sqlbuild-core.
  • About 2,500 lines of Python deleted; the boundary baseline drops from 146,314 to 143,789.

Verification

  • Compiler differential: all 368 projects match their goldens, with full discovery and render coverage.
  • Identical compile output to main on a generated 1k-model project and a private project (cold and edit).
  • Existing source/schema YAML unit tests run unchanged through the native parsers; new integration tests pass identically against main.
  • New Rust tests, including 127 CSV header reads recorded from CPython.
  • make test, make check-ci, one review plus a follow-up pass.
  • 10k benchmark: cold compile about 1.5% faster, edit compile unchanged.

@kvlonge
kvlonge enabled auto-merge (squash) October 12, 2026 17:12
@kvlonge
kvlonge merged commit 9ff80db into main Oct 12, 2026
40 checks passed
@kvlonge
kvlonge deleted the feat/native-discovery-stage branch October 12, 2026 17:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant