Skip to content

feat(native): add core types, exact JSON emitter and config parsing - #992

Merged
kvlonge merged 4 commits into
mainfrom
feat/native-foundations
Oct 7, 2026
Merged

kvlonge merged 4 commits into
mainfrom
feat/native-foundations

Conversation

@kvlonge

@kvlonge kvlonge commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Why

Later native milestones need Rust building blocks that match Python exactly: identities and diagnostics, text positions, JSON text that equals json.dumps/orjson byte for byte, and configuration reading that matches tomllib and PyYAML. This PR adds those foundations only. Nothing is wired into the compile path, and Python behaviour does not change.

Changes

  • sqlbuild-core: passive identity, diagnostic and SQL value types; LineIndex positions with Python universal-newline decoding; an exact json.dumps/orjson emitter. The orjson floor rises to 3.11.7, whose float layout it reproduces.
  • New sqlbuild-config crate:
    • tomllib-compatible TOML (creation order, namespace rules, CRLF, TOML 1.1 rejection);
    • PyYAML-compatible YAML 1.1 (PyYAML's block scalar scanner, resolver and SafeConstructor);
    • typed project/local config readers.
  • Contract: native either returns exactly Python's value or defers to Python.
    • YAML that SafeLoader and CSafeLoader read differently is deferred.
    • Alias, merge and nesting expansion are bounded, as are TOML depth and JSON depth/digits, so hostile input defers instead of crashing; large single-line collections load in linear time.
  • Internal _oracle_* hooks (not an API) and seeded oracles:
    • JSON: 9 × 1500 values;
    • text positions;
    • YAML: 7000 documents vs both loaders;
    • TOML: 5000 general and 5000 namespace documents;
    • repository files;
    • hostile documents with a time bound.

Verification

  • cargo test --workspace --all-features; clippy -D warnings and fmt on 1.98.0; make check-native-layering
  • uv sync --all-extras --reinstall-package sqlbuild, then oracle tests and review repro scripts: no crash, no harmful mismatch
  • 2.6M fuzzed YAML documents: zero accepts that differ from either loader
  • 80k fuzzed TOML documents: zero wrong results
  • make test (11204 passed), make check-ci, public-repository scanner

The emitter refuses integers beyond Python's 4300-digit string limit and values nested deeper than 128 levels instead of exhausting the stack. orjson is required at 3.11.7 or later, whose float layout the emitter reproduces, and a test pins that layout.
…L and project config reading

Adds TOML loading that matches tomllib, rejecting TOML 1.1-only syntax, and typed project and local config readers for the fields discovery needs. Any native error defers to the Python loader. Internal oracle hooks compare both against Python on seeded and repository documents.

Tables keep tomllib's order of first creation even when an implicit table is declared later, CRLF is normalized and a leading byte order mark is rejected as tomllib does. Container depth and dotted key length are bounded before parsing so deep documents are refused instead of exhausting the stack.

tomllib's namespace rules are replayed over the parser events, so dotted keys into arrays of tables or declared tables, headers over dotted or inline tables, and extended inline tables are rejected as tomllib rejects them.
Composes saphyr-parser events into PyYAML's node graph and emulates the YAML 1.1 resolver and SafeConstructor, including merge keys, base-60 numbers, timestamps and duplicate-key handling. Constructs PyYAML or LibYAML handle differently defer to Python. Oracle tests compare against SafeLoader and CSafeLoader on seeded and repository documents.

Literal and folded scalars are read by a port of PyYAML's block scalar scanner, cross-checked against the parser, so chomping matches PyYAML with or without a final line break. A leading byte order mark is skipped. Constructs the two PyYAML loaders read differently, such as tabs, '?' inside flow scalars and directives, are deferred, and the oracle tests require deferral wherever SafeLoader and CSafeLoader disagree. Shared nodes are built once, and construction depth plus a size budget proportional to the input bound alias and merge expansion. Property lookups use code positions computed once per document and mapping keys are hashed by Python equality, so large single-line flow collections load in linear time.
@kvlonge
kvlonge enabled auto-merge (squash) October 7, 2026 00:40
@kvlonge
kvlonge merged commit 6794064 into main Oct 7, 2026
34 checks passed
@kvlonge
kvlonge deleted the feat/native-foundations branch October 7, 2026 01:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant