Skip to content

perf(native): evaluate model contracts without repeated failing type deserialization - #1071

Closed
kvlonge wants to merge 2 commits into
mainfrom
fix/native-contract-evaluation-speed
Closed

kvlonge wants to merge 2 commits into
mainfrom
fix/native-contract-evaluation-speed

Conversation

@kvlonge

@kvlonge kvlonge commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Why

Under native-preview, native model contract evaluation took ~15.7 s CPU on the dense 3,000-model project, against ~0.2 s for the Python engine (the same on main). That made the whole preview compile ~2.4× slower and blocks promoting the lane.

Changes

  • type_system/_helpers/parsed.rs: is_python_expression returns false for scalars before trying to deserialize them as Expression. Every Expression variant carries data, so a scalar can never deserialize. The failing deserialization built a large error message each time, costing ~100 µs per normalize_type call; now ~5 µs.
  • contracts: a request-scoped TypeComparer normalizes each type string once, as Python's lru_cache on normalize_type does. No global state.
  • compile_timings gains contracts_cpu_ms (CPU time of the calling thread).
  • Same-runner ratio tool: --base-engine/--head-engine and --gate-phase (gate one reported phase, plus --phase-noise-floor-ms). A new matrix entry in the ratio job gates native-preview contracts_cpu_ms at ≤1.00× python + 50 ms on dense 3,000 models.
  • Ratio tool: a phase missing from any run is reported missing, not counted as 0.
  • Changed expectation: test_compile_json_behavior timing keys include contracts_cpu_ms.

Verification

  • Dense 3,000 cold, 3 alternating runs: contracts CPU 124 ms python vs 90 ms native-preview (was ~15.7 s).
  • Guard sabotage (reverted):
    • original slow path: 14,771 ms, fails;
    • memo removed only: 334 ms, fails.
  • New Rust test: a batch sharing types equals per-model requests, including repeated deferrals.
  • Contract and type-normalization parity tests, ratio-tool E2E, test_compile_json_behavior.
  • CI-flag differential: 320 identical, no contract deferrals.
  • cargo test/clippy/fmt on 1.99.0, make check-ci, make test (11938 passed), hygiene.
  • No local real project compiles against 0.151.0, so dense parity and timing stand in.

…deserialization

Scalars skip the Expression deserialization that always failed and built a large error per call, and contract evaluation normalizes each type string once per request. Adds contracts_cpu_ms to compile timings and a same-runner python vs native-preview gate on it.
@kvlonge
kvlonge enabled auto-merge October 9, 2026 15:24
@kvlonge
kvlonge added this pull request to the merge queue Oct 9, 2026
@kvlonge
kvlonge removed this pull request from the merge queue due to a manual request Oct 9, 2026
@kvlonge
kvlonge added this pull request to the merge queue Oct 9, 2026
@kvlonge

kvlonge commented Oct 9, 2026

Copy link
Copy Markdown
Contributor Author

Landed together in #1075 to cut merge-queue CI runs; review history stays here.

@kvlonge kvlonge closed this Oct 9, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a manual request Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant