Repository navigation
DDIR: projection compiles each lambda once and walks the program once - #1086
Draft
frankmcsherry wants to merge 1 commit into
Draft
frankmcsherry wants to merge 1 commit into
frankmcsherry wants to merge 1 commit into
Conversation
Projection compiled and analyzed a lambda once for every output field, and alternated with the general optimizer until nothing changed. It now caches each typed lambda graph by its input shape, and reuses it for every output field's demand. Demand is built once over the program with the existing worklist closure. Masks and reduce-map fusions apply together, and one whole-program inference validates the result, keeping the rollback. Local cleanup carries the checked shapes, so projection no longer calls the general optimizer. Dead computations are removed from a typed graph before an upstream mask can turn their input into unit. This fixes generated LDBC BI19 and IC14, which failed with `() vs Int` and rolled back. Final plans are byte-identical for the authored LDBC plans, the replay halves, 39 of 41 generated plans and the plan-pair corpus. BI19 and IC14 now project. Co-Authored-By: GPT-6 Sol (Codex) <noreply@openai.com> Claude-Session: https://claude.ai/code/session_01H6wB4H99TLw4tqu4x48pXL
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Projection now produces the same plans 2.8× faster on the authored LDBC plans and 6.8× faster on generated ones, and two generated plans it used to reject now project.
Projection stays off by default; this changes its cost, not the default.
Why it was slow
Projection compiled each lambda with corgi and ran demand analysis once per output field, not once per lambda: 4,734 compile/analyze pairs across the 41 authored LDBC plans, and 31,772 across the generated ones.
Compilation and demand analysis were 53% of its time on authored plans and 63% on generated ones.
It also ran two nested fixed-point loops, re-inferring shapes for the whole program 4 times on a plan it did not change and 11 to 14 times on one it did, and called the general optimizer to confirm stability.
The demand closure itself was about 1%.
What changes
() vs Intand rolled back.Same plans
Final emitted DDP is byte-identical for all 41 authored LDBC plans, all 22 one-off/standing replay halves, 39 of 41 generated plans, and the 60-case plan-pair corpus.
The other two are BI19 and IC14, which now project and match the authored answers on both witness bindings.
The eight programs of the join-fragment spike (#1085) also emit identical plans.
All 184 library tests pass.
Cost
Projection stage time, sums of five-run per-program medians from
ddir-plan:The LDBC SF1 one-off replay (808 ticks, 6,850 one-off installs, one worker), with projection on, three alternating trials of each binary:
Answers match at every tick.
With projection off, install is 0.71 ms per read; on this workload projection still costs more than it saves, so off remains the right default here.
A companion corgi prototype,
analyze_demands, answers every output field of a lambda in one backward walk and saves a further 28% of generated-plan projection time (10–12% elsewhere). It is a separate draft in corgi and is not needed by this change.Implemented by GPT-6 Sol (Codex); reviewed, tested and opened by Claude Code.
🤖 Generated with Claude Code
https://claude.ai/code/session_01H6wB4H99TLw4tqu4x48pXL