Repository navigation
fix(compiler): keep shared-analysis lineage and stage captures deterministic - #1005
Merged
Merged
Conversation
…le-independent A model that reused another model's shared analysis lost its own resource names when the template facts were already projected, so fast column lineage named the representative's source. Pass the member's resource names on the cached-template path too. Compiled-project stage captures also depended on how the binding dataflow batched models: compact lineage facts index a batch-wide string pool, and the shared-analysis memo keeps whichever representative and lazy caches a schedule produced. Capture compact lineage as its decoded facts and leave the memo out, so python-vs-python captures are identical under any schedule. Bump the analysis cache version so entries written with the wrong names are recomputed.
Captures embedded every repeated subtree in full, so their size grew
quadratically with sources per file and the harness held and parsed both
engines' captures in memory; dense 3k ran out of memory.
The capture writer now stores every container or string of at least 1 KiB
once under the SHA-256 of its compact JSON, with invocation ids and native
store suffixes masked, and refers to it as {"__shared__": digest}. Sets and
unordered maps sort by the same masked form, so ordering never depends on
noise the comparison ignores, and authored mappings using a reserved marker
key are encoded as pairs. Each shared node and the root sit on their own
line, so the harness indexes a capture by digest, treats equal digests as
equal without reading them, and parses only the nodes on a differing path
while still reporting the logical JSON pointer and resolved previews.
Captures stay on disk, one directory per compared side, instead of in memory.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
CHI-539: compiled-project stage captures differed python-vs-python (seeds 14, 29, 34), and dense 3k with
--stage-capturesran out of memory. Native batch sizes depend on timing and each batch shares one string pool, so encoded lineage rows and the memo's cached entries changed with batching. This hid a bug: a member reusing a cached shared analysis lost its own resource names, sosqb lineage --mode fastcould name another model's source.Changes
resource_nameson the cached-template path; the analysis cache version is bumped so wrong entries are recomputed.CompactLineageFactsas decoded facts and omit theshared_analysesreuse cache;schemas,analysis_shapes,expression_shapesstay unordered.Verification
make compiler-differentialwith captures and coverage passes 139/139, python vs native and python vs python. Seeds 14–34 pass both ways.--stage-captures --jobs 1: harness peak RSS 0.93 GB, compile 1.44 GB; captures 0.5 GB per side, compared in 1.5 s with 39 MB.make check-ci,make test(11,445 passed) and the hygiene scan pass.