Repository navigation
Conversation
Trace imports, and parent imports of arranged parent items, are no longer wrapped in another Arrange, so printed plans replan identically with passes off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Node::Reduce names its inputs with a ReduceInput, either a signed merge or a cogroup. Nothing builds a non-single input yet. Ref walkers handle every input, consumers that need one input call unimplemented!. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A list [a, -b] before min, distinct, count, collect or reduce lowers to one reduce over Merged inputs, each arranged on its own with its sign. Shape inference takes the row shape from any known element. The runtime renders a merged reduce as concat, arrange, then the unary reduce, until an N-ary operator replaces it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A reduce driver over several arrangements of one trace type. It mirrors reduce_with_tactic and hands the tactic batches tagged by input, retiring the interval bounded by the meet of the inputs' progress. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A merged reduce over several inputs renders as one co_reduce_with_tactic over the inputs' arrangements, rather than arranging their signed concatenation. The corgi reduce backend reads batches tagged by input and counts each input's diffs times its sign. Grouped reduces remain unimplemented. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One input runs twenty ticks ahead of the other, so the driver hands over batches holding times beyond the interval it retires. The test counts those batches and checks the output against the reduce of the concatenation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
reduce_many checks that it has one sign and one shape per input, and that a merged reduce's inputs share a shape. The backend builds per-chunk signs only when some input is negated, so a unary reduce reads its chunks as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`a | reduce(b, c, (k, la, lb, lc) -> ..)` lowers to a Grouped reduce over every input's arrangement. The lambda takes the key and one (val, diff) list per input. Shape inference revisits the reduce until every input's shape is known, and the corgi backend presents each input on its own, ordered by real key and then input. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The generator emits bracket lists before a reduce and grouped reduces over inputs of different value shapes, and the pass-subset oracle draws merge-reduce as a bit. Close the Task 7 test gaps: the concat form with default passes, the pass firing on and off, a shared negation left alone, and a repeated arranged leaf. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The difftest grouped arm also picks a first extra input of another value shape when one exists, and a lambda body reads the contents of the last list. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
With arranged imports off, a trace import renders as rows. Lowering treats it as arranged and inserts no arrange, so each index reader arranged it again. The first index reader now arranges it, and the import keeps both forms for every later reader. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Print a one-element positive list's distinct with the by-record re-key, reuse the arrangement helpers in merge-reduce, list merge-reduce in the README, and tidy the pipe syntax docs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Plan with the default passes, print the optimized stage, and replan the text with none: shared and single unarranged leaves, a negated and a repeated arranged leaf, a merged reduce in a loop, and one hoisted out of a child scope. Also check that a one-element list's distinct prints keyed by record. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An input entered into an iterative scope and arranged there runs several outer epochs ahead of the loop's feedback. The driver hands over its batches beyond the retired interval, and the reduce's output at every iteration matches the reduce of the concatenation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A DDIR reduce over a concatenation arranged the concatenation, even when its parts were already arranged, so
(a - b) | distinctcopied both inputs. This change lets one reduce read several arrangements directly.Node::Reducetakes aReduceInput:Merged(the reducer sees the signed sum of its inputs) orGrouped(one value list per input, value shapes may differ). Today's reduce isMergedwith one input.[a, -b] | distinctwrites a merged reduce. A list is part of the reduce's syntax, not a value.a | reduce(b, c, (k, la, lb, lc) -> ..)writes a grouped reduce, extendingreducethe wayjointakes its other input. The printer emits both forms, so optimized DDP replans to the same plan with every pass off.co_reduce_with_tacticin differential-dataflow drives a reduce tactic over several arrangements, handing it batches tagged by input. The corgi backend reads them with per-input signs or per-input lists.merge-reducepass, on by default, rewrites a reduce over a concat with an already arranged leaf. Unarranged leaves share one new arrangement.arrangesteps from many expected plans.At two inputs of 50M rows and 8 workers,
EXCEPT ALLover two imported arrangements holds 0.7-0.9 GB of steady consumer memory against 4.3-4.4 GB before, with identical output. Install peak memory does not drop, because the corgi reduce backend's per-retire scratch over all history dominates it.🤖 Generated with Claude Code