Repository navigation
Chunk: merge and advance as one step - #1083
Draft
frankmcsherry wants to merge 1 commit into
Draft
frankmcsherry wants to merge 1 commit into
frankmcsherry wants to merge 1 commit into
Conversation
A spine merge of chunked batches merges two chunks, then advances and consolidates the result. Each step writes its output, so an implementor for whom writing a chunk is the cost pays it twice. `Chunk` gains `merge_advance`, which does both. Its default is `merge` followed by `advance`, so no implementor has to change. `ChunkBatchMerger` calls it while both inputs have data, and calls `advance` as before for the tails and the final flush. The merger now counts a step's fuel by the updates it takes from the inputs, where it counted the updates `merge` produced. DDIR's `CorgiChunk` implements it. The body of its `merge` moves to `merge_fronts`, which takes an optional frontier and carry. With them, the merged order stays as indices into the source columns. The times of the complete groups are advanced, each group of several rows is sorted and summed on its times, and only the surviving rows are gathered. The last group may continue in a later chunk, so it is held back, and joins the next step as a third source. On a merge-heavy DDIR workload the second gather was 15% of merge time, and this removes it. The transaction benchmark's maintained program runs 2 to 4% more transactions a second, and the graph examples are unchanged. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H6wB4EnzBnYpvwz5E2ZGmD
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A spine merge of chunked batches merges two chunks, then advances and consolidates the result. Each step writes its output, so an implementor for whom writing a chunk is the cost pays it twice.
Chunkgainsmerge_advance, which does both.Its default is
mergefollowed byadvance, so no implementor has to change.ChunkBatchMergercalls it while both inputs have data, and callsadvanceas before for the tails and the final flush. The merger now counts a step's fuel by the updates it takes from the inputs, where it counted the updatesmergeproduced.DDIR's
CorgiChunkimplements it.The body of its
mergemoves tomerge_fronts, which takes an optional frontier and carry. With them, the merged order stays as indices into the source columns. The times of the complete groups are advanced, each group of several rows is sorted and summed on its times, and only the surviving rows are gathered. The last group may continue in a later chunk, so it is held back, and joins the next step as a third source.On a merge-heavy DDIR workload the second gather was 15% of merge time, and this removes it. The transaction benchmark's maintained program runs 2 to 4% more transactions a second, and the graph examples are unchanged.
Claude-Session: https://claude.ai/code/session_01H6wB4EnzBnYpvwz5E2ZGmD