Skip to content

Chunk: merge and advance as one step - #1083

Draft
frankmcsherry wants to merge 1 commit into
TimelyDataflow:master-nextfrom
frankmcsherry:chunk-merge-advance
Draft

frankmcsherry wants to merge 1 commit into
TimelyDataflow:master-nextfrom
frankmcsherry:chunk-merge-advance

Conversation

@frankmcsherry

Copy link
Copy Markdown
Member

A spine merge of chunked batches merges two chunks, then advances and consolidates the result. Each step writes its output, so an implementor for whom writing a chunk is the cost pays it twice.

Chunk gains merge_advance, which does both.
Its default is merge followed by advance, so no implementor has to change. ChunkBatchMerger calls it while both inputs have data, and calls advance as before for the tails and the final flush. The merger now counts a step's fuel by the updates it takes from the inputs, where it counted the updates merge produced.

DDIR's CorgiChunk implements it.
The body of its merge moves to merge_fronts, which takes an optional frontier and carry. With them, the merged order stays as indices into the source columns. The times of the complete groups are advanced, each group of several rows is sorted and summed on its times, and only the surviving rows are gathered. The last group may continue in a later chunk, so it is held back, and joins the next step as a third source.

On a merge-heavy DDIR workload the second gather was 15% of merge time, and this removes it. The transaction benchmark's maintained program runs 2 to 4% more transactions a second, and the graph examples are unchanged.

Claude-Session: https://claude.ai/code/session_01H6wB4EnzBnYpvwz5E2ZGmD

A spine merge of chunked batches merges two chunks, then advances and consolidates the result.
Each step writes its output, so an implementor for whom writing a chunk is the cost pays it twice.

`Chunk` gains `merge_advance`, which does both.
Its default is `merge` followed by `advance`, so no implementor has to change.
`ChunkBatchMerger` calls it while both inputs have data, and calls `advance` as before for the tails and the final flush.
The merger now counts a step's fuel by the updates it takes from the inputs, where it counted the updates `merge` produced.

DDIR's `CorgiChunk` implements it.
The body of its `merge` moves to `merge_fronts`, which takes an optional frontier and carry.
With them, the merged order stays as indices into the source columns.
The times of the complete groups are advanced, each group of several rows is sorted and summed on its times, and only the surviving rows are gathered.
The last group may continue in a later chunk, so it is held back, and joins the next step as a third source.

On a merge-heavy DDIR workload the second gather was 15% of merge time, and this removes it.
The transaction benchmark's maintained program runs 2 to 4% more transactions a second, and the graph examples are unchanged.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6wB4EnzBnYpvwz5E2ZGmD

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant