Skip to content

Yield to other work before each block (WithYield) - #35

Open
tamirms wants to merge 1 commit into
build-block-errorfrom
build-yield
Open

tamirms wants to merge 1 commit into
build-block-errorfrom
build-yield

Conversation

@tamirms

@tamirms tamirms commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

stellar-rpc rebuilds its tx-hash window index in the same process that ingests ledgers. The build's workers share cores, caches and memory bandwidth with ingestion, and ingestion's tail latency grows beside the build however few workers it has. But ingestion works in bursts, one per ledger, and a Bijection block of about 3,000 keys takes one worker about half a millisecond. So the build can run in the gaps, if the caller can pause it between blocks.

func WithYield(yield func(ctx context.Context) error) BuildOption

Before each block, empty blocks included, the goroutine about to build it calls yield and goes on when it returns. That holds in every build mode: sorted and unsorted, one worker or several, and BuildFromSortedFiles.

  • A yield that blocks holds the build back. The rest of the build stops behind the parked workers: what feeds them stops once their queue is full, and the writer once it has committed the blocks already solved. A parked build holds only its buffers and open files.
  • A yield that returns an error fails the build with that error, as a block that fails to build does.
  • The context is the build's. It is done once the build is canceled or has failed, so a yield that waits on it returns then.
  • WithYield(nil) is the same as not passing the option. Without a yield, the cost is a nil check per block, and the builder benchmarks don't move.

A failed block's result is no longer sent to the writer once the pipeline's context is done. Otherwise a yield that returns its context's error stands in for the real failure: with one, 117 of 200 four-worker builds over a sorted file with a record out of order reported context.Canceled instead of the out-of-order error.

The option is §6.4 of the spec. The tests cover every build mode, the unsorted builder's spill to disk included.

In stellar-rpc, the ingest loop holds a gate while it ingests a ledger, and the build gets WithYield(gate.Wait). At sac-9000 load, with diagnostic events stripped from the ledgers, ingestion's p99 then stays within about 4 ms of its value alone while the build runs 16 workers at 99.4M keys/s. Without the gate, an 8-worker build adds about 26 ms to it.

🤖 Generated with Claude Code

stellar-rpc rebuilds its tx-hash window index while it ingests ledgers,
and the build's workers share cores, caches and memory bandwidth with
ingest. On a 32-vCPU box, ingest loses about 20 to 26 ms of p99 beside
the build, and fewer workers do not win it back. Ingest is bursty, and a
block is about half a millisecond of one worker, so the build can run in
the gaps: with a gate that the ingest loop holds while it ingests a
ledger, passed as WithYield(gate.Wait), ingest comes back to within 0.3
to 4 ms of running alone, while the build runs at 16 workers and 99M
keys/s.

WithYield(yield) makes the goroutine about to build a block call yield
first, in every build mode, inline or parallel, empty blocks included. A
yield that blocks holds the build back, one that returns an error fails
the build with it, and WithYield(nil) is the same as not passing the
option. The workers yield in the block build they share, so a yield
error is the block's result and reaches the writer like a build error;
the unsorted builder's parallel finish waits on the writer while it
reads partitions. A failed block's result is no longer sent once the
pipeline's context is done, since a yield that returns its context's
error would otherwise stand in for a failure that skipped the writer:
117 of 200 four-worker builds over a sorted file with a record out of
order reported context.Canceled. Without a yield the cost is a nil check
per block, and the builder benchmarks do not move. The tests cover every
mode, the unsorted builder's spill to disk included.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DjQed1VX9uGU9PWpAjBiz7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant