Skip to content

Add a sorted-file format for index build inputs - #32

Open
tamirms wants to merge 2 commits into
ram-index-streamingfrom
sorted-files
Open

tamirms wants to merge 2 commits into
ram-index-streamingfrom
sorted-files

Conversation

@tamirms

@tamirms tamirms commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

stellar-rpc builds its tx-hash index from keys that arrive over a long period, so it writes them to sorted files of its own, merges the files into one stream, and feeds the stream to a SortedBuilder. The merge is the slowest part of the build. A following change adds BuildFromSortedFiles, which builds an index from many sorted files without merging them. That build relies on its inputs being well formed: fixed-width records, keys in order, and a count it can check. It can only check those if streamhash owns the format, the way RocksDB owns the format that SstFileWriter writes and IngestExternalFile takes. Letting callers describe their own layout, or pass their own readers, would leave the library unable to validate a file or promise speed and memory.

A sorted file is a header padded to 4 KiB, then fixed-width records in key order. Each record is a key's first 16 bytes, the bytes the index uses, then a payload of 0 to 8 bytes. The header holds a magic, a version, the payload size, the record count and a caller's metadata blob of up to 64 KiB. The padding keeps records aligned for direct reads. The format is §3.9 of the spec.

  • SortedFileWriter writes one. It refuses keys out of order and payloads that don't fit. Finish writes the count and syncs, and Close without Finish removes the file. BytesPerSync starts writeback as the file grows, as WithBytesPerSync does for an index.
  • ReadSortedFileHeader checks a file's magic, version and payload size, and that its count matches its length.
  • MergeSortedFiles merges several into one, for writers that spill sorted pieces. It runs one heap over the inputs on its own goroutine, reads directly into aligned buffers, and hands batches of whole records to a writing goroutine. It checks each input's order as it reads, so a corrupt input fails the merge instead of producing unsorted output.
  • OpenSortedFile reads one record at a time, for tools and tests.

ErrTruncatedFile and ErrCorruptedIndex no longer say "index" in their messages, since sorted files return them too.

Most of this code comes from stellar-rpc's tx-hash index build, where it is already tuned. MergeSortedFiles, with its direct reads, is stellar-rpc's spill merge, which began as this repo's cmd/bench/bench_merge.go. SortedFileWriter keeps the mechanics of stellar-rpc's .bin writer: a 1 MiB buffer with writeback, the count patched in at the end, then a sync and a checked close. ReadSortedFileHeader's count check is stellar-rpc's too. stellar/stellar-rpc#1139 switches stellar-rpc to these functions and deletes its own merge.

It differs from stellar-rpc's code in these ways:

  • Payload size. Payloads are 0 to 8 bytes rather than stellar-rpc's 4-byte ledger sequence, since an index takes a payload of any of those sizes.
  • Header. It is padded to 4 KiB rather than stellar-rpc's 32 bytes, for the reason above, and holds the caller's metadata instead of stellar-rpc's index secret, since streamhash doesn't know what a caller needs to record. For the same reason, the merge takes the output's metadata from the caller. stellar-rpc's merge checked that its inputs shared one secret and copied it.
  • Order checks. Add checks key order, and the merge checks each input's order as it reads. stellar-rpc's writer trusted its callers, and its merge checked only headers. BuildFromSortedFiles relies on sorted inputs, and a library can't count on every caller to sort correctly. The checks cost one comparison per record, and the rates below still match stellar-rpc's.
  • Unfinished files. Close without Finish, a failed Finish and a failed merge delete the file. stellar-rpc left a partial file for its retry to overwrite and relied on its own record of finished files, which another caller may not keep.
  • Merging into an input. The merge refuses an output path that is one of its inputs, which stellar-rpc's did not check. Creating the output would truncate that input before the merge read it.
  • One heap. stellar-rpc's merge can also spread its inputs over a tree of heaps on several goroutines. Only its index build used the tree, and BuildFromSortedFiles builds without merging, so only the single heap of stellar-rpc's spill merge came over.

Merging 90 files of 1M records with 3-byte payloads, the merge matches stellar-rpc's: 23.0M records/s against 23.2M from the page cache, and 11.95M against 11.77M with direct reads on Linux. Both wrote identical output. The writer runs at the same rate as stellar-rpc's.

🤖 Generated with Claude Code

tamirms and others added 2 commits November 17, 2026 07:30
stellar-rpc keeps its txhash build inputs in sorted .bin files of its own
and merges them with code ported from cmd/bench. This moves the format and
the merge into streamhash, so that a later change can build an index
straight from many sorted files. A sorted file is a header padded to 4096
bytes, then fixed-width records of a key's first 16 bytes and a payload of
0 to 8 bytes, in key order (spec §3.9). SortedFileWriter writes one and
refuses keys out of order, ReadSortedFileHeader and OpenSortedFile check
and read one, and MergeSortedFiles merges several, checking each input's
order as it reads. ErrTruncatedFile and ErrCorruptedIndex no longer say
"index" in their messages, since sorted files return them too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DjQed1VX9uGU9PWpAjBiz7
TestMergeSortedFilesRejectsOutputAmongInputs read its input with
readFileBytes, a helper the BuildFromSortedFiles change adds, so this
change's tests did not build on their own. It now reads the input with
os.ReadFile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DjQed1VX9uGU9PWpAjBiz7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant