Repository navigation
Conversation
tamirms
force-pushed
the
sorted-files
branch
from
November 10, 2026 12:17
56be8f7 to
b2bd427
Compare
tamirms
force-pushed
the
sorted-files
branch
from
November 17, 2026 07:25
b2bd427 to
818d357
Compare
stellar-rpc keeps its txhash build inputs in sorted .bin files of its own and merges them with code ported from cmd/bench. This moves the format and the merge into streamhash, so that a later change can build an index straight from many sorted files. A sorted file is a header padded to 4096 bytes, then fixed-width records of a key's first 16 bytes and a payload of 0 to 8 bytes, in key order (spec §3.9). SortedFileWriter writes one and refuses keys out of order, ReadSortedFileHeader and OpenSortedFile check and read one, and MergeSortedFiles merges several, checking each input's order as it reads. ErrTruncatedFile and ErrCorruptedIndex no longer say "index" in their messages, since sorted files return them too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DjQed1VX9uGU9PWpAjBiz7
TestMergeSortedFilesRejectsOutputAmongInputs read its input with readFileBytes, a helper the BuildFromSortedFiles change adds, so this change's tests did not build on their own. It now reads the input with os.ReadFile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DjQed1VX9uGU9PWpAjBiz7
tamirms
force-pushed
the
sorted-files
branch
from
November 17, 2026 07:30
818d357 to
e8db257
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
stellar-rpc builds its tx-hash index from keys that arrive over a long period, so it writes them to sorted files of its own, merges the files into one stream, and feeds the stream to a
SortedBuilder. The merge is the slowest part of the build. A following change addsBuildFromSortedFiles, which builds an index from many sorted files without merging them. That build relies on its inputs being well formed: fixed-width records, keys in order, and a count it can check. It can only check those if streamhash owns the format, the way RocksDB owns the format thatSstFileWriterwrites andIngestExternalFiletakes. Letting callers describe their own layout, or pass their own readers, would leave the library unable to validate a file or promise speed and memory.A sorted file is a header padded to 4 KiB, then fixed-width records in key order. Each record is a key's first 16 bytes, the bytes the index uses, then a payload of 0 to 8 bytes. The header holds a magic, a version, the payload size, the record count and a caller's metadata blob of up to 64 KiB. The padding keeps records aligned for direct reads. The format is §3.9 of the spec.
SortedFileWriterwrites one. It refuses keys out of order and payloads that don't fit.Finishwrites the count and syncs, andClosewithoutFinishremoves the file.BytesPerSyncstarts writeback as the file grows, asWithBytesPerSyncdoes for an index.ReadSortedFileHeaderchecks a file's magic, version and payload size, and that its count matches its length.MergeSortedFilesmerges several into one, for writers that spill sorted pieces. It runs one heap over the inputs on its own goroutine, reads directly into aligned buffers, and hands batches of whole records to a writing goroutine. It checks each input's order as it reads, so a corrupt input fails the merge instead of producing unsorted output.OpenSortedFilereads one record at a time, for tools and tests.ErrTruncatedFileandErrCorruptedIndexno longer say "index" in their messages, since sorted files return them too.Most of this code comes from stellar-rpc's tx-hash index build, where it is already tuned.
MergeSortedFiles, with its direct reads, is stellar-rpc's spill merge, which began as this repo'scmd/bench/bench_merge.go.SortedFileWriterkeeps the mechanics of stellar-rpc's.binwriter: a 1 MiB buffer with writeback, the count patched in at the end, then a sync and a checked close.ReadSortedFileHeader's count check is stellar-rpc's too. stellar/stellar-rpc#1139 switches stellar-rpc to these functions and deletes its own merge.It differs from stellar-rpc's code in these ways:
Addchecks key order, and the merge checks each input's order as it reads. stellar-rpc's writer trusted its callers, and its merge checked only headers.BuildFromSortedFilesrelies on sorted inputs, and a library can't count on every caller to sort correctly. The checks cost one comparison per record, and the rates below still match stellar-rpc's.ClosewithoutFinish, a failedFinishand a failed merge delete the file. stellar-rpc left a partial file for its retry to overwrite and relied on its own record of finished files, which another caller may not keep.BuildFromSortedFilesbuilds without merging, so only the single heap of stellar-rpc's spill merge came over.Merging 90 files of 1M records with 3-byte payloads, the merge matches stellar-rpc's: 23.0M records/s against 23.2M from the page cache, and 11.95M against 11.77M with direct reads on Linux. Both wrote identical output. The writer runs at the same rate as stellar-rpc's.
🤖 Generated with Claude Code