Skip to content

Write the RAM index to the file as blocks commit - #31

Open
tamirms wants to merge 1 commit into
mainfrom
ram-index-streaming
Open

tamirms wants to merge 1 commit into
mainfrom
ram-index-streaming

Conversation

@tamirms

@tamirms tamirms commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

The index writer keeps every block's RAM index entry in memory until the build finishes. It then encodes them all and writes them in one piece, just before the final sync. Everything else the writer holds is a fixed size, but this part grows with the key count: 16 bytes per block, plus a 10-byte-per-block encode buffer at the end. For a 90 billion key bijection index, which has 29 million blocks, that is about 760 MB. That final write, about 290 MB, comes right before the sync, so WithBytesPerSync can't spread it out.

The RAM index's position and size in the file are fixed when the file is created, and its entries are produced in block order. So the writer can write them as blocks commit. It now encodes the entries into a 64 KiB buffer and writes the buffer to its place in the file each time it fills. WithBytesPerSync counts those writes.

The file layout does not change. Indexes built before and after the change are byte-identical across both algorithms, sorted and unsorted builds, a range of payload sizes, fingerprint sizes, worker counts and key counts, and builds with and without user metadata. A new test commits block counts just under, at and over the buffer's size and checks the RAM index byte for byte.

Measured with 4.3 billion keys (1.4 million blocks), 16 workers, a 3-byte payload and a 1-byte fingerprint, two runs each:

Before After
Build rate 73.8M, 73.1M keys/s 73.3M, 73.4M keys/s
Peak RSS 115, 113 MB 63, 64 MB
Finish 29, 25 ms 2, 2 ms

🤖 Generated with Claude Code

Comment thread index_writer.go

// flushRAMIndex writes the buffered RAM index entries after those already
// written and empties the buffer.
func (iw *indexWriter) flushRAMIndex() error {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking, and not caused by this PR: the same result occurs on main.

The RAM index can get one more entry than numBlocks + 1, and the write then goes into the payload region with no error.

How it occurs (sorted, single-threaded build):

  1. commitBlockWithData appends the RAM index entry before the writes that can fail.
  2. A write fails. The entry stays in the buffer, but blockCount does not advance.
  3. SortedBuilder.AddKey returns the error but does not mark the builder as failed.
  4. The caller calls AddKey again. The builder commits that block a second time and appends a second entry.
  5. Finish returns nil. The sentinel is written 10 bytes into the payload region, and all later entries are shifted by one.

I reproduced this with 40,000 keys (14 blocks). I replaced iw.file with a read-only handle for one AddKey call at a block boundary, restored it, and repeated the call. Result on main and on this branch: Finish nil, Open nil, Verify returns a checksum error, and most queries return errors or duplicate ranks.

The parallel and unsorted paths are not affected. A writer error stops the writer goroutine there, and each Finish error removes the output.

Now that the write position is known at each flush, one comparison stops the write:

func (iw *indexWriter) flushRAMIndex() error {
	if iw.ramIndexWriteOffset+uint64(len(iw.ramIndexBuf)) > iw.payloadRegionOffset {
		return errors.New("RAM index overflow: more entries than numBlocks+1")
	}
	...

With this check, Finish fails in the reproduction and the builder removes the file. The short test suite still passes.

The check is a backstop. The root fix is to make the builder fail permanently after a commit error. Adding the RAM index entry after the writes is not enough on its own, because a failed payload write leaves the metadata offset and the metadata checksum already advanced.

@tamirms
tamirms force-pushed the ram-index-streaming branch from 969eaf1 to 515cf31 Compare November 17, 2026 07:25
Base automatically changed from index-writeback to main November 17, 2026 07:30
The index writer kept every block's RAM index entry in memory until
finalize, then encoded them all and wrote them in one piece just before
the final sync. For a 90B-key bijection index it held about 760 MB, the
only part of the writer's memory that grew with the key count. That last
write was 290 MB, with the sync right after it, so WithBytesPerSync could
not spread it out. The writer now encodes the entries into a 64 KiB
buffer and writes the buffer out each time it fills, and WithBytesPerSync
counts these writes. The index built is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DjQed1VX9uGU9PWpAjBiz7
@tamirms
tamirms force-pushed the ram-index-streaming branch from 515cf31 to 7ef8427 Compare November 17, 2026 07:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants