Repository navigation
Conversation
04ba306 to
969eaf1
Compare
|
|
||
| // flushRAMIndex writes the buffered RAM index entries after those already | ||
| // written and empties the buffer. | ||
| func (iw *indexWriter) flushRAMIndex() error { |
There was a problem hiding this comment.
Non-blocking, and not caused by this PR: the same result occurs on main.
The RAM index can get one more entry than numBlocks + 1, and the write then goes into the payload region with no error.
How it occurs (sorted, single-threaded build):
commitBlockWithDataappends the RAM index entry before the writes that can fail.- A write fails. The entry stays in the buffer, but
blockCountdoes not advance. SortedBuilder.AddKeyreturns the error but does not mark the builder as failed.- The caller calls
AddKeyagain. The builder commits that block a second time and appends a second entry. Finishreturns nil. The sentinel is written 10 bytes into the payload region, and all later entries are shifted by one.
I reproduced this with 40,000 keys (14 blocks). I replaced iw.file with a read-only handle for one AddKey call at a block boundary, restored it, and repeated the call. Result on main and on this branch: Finish nil, Open nil, Verify returns a checksum error, and most queries return errors or duplicate ranks.
The parallel and unsorted paths are not affected. A writer error stops the writer goroutine there, and each Finish error removes the output.
Now that the write position is known at each flush, one comparison stops the write:
func (iw *indexWriter) flushRAMIndex() error {
if iw.ramIndexWriteOffset+uint64(len(iw.ramIndexBuf)) > iw.payloadRegionOffset {
return errors.New("RAM index overflow: more entries than numBlocks+1")
}
...With this check, Finish fails in the reproduction and the builder removes the file. The short test suite still passes.
The check is a backstop. The root fix is to make the builder fail permanently after a commit error. Adding the RAM index entry after the writes is not enough on its own, because a failed payload write leaves the metadata offset and the metadata checksum already advanced.
969eaf1 to
515cf31
Compare
The index writer kept every block's RAM index entry in memory until finalize, then encoded them all and wrote them in one piece just before the final sync. For a 90B-key bijection index it held about 760 MB, the only part of the writer's memory that grew with the key count. That last write was 290 MB, with the sync right after it, so WithBytesPerSync could not spread it out. The writer now encodes the entries into a 64 KiB buffer and writes the buffer out each time it fills, and WithBytesPerSync counts these writes. The index built is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DjQed1VX9uGU9PWpAjBiz7
515cf31 to
7ef8427
Compare
The index writer keeps every block's RAM index entry in memory until the build finishes. It then encodes them all and writes them in one piece, just before the final sync. Everything else the writer holds is a fixed size, but this part grows with the key count: 16 bytes per block, plus a 10-byte-per-block encode buffer at the end. For a 90 billion key bijection index, which has 29 million blocks, that is about 760 MB. That final write, about 290 MB, comes right before the sync, so
WithBytesPerSynccan't spread it out.The RAM index's position and size in the file are fixed when the file is created, and its entries are produced in block order. So the writer can write them as blocks commit. It now encodes the entries into a 64 KiB buffer and writes the buffer to its place in the file each time it fills.
WithBytesPerSynccounts those writes.The file layout does not change. Indexes built before and after the change are byte-identical across both algorithms, sorted and unsorted builds, a range of payload sizes, fingerprint sizes, worker counts and key counts, and builds with and without user metadata. A new test commits block counts just under, at and over the buffer's size and checks the RAM index byte for byte.
Measured with 4.3 billion keys (1.4 million blocks), 16 workers, a 3-byte payload and a 1-byte fingerprint, two runs each:
Finish🤖 Generated with Claude Code