Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,7 @@ rank, err := idx.QueryRank(streamhash.PreHash(originalKey))
| `WithGlobalSeed(seed)` | Hash seed (change on build failure) | fixed default |
| `WithMetadata(data)` | Arbitrary metadata stored in the file | none |
| `WithBytesPerSync(n)` | Start writeback of the index every `n` bytes written (Linux) | 0 (left to the kernel) |
| `WithYield(yield)` | Call `yield` before each block: one that blocks holds the build back, an error fails it | none (never yields) |

## Choosing an Algorithm

Expand Down
15 changes: 15 additions & 0 deletions builder.go
Original file line number Diff line number Diff line change
Expand Up @@ -246,9 +246,21 @@ func (b *builder) resetCurrentBuilder() {
b.builder.Reset()
}

// yield calls the WithYield function, if one was given. It runs before every
// block is built, empty blocks included.
func (b *builder) yield(ctx context.Context) error {
if b.cfg.yield == nil {
return nil
}
return b.cfg.yield(ctx)
}

// commitEmptyBlock writes an empty block's metadata directly to the index.
// Resets the builder, builds empty metadata, and commits via pwrite.
func (b *builder) commitEmptyBlock() error {
if err := b.yield(b.ctx); err != nil {
return err
}
b.builder.Reset()
metadataNeeded := b.estimateMetadataSize()
if cap(b.metadataBuf) < metadataNeeded {
Expand Down Expand Up @@ -309,6 +321,9 @@ func (b *builder) commitTrailingEmptyBlocksAndFinalize() error {

// buildBlock builds the current block into reusable buffers and writes via pwrite.
func (b *builder) buildBlock() error {
if err := b.yield(b.ctx); err != nil {
return err
}
numKeys := b.currentBlockKeysAdded()
entrySize := b.cfg.payloadSize + b.cfg.fingerprintSize

Expand Down
16 changes: 16 additions & 0 deletions builder_options.go
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
package streamhash

import "context"

const (
maxPayloadSize = 8
maxFingerprintSize = 4
Expand All @@ -18,6 +20,7 @@ type buildConfig struct {
unsortedTempDir string // temp directory for unsorted partition files
algorithm Algorithm
bytesPerSync int // start writeback of the index every this many bytes written; 0 = off
yield func(context.Context) error

totalKeys uint64 // Pre-known key count for single-pass builds
}
Expand Down Expand Up @@ -89,6 +92,19 @@ func WithBytesPerSync(n int) BuildOption {
}
}

// WithYield makes the build yield to other work: before each block, the
// goroutine about to build it calls yield and goes on when it returns. A
// yield that blocks holds the build back, and one that returns an error
// fails the build with it. The context is the build's, done once the build
// is canceled or has failed. Nothing is held while yield blocks: a build at
// rest holds only its buffers and its open files. The default never yields,
// and WithYield(nil) never yields: it is the same as not passing the option.
func WithYield(yield func(ctx context.Context) error) BuildOption {
return func(c *buildConfig) {
c.yield = yield
}
}

// WithAlgorithm sets the block construction algorithm.
// Default is AlgoBijection.
func WithAlgorithm(algo Algorithm) BuildOption {
Expand Down
15 changes: 13 additions & 2 deletions builder_parallel.go
Original file line number Diff line number Diff line change
Expand Up @@ -202,9 +202,13 @@ func (b *builder) loadBlock(blk blockBuilder, entries []routedEntry) {

// build solves the loaded block as blockID, writes its payloads after the
// keysBefore keys of the blocks before it, and returns the result for the
// writer. The metadata buffer comes from the pool; the writer returns it.
// writer. The metadata buffer comes from the pool; the writer returns it. It
// yields first, and a yield error is the block's result.
func (s *blockSolver) build(blockID uint32, keysBefore uint64) blockResult {
b := s.b
if err := b.yield(b.workerCtx); err != nil {
return blockResult{blockID: blockID, err: err}
}
entrySize := b.cfg.payloadSize + b.cfg.fingerprintSize
payloadsNeeded := s.blk.KeysAdded() * entrySize
if cap(s.payloadsBuf) < payloadsNeeded {
Expand All @@ -228,8 +232,15 @@ func (s *blockSolver) build(blockID uint32, keysBefore uint64) blockResult {
// returning it cancels workerCtx, which releases any peer workers blocked
// sending on resultChan after the writer exits on the first error --
// otherwise they (and drainParallelPipeline's workerGroup.Wait) deadlock.
// Wait then surfaces this error to the build.
// Wait then surfaces this error to the build. An error once workerCtx is
// done is not sent: it may only echo the cancellation, as a yield's does,
// and the writer must not report it in place of the failure.
func (b *builder) sendResult(result blockResult) error {
if result.err != nil {
if err := b.workerCtx.Err(); err != nil {
return err
}
}
select {
case b.resultChan <- result:
return result.err
Expand Down
8 changes: 7 additions & 1 deletion streamhash-spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -816,7 +816,13 @@ With `WithWorkers(n)`, both sorted and unsorted builds solve blocks concurrently

Throughput scales near-linearly because the serialized step (writing metadata, folding hashes) is cheap relative to solving. The pipeline is carefully torn down on error or cancellation so a failing block, a writer-side I/O error, or a context cancel never deadlocks the pool — any internal failure cancels the worker context exactly once, releasing every parked goroutine.

### 6.4. Thread safety
### 6.4. Yielding to other work

`WithYield(yield)` lets a build give way to other work between blocks. Before each block, empty blocks included, the goroutine about to build it calls `yield` with the build's context and goes on when it returns. With one worker that is the calling goroutine; with several, it is a worker. A block is about 3,000 keys with Bijection and 31,600 with PTRHash (§7.3), and a `yield` that starts blocking stops each worker at its next block.

While `yield` blocks, the rest of the build stops behind the parked workers: what feeds them stops once their queue is full, and the writer once it has committed the blocks already solved. A parked build holds only its buffers and open files. A `yield` that returns an error fails the build with that error, as a block that fails to build does. The context is done once the build is canceled or has failed, so a `yield` that waits on it returns then.

### 6.5. Thread safety

- **Queries** are thread-safe and lock-free once the index is open: the data is immutable and each query reads an independent block. The RAM index must be fully loaded and visible before concurrent queries begin.
- **Solvers** are single-threaded; each parallel worker has its own.
Expand Down
Loading
Loading