Skip to content

Performance: shard layout, call-free wrapped reads, fixed index load - #6

Merged
floatdrop merged 4 commits into
mainfrom
perf-review
Nov 12, 2026
Merged

Performance: shard layout, call-free wrapped reads, fixed index load#6
floatdrop merged 4 commits into
mainfrom
perf-review

Conversation

@floatdrop

@floatdrop floatdrop commented Nov 12, 2026

Copy link
Copy Markdown
Owner

Three changes from a performance review, each its own commit, then a fresh throughput run.

1. Lay shards out in whole cache lines (9bcafbc)
Shard sizes were not multiples of 128 B, so the slab header that every lock-free read touches straddled two lines in some shards. A per-package padded wrapper rounds shards to whole lines, and the cslab fields only writers use move below its pad (readers' part: 113 → 81 B).

  • 8 CPUs, 1% writes: tinylfu 5.32 → 4.62 ns/op (level with s3fifo), sieve −6%. This was the "22% slower, reason not found" gap.

2. Keep reads free of calls when a cache is wrapped in a generic type (c6e0e81)
Reached through a generic type of another package that instantiates nothing (as bench/caches does), the read path made ~10 real calls (typed-atomic methods, Readers.Pin), one of them while a node was pinned. The words readers touch are now plain integers accessed with sync/atomic functions (intrinsics); Node.Meta()/SetMeta() are the accessors; pinning moved to generic Slab methods. internal/inlinecheck gains a wrapper probe that fails on the old code.

  • Through the wrapper: reads −2–7%, tinylfu with 1% writes −9%. Direct use: 25% writes on 8 CPUs ≈ −1%; single-thread string-key reads +1–3% (same instructions, different layout).

3. Give every hash index three slots per entry (5c9de57)
Power-of-two tables let the load swing between 1/3 and 2/3 with capacity; lookups were up to 40% slower at 2/3. Tables in slab, cslab and ghost now have max(3·cap, 1) slots of any size, with a multiply-shift home slot and wrap-around probing and deletion.

  • Where the load was 2/3 or 1/2: misses 25–40% faster, evicting writes up to 29% faster; unchanged where it was 1/3.
  • Memory +3 to +18 B per entry; s3fifo and tinylfu are now about level with otter. memory.md and the README say so.
  • Collision tests place hashes with homeAt so they still cover wrap-around and the stay branch of deletion.

4. Measure throughput again (edb24d7)
5 runs on 1/2/4/8 CPUs. On 8 CPUs the lock-free caches read 50–55% faster than otter (was 28–36%); the README's throughput table and claims follow.

Tried and dropped: lazy promotion in lru (15–23% faster at 4–8 CPUs, 12% slower on 1, ends exact LRU), spinning writers before they park (no gain beyond noise), a maphash.String fast path (slower), and a ghost index that holds the hash (would cost 7–11 B per entry).

Each commit was reviewed separately; go test -race and golangci-lint (incl. 386) pass on each.

🤖 Generated with Claude Code

@floatdrop floatdrop mentioned this pull request Nov 12, 2026
Merged
floatdrop and others added 4 commits November 12, 2026 20:13
@floatdrop @claude
Lay shards out in whole cache lines
1a60277 
The shards of a cache lie side by side, and their sizes were not multiples
of 128 bytes, so each one sat at another place in a line, and in some of
them the slab header that every read touches lay across two lines. Each
cache now wraps its shard in a type that pads it with a line or more and
rounds it up to whole lines, so that every shard starts at the same place
in one. The fields of a cslab that only writers use move below its pad,
which leaves what a read needs at 81 bytes: on one line even where the
array starts a word into one, behind the allocator's header, as most do.

With 8 CPUs and 1% writes, tinylfu goes from 5.32 to 4.62 ns per
operation, level with s3fifo, and sieve from 4.82 to 4.55. Nothing else
moves measurably.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@floatdrop @claude
Keep reads free of calls when a cache is wrapped in a generic type
538cd5b 
The methods of a cache are generic, so they are compiled in the package
that instantiates them. Where that package gets to the cache through a
generic type of another package that instantiates nothing, as the
benchmarks do, the compiler does not have the bodies of the atomic types'
methods or of Readers.Pin, and a read made about ten calls, one of them
while a node was pinned.

The words that readers touch are now plain integers accessed with the
functions of sync/atomic, which the compiler builds in: the index, seq
and overdue of a slab, the meta of a node, which gets Meta and SetMeta,
and the pin and the hits of a slot. seq and overdue become 32 bits wide,
and the 64-bit words all lie in slices, which keeps them aligned on 32-bit
platforms. Pinning moves from Readers to generic methods of the slab. The
inline check gains a wrapper and a package that uses it, and makes sure
that reads there call nothing but the lock they rarely take.

Through the wrapper of the benchmarks, reads are 2-7% faster and, with 1%
writes on 8 CPUs, tinylfu is 9% faster. Used directly, single-threaded
reads with string keys are 1-3% slower, the same instructions laid out
differently, and 25% writes on 8 CPUs are about 1% faster.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@floatdrop @claude
Give every hash index three slots per entry
5f5c220 
The indexes of the slabs and of the ghosts were tables of a power of two
in size, the smallest above 1.5 times the capacity, so that their load
went from a third to two thirds as the capacity grew to the next power
of two. Lookups were up to 40% slower at two thirds than at a third, at
capacities one entry apart.

A table now has three slots for every entry, whatever the capacity, and
a hash's probe starts at the hash scaled to the size of the table rather
than at its lower bits. Probes and the moves that close a gap wrap around
the end of a table of any size. Sizes are worked out in uint, so the
largest still fits 32-bit platforms.

Where the load was two thirds or a half, misses are 25-40% faster and
writes that evict up to 29% faster; where it was a third, nothing
changes. With 8 CPUs, reads in the benchmarks are 5-9% faster. An entry
takes 3 to 18 bytes more, a ghost about twenty, and memory.md and the
README say so.

Tests that took keys for hashes would now have every entry in one slot,
so they spread their keys, and those that collide on purpose put their
hashes where they want them, across the end of the table among others.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@floatdrop @claude
Measure throughput again, after the three changes
f9d0d36 
The benchmarks ran five times on 1, 2, 4 and 8 CPUs on the branch, and
the README follows. On 8 CPUs sieve, s3fifo and tinylfu now read 50-55%
faster than otter, against 28-36% before. With 1% writes on 8 CPUs,
tinylfu is level with s3fifo: the gap of 22% that the README could not
explain was the layout of the shards, and is gone, so the README no
longer mentions it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@floatdrop
floatdrop merged commit c29254e into main Nov 12, 2026
7 checks passed
@floatdrop
floatdrop deleted the perf-review branch November 12, 2026 16:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@floatdrop