Performance: shard layout, call-free wrapped reads, fixed index load

README.md

@@ -95,7 +95,7 @@ counters take 2–4 KiB per CPU.
 
 | | Use it when | Watch out for |
 |:---|:---|:---|
-| `tinylfu` | **in doubt**: the workload is unknown, or has loops or scans, and is read far more than written | Evicting writes are 5–10% slower than in `s3fifo`, and with 1% writes on 8 CPUs it was 22% slower, for a reason not yet found. Hit ratios vary between runs by up to 5.6 points. |
+| `tinylfu` | **in doubt**: the workload is unknown, or has loops or scans, and is read far more than written | Evicting writes are 8–13% slower than in `s3fifo`. Hit ratios vary between runs by up to 5.6 points. |
 | `s3fifo` | you want the same results on every run, or the workload has no loops | Falls far behind on looping workloads: on Mixed, at some sizes, it hits as rarely as `lru`. Writes cost about twice what they do in `lru`. |
 | `sieve` | reading through has to be as fast as possible, and you have checked your workload | It can collapse: on Mixed it answers under 1% of requests where `lru` answers 44%; see [when not to use it](https://pkg.go.dev/github.com/floatdrop/evict/sieve#hdr-When_not_to_use_it). |
 | `twoq` | a quarter or more of the operations are writes, or one goroutine uses the cache | Reads take a lock; more memory than `lru`. |
@@ -158,21 +158,21 @@ Millions of operations per second on one CPU and on eight (`1 → 8`), against a
 
 | Benchmark | evict/lru | evict/twoq | evict/sieve | evict/s3fifo | evict/tinylfu | otter | golang-lru |
 |:---|---:|---:|---:|---:|---:|---:|---:|
-| `Get`: reads, all of them hits | 39.7 → 35.9 | 46.7 → 44.4 | 35.6 → 265.8 | 36.1 → 274.7 | 35.0 → 258.1 | 27.2 → 202.3 | 34.2 → 4.3 |
-| `Mixed`: the same, 10% of them writes | 37.6 → 34.0 | 42.5 → 42.8 | 32.9 → 68.6 | 33.4 → 69.9 | 33.1 → 62.6 | 21.7 → 80.8 | 33.5 → 4.4 |
-| `Set`: writes that replace | 36.5 → 24.9 | 43.3 → 28.4 | 22.4 → 13.8 | 22.5 → 13.6 | 22.2 → 12.8 | 8.9 → 15.9 | 32.1 → 4.3 |
-| `Evict`: writes that evict | 16.5 → 31.3 | 16.9 → 30.9 | 13.4 → 12.0 | 8.6 → 10.5 | 8.2 → 9.8 | 3.6 → 5.9 | 7.5 → 3.3 |
-| `ReadThrough`: read, store on a miss | 30.3 → 34.1 | 31.9 → 33.7 | 28.6 → 53.3 | 22.1 → 43.1 | 21.7 → 46.0 | 12.7 → 23.6 | 16.7 → 3.6 |
-| `ReadThrough`, a million entries | 8.8 → 30.1 | 8.4 → 28.7 | 8.9 → 41.5 | 7.2 → 36.8 | 7.5 → 37.8 | 4.3 → 17.9 | 6.3 → 2.9 |
+| `Get`: reads, all of them hits | 40.2 → 34.7 | 47.3 → 44.9 | 40.1 → 317.8 | 40.7 → 308.7 | 38.9 → 307.0 | 26.1 → 204.7 | 33.1 → 4.2 |
+| `Mixed`: the same, 10% of them writes | 40.3 → 32.4 | 46.4 → 38.9 | 35.7 → 71.6 | 36.6 → 64.2 | 35.3 → 61.4 | 19.7 → 80.3 | 33.3 → 3.8 |
+| `Set`: writes that replace | 38.1 → 24.1 | 44.0 → 26.9 | 22.0 → 11.8 | 22.4 → 12.6 | 22.5 → 11.6 | 8.7 → 15.3 | 31.8 → 3.6 |
+| `Evict`: writes that evict | 17.8 → 31.6 | 16.9 → 30.3 | 14.4 → 12.0 | 9.6 → 10.3 | 8.4 → 9.5 | 3.4 → 5.6 | 7.1 → 3.2 |
+| `ReadThrough`: read, store on a miss | 31.3 → 35.3 | 30.6 → 35.9 | 31.3 → 55.1 | 23.8 → 42.2 | 22.7 → 46.0 | 10.9 → 22.8 | 16.2 → 3.4 |
+| `ReadThrough`, a million entries | 7.2 → 28.6 | 7.6 → 27.3 | 7.3 → 41.1 | 6.8 → 35.5 | 6.8 → 37.3 | 4.0 → 17.2 | 5.9 → 2.7 |
 
 ![Throughput of ReadThrough with a million entries by number of CPUs](bench/results/throughput-readthrough-capacity-1m.svg)
 
-- Lock-free reads scale: on 8 CPUs `sieve`, `s3fifo` and `tinylfu` read 28–36% faster than
+- Lock-free reads scale: on 8 CPUs `sieve`, `s3fifo` and `tinylfu` read 50–55% faster than
   otter, while `lru` and `twoq` slow down past 4 CPUs.
-- On one CPU all five caches beat otter at everything, by 1.3 to 5 times.
+- On one CPU all five caches beat otter at everything, by 1.5 to 5 times.
 - `lru` and `twoq` change entries in place and write fastest. The lock-free caches move a
   replaced entry to a fresh node so that readers need no lock, and their writes cost 1.2 to
-  3.4 times as much.
+  3.3 times as much.
 - Otter wins on 8 CPUs when 10–25% of the operations overwrite entries.
 - These are numbers from one machine. Otter's own, from Linux on x86, show its writes
   scaling five-fold from one CPU to eight, against less than two-fold here.
@@ -184,25 +184,24 @@ Bytes per entry with `uint64` keys and values, over caches of 10,000 to 10 milli
 
 | | Bytes per entry | of which the GC scans | One GC cycle, 1M entries | 10M entries |
 |:---|---:|---:|---:|---:|
-| evict/lru | 46–58 | 0–1 | 0.2 ms | 0.2 ms |
-| evict/sieve | 54–72 | 0–2 | 0.4 ms | 0.2 ms |
-| evict/twoq | 74–82 | 0–1 | 0.4 ms | 0.2 ms |
-| evict/s3fifo | 68–90 | 0–2 | 0.3 ms | 0.2 ms |
-| evict/tinylfu | 67–82 | 0–2 | 0.4 ms | 0.2 ms |
-| otter | 83–94 | 40–50 | 64 ms | 634 ms |
-| golang-lru | 110–127 | 93–111 | 65 ms | 740 ms |
-| golang-lru/2q | 156–183 | 136–163 | 97 ms | 1,167 ms |
-
-Sizes swing with capacity because indexes come in powers of two. The garbage collector's
-work does not: it grows with caches built from pointers and stays near zero for these. That
-holds for keys and values without pointers; what strings cost the collector was not
-measured.
+| evict/lru | 56–63 | 0–2 | 0.2 ms | 0.2 ms |
+| evict/sieve | 64–81 | 0–2 | 0.3 ms | 0.3 ms |
+| evict/twoq | 84–93 | 0–1 | 0.3 ms | 0.2 ms |
+| evict/s3fifo | 82–100 | 0–2 | 0.5 ms | 0.2 ms |
+| evict/tinylfu | 76–95 | 0–2 | 0.4 ms | 0.2 ms |
+| otter | 83–95 | 40–50 | 65 ms | 530 ms |
+| golang-lru | 110–127 | 94–111 | 66 ms | 719 ms |
+| golang-lru/2q | 156–183 | 136–163 | 101 ms | 1,157 ms |
+
+The garbage collector's work grows with caches built from pointers and stays near zero for
+these. That holds for keys and values without pointers; what strings cost the collector
+was not measured.
 
 ## How it works
 
 - **Arrays, not pointers.** Each shard allocates its nodes once and links them by 32-bit
-  index. An open-addressing index maps a 32-bit hash to a node, so a lookup compares hashes
-  before it touches an entry.
+  index. An open-addressing index with three slots per entry maps a 32-bit hash to a node,
+  so a lookup compares hashes before it touches an entry.
 - **Shards.** The upper half of a key's hash picks a shard, which writers lock on its own;
   the lower half indexes the key within it. Each shard evicts from its own share of the
   capacity. A cache under 512 entries is one shard; a larger one has up to eight per CPU, of

bench/cmd/tune/main.go

@@ -380,7 +380,7 @@ The caches are in one piece. A setting is written as the options that make it:
 `+"`small`"+` is WithSmallRatio, `+"`ghosts`"+` WithGhostRatio, `+"`promotion`"+` WithPromotionReads and
 `+"`returns`"+` WithGhostReturns. Where settings do equally well, the standard one is named
 before the others, and one with fewer ghosts before one with more: a ghost takes about
-sixteen bytes.
+twenty bytes.
 
 Produced by `+"`%s`"+` with %s.
 

bench/results/memory.md

@@ -13,14 +13,14 @@ caches of this module store them in place.
 
 | cache | 10,000 | 100,000 | 1,000,000 | 10,000,000 |
 |:---|---:|---:|---:|---:|
-| evict/lru | 47.5 | 57.9 | 49.3 | 45.5 |
-| evict/twoq | 79.6 | 73.6 | 81.8 | 74.9 |
-| evict/sieve | 67.9 | 71.5 | 59.8 | 53.7 |
-| evict/s3fifo | 82.3 | 89.9 | 75.6 | 67.7 |
-| evict/tinylfu | 81.4 | 82.0 | 68.2 | 67.2 |
-| otter | 94.0 | 82.9 | 90.5 | 89.2 |
+| evict/lru | 61.4 | 63.1 | 56.6 | 56.0 |
+| evict/twoq | 92.7 | 89.3 | 84.4 | 84.0 |
+| evict/sieve | 81.4 | 76.7 | 67.2 | 64.3 |
+| evict/s3fifo | 100.2 | 96.4 | 85.5 | 82.4 |
+| evict/tinylfu | 94.9 | 87.3 | 75.6 | 77.8 |
+| otter | 94.9 | 83.1 | 90.5 | 89.2 |
 | golang-lru | 109.6 | 127.3 | 117.8 | 110.3 |
-| golang-lru/2q | 156.4 | 182.9 | 168.8 | 157.5 |
+| golang-lru/2q | 156.4 | 182.9 | 168.8 | 157.4 |
 
 ## Strings
 
@@ -29,14 +29,14 @@ overhead of the cache alone, including the 32 bytes of the two string headers.
 
 | cache | 10,000 | 100,000 | 1,000,000 | 10,000,000 |
 |:---|---:|---:|---:|---:|
-| evict/lru | 66.2 | 73.5 | 65.0 | 61.4 |
-| evict/twoq | 105.8 | 94.6 | 105.9 | 98.9 |
-| evict/sieve | 83.4 | 87.1 | 77.1 | 69.8 |
-| evict/s3fifo | 97.8 | 105.6 | 92.9 | 83.8 |
-| evict/tinylfu | 97.0 | 97.7 | 85.5 | 83.3 |
-| otter | 106.5 | 97.6 | 106.4 | 105.2 |
+| evict/lru | 79.6 | 78.8 | 72.4 | 72.0 |
+| evict/twoq | 118.9 | 110.3 | 108.5 | 108.1 |
+| evict/sieve | 96.9 | 92.4 | 84.5 | 80.4 |
+| evict/s3fifo | 115.7 | 112.2 | 102.8 | 98.5 |
+| evict/tinylfu | 110.4 | 103.0 | 92.9 | 93.9 |
+| otter | 106.5 | 97.6 | 106.5 | 105.2 |
 | golang-lru | 139.7 | 165.9 | 151.9 | 140.8 |
-| golang-lru/2q | 201.6 | 240.9 | 219.9 | 203.1 |
+| golang-lru/2q | 201.6 | 240.9 | 219.9 | 203.2 |
 
 ## Garbage collection
 
@@ -50,14 +50,14 @@ one run to the next.
 
 | cache | 10,000 | 100,000 | 1,000,000 | 10,000,000 |
 |:---|---:|---:|---:|---:|
-| evict/lru | 0.7 | 0.2 | 0.0 | 0.0 |
-| evict/twoq | 0.7 | 0.2 | 0.0 | 0.0 |
-| evict/sieve | 1.6 | 0.3 | 0.0 | 0.0 |
-| evict/s3fifo | 1.9 | 0.4 | 0.0 | 0.0 |
-| evict/tinylfu | 1.9 | 0.4 | 0.0 | 0.0 |
-| otter | 48.3 | 40.1 | 50.1 | 43.8 |
-| golang-lru | 93.4 | 111.3 | 101.8 | 94.3 |
-| golang-lru/2q | 136.3 | 162.9 | 148.7 | 137.4 |
+| evict/lru | 1.5 | 0.2 | 0.0 | 0.0 |
+| evict/twoq | 0.9 | 0.2 | 0.0 | 0.0 |
+| evict/sieve | 1.8 | 0.3 | 0.0 | 0.0 |
+| evict/s3fifo | 2.2 | 0.4 | 0.0 | 0.0 |
+| evict/tinylfu | 2.2 | 0.4 | 0.0 | 0.0 |
+| otter | 49.0 | 40.3 | 50.1 | 43.8 |
+| golang-lru | 93.5 | 111.3 | 101.8 | 94.3 |
+| golang-lru/2q | 136.1 | 162.9 | 148.7 | 137.4 |
 
 The second has the CPU time that one collection takes while the cache is alive and
 little else is, in milliseconds: the least and the most of seven collections. It is the
@@ -67,11 +67,11 @@ lower number is the one to go by.
 
 | cache | 10,000 | 100,000 | 1,000,000 | 10,000,000 |
 |:---|---:|---:|---:|---:|
-| evict/lru | 0.36–0.63 | 0.27–0.53 | 0.16–0.31 | 0.20–0.34 |
-| evict/twoq | 0.36–0.53 | 0.35–0.56 | 0.35–0.47 | 0.22–0.39 |
-| evict/sieve | 0.39–0.51 | 0.28–0.55 | 0.42–0.53 | 0.19–0.33 |
-| evict/s3fifo | 0.43–0.56 | 0.33–1.03 | 0.34–0.61 | 0.23–0.37 |
-| evict/tinylfu | 0.39–0.61 | 0.36–0.48 | 0.38–0.56 | 0.19–0.38 |
-| otter | 0.67–1.12 | 5.94–6.60 | 64.28–68.37 | 633.72–653.00 |
-| golang-lru | 1.30–1.61 | 8.50–9.00 | 64.80–67.87 | 740.43–863.89 |
-| golang-lru/2q | 1.26–2.21 | 11.84–12.96 | 97.42–102.44 | 1167.18–1230.22 |
+| evict/lru | 0.43–0.55 | 0.25–0.41 | 0.17–0.28 | 0.19–0.37 |
+| evict/twoq | 0.34–0.48 | 0.32–0.60 | 0.34–0.49 | 0.22–0.34 |
+| evict/sieve | 0.28–0.62 | 0.32–0.53 | 0.29–0.61 | 0.29–0.83 |
+| evict/s3fifo | 0.32–0.74 | 0.48–1.20 | 0.46–0.98 | 0.24–0.40 |
+| evict/tinylfu | 0.41–0.59 | 0.33–0.59 | 0.35–0.59 | 0.22–0.34 |
+| otter | 0.63–0.88 | 5.48–6.25 | 65.18–69.80 | 530.03–584.60 |
+| golang-lru | 1.25–1.69 | 7.34–9.69 | 66.21–72.52 | 719.26–828.59 |
+| golang-lru/2q | 1.63–2.02 | 11.80–13.09 | 100.53–109.53 | 1157.10–1312.65 |