diff --git a/Makefile b/Makefile index 3bf19e5..df443f5 100644 --- a/Makefile +++ b/Makefile @@ -35,6 +35,7 @@ fuzz: ## Fuzz every target for FUZZTIME (default 30s) each. $(GO) test -run='^$$' -fuzz=FuzzAgainstModel -fuzztime=$(FUZZTIME) ./twoq $(GO) test -run='^$$' -fuzz=FuzzAgainstModel -fuzztime=$(FUZZTIME) ./sieve $(GO) test -run='^$$' -fuzz=FuzzAgainstModel -fuzztime=$(FUZZTIME) ./s3fifo + $(GO) test -run='^$$' -fuzz=FuzzAgainstModel -fuzztime=$(FUZZTIME) ./tinylfu cd bench && $(GO) test -run='^$$' -fuzz=FuzzRead -fuzztime=$(FUZZTIME) ./trace .PHONY: bench diff --git a/README.md b/README.md index 31e425c..1c4c36d 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ [![Go Reference](https://pkg.go.dev/badge/github.com/floatdrop/evict.svg)](https://pkg.go.dev/github.com/floatdrop/evict) [![CI](https://github.com/floatdrop/evict/actions/workflows/ci.yml/badge.svg)](https://github.com/floatdrop/evict/actions/workflows/ci.yml) -Fixed-capacity in-memory caches for Go: one API, four eviction policies. +Fixed-capacity in-memory caches for Go: one API, five eviction policies. | Package | Evicts | Reads | |:---|:---|:---| @@ -11,6 +11,7 @@ Fixed-capacity in-memory caches for Go: one API, four eviction policies. | [`twoq`](https://pkg.go.dev/github.com/floatdrop/evict/twoq) | with [2Q](http://www.vldb.org/conf/1994/P439.PDF), which keeps new entries apart from proven ones | take the lock of a shard | | [`sieve`](https://pkg.go.dev/github.com/floatdrop/evict/sieve) | with [SIEVE](https://www.usenix.org/conference/nsdi24/presentation/zhang-yazhuo), a queue with lazy promotion | take no lock | | [`s3fifo`](https://pkg.go.dev/github.com/floatdrop/evict/s3fifo) | with [S3-FIFO](https://dl.acm.org/doi/10.1145/3600006.3613147), which makes new entries prove themselves in a small queue first | take no lock | +| [`tinylfu`](https://pkg.go.dev/github.com/floatdrop/evict/tinylfu) | with S3-FIFO's queues and the admission of [TinyLFU](https://arxiv.org/abs/1512.00727): once the cache is full, a new entry gets into its main queue only if its key has been asked for more often of late than the entry it would push out | take no lock | All of them @@ -21,7 +22,7 @@ All of them - **tell you what they dropped**: `Set` returns the entry it replaced or evicted, so that whatever the entry held on to can be released; - **are safe for concurrent use**: entries are spread over shards that are locked one by one, - and `sieve` and `s3fifo` read without a lock at all; + and `sieve`, `s3fifo` and `tinylfu` read without a lock at all; - depend on nothing but the standard library. ``` @@ -38,11 +39,11 @@ package main import ( "fmt" - "github.com/floatdrop/evict/s3fifo" + "github.com/floatdrop/evict/tinylfu" ) func main() { - cache := s3fifo.New[string, int](10_000) + cache := tinylfu.New[string, int](10_000) cache.Set("answer", 42) @@ -62,7 +63,7 @@ func main() { } ``` -The four packages have the same methods, which are also the interface +The five packages have the same methods, which are also the interface [`evict.Cache`](https://pkg.go.dev/github.com/floatdrop/evict#Cache): | Method | | @@ -75,36 +76,41 @@ The four packages have the same methods, which are also the interface | `Clear()` | Removes every entry. | | `All()` | Iterates over the entries. | -`New` takes options. All four packages have `WithShards`. `twoq` has `WithRecentRatio` and +`New` takes options. All five packages have `WithShards`. `twoq` has `WithRecentRatio` and `WithGhostRatio` for the sizes of its queues, and `s3fifo` has `WithSmallRatio`, `WithGhostRatio`, `WithPromotionReads` and `WithGhostReturns` for tuning it to a workload -that is known; see [what that is worth](bench/results/tuning.md). +that is known; see [what that is worth](bench/results/tuning.md). `tinylfu` has nothing else +to set: it finds the size of its small queue by itself. ## Choosing a policy -**If in doubt, use `s3fifo`.** Of the four it answers the most requests on nine of the eleven -traces that the benchmarks replay, its reads take no lock, and it never does much worse -than `lru`: about two points at the most, where `sieve` can lose everything. It has a weak spot -of its own, though. On the trace called Mixed, at some capacities, it answers what `lru` -answers while `twoq` and otter answer four to six times as much. +**If in doubt, use `tinylfu`.** On the eleven traces that the benchmarks replay it answers +more requests on average than any other cache here, otter included: 47.9% of them, against +47.5% for otter and 44.8% for `s3fifo`. At no capacity is it more than 5.3 points behind +the best cache, and its reads take no lock. Two things temper that. It was designed with +these very traces at hand; on fourteen others, in a simulation of the policy that is not +part of this repository, it came out level with S3-FIFO and W-TinyLFU. And it has weak +spots of its own: on the trace called Mixed, at its larger capacities, it answers up to four +points less than `lru`, and where every key comes back once or twice after much the same +while and never again, it can answer far less than `lru` does; see +[its documentation](https://pkg.go.dev/github.com/floatdrop/evict/tinylfu). | | Use it when | Mind | |:---|:---|:---| -| `s3fifo` | you do not know the workload, or it is read far more than it is written | Writes cost about twice what they do in `lru`, and at worst three times. A cache of fewer than ten entries per shard is too small for it. | -| `sieve` | several CPUs read through the cache and it has to be as fast as can be, in less memory than `s3fifo` takes, and you have measured that your workload is one it copes with | It can fail outright. On one of the traces a SIEVE cache in one piece answers less than 1% of the requests where LRU answers 44%; see [when not to use it](https://pkg.go.dev/github.com/floatdrop/evict/sieve#hdr-When_not_to_use_it). On the traces here its hit ratio is ahead of that of `s3fifo` on one only. | +| `tinylfu` | you do not know the workload, or it has loops or scans in it, and it is read far more than it is written | Writes that evict get 5 to 10% less done than in `s3fifo`, and with eight CPUs and 1% writes among the reads it was 22% slower than `s3fifo` here. Its hit ratio varies from one run to the next, by up to 5.6 points on these traces: keys share counters in its sketch by a hash that is seeded at random. | +| `s3fifo` | you want the same answers on every run, or the workload is read far more than it is written and has no loops in it | It falls far behind on Mixed, a trace with loops, and a few points behind on Glimpse. On Mixed, at some capacities, it answers what `lru` answers while `tinylfu`, `twoq` and otter answer four to six times as much. Writes cost about twice what they do in `lru`, and at worst three times. A cache of fewer than ten entries per shard is too small for it. | +| `sieve` | several CPUs read through the cache and it has to be as fast as can be, in less memory than `s3fifo` and `tinylfu` take, and you have measured that your workload is one it copes with | It can fail outright. On one of the traces a SIEVE cache in one piece answers less than 1% of the requests where LRU answers 44%; see [when not to use it](https://pkg.go.dev/github.com/floatdrop/evict/sieve#hdr-When_not_to_use_it). On the traces here its hit ratio is ahead of that of `s3fifo` on one only. | | `twoq` | a quarter or more of the operations are writes, or one goroutine at a time uses the cache | It takes more memory than `lru` and `sieve`, because it keeps a node for every key it remembers. Reads take a lock. | -| `lru` | memory is what counts, there are a great many small caches, or you want the behavior that everyone knows | The lowest hit ratio of the four on nine traces of eleven. Reads take a lock. | +| `lru` | memory is what counts, there are a great many small caches, or you want the behavior that everyone knows | The lowest hit ratio of the five on nine traces of eleven. Reads take a lock. | And when to use [otter](https://github.com/maypok86/otter) instead: when you need what -these caches do not have (expiry, weights, loading, statistics); when the worst case matters -more than the usual one, since otter is never more than six points behind the best cache -on these traces and each of the four here is thirty or more behind somewhere; or when eight -CPUs read a cache while a tenth to a quarter of their operations overwrite entries. (For -writes alone, `twoq` and `lru` stay ahead.) The numbers are below. +these caches do not have (expiry, weights, loading, statistics), or when eight CPUs read a +cache while a tenth to a quarter of their operations overwrite entries. (For writes alone, +`twoq` and `lru` stay ahead.) The numbers are below. ## Benchmarks -The benchmarks are in [`bench`](bench), a module of its own, and compare the four caches +The benchmarks are in [`bench`](bench), a module of its own, and compare the five caches with otter v2, the state of the art among Go caches, and with hashicorp's `golang-lru`, a cache behind a single lock. [bench/README.md](bench/README.md) says what is measured and how; `make bench` runs it all again. What follows is a summary of @@ -115,56 +121,63 @@ how; `make bench` runs it all again. What follows is a summary of The share of requests, in percent, that a cache answers when a trace is replayed against it, averaged over the eight capacities at which each trace is replayed. Ten of the traces are public recordings of real systems, checked against a SHA-256 before they are used; -`optimal` is the bound that no cache can beat. Otter's hit ratio differs from one replay to -the next, by up to 3.8 points, so it is replayed eleven times: its number is the median, -and a number is in bold only if the cache leads whichever of those replays it is compared -with. - -| trace | evict/lru | evict/twoq | evict/sieve | evict/s3fifo | otter | golang-lru/2q | optimal | -|:---|---:|---:|---:|---:|---:|---:|---:| -| Zipf | 29.97 | 36.53 | 37.55 | **38.25** | 37.86 | 37.61 | 44.07 | -| S3 | 23.33 | 32.78 | 35.54 | 38.90 | **44.57** | 33.15 | 59.37 | -| DS1 | 23.70 | 28.88 | 38.87 | 37.05 | **46.52** | 34.47 | 50.21 | -| OLTP | 32.40 | 39.07 | 30.60 | 39.11 | 38.92 | 37.60 | 52.77 | -| P8 | 24.10 | 32.11 | 27.66 | 35.31 | 36.22 | 31.45 | 53.53 | -| Glimpse | 21.81 | 27.40 | 30.14 | 43.24 | **45.21** | 33.20 | 47.81 | -| Mixed | 44.11 | 56.80 | 0.43 | 43.84 | **61.62** | 35.97 | 64.22 | -| Scarab | 59.10 | 61.00 | 60.53 | **61.60** | 58.97 | 61.14 | 70.30 | -| Twitter | 58.37 | 59.72 | 58.99 | **60.32** | 59.28 | 58.80 | 65.46 | -| MSR | 64.40 | 66.31 | 66.16 | **67.22** | 66.56 | 66.57 | 73.60 | -| CloudPhysics | 20.90 | 24.41 | 24.70 | **27.59** | 26.73 | 25.59 | 34.23 | - -- **`s3fifo` and otter share the wins.** Of the 88 combinations of trace and capacity, - `s3fifo` wins 37 outright and otter 31. Six go to one of the 2Q caches, and 14 have no - clear winner. -- **`s3fifo` is ahead** on Zipf and on the traces of a recommendation service, a key-value - cache and block storage (Scarab, Twitter, MSR, CloudPhysics), by 0.4 to 2.6 points on - average and by no more than 4.8 at any capacity. -- **Otter is ahead by more where it is ahead**: on two of the disk traces of the ARC paper - and on the two traces with loops in them (S3, DS1, Glimpse, Mixed), by 2 to 18 points - over `s3fifo` on average, and by 61 at the capacity where `s3fifo` does worst. Against the - best of the four on each of those traces, which is `sieve` on DS1 and `twoq` on Mixed, it - is 2 to 8 points. On OLTP and P8 the two are within a point of each other. +`optimal` is the bound that no cache can beat. The hit ratios of otter and of `tinylfu` +differ from one replay to the next, by up to 4.1 and 5.6 points, so they are replayed eleven +times: their numbers are medians, and a number is in bold only if the cache leads whichever +of those replays it is compared with. + +| trace | evict/lru | evict/twoq | evict/sieve | evict/s3fifo | evict/tinylfu | otter | golang-lru/2q | optimal | +|:---|---:|---:|---:|---:|---:|---:|---:|---:| +| Zipf | 29.97 | 36.53 | 37.55 | **38.25** | 37.66 | 37.90 | 37.61 | 44.07 | +| S3 | 23.33 | 32.78 | 35.54 | 38.90 | **44.83** | 44.57 | 33.15 | 59.37 | +| DS1 | 23.70 | 28.88 | 38.87 | 37.05 | **47.07** | 46.54 | 34.47 | 50.21 | +| OLTP | 32.40 | 39.07 | 30.60 | 39.11 | 38.76 | 38.97 | 37.60 | 52.77 | +| P8 | 24.10 | 32.11 | 27.66 | 35.31 | **37.86** | 36.20 | 31.45 | 53.53 | +| Glimpse | 21.81 | 27.40 | 30.14 | 43.24 | 45.20 | 45.28 | 33.20 | 47.81 | +| Mixed | 44.11 | 56.80 | 0.43 | 43.84 | 60.32 | **61.61** | 35.97 | 64.22 | +| Scarab | 59.10 | 61.00 | 60.53 | **61.60** | 60.31 | 58.99 | 61.14 | 70.30 | +| Twitter | 58.37 | 59.72 | 58.99 | **60.32** | 59.59 | 59.29 | 58.80 | 65.46 | +| MSR | 64.40 | 66.31 | 66.16 | **67.22** | 67.06 | 66.54 | 66.57 | 73.60 | +| CloudPhysics | 20.90 | 24.41 | 24.70 | 27.59 | 27.80 | 26.77 | 25.59 | 34.23 | + +- **`s3fifo` and `tinylfu` share the wins.** Of the 88 combinations of trace and capacity, + `s3fifo` wins 32 outright and `tinylfu` 25. Six go to one of the 2Q caches, one to otter, + and 24 have no clear winner. +- **`s3fifo` is ahead** on Zipf and OLTP and on the traces of a recommendation service, a + key-value cache and block storage (Scarab, Twitter, MSR), by 0.2 to 1.3 points over + `tinylfu` on average and by no more than 2.5 at any capacity. +- **`tinylfu` and otter are ahead by more where they are ahead**: on two of the disk traces + of the ARC paper and on the two traces with loops in them (S3, DS1, Glimpse, Mixed), by 2 + to 18 points over `s3fifo` on average, and by 58 and 61 at the capacity where `s3fifo` + does worst. On P8 and CloudPhysics `tinylfu` is ahead of every other cache. +- **`tinylfu` and otter are close.** `tinylfu` is ahead on average on seven of the traces, + by up to 1.7 points (P8), and otter on the other four, by up to 1.3 (Mixed). Of the 88 + combinations, `tinylfu` is ahead on every replay in 42 and behind in 14. These are the + traces that `tinylfu` was designed with, which favors it. - **`sieve` is close behind `s3fifo` where it works**: about a point on Zipf, Scarab, Twitter and MSR, about three on S3 and CloudPhysics, and ahead of it on DS1. Where it does not work it is far behind: 8 to 13 points on OLTP, P8 and Glimpse, and on Mixed it answers next to nothing. - **`twoq` is ahead of `lru` on the average of every trace**, though not at every capacity: - `lru` leads in 10 of the 88 combinations. It is the best of the four on Mixed. -- **Otter has no bad trace.** In none of the 88 combinations is it more than 5.9 points - behind the best cache. Each of the four here is somewhere: `twoq` by 32 points, `s3fifo` - and `lru` by 63, `sieve` by 76. + `lru` leads in 10 of the 88 combinations. Of the five here only `tinylfu` beats it on Mixed. +- **Neither otter nor `tinylfu` has a bad trace.** In none of the 88 combinations is otter + more than 5.9 points behind the best cache, or `tinylfu` more than 5.3. Each of the other + four here is somewhere: `twoq` by 32 points, `s3fifo` and `lru` by 63, `sieve` by 76. - Averages hide a lot. [hitratio.md](bench/results/hitratio.md) has every capacity, with a chart for each trace. These numbers are for caches in one piece, so that the policy is all that is compared. A program gets shards by default, and [sharding.md](bench/results/sharding.md) has what that -changes to the median of eleven runs: less than a tenth of a point for `lru`, about a -tenth at the most for `s3fifo`, the same for `twoq` but for two cases of half a point and -of a point. For `sieve` it changes a great deal on two of the traces it does badly on, for -the better: 3 to 4 points on OLTP and anything from 2 to 41 on Mixed, depending on the -capacity. On P8 it is a point at the most, and on Glimpse nothing. It was measured for a machine with eight CPUs; one with more gives a cache of -some size more and smaller shards. +changes to the median of eleven runs: about a tenth of a point at the most for `s3fifo`; +the same for `lru` but for one case of more than a point on Glimpse, a short trace; and the +same for `twoq` but for two cases of half a point and of a point. For `sieve` it changes a +great deal on two of the traces it does badly on, for the better: 3 to 4 points on OLTP and +anything from 2 to 41 on Mixed, depending on the capacity. On P8 it is a point or so at the +most, and on Glimpse nothing. `tinylfu` loses a little in shards: about a point on DS1 at a +million entries, half a point on S3 and P8 at their smaller capacities, and 2.6 points on +Mixed at 750 entries; some of what the report shows for it is not the shards but how much +it varies from one replay to the next anyway. It was measured for a machine with eight CPUs; +one with more gives a cache of some size more and smaller shards. ### Throughput @@ -174,35 +187,46 @@ numbers are medians of five runs on an Apple M3 Max with Go 1.27.3; [throughput.md](bench/results/throughput.md) has two and four CPUs as well, the range of the runs, and which differences are beyond noise. -| Benchmark | evict/lru | evict/twoq | evict/sieve | evict/s3fifo | otter | golang-lru | -|:---|---:|---:|---:|---:|---:|---:| -| `Get`: reads, all of them hits | 37.0 → 36.4 | 48.3 → 47.3 | 35.0 → 278.0 | 36.2 → 276.6 | 26.6 → 205.3 | 33.6 → 4.1 | -| `Mixed`: the same, 10% of them writes | 37.3 → 32.2 | 44.5 → 41.5 | 33.0 → 67.7 | 33.0 → 67.8 | 21.4 → 81.9 | 33.6 → 4.2 | -| `Set`: writes that replace | 35.9 → 25.3 | 43.4 → 29.3 | 21.9 → 12.9 | 21.7 → 13.1 | 8.6 → 15.7 | 31.7 → 4.2 | -| `Evict`: writes that evict | 16.5 → 31.4 | 16.8 → 31.4 | 13.3 → 12.0 | 8.5 → 10.5 | 3.5 → 5.8 | 7.4 → 3.2 | -| `ReadThrough`: read, store on a miss | 30.1 → 34.5 | 31.6 → 34.8 | 28.5 → 52.1 | 21.9 → 43.2 | 12.4 → 23.6 | 16.5 → 3.5 | -| `ReadThrough`, a million entries | 8.9 → 29.4 | 7.8 → 28.6 | 8.4 → 41.6 | 7.3 → 37.0 | 4.1 → 17.7 | 6.2 → 2.8 | +| Benchmark | evict/lru | evict/twoq | evict/sieve | evict/s3fifo | evict/tinylfu | otter | golang-lru | +|:---|---:|---:|---:|---:|---:|---:|---:| +| `Get`: reads, all of them hits | 39.7 → 35.9 | 46.7 → 44.4 | 35.6 → 265.8 | 36.1 → 274.7 | 35.0 → 258.1 | 27.2 → 202.3 | 34.2 → 4.3 | +| `Mixed`: the same, 10% of them writes | 37.6 → 34.0 | 42.5 → 42.8 | 32.9 → 68.6 | 33.4 → 69.9 | 33.1 → 62.6 | 21.7 → 80.8 | 33.5 → 4.4 | +| `Set`: writes that replace | 36.5 → 24.9 | 43.3 → 28.4 | 22.4 → 13.8 | 22.5 → 13.6 | 22.2 → 12.8 | 8.9 → 15.9 | 32.1 → 4.3 | +| `Evict`: writes that evict | 16.5 → 31.3 | 16.9 → 30.9 | 13.4 → 12.0 | 8.6 → 10.5 | 8.2 → 9.8 | 3.6 → 5.9 | 7.5 → 3.3 | +| `ReadThrough`: read, store on a miss | 30.3 → 34.1 | 31.9 → 33.7 | 28.6 → 53.3 | 22.1 → 43.1 | 21.7 → 46.0 | 12.7 → 23.6 | 16.7 → 3.6 | +| `ReadThrough`, a million entries | 8.8 → 30.1 | 8.4 → 28.7 | 8.9 → 41.5 | 7.2 → 36.8 | 7.5 → 37.8 | 4.3 → 17.9 | 6.3 → 2.9 | ![Throughput of ReadThrough with a million entries by number of CPUs](bench/results/throughput-readthrough-capacity-1m.svg) -- **Reads scale in `sieve` and `s3fifo`**, as they do in otter: with eight CPUs both read - about a third faster than otter does. `lru` and `twoq` read under a lock, and with eight - CPUs they are slower than with four. -- **With one CPU all four are faster than otter** at everything measured, by a factor of +- **Reads scale in `sieve`, `s3fifo` and `tinylfu`**, as they do in otter: with eight CPUs + they read 28 to 36% faster than otter does. `lru` and `twoq` read under a lock, + and with eight CPUs they are slower than with four. +- **With one CPU all five are faster than otter** at everything measured, by a factor of 1.3 to 5. -- **Reading through is what `sieve` and `s3fifo` are best at.** `ReadThrough` is the - benchmark that comes closest to a cache in front of a database. From two CPUs on `sieve` - is the fastest in it, and from four on `s3fifo` is the second; with one CPU it is `twoq` - or `lru`. With eight, `sieve` and `s3fifo` do about twice what otter does. -- **Writes are where `lru` and `twoq` are at home.** They change an entry in place. `sieve` - and `s3fifo` move a replaced entry to a fresh node so that readers need no lock. A write - that replaces costs them 1.6 to 2.5 times as much, and up to three times with four CPUs; - one that evicts costs 1.2 to 3 times as much. Their replacing writes are fastest with one - CPU, and every cache here writes slower with eight CPUs than with four. -- **Otter wins when eight CPUs keep overwriting the entries that they read**: by a fifth - over `sieve` and `s3fifo` with 10% writes, and by nearly a quarter over `twoq` with 25%. - With four CPUs the fastest of the caches here are still well ahead, though `lru` is - behind otter already with 10% writes. +- **Reading through is what the caches that read without a lock are best at.** + `ReadThrough` is the benchmark that comes closest to a cache in front of a database. From + two CPUs on `sieve` is the fastest in it, and from four on `s3fifo` or `tinylfu` is the + second; with one CPU it is `twoq`, or `sieve` and `lru` with a million entries. With + eight CPUs, those three do about twice what otter does. +- **`tinylfu` reads about as fast as `s3fifo`, and writes a little slower.** With one, two + and eight CPUs its `Get` is within 6% of that of `s3fifo`; with four, both vary by about a + third from run to run. With eight CPUs it reads through faster, because it misses less. + Its writes do more, for its sketch: those that evict get 5 to 10% less done, and those + that replace about as much but for 17% less with four CPUs. With 1% writes among the + reads and eight CPUs it falls further behind, by 22%; with 10% the medians differ by 10%, + but the runs overlap. In an experiment that is not part of this repository, taking out + what `tinylfu` does differently on that path did not close the gap, and it has not been + explained yet. +- **Writes are where `lru` and `twoq` are at home.** They change an entry in place. `sieve`, + `s3fifo` and `tinylfu` move a replaced entry to a fresh node so that readers need no lock. + A write that replaces costs them 1.6 to 2.6 times as much, and up to 3.4 times with four + CPUs; one that evicts costs 1.2 to 3.2 times as much. Their replacing writes are fastest + with one CPU, and every cache here writes slower with eight CPUs than with four. +- **Otter wins when eight CPUs keep overwriting the entries that they read**: by a sixth + over `sieve` and `s3fifo` and by more than a quarter over `tinylfu` with 10% writes, and + by nearly a quarter over `twoq` with 25%. With four CPUs the fastest of the caches here are + still well ahead, though `lru` is behind otter up to 10% writes and ahead of it only with + 25%. - **One machine is one machine.** In the numbers that otter publishes, from Linux on x86, its writes get five times as fast between one CPU and eight; here they do not get twice as fast. How the caches compare on such a machine may well differ, and the `Benchmarks` @@ -219,16 +243,17 @@ entries. [memory.md](bench/results/memory.md) has every size, and strings. | | Bytes per entry | of which the collector has to scan | CPU time of one collection, a million entries | ten million | |:---|---:|---:|---:|---:| | evict/lru | 46–58 | 0–1 | 0.2 ms | 0.2 ms | -| evict/sieve | 54–71 | 0–2 | 0.3 ms | 0.2 ms | -| evict/twoq | 74–82 | 0–1 | 0.3 ms | 0.2 ms | +| evict/sieve | 54–72 | 0–2 | 0.4 ms | 0.2 ms | +| evict/twoq | 74–82 | 0–1 | 0.4 ms | 0.2 ms | | evict/s3fifo | 68–90 | 0–2 | 0.3 ms | 0.2 ms | -| otter | 84–95 | 41–50 | 64 ms | 631 ms | -| golang-lru | 110–127 | 94–111 | 65 ms | 736 ms | -| golang-lru/2q | 156–183 | 136–163 | 99 ms | 1,229 ms | +| evict/tinylfu | 67–82 | 0–2 | 0.4 ms | 0.2 ms | +| otter | 83–94 | 40–50 | 64 ms | 634 ms | +| golang-lru | 110–127 | 93–111 | 65 ms | 740 ms | +| golang-lru/2q | 156–183 | 136–163 | 97 ms | 1,167 ms | The bytes per entry go up and down with the capacity, because the index of a cache has a power of two of slots, and the ranges overlap: at 100,000 entries `s3fifo` takes 90 bytes -and otter 84. What does not overlap is the work for the garbage collector. A cache that is +and otter 83. What does not overlap is the work for the garbage collector. A cache that is made of pointers gives it more to do the larger it is, and a cache of this module gives it next to nothing at any size. That is for keys and values that hold no pointers. If they do, as strings do, the collector has to go through the entries to follow them, which was not @@ -249,14 +274,15 @@ bound that it rarely reaches exactly. By default a cache of fewer than 512 entri shard, and a larger one has up to eight per CPU, of 256 entries or more. `WithShards(1)` gives the textbook behavior of a policy. -**Reads without a lock** (`sieve`, `s3fifo`). The index is made of atomic words. A reader +**Reads without a lock** (`sieve`, `s3fifo`, `tinylfu`). The index is made of atomic words. A reader that finds its key says which node it is about to read by writing the node's number into a slot of its own, checks that the index still points there, and reads. A writer never changes a node that readers can reach: replacing a value moves the entry to a fresh node, and a node that has left the index is reused only after a look at the slots shows that nobody has it. A read thus costs two writes to a cache line that the reading CPU has to itself, and nothing -that all CPUs share. Both policies were chosen for this: reading an entry sets a mark or -counts to three, and writes nothing once that is done. +that all CPUs share. The three policies were chosen for this: reading an entry sets a mark or +counts to three, and writes nothing once that is done. `tinylfu` also adds the read to a +count of hits in the reader's slot, a third write to the same line. Two consequences are worth knowing: @@ -267,6 +293,18 @@ Two consequences are worth knowing: - A cache has a table of those slots, of 2 to 4 KiB per CPU. For a great many small caches, use `lru`. +**Admission by frequency** (`tinylfu`). Each shard keeps a count-min sketch of how often +keys have been asked for of late, whether it holds them or not: four 4-bit counters per key +in one 64-byte block, halved with the tally of what they have counted whenever that tally +reaches ten requests per entry. Only writers touch it. A read counts on its entry, and the +next write that comes by the entry tells the sketch. A new entry waits in a small queue, +and once the shard is full and the small queue has its share, an entry whose turn comes +gets into the main queue only if the sketch has its key down as asked for more often than +the entry that the main queue would evict next. How much room the small queue gets, the +cache finds out as Caffeine does, by hill climbing on the share of requests it answers. It +takes that share from the hits that readers count in their slots and, as Caffeine does, +counts a write of a key that it does not hold as a miss. + **Tests.** Every cache passes the same conformance suite, is compared operation by operation against a simple model of its policy, and is fuzzed against that model. The lock-free path is tested for values that tear, keys that go missing while they are present, diff --git a/bench/README.md b/bench/README.md index fe5011b..1d6b1ca 100644 --- a/bench/README.md +++ b/bench/README.md @@ -22,7 +22,7 @@ libraries they are compared with. | Name | What it is | |:---|:---| -| `evict/lru`, `evict/twoq`, `evict/sieve`, `evict/s3fifo` | The caches of this module. | +| `evict/lru`, `evict/twoq`, `evict/sieve`, `evict/s3fifo`, `evict/tinylfu` | The caches of this module. | | `otter` | [otter](https://github.com/maypok86/otter) v2, the state of the art among Go caches. It evicts with adaptive W-TinyLFU, reads without locks, and writes through buffers that a background task drains. | | `golang-lru`, `golang-lru/2q` | [hashicorp/golang-lru](https://github.com/hashicorp/golang-lru) v2: an LRU and a 2Q cache behind a single lock, the way most caches used to be built. They are here for perspective, and `golang-lru` also as a check: the tests require `evict/lru` to agree with it on every single request of a trace. Its 2Q is not the same algorithm as `evict/twoq`: it promotes an entry that is read again while it is still recent, where the 2Q of the paper waits for it to come back after it has been evicted. | | `optimal` | Not a cache but the bound on hit ratios: what a cache would score that knew the requests to come ([Belady's algorithm](https://en.wikipedia.org/wiki/Cache_replacement_policies#Belady's_algorithm), with the option of not storing a key at all). | @@ -43,14 +43,16 @@ background, as it is in its own simulator. The traces are downloaded on first use, into the user's cache directory unless `-dir` says otherwise, and each file is checked against the SHA-256 that [dataset/dataset.go](dataset/dataset.go) has on record for it. A run on another machine -therefore replays the very same requests, and the numbers for the caches of this module -and for golang-lru come out the same to the last digit. +therefore replays the very same requests, and the numbers for golang-lru and for the caches +of this module but `evict/tinylfu` come out the same to the last digit. Otter's do not. It admits some entries at random, and its hit ratio moves from one replay to the next because of it: by several percentage points at some capacities, and hardly at -all at others. A cache like that is replayed eleven times (`-runs`), and the report has the -median and the range. A cache is only called the winner of a row if it beat the others on -every replay. +all at others. Nor do those of `evict/tinylfu`, whose sketch has keys share counters by a +hash that is seeded at random, so that which keys it confuses differs from one replay to the +next. A cache like that is replayed eleven times (`-runs`), and the report has the median and +the range. A cache is only called the winner of a row if it beat the others on every +replay. | Dataset | Workload | Requests | Keys | Download | |:---|:---|---:|---:|---:| @@ -149,7 +151,7 @@ measures comes after that. With a million entries it is not: the share of reads `ReadThrough` reports for those caches is that of a cache which is still learning, and a few tenths of a percentage point higher where a cache got through more requests in its second, as it does with more CPUs. Otter's share moves with the number of CPUs in the small -cache as well, from 87.0% to 87.6%. That is not the warm-up, and why it does was not looked +cache as well, between 86.9% with two CPUs and 87.6% with eight. That is not the warm-up, and why it does was not looked into. | Benchmark | What every goroutine does | diff --git a/bench/caches/caches.go b/bench/caches/caches.go index 02c0502..8f6205c 100644 --- a/bench/caches/caches.go +++ b/bench/caches/caches.go @@ -9,6 +9,7 @@ import ( "github.com/floatdrop/evict/lru" "github.com/floatdrop/evict/s3fifo" "github.com/floatdrop/evict/sieve" + "github.com/floatdrop/evict/tinylfu" "github.com/floatdrop/evict/twoq" ) @@ -104,6 +105,21 @@ func All[K comparable, V any]() []Kind[K, V] { return s3fifoCache[K, V]{s3fifo.New[K, V](capacity, s3fifo.WithShards(shards))} }, }, + { + Name: "evict/tinylfu", + New: func(capacity int) Cache[K, V] { + return tinylfuCache[K, V]{tinylfu.New[K, V](capacity)} + }, + Exact: func(capacity int) Cache[K, V] { + return tinylfuCache[K, V]{tinylfu.New[K, V](capacity, tinylfu.WithShards(1))} + }, + Shards: func(capacity, shards int) Cache[K, V] { + return tinylfuCache[K, V]{tinylfu.New[K, V](capacity, tinylfu.WithShards(shards))} + }, + // Which keys share counters in its sketch is up to a hash that + // is seeded at random. + Varies: true, + }, { Name: "otter", // Setting the initial capacity is not a default, but it is what @@ -170,6 +186,11 @@ type s3fifoCache[K comparable, V any] struct{ c *s3fifo.Cache[K, V] } func (c s3fifoCache[K, V]) Get(key K) (V, bool) { return c.c.Get(key) } func (c s3fifoCache[K, V]) Set(key K, value V) { c.c.Set(key, value) } +type tinylfuCache[K comparable, V any] struct{ c *tinylfu.Cache[K, V] } + +func (c tinylfuCache[K, V]) Get(key K) (V, bool) { return c.c.Get(key) } +func (c tinylfuCache[K, V]) Set(key K, value V) { c.c.Set(key, value) } + type otterCache[K comparable, V any] struct{ c *otter.Cache[K, V] } func (c otterCache[K, V]) Get(key K) (V, bool) { return c.c.GetIfPresent(key) } diff --git a/bench/cmd/hitratio/main.go b/bench/cmd/hitratio/main.go index 45b2115..8f4374a 100644 --- a/bench/cmd/hitratio/main.go +++ b/bench/cmd/hitratio/main.go @@ -166,7 +166,7 @@ type input struct { // defaultCaches are the columns of the report: the caches of this module, // the ones they are compared with, and the best that any cache could do. -var defaultCaches = []string{"evict/lru", "evict/twoq", "evict/sieve", "evict/s3fifo", "otter", "golang-lru/2q", report.Bound} +var defaultCaches = []string{"evict/lru", "evict/twoq", "evict/sieve", "evict/s3fifo", "evict/tinylfu", "otter", "golang-lru/2q", report.Bound} func allPlayers() []player { var players []player @@ -246,7 +246,7 @@ func shardingPlayers() []player { continue } players = append(players, - player{kind.Name, false, func(t *input, capacity int) int { + player{kind.Name, kind.Varies, func(t *input, capacity int) int { return replay.Hits(kind.Exact(capacity), t.keys) }}, player{kind.Name + " in shards", true, func(t *input, capacity int) int { diff --git a/bench/cmd/hitratio/main_test.go b/bench/cmd/hitratio/main_test.go index c7d01f0..37a0368 100644 --- a/bench/cmd/hitratio/main_test.go +++ b/bench/cmd/hitratio/main_test.go @@ -48,7 +48,7 @@ func TestRun(t *testing.T) { md := read(t, dir, "hitratio.md") for _, want := range []string{ "# Hit ratios\n", - "the median of 3 replays", + "the medians of 3 replays", "in this run by up to ", "Produced by `go run ./cmd/hitratio -caches evict/lru,evict/sieve@4,otter,optimal -runs 3 ", "## At a glance\n", @@ -118,9 +118,10 @@ func TestSharding(t *testing.T) { "as on a machine with 8 CPUs", "up to 64.", "### Small\n", - "| capacity | evict/lru | evict/twoq | evict/sieve | evict/s3fifo |\n", - // A cache of 100 entries is in one piece either way. - "| 100 | 0.00 | 0.00 | 0.00 | 0.00 |\n", + "| capacity | evict/lru | evict/twoq | evict/sieve | evict/s3fifo | evict/tinylfu |\n", + // A cache of 100 entries is in one piece either way. The last one + // is not the same on every replay even so. + "| 100 | 0.00 | 0.00 | 0.00 | 0.00 | ", } { if !strings.Contains(md, want) { t.Errorf("sharding.md lacks %q:\n%s", want, md) diff --git a/bench/cmd/hitratio/preamble.go b/bench/cmd/hitratio/preamble.go index 27ff989..3ff65e8 100644 --- a/bench/cmd/hitratio/preamble.go +++ b/bench/cmd/hitratio/preamble.go @@ -45,16 +45,17 @@ it is missing. The column `+"`optimal`"+` is not a cache but the bound that no c beat: what one would score that knew the requests to come. The caches of this module run in one piece here, with a single shard, and otter evicts on -the spot rather than in the background, as it does in its own simulator. The caches of -this module and those of golang-lru then evict the same way every time, and their numbers -are the same on every run. Otter's are not: it admits some entries at random, which moves -its hit ratio from one replay to the next, in this run %s. -Its number is the median of %d replays, which is also what the charts show, and the column -next to it has the lowest and the highest of them. That is not all the range there is: -of further replays, about %d in a hundred would come out below it or above it. +the spot rather than in the background, as it does in its own simulator. Most of the +caches then evict the same way every time, and their numbers are the same on every run. +Two do not. Otter admits some entries at random, and `+"`evict/tinylfu`"+` counts requests in a +sketch in which keys share counters, by a hash that is seeded at random. That moves their +hit ratios from one replay to the next, in this run %s. +Their numbers are the medians of %d replays, which is also what the charts show, and the +column next to each has the lowest and the highest of them. That is not all the range +there is: of further replays, about %d in a hundred would come out below it or above it. A number is in bold if the cache did better in that row than every other one, on each of -their replays. Where the range of otter reaches the best of the others, the row has no +their replays. Where the range of one cache reaches the best of the others, the row has no winner. A program does not get the caches of this module in one piece unless it asks for that: by @@ -101,6 +102,11 @@ so the hit ratio of a cache in shards differs from run to run. The number is the difference for the median of %d replays, and the brackets hold the differences for the worst and for the best of them. +`+"`evict/tinylfu`"+` differs from run to run in one piece as well: which keys share counters in +its sketch is up to the same hash. What its replays in shards are compared with is the +median of as many replays in one piece, so its brackets are never empty, not even where a +cache is too small to be split. + %s`, sharedCPUs, 8*sharedCPUs, runs, produced(command)) } diff --git a/bench/report/chart.go b/bench/report/chart.go index 248f5e3..e3318f3 100644 --- a/bench/report/chart.go +++ b/bench/report/chart.go @@ -51,6 +51,7 @@ var ( "otter": 4, "golang-lru": 5, "golang-lru/2q": 6, + "evict/tinylfu": 7, } light = [...]string{"#2a78d6", "#eb6834", "#1baf7a", "#eda100", "#e87ba4", "#008300", "#6250d6", "#e34948"} dark = [...]string{"#3987e5", "#d95926", "#199e70", "#c98500", "#d55181", "#008300", "#9085e9", "#e66767"} diff --git a/bench/results/hitratio-cloudphysics.svg b/bench/results/hitratio-cloudphysics.svg index e50ca7f..2958caa 100644 --- a/bench/results/hitratio-cloudphysics.svg +++ b/bench/results/hitratio-cloudphysics.svg @@ -1,6 +1,6 @@ CloudPhysics -Line chart of Hit ratio, % by Cache capacity, entries for evict/lru, evict/twoq, evict/sieve, evict/s3fifo, otter, golang-lru/2q, optimal. The numbers are in the table that goes with it. +Line chart of Hit ratio, % by Cache capacity, entries for evict/lru, evict/twoq, evict/sieve, evict/s3fifo, evict/tinylfu, otter, golang-lru/2q, optimal. The numbers are in the table that goes with it.